At Actrix we are interested in practical AI for rates markets not just impressive benchmark scores.

Public LLM benchmarks measure performance on someone else’s test. For business-specific outcomes from LLMs, you need business-specific benchmarks.

For my first Actrix blog, I wanted to test what this means in practice. Gemini 2.5 Flash scores 70% on the FACTS Grounding Benchmark. On our task of extracting orders from trader chats, it scored 52% fully correct. Okay, different tasks, but how do we get to 95%+ accuracy in our business?

That gap is the point of this post.

At Actrix, we are using LLMs for real rates-market workflows, where “mostly right” is not good enough. This post looks at how we moved from vibes-based prompt evaluation to a proper scorecard and A/B testing framework for order extraction.

What do we mean by “performance”?

Answer: Some metric to assess the quality (accuracy, speed and cost are typical) of an LLM’s output.

This is made easier with a scorecard which will give details over a handful of metrics so when comparing across tests we can quickly see which one performed better. We need to ask ourselves: “what metrics do we care about?”.

For our specific task we want an LLM to:

• Extract every order from a chat – never miss an order!
• Find every field and record the correct value.
• NEVER hallucinate an order – imagine filling a fake order. Ooops….
• Be relatively speedy, but not in the realm of “low-latency” trading.

That leads us to our scorecard below:

Metric	Weight
% Orders Found	35%
% Fields Correct	35%
% Precision (1 – Hallucination Rate)	20%
Speed (relative to the fastest variant)	10%

The scorecard gives a single headline score for comparing prompts, or models, weighted toward what matters most for this problem. Each metric points to a different type of problem. Low orders found means the prompt needs more context. Low fields correct means either a normalisation issue or an actual incorrect model output. Low precision means hallucinated orders.

Knowing how well a system works is good, but knowing how well it is working and how we can improve it is even better, which has been my goal for this project. This means in the future the scorecard and test harness can be used to compare the underlying software itself, different LLM models, prompts and post-trained vs pre-trained open weight models. So we know what we are measuring now, but let’s explore how we measure it.

Building Ground Truth: Labelling

The way to truly know if an order has been correctly found or a field is correct is to know what it was supposed to be:

• Fields that make up an order 
• Earliest time in the chat that all of the required fields to construct an order could be extracted

I approached this challenge of labelling the data by having an LLM do a first pass. After that, I asked Chris to review every label to correct/confirm each one (lucky Chris!).

After our expert verifies or fixes a label for a set of raw data it becomes “human reviewed”.

This is slow and expensive but pure LLM labelling has no way to catch systemic errors (although we might still have human errors we can aim to catch later on).

Benchmarking a prompt against labels it helped create is circular, you need independent ground truth or you have no idea how far you actually are from your target accuracy. Performance here is simple: does the model’s output match the label or not?

A/B Testing

For this problem we have a high goal of ≥95% accuracy, this is important for a trading context, if the tool isn’t reliable and convenient it won’t get used. A hallucinated order is unthinkable. Doing an A/B test with our scorecard and using the original prompt for order extraction and an improved version based on the failings of the V1 prompt we get:

Dashboard Results using V1 prompt from before evaluation system was built:

Comparing key results between the V1 prompt run and V2 we get:

This is cool! We can now:

• Quantify prompt A vs prompt B
• Confidently say “I have improved the prompt by 14.8 points according to our priorities!”
• We can see where that improvement has come from – missing orders, field correctness, speed/cost.
• Weigh up cost and latency vs quality improvements

Anyone who has built with AI will tell you of the pitfalls of changing a prompt and testing: it is expensive in tokens, probabilistic in nature and hard to quantify. The scorecard fixes that. V2 improved overall score by 14.8 points, driven by the improvements in Orders Found and a fair portion from Fields Correct which still remains the biggest gap to close, requiring more nuanced prompt rules.

That improvement isn’t free though, the longer V2 prompt means higher cost and slower response times per API call, both of which need factoring in alongside the quality of the models output.

Currently the Fully Correct rate (combination of Orders Found and Fields Correct) is lower than our target but this is a work in progress first pass.

As an example of how we can use what has been discussed to diagnose and fix an issue with extraction we can look at a portion of the evaluation dashboard I created for this above. I detail the accuracy per field that the model extracts and then the orders with mismatched fields.

When we look at order level detail for orders that have mismatched fields between the expected label and LLM output we can see how they differ and look at the chat from the test case to try and figure out the issue: is the label wrong? Are they formatted differently? Or is the LLM “misunderstanding” the chat?

In Summary

• Gemini didn’t become a better model during this project. What improved was our ability to measure it and improve our prompts. Once we had task-specific labels, a scorecard and A/B testing, we could identify failures, prioritise fixes and quantify progress.
• LLM assisted labelling is only a starting point as you still need domain expert reviewers to catch errors, you can’t assess anything accurately if you don’t have a ground truth data set.
• A composite scorecard forces better decisions. A single value that is weighted in the direction of your priorities makes it clear whether a model or prompt is holistically better or not.