LLM cost–quality benchmark

The question, and the version of it we can answer

“Can we use a cheaper model?” is usually asked as if it had one answer. It does not. A smaller model may match the current one on routine requests and fall apart on the minority of requests that carry most of the business risk. A routed mix may cut spend while adding a classification step that worsens tail latency. A shorter prompt may save tokens and quietly drop an instruction a downstream system relies on. Every one of those trade-offs is real, and every one is invisible in an average.

So we turn the question into one that can be answered with evidence: on a defined sample of your traffic, which candidates deliver results that pass a quality bar you set in advance, at what cost per accepted result, and at what latency? The answer can be “this candidate, for these task types, with this fallback”. It can also be “none of them”, and that is a useful answer.

AI spend has become routine work for finance and platform teams. The FinOps Foundation’s State of FinOps 2026 reports that 98% of respondents now manage AI spend and names AI cost management as the top skill set teams need to develop (data.finops.org). The difficulty is rarely seeing the bill; it is knowing what a cheaper configuration would do to quality before you commit to it.

Step one: fix the quality bar before measuring anything

The most common way a model-cost study goes wrong is choosing the threshold after seeing the results. We prevent that by agreeing, in writing, three things before any candidate runs:

1. What counts as an accepted result for each task type: an exact match, a passing validator, a rubric score above a set level, or a human reviewer’s sign-off.
2. The minimum acceptance rate each task type must reach, and whether any single failure type disqualifies a candidate outright.
3. The latency budget, stated as a percentile, because a p50 inside budget says nothing about the users who wait at p95.

Where outputs are open-ended and no reference answer exists, absolute scores are unreliable. Pairwise judgements (“which of these two is better for this request?”) are easier to make consistently. Compere chooses which pairs to compare with UCB1 and turns outcomes into Elo ratings, which assumes reasonably transitive preferences; we use it when that assumption suits the task and a simpler rubric when it does not.

The candidates

Candidate type	What it tests	Typical risk
Smaller or cheaper model	Whether quality holds on your task mix	Collapse on the hard slice
Routed mix	Whether easy requests can go cheap and hard ones stay expensive	The router misclassifies; extra hop adds latency
Prompt change	Whether fewer tokens or a different structure keep quality	Silent loss of an instruction downstream code needs
Caching	Whether repeated or near-repeated work can be served without a model call	Stale or wrong-context answers
Current configuration	The baseline every candidate must beat	None; it is the control

For embedding-heavy workloads, EmbedCache tests a specific cost lever: generating embeddings locally with ONNX models on CPU and caching them by content hash and model id, so the same text is never embedded twice. Whether that beats a hosted embedding API depends on your volume, your hardware and whether a local model’s retrieval quality is good enough, which is part of what the benchmark measures.

Why the hard slice is reported on its own

Averages hide the failures that matter. We split results by task type and by difficulty, and we always report the hardest slice separately, defined before the run, not chosen afterwards. If a candidate is within threshold overall but fails the hard slice, the recommendation says so and usually proposes routing that slice to a stronger model rather than rejecting the candidate entirely.

Where the current model and a candidate disagree, we read the disagreements. Some are cases where the candidate is wrong. Some are cases where the current model was wrong and nobody had noticed. Both go into the failure analysis, because the second kind changes what “keeping the current model” actually means.

Instruments, and their limits

Route-Switch is a self-hosted, OpenAI-compatible gateway that routes by configured strategy, logs per-prompt success, cost and latency to DuckDB, and can rerun MIPROv2 prompt optimisation on captured traces (claim record). It is useful when you have no per-request logging of cost and outcome, or when the candidate under test is a routed mix. It publishes no latency, cost-saving or quality benchmark, and we will not quote one. The whole point of this benchmark is that the trade-off has to be measured on your traffic; the routing-triangle essay sets out why.

If your existing gateway, tracing or observability stack already records what we need, we use that instead. The harness we leave behind should run on the infrastructure you already operate.

What we need from you

• A sample of production requests large enough to cover each task type, with the hard slice represented. We help size it; redaction is fine if it preserves the task.
• Current pricing, including any negotiated rates, so cost figures reflect your contracts and not list prices.
• API access or credentials to the candidate models, held by you where possible.
• One or two people who can say whether an output is acceptable, for grading and for checking any model-based grader.
• A sponsor who owns the budget line and will act on the recommendation.

Limits

The benchmark tells you how candidates performed on the sample, at the prices and model versions in force during the run. It does not predict how providers will change prices or models, and it does not cover task types missing from the sample. If your traffic mix shifts materially, the result needs re-running. Our measured, inferred and unknown findings are kept separate in the report, following the methods we apply to all published work.

Typical shape

When the quality bar and traffic sample already exist, the work starts as an evaluation sprint (£8,000–£20,000): candidates run, cost per accepted result calculated, slices and disagreements analysed, recommendation written. When they do not, a technical diagnostic (£2,500–£5,000) comes first to agree the threshold, size the sample and choose candidates. Recurring re-evaluation reruns the harness when a provider ships a new model, changes pricing, or your prompts change. The smaller-model replacement guide covers the decision in more depth, and procurement covers contracting and data terms.