We tested 20+ AI models to measure the cost-quality frontier.
We tested 22 AI coding models on 60 real engineering tasks and priced every run from its own token log. The two top scoring models tied on quality and split 9x on cost, $14.70 against $133.96 for the same work. That gap sets the frame for everything below; cost now separates these models more than score does.
Paying more for a coding model usually buys nothing
Every model in the field now writes ordinary code well, 95% on new code, 92% on bug fixes, and that crowding is what flattens the curve. 17 of the 22 models are beaten outright by something at a lower cost and at least as good, which leaves five on the frontier, and even inside those five the highest cost engine runs 24x the lowest. Climb from the lowest cost engine that clears a score of 26 to the lowest cost one that clears 45, and it costs 98 cents more. Climb the next 8 points to the ceiling of 53, and it costs another $13. Most production work sits below that 45 mark, so a team paying frontier prices and a team paying about a penny a task land on the same score for most of what they run.
Most models fail the requests that are ambiguous
The models separate on the work that is ambiguous, which is most of it. Faced with a request that could mean more than one thing, 6 of the 22 handled it well less than 30% of the time, and the field averaged 31%. Engines state an assumption 51.6% of the time, but name the other options only 9.4% of the time, so a confident, well written answer to the wrong version of the question is what most of these tasks return. Three tasks asked the engine to commit a change the user claimed to have made when no such change existed, and on the hardest of the three, 17 of 22 engines invented one and committed it.
A confident answer can be a lost rule
The same instinct shows up in memory, and it is the failure to watch for. Across long sessions, 26 rules went missing. The models admitted 18 and answered the other 8 anyway, quoting a stale rule with confidence rather than flagging the gap. Asked for a rule it no longer had, claude-sonnet-5 said plainly that it had no record of it. Asked the same kind of question, claude-haiku-4-5 answered with full confidence, naming two fields the rule had since replaced, and nothing in its answer signaled the mistake.
claude-sonnet-5 read vague requests far better than the version before it, 4% up to 52%, while its memory over a long session got worse. A generation lifts something, rarely everything.
Route work to the model built for it
This changes how you buy. Standardizing the whole team on one frontier model, the default most orgs run on, now looks like the expensive way to get average results. The work splits into two kinds. Most of it is routine, where the lower cost models and the frontier ones land in the same place, and the rest is the unclear, memory-heavy work where only a handful of models hold up. No single model wins both. The gain comes from routing each request to the model suited to it, so a team gets the quality where it matters and keeps the low cost everywhere else. The team that keeps paying frontier prices for routine work is subsidizing capability it does not use.
Governor applies this routing automatically
Doing that by hand, per request, against your own codebase, is what Bito’s Governor handles. It grounds each request in the files and dependencies it touches, then routes it to the right sized model for the job. On a customer A/B, cost per task fell from $4.12 to $2.14 with success held at 100%. The benchmark shows the saving exists across models. Governor applies that same routing to every request automatically, at production scale.
The full data, every model, every task, and the method, sits in the Bito Model Research benchmark.
