We tested 20+ AI models to measure the cost-quality frontier.
AI model research
What a Crowded AI Frontier Means for Engineering Teams
How real-world scenarios and a plethora of options dramatically change the cost-quality frontier.
Based on 22 AI models, 60 real engineering jobs, and 1,320 graded answers.
We tested 22 AI models on the same 60 real engineering tasks, graded every one of the 1,320 answers against criteria written before any model ran, and priced every run. That combination, quality and cost measured together on the same work, is what this research is about.
Three findings matter for anyone deciding what AI models to buy for their engineering teams.
Most of the market is overpriced for what it delivers
17 of the 22 models are beaten by something cheaper that scores at least as well. The two best scores tied at 53 out of 60, and one costs 9x as much to run as the other. grok-4.5 at $14.70 for the full run, claude-opus-5 at $133.96.
Nearly all of the quality, for almost none of the money
deepseek-v4-flash cost $1.58 for the full run and reached 85% of the best score anyone achieved. The most expensive route to the top score costs $133.96. For a few years, buying the newest flagship made sense. That era appears to be over.
The raw score alone won't tell you
Two behaviors separate the models that hold up in daily use. What a model does when your request is vague, and how it handles an instruction it has forgotten. We scored both, model by model, and the results don’t follow price. 10 of the 22 models pass both tests.
WHAT YOU CAN BUY, AND WHAT IT COSTS
17 of the 22 engines are beaten outright by something both cheaper and at least as good. They are not on the Pareto efficient frontier.
Five are left. Those five are the only ones where paying more actually buys you a higher score, and the gap between the cheapest and the most expensive of them is a factor of 24.
Ignore the dots for a moment and follow the line. It runs almost flat from $0.60 to about $1.60, picking up 19 points on the way, and then it turns and goes nearly straight up.
98 incremental cents takes you from a score of 26 out of 60 to a score of 45. The next 8 points cost another $13, and 53 is the ceiling, so there is no further rung to climb. A lot of production work sits in that flat stretch, where the engines cost about a penny a task and the extra money buys nothing you can measure.
These are list prices and they keep moving. Two of the three OpenAI tiers were cut on 30 July, and claude-sonnet-5 is a third cheaper than the model it replaced. The dollar figures are a snapshot.
WHY THE SCORE CANNOT PICK FOR YOU
22 of the 60 tasks were fully passed by 17 or more of the 22 models. Writing new code averages 95% across the field. Fixing bugs, 92%. Public benchmarks are assembled mostly from that band, which is how you end up with rankings that do not survive the first week you run the model.
Tasks grouped by how many of the 22 engines fully passed them.
Where the field actually comes apart is 14 tasks that only one to four engines passed, plus two that nobody passed. Both of those two were vague requests, requests that were ambiguous and it was hard to really ascertain the meaning. So a tie at 53.0 is mostly a tie on the part of the job that is already solved.
But two things do separate them, and a score will not show you either one.
1➞ WHAT THE AI DOES WHEN YOUR REQUEST IS VAGUE
Half the messages people send an assistant do not say what they are about. We wrote 14 requests like that, each one ambiguous enough to mean several jobs in its codebase, and watched what came back.
the retries aren’t behaving. can you look into it?
O01, BULLMQ · WRITTEN AS AN ENGINEER
this isn’t working the way we want. can you fix it before the demo?
O13, RESQUE · THE LOWEST-SCORING TASK IN THE STUDY, 11%
We mark two things here and keep them apart. Did the engine spot that the job could mean more than one thing, and either ask or say out loud which meaning it was going with. And separately, was the work it did do any good.
68 points separate the top of that chart from the bottom, and the order barely resembles the leaderboard. claude-sonnet-4-6 has great memory, scores a respectable 38.0 overall, and bombs on handling vague requests (it only got a score of 0.4 out of 10).
This is also where the money is
Because every run was priced from its own token log, we can ask which kind of work is actually expensive. The answer is lopsided enough to be worth a chart.
14 vague requests take 46% of the entire bill while scoring worst of the 13 categories. Planning takes another 18%. Writing code and fixing bugs, the benchmark staples, are 13 tasks and 8% of the spend. An unclear request burns four to seven times the tokens of a clear one and then usually does not work, so you pay twice. A failed answer costs about what a good one does, $0.38 against $0.33, which means the bill will not tell you which you got.
Engines state an assumption 51.6% of the time. They name the other options only 9.4% of the time.
They commit to an option and say so. But the other four choices they could have made go unmentioned, and that is the part you would have wanted, because a confident, well-sourced answer to the wrong version of your question looks the same as a right one.
The same behaviour has a louder version. Three tasks ask the engine to commit a change the user says they made, when no such change exists in the working tree. Two words would do it: nothing here. On the hardest of the three, 17 of 22 engines invented a change and committed it. claude-opus-5 was the only engine to get all three right. Whatever makes an assistant manufacture a README edit will make it manufacture a stack trace.
The bar we would set: 30% of ambiguous tasks handled appropriately
Our data set has 14 vague jobs, two chances on each, two chances on each to show it had noticed, so 28 marks per engine. Six engines come in under 30%, and those are the ones that essentially never notice. Above 30% the gaps get small enough that 28 marks cannot reliably tell 37% from 40%, so read an engine’s number rather than whether it scraped over a line.
2➞ WHAT IT DOES WHEN IT FORGETS A RULE YOU GAVE IT
We gave each model 10 rules, five at the start and five midway, then worked the session hard enough that it had to compress its own history before we asked for the rules back. 10 of 22 returned all 10. Rules given at the start came back 97% of the time against 79% for rules given midway, so the rule most likely to vanish is the correction you made an hour in, after watching it get something wrong.
The count is the less interesting half. 26 rules went missing across the field. 18 of those the engine admitted. Eight it answered anyway, confidently and wrongly.
Ten rules per engine, a held rule is worth a point, a rule lost and owned up to a quarter of a point, a rule lost and answered wrongly costs a point.
Asked for a rule it had lost, claude-sonnet-5 said “I don’t have a record of an API base URL being part of the project rules I was given”, and on a second one Sonnet carefully separated what it could trace to a source from what it could not. You repost the rule. Two minutes gone.
claude-haiku-4-5, asked which two fields must never be logged: “I can answer this one with confidence. From phase 1 rules, rule 3: the two specific fields that must never be logged are ssn and card_number.” By then the rule had been changed to auth_token and device_id. Nothing in the answer signals a problem, so the fields it was told to protect keep getting logged.
kimi-k2.7-code produced the same superseded answer and cited project memory as its source, then invented a feature flag called enable_turbo_mode that has never existed. kimi-k3 and gemini-3.6-flash both came up with enable_payout_v2 where the real rule said enable_fast_payout, and kimi-k3 went on to explain why its invention differed from the original.
Count the rules and kimi-k3 beats claude-sonnet-5, nine to eight. Ask which you would rather leave running unattended and the order reverses, because kimi-k3’s one loss was an invention and sonnet-5 flagged both of hers. Memory correlates with overall score at 0.36 and with ambiguity handling at 0.25, so it’s truly another lever to consider.
The bar we would set: a score of 8.5 out of 10
Each of the ten rules is worth a point if the engine still has it. A rule it lost but owned up to is worth a quarter of a point, because you can recover from that in about two minutes. A rule it lost and answered anyway, wrongly and with confidence, costs a point, because you will not find out. Eight and a half is roughly nine rules held with the rest owned up to. Claude-haiku-4-5 is a loser at 6.50, gpt-oss-120b at 3.75 and kimi-k2.7-code at 2.50. It also removes kimi-k3 and gemini-3.6-flash, which both look fine on a straight count, but both made a rule up.
WHAT A NEW VERSION AND A BIGGER TIER ACTUALLY BUY
A new release does not lift memory and ambiguity together, and a more expensive tier does not always lift either.
Same tier, one generation apart
Ambiguity handling rests on 28 judgements per engine, so a move under 9 points is inside the noise. The sonnet move clears it, the opus move is smaller, and the kimi move sits within it.
claude-sonnet-4-6 to sonnet-5 is the move that should make an engineer sit up. 5.5 points more, a quarter of the price, and ambiguity handling from 4% to 52% in one generation. Memory went the other way over the same step, 10 rules down to 8, so even a clear upgrade trades one quality for another.
Same generation, different tier
gpt-5.6 shipped three tiers in one release, a clean test of what a flagship is for. The top tier came last of the three on score, at 20x the price of the cheapest, and handled ambiguity worst as well, 25% against terra’s 60%. DeepSeek’s v4 line runs the same shape, pro scores about what flash scores for 6x the money. Anthropic’s ladder is the one that behaves the way its pricing implies.
A generation reliably improves something, rarely everything, and a step up the tier ladder inside one generation can buy almost nothing.
HOW WE WOULD ACTUALLY PICK
Say you are picking one model to put in front of your team on Monday. You want three things from it. It should remember the things you tell it, or tell you when it has not. It should work out what you meant when you were vague. And it should score well enough for a price you are willing to pay.
10 models of the 22 clear the first two. The ones that do not are worth naming, because several of them look perfectly reasonable if you only read the score and the price:
- claude-haiku-4-5 never once spotted a vague request across 14 tries, and quoted an out-of-date rule back at us with confidence.
- claude-sonnet-4-6 spotted 4% of them, and glm-5.2 the same, despite both being solid on score.
- gpt-5.6-sol managed 25% on vagueness, while costing $62.30, the third most expensive engine on the board.
- kimi-k3 and gemini-3.6-flash both look fine on a straight rule count, and both invented a rule rather than admit losing it.
That leaves 10 to shop among. From those 10, two combinations are worth knowing about.
Each roster routes by kind of work. A session picks one of the three engines at the start, based on what is being asked, and keeps it for the whole job. Nothing switches mid-task. Scores are leave-one-out cross-validated, so each task is assigned using only the other 59.
Who does what in the $18.49 roster
grok-4.5 takes 44 of the 60 jobs: vague requests, investigation, triage, bug fixes, writing code, refactoring, end-to-end work, plan reviews and refusing dangerous instructions. gpt-5.6-terra takes 9: planning, and anything coming from someone non-technical. claude-opus-5 takes 7: code review, and git work.
Who does what in the $12.22 roster
grok-4.5 takes 31: vague requests, non-technical requests, investigation, triage, plan reviews and end-to-end work. minimax-m3 takes 16: writing code, fixing bugs, refactoring. claude-sonnet-5 takes 13: planning, code review, git work and refusing dangerous instructions.
Both beat every single engine on total score, and the cheaper roster comes in under what grok-4.5 costs alone. Look at what the top roster is made of. claude-opus-5, gpt-5.6-terra and grok-4.5 are the three best engines in the study at reading a vague request, at 63%, 60% and 68%. The winning combination is built entirely from them.
We would weight that more heavily than score alone justifies. People who favour claude-opus-5 describe it as getting what they were driving at, and 63% against a field average of 31% is the measure of exactly that. If your requests arrive as precise tickets written by engineers, take the cheaper roster and optimise for price. If they arrive as a sentence somebody typed in a hurry, the gap between a 63% engine and a 33% engine will show up in your week in a way that a point and a half of score never will.
WHAT WE WILL MEASURE NEXT TIME
We would not start with the score. It moved less than anything else here and explained less. The two questions worth putting to the next model are whether it notices an ambiguous request and says so, and whether it admits losing a rule instead of inventing one. You can test both against your own traffic in an afternoon, and neither appears on any leaderboard we know of.
Every model, every task, priced from its own token log, sits in the full benchmark. Each model saw each task once, so a single cell carries noise while the shape of these findings holds. 34 of the 60 tasks run against public open-source repositories these models may have seen in training. The other 26 run against private code that exists nowhere public. We re-run this every quarter.
BITO MODEL INDEX
22 AI models, 60 real jobs, 1,320 graded results
Every AI model ran the same 60 pieces of work against 11 open-source job and queue repositories plus purpose-built private Bito code. All 1,320 answers were graded and every run was priced from its own token log. This page is the facts and rankings. Our companion piece argues about what it means.
TOP SCORE
53.0 / 60
A tie between grok-4.5 and claude-opus-5.
PRICE OF THAT TIE
9 x
$14.70 on Grok, $133.96 on Opus.
85% OF THE TOP SCORE
$1.58
deepseek-v4-flash reaches 45.0 for less than the price of a coffee.
ON THE EFFICIENT FRONTIER
5/22
17 models are beaten by something cheaper.
SCORE AGAINST COST
Where an engine sits relative to the line is the whole question. A dot above and to the left of another one is cheaper and better at the same time, and 17 of the 22 have at least one of those sitting over them. The upper left quadrant is the best place to be.
THE COST CLIFF
Same data as a ladder: the cheapest engine that reaches each score, and what the next rung costs.
WHERE THE MONEY GOES
Spend concentrates significantly within certain types of requests (e.g., planning, vague requests, etc). Two categories account for two thirds of the bill, and those two categories are where the models, in general, do quite poor.
WHERE MODELS ACTUALLY DIFFER
The real heatmap. You can see here every model against every kind of work, darker being better, columns ordered by what each category costs (left is the cheapest category, right is most expensive). Writing code and fixing bugs are near-black for almost everyone, which is why a benchmark assembled from them cannot separate this field. And that is actually what so many benchmarks focus on. The two columns on the right take two thirds of the money and are the palest on the board.
VAGUE AND AMBIGUOUS REQUESTS
14 tasks are ambiguous, lacking specifics to help a model nail down exactly what you want to do. Two things get scored separately: whether the engine spotted that the job could mean more than one thing, and whether the work underneath was any good.
the retries aren’t behaving. can you look into it?
O01, BULLMQ · WRITTEN AS AN ENGINEER
this isn’t working the way we want. can you fix it before the demo?
O13, RESQUE · WRITTEN BY SOMEONE NON-TECHNICAL, AND THE LOWEST-SCORING TASK IN THE STUDY AT 11%
MEMORY ACROSS A LONG SESSION
We asked each model to remember 10 rules – five given at the start of a session and five dropped in partway. The session was then worked hard enough that the engine had to compress its own history to keep going, and we asked for the rules back. Across the field 26 rules went missing. 18 times the engine said so. Eight times it answered anyway, wrongly and with confidence, which is probably more concerning.
WHAT THE MONEY BOUGHT
Cost per full run
Share of spend on failed answers
TOKEN TYPES
Input tokens bill at two very different rates depending on whether it came from cache. Cache reads are 77% of all volume here and cost 10% of fresh input tokens, so a figure that lumps them together overstates the bill several times over. Priced without caching, this study would have run to $1,153 instead of $472.
HOW DEMANDING THE WORK IS
Not every job here is equally revealing. We grouped all 60 by how many of the 22 models managed a full pass. A task that 17 or more of them get right tells you almost nothing about which model to pick, and there are 22 of those. The ones that separate the field are the 14 only a handful cleared. Two were passed by nobody at all, and both of those were vague requests.
| Engine | Vendor | Score | Cost | $/Point | Vague | Memory | Seconds | Wasted | Weights |
|---|---|---|---|---|---|---|---|---|---|
| grok-4.5 | xAI | 53.0 | $14.70 | $0.277 | 68% | 10.00 | 137s | 5% | closed |
| claude-opus-5 | Anthropic | 53.0 | $133.96 | $2.528 | 63% | 10.00 | 187s | 0% | closed |
| kimi-k3 | Moonshot | 47.0 | $12.75 | $0.271 | 38% | 8.00 | 70s | 10% | open |
| deepseek-v4-flash | DeepSeek | 45.0 | $1.58 | $0.035 | 37% | 10.00 | 116s | 18% | open |
| claude-opus-4-8 | Anthropic | 45.0 | $29.73 | $0.661 | 40% | 10.00 | 49s | 13% | closed |
| gpt-5.6-terra | OpenAI | 44.0 | $27.12 | $0.616 | 60% | 9.25 | 66s | 7% | closed |
| claude-sonnet-5 | Anthropic | 43.5 | $17.63 | $0.405 | 52% | 8.50 | 77s | 10% | closed |
| deepseek-v4-pro | DeepSeek | 42.5 | $9.49 | $0.223 | 39% | 10.00 | 143s | 36% | open |
| gpt-5.6-luna | OpenAI | 42.0 | $3.17 | $0.075 | 40% | 10.00 | 65s | 14% | closed |
| gpt-5.6-sol | OpenAI | 41.5 | $62.30 | $1.501 | 25% | 9.25 | 102s | 21% | closed |
| minimax-m3 | MiniMax | 40.0 | $3.48 | $0.087 | 48% | 10.00 | 51s | 34% | open |
| kimi-k2.7-code | Moonshot | 39.5 | $12.59 | $0.319 | 27% | 2.50 | 95s | 43% | open |
| claude-sonnet-4-6 | Anthropic | 38.0 | $71.67 | $1.886 | 4% | 10.00 | 146s | 54% | closed |
| gemini-3.6-flash | 37.0 | $16.60 | $0.449 | 33% | 8.00 | 102s | 35% | closed | |
| gemini-3.1-pro | 35.0 | $13.03 | $0.372 | 34% | 7.75 | 109s | 58% | closed | |
| glm-5p2 | Z.ai | 34.5 | $12.08 | $0.350 | 4% | 9.25 | 236s | 48% | open |
| qwen3.7-plus | Alibaba | 34.0 | $3.89 | $0.114 | 17% | 8.50 | 47s | 53% | open |
| gemini-3.1-flash-lite | 31.5 | $1.12 | $0.036 | 44% | 10.00 | 21s | 41% | closed | |
| claude-haiku-4-5 | Anthropic | 31.0 | $12.96 | $0.418 | 0% | 6.50 | 46s | 78% | closed |
| nemotron-3-ultra | NVIDIA | 30.0 | $8.79 | $0.293 | 13% | 10.00 | 109s | 63% | open |
| gpt-oss-120b | OpenAI | 26.0 | $0.60 | $0.023 | 31% | 3.75 | 58s | 57% | open |
| inkling |
Thinking Machines |
15.5 | $2.70 | $0.174 | 6% | 9.25 | 16s | 62% | open |
Two things worth checking before you go directly to start comparing prices and scores. First, look how the model handles vague requests (6 of the 22 models do a horrible job here), and if the model has a good memory. 10 of the 22 models pass both gates.
THE FULL BOARD
Not every job here is equally revealing. We grouped all 60 by how many of the 22 models managed a full pass. A task that 17 or more of them get right tells you almost nothing about which model to pick, and there are 22 of those. The ones that separate the field are the 14 only a handful cleared. Two were passed by nobody at all, and both of those were vague requests.
Tasks grouped by how many of the 22 engines fully passed them.
IF YOU ONLY WANT A SHORTLIST OF MODELS
Two things worth checking before you go directly to start comparing prices and scores. First, look how the model handles vague requests (6 of the 22 models do a horrible job here), and if the model has a good memory. Ten of the 22 models pass both gates.
Three engines, chosen between by kind of work, leave-one-out cross-validated. A session picks one at the start and keeps it, so nothing switches mid-task. The top roster is also the three best engines in the study at handling ambiguity.
CONSTRAINED TO A VENDOR OR A JURISDICTION
Most teams cannot shop freely. Best routed result available inside each constraint, choosing a different engine per kind of work, against the unconstrained baseline.
Leave-one-out cross-validated: the choice for each task uses only the other 59. Wider pools score worse here, because a 13-category policy fitted on 59 tasks overfits badly when it can pick from 22 engines and hardly at all when it can pick from three.
METHOD
Each request is worth one point, half for a partial. Every request’s criteria were written before any engine ran. Answers were graded by running the engine’s code against real tests, by planting a known fault and checking whether it was found, against lists of facts confirmed in the source beforehand, and by an AI judge working from criteria fixed in advance. Answers are blinded before judging: engine names are stripped from the answer and the diff, and each judgement runs in its own throwaway session.
Each engine saw each request once, so differences of a point or two sit inside the noise of a single run. Prices are public list rates checked on 18 August 2026 and they move; token counts are exact, so current rates can be reapplied.
Three caveats worth reading before quoting a number.
34 of the 60 tasks run against public open-source repositories that these engines may well have seen in training. The other 26 run against Bito code that exists nowhere public, and those are the ones graded by running code.
16 records were killed mid-turn by a tool the runner advertised without credentials. They are excluded from both numerator and denominator. The published scoreboard scales each engine’s score up to 60; this page uses the unscaled score over records that produced an answer, which is why the top reads as a tie.
Splitting a memory failure into admitted and invented was classified by hand from the 26 answers involved.
Where this argument leads: Governor
The conclusion lands on routing: send each request to the right-sized engine instead of paying frontier prices for everything. That is the decision Bito’s Governor makes automatically, in production, against your own codebase, rather than a roster fixed in a spreadsheet.
CODE CONTEXT ENGINE
Agents stop paying to find your code.
Governor attaches a map of the relevant files, symbols, and dependencies to each request, so the agent skips the grep-and-read spiral before it writes anything.
MODEL ROUTER
Simple work stops paying frontier prices.
Governor scores each request by structural complexity, services touched, dependency depth, blast radius, and sends it to the right-sized model, reserving frontier engines for the hard tasks.
Customer A/B
Same tasks, same harness
25 min → 8.5 min