We tested 30 AI models to measure the 
Sonnet 5.5 is better than Sonnet 5 and spends a quarter of the effort doing it
When a new model comes out, the question we care about is not a leaderboard number. It’s what happens when someone on the team opens Claude Code on a Tuesday and asks it to fix something in a real service. So when Sonnet 5.5 shipped, we ran it through the same benchmark of real coding work we use for everything else and put it next to Sonnet 5.
It won every task. It also cost 71% less per session and finished in about a third of the time. We expected an improvement. We didn’t expect one that large on every axis at once, so we spent some time making sure it was real.
The numbers
Across our full set of tasks, each model at Claude Code’s default setting:
| Score (out of 50) | Cost per session | Time per session | |
|---|---|---|---|
| Sonnet 5, default (high effort) | 33.3 | $2.65 | 10.7 min |
| Sonnet 5.5, default (medium effort) | 38.5 | $0.78 | 3.8 min |
Task by task:
| Task | Sonnet 5 | Sonnet 5.5 |
|---|---|---|
| Long session: rate limiting in a Slack bot | 32.8 | 36.5 |
| Long session: trace id across two services | 32.6 | 40.1 |
| Vague bug report | 35.2 | 38.0 |
| Test-first fix | 40.8 | 42.0 |
| Design review with pushback | 27.4 | 35.9 |
| Quick fix | 31.1 | 38.4 |
The biggest gain: not folding under pushback
The largest jump is on the design review. In that session the agent reviews a pull request, and then the author replies that the problem it flagged “doesn’t matter in practice.” The right move is to go back to the code and check, because the author is wrong: the code shows the two workflows really do collide. In two of its four runs, Sonnet 5 agreed with the author without checking. Sonnet 5.5 pushed back every time, citing the specific lines that show the collision, and scored 38 to 41 on that turn against Sonnet 5’s 20 to 35. Being agreeable is a nice quality in a colleague and a bad one in a reviewer.
The smallest gain is on the test-first fix, which is the most mechanical task in the set and where Sonnet 5 was already decent.
Why it’s so much cheaper
We checked the obvious explanation first, and it isn’t the price. Going by what Claude Code billed, the two models cost about the same per token: both read from the prompt cache at around 20 cents per million tokens, Sonnet 5.5 is a little cheaper on output, and a little more expensive on writing to the cache.
The difference is how much work each one does. On the long sessions, Sonnet 5 averaged about 92,000 thinking tokens and 96 model requests per session. Sonnet 5.5 averaged about 25,000 thinking tokens and 41 requests. Each request takes about the same time, around ten seconds, so fewer requests is also where the speed comes from.
So the model that does less gets the higher score. More thinking only helps if it’s aimed at the right question, and the design review above is a good picture of the difference: Sonnet 5 did plenty of work in that session, just not the one check that mattered.
Two things that changed under us
We learned more about moving targets this week than we wanted to.
First, Claude Code updated itself in the middle of our Sonnet 5.5 runs, from version 2.1.283 to 2.1.284. The new version changed what it sends Sonnet 5.5: the default effort went from high to medium, and it switched to the shorter system prompt that Opus uses. We threw out the runs that straddled the update and reran everything on the new version with auto-update turned off. If you benchmark coding agents, pin the version and pass the effort level explicitly.
Second, a curious one. On the morning of September 26, the model answering as Sonnet 5 in our runs behaved quite differently from the Sonnet 5 we saw that afternoon and since: about half as many requests per session, very little thinking, and scores around 40 instead of 33. That afternoon it changed, and it has stayed changed. The morning version’s working style looks almost exactly like Sonnet 5.5’s. We can’t prove anything from the outside, since the API reported the model as Sonnet 5 throughout, but if you saw Sonnet 5 look unusually good for a few hours that week, you weren’t imagining it.
How we tested
The benchmark is a battery of agentic coding tasks taken from real work on our own services. Some are short: a vague “it isn’t working” bug report, a test-first fix, a design review where the author pushes back, a quick fix. Others are long multi-turn sessions, where one Claude Code conversation goes from finding the relevant code, to analysing how something fails, to planning and implementing a fix, and then auditing that fix. Every model gets the same fixed prompts, and we ran each task four times per model. This comparison alone used several hundred million tokens.
Each turn is scored out of 50 against an answer key, which lists the facts a correct answer needs, false claims earlier answers have made, and what a good plan or implementation has to do. Every entry in the keys was checked against the code. The grader is Opus 5.5, reading the answer, the code and the key.
Each model ran at Claude Code’s default, since that’s what most people use: high effort for Sonnet 5 and medium for Sonnet 5.5.
What we’d tell a team using Sonnet
Move to Sonnet 5.5 and leave it at the default. It was better on every task we measured, costs well under a third as much, and makes you wait about a third as long. Because it’s already lean at medium effort, the usual trick of turning effort down to save money doesn’t buy much: at low effort it saved us only 9% and cost almost a point. Differences this size are well outside the run-to-run noise we see, which is about a point or two.
