We tested 30 AI models to measure the 
Opus 5.5 does more with less, and it shows up in the bill
We spend a lot of time watching coding agents work on our own code, and over the past few weeks we built a comprehensive benchmark out of that habit, mostly so we could stop arguing from vibes. When Opus 5.5 came out, the obvious thing to do was to run it next to Opus 5 on the same work and see what actually changed.
Opus 5.5 scored higher, finished in about half the time, and cost about a third as much.
The numbers
Across our full set of tasks, each model at Claude Code’s default setting:
| Score (out of 50) | Cost per session | Time per session | |
|---|---|---|---|
| Opus 5, default (high effort) | 40.3 | $7.23 | 16.5 min |
| Opus 5.5, default (medium effort) | 42.7 | $2.71 | 8.5 min |
Task by task:
| Task | Opus 5 | Opus 5.5 |
|---|---|---|
| Long session: rate limiting in a Slack bot | 36.1 | 44.5 |
| Long session: trace id across two services | 39.6 | 42.1 |
| Vague bug report | 38.5 | 40.8 |
| Test-first fix | 44.1 | 45.0 |
| Design review with pushback | 41.2 | 42.9 |
| Quick fix | 42.1 | 41.0 |
Opus 5 won the quick fix by about a point, which is inside the noise we see between runs of the same model, so we’d call it a tie. Everywhere else Opus 5.5 was ahead, and on the rate-limiting session it was ahead by eight points.
Where the eight points came from
The rate-limiting session has a trap in it. The service’s Redis client has three states that matter: connected, disconnected, and reconnecting. When Redis goes down and the client is reconnecting, it still reports itself as open, and requests quietly queue up with no timeout. So “what happens to a real user’s event when Redis is down” has a subtle answer, and the last turn of the session asks the agent to audit its own fix by walking through exactly that.
Opus 5 walked only the easy failure path. In all four of its audits of that session it traced the disconnected case, declared things fine, and never looked at the reconnecting state. The grader’s note on one of them says the audit “never looks at the reconnecting state.” Its audit turns scored 26 to 30. Opus 5.5’s scored 38 to 42, and the grader’s only complaint about Redis in any of them was that one audit lumped the connection states into a single row instead of tracing each. That one habit accounts for most of the gap on that task.
Where the money went
Part of it is price. Going by what Claude Code billed for each session, Opus 5.5 costs less per token than Opus 5: $4 per million input tokens against $5, $20 per million output tokens against $25, and 20 cents per million for reading from the prompt cache against 50 cents. That last one matters more than it looks. An agent re-reads the whole conversation from the cache on every request, so a long session reads millions of cached tokens, and a cheaper cache read adds up.
The rest is doing fewer things. On the long sessions Opus 5 made about 107 model requests per session, against 87 for Opus 5.5. It wasn’t that Opus 5.5 thought less. It actually used slightly more thinking tokens per session (about 53,000 against 49,000). It just thought and then did fewer things.
The vague bug report shows it most clearly. Opus 5 spent an average of $13 and 25 minutes per session on it. Opus 5.5 spent $3.13 and just over 8 minutes, and scored slightly higher. If you’ve ever watched someone search every log file on a machine before checking the one the error message names, you know the shape of it.
How we tested
The benchmark is a battery of agentic coding tasks taken from real work on our own services. Some are short: a vague “it isn’t working” bug report, a test-first fix, a design review where the author pushes back on your comments, a quick fix. Others are long multi-turn sessions of about 20 minutes, where one Claude Code conversation goes from finding the relevant code, to analysing how something fails, to planning and implementing a fix, and then auditing that fix. Every model gets the same fixed prompts, and we ran each task several times per model. This comparison alone used several hundred million tokens, most of it the agents re-reading their own conversations.
Each turn is graded on its own against an answer key: the facts a correct answer has to get right, false claims that earlier answers have made, and what a good plan, implementation or audit needs to do. Every entry in the keys was checked against the actual code before we used it. The grader is Opus 5.5 reading the answer, the code and the key, and it scores five things out of 10 each (grounding, correctness, completeness, specificity, usefulness), so a turn is out of 50.
Yes, the grader is one of the models being tested. We think the answer keys keep it honest, because it’s checking claims against code rather than judging style, but you should know. It also marked Opus 5.5 down on plenty of turns, which is reassuring in the way a strict referee is reassuring.
Both models ran at Claude Code’s default settings, because that’s what most people get: high effort for Opus 5, medium for Opus 5.5.
What we take from it
If you’re on Opus 5 today, moving to Opus 5.5 at its default is an easy call: better answers, half the wait, about a third of the cost. The gain is biggest on long sessions where a subtle failure mode decides whether the fix works, which is also where a wrong answer costs you the most. We’ll keep rerunning as new versions ship, since, as we learned this month, models and their defaults can change under you without much warning.
