Summary
This case study comes from the SWE-Bench Pro Evaluation, an independent benchmark conducted by The Context Lab that tests AI coding agents on real-world codebases. It shows how, in a ProtonMail webclients task where the local code had partially deviated from a reference architecture, Claude Sonnet 4.5 (baseline agent) patched the visible symptoms and shipped an incomplete fix, while Claude Opus 4.6 augmented with deep codebase context from Bito’s AI Architect compared local files against the indexed reference, identified the missing structural pieces, and delivered a complete patch that passed all tests.
The challenge
The PassAliases drawer in ProtonMail’s Security Center had several rendering bugs. Aliases sometimes failed to display, the empty state did not appear when it should have, and the “Get an alias” button did not reliably open the modal. On the surface, the task looked like three concrete UI defects waiting for surgical fixes.
The hidden complexity sat below the symptoms. The local checkout had drifted into a partially broken intermediate state relative to the reference architecture. The provider had not been split into its setup hook and provider component, error handling had not been generalized into a MISSING_SCOPE enum, and several helper files including PassAliasesProvider.helpers.ts and SecurityCenter/constants.ts were missing entirely.
Patching the visible bugs produced a partial fix that addressed each defect on its own terms. Restoring the reference architecture produced a complete fix that the verifier would actually accept.
Why the baseline agent failed
Claude Sonnet 4.5 explored the directory carefully. It performed 30 file reads across PassAliases.tsx, PassAliasesProvider.tsx, PassAliasesContainer.tsx, interface.ts, usePassAliasesProviderSetup.ts, AliasesList.tsx, and the modal forms. From that exploration, it correctly diagnosed three concrete bugs.
First, hasAliases was computed from passAliasesItems.length, while the UI filtered out items without aliasEmail, creating an inconsistency. Second, the “Get an alias” callback captured a stale hasAliases closure value. Third, AliasesList rendered a partial empty UI with a headline and no items instead of returning null.
The agent applied 5 surgical edits across usePassAliasesProviderSetup.ts, PassAliases.tsx, and AliasesList.tsx to fix each diagnosed bug. It then ran roughly 10 bash commands trying to execute jest, all of which failed with sh: 1: jest: not found and Unrecognized configuration: enableLegacyPeerDeps. Without the ability to verify, the agent wrote summary notes and stopped.
The patch worked at the bug level. Each fix addressed a real defect. The verifier, however, was checking a broader architectural state, including the existence of PassAliasesProvider.helpers.ts, the MISSING_SCOPE enum, the customImage prop on GenericError, and the split between usePassAliasesProviderSetup and PassAliasesProvider. Sonnet 4.5 had no way to know any of this existed in the reference state, so its patch was orthogonal to what the verifier was actually checking. All tests failed.
⚠ ROOT CAUSE: The baseline agent operated on the symptoms it could observe in the local code, not on the gap between the local code and the reference state. When the bug is a partial reversion of a previous change, surgical fixes leave the broader architectural drift untouched.
How Bito’s AI Architect solved it
Bito’s AI Architect builds a knowledge graph of the codebase, a structured map of files, symbols, and their indexed reference content. The agent can ask the graph not only “where is this defined” but also “what does this file look like in the reference,” then compare against the local copy.
For this task, Claude Opus 4.6 made 15 calls to Bito’s AI Architect, almost every one of them an explicit local-versus-indexed comparison. The pattern shows up clearly in the purpose parameter the agent passed to each call.
In call 3, the agent asked for getCode(file="PassAliases.tsx", lines=1-100) to compare the indexed version with the local version and identify rendering bugs in the drawer. In call 5, it requested getCode(file="usePassAliasesProviderSetup.ts", lines=1-100) to see the original separate setup hook against the merged version it had locally. In call 11, it pulled getCode(file="PassAliasesProvider.tsx", lines=1-30) to see the indexed provider, which should be lean and context-only.
The agent was not asking which file to edit. It was asking what the file should look like, then diffing against local to surface what had been removed or changed.
What each Architect call surfaced
| Call | Asked for | What the indexed version revealed (vs local) |
|---|---|---|
| 1 | searchSymbols("PassAliases") | All PassAliases symbols including PassAliasesProvider.helpers.ts, missing locally |
| 2 | searchCode("PassAliases") | Component tree references including SecurityCenter/constants.ts, missing locally |
| 3, 4 | getCode(PassAliases.tsx, lines 1-200) | The reference component is significantly leaner than the local version |
| 5–7 | getCode(usePassAliasesProviderSetup.ts, lines 1-225) | A dedicated hook with initPassBridge extracted, locally merged into the provider |
| 8 | getCode(PassAliasesProvider.helpers.ts) | This file should exist and contain fetchPassAliases, locally absent |
| 9 | getCode(interface.ts) | The interface should include PassAliasesProviderReturnedValues, locally incomplete |
| 10 | searchCode("DRAWER_PASS_ALIASES_CREATE_ALIAS_MODAL_CTA_ID") | Constant exists in indexed code, missing locally |
| 11 | getCode(PassAliasesProvider.tsx, lines 1-30) | The reference provider is around 30 lines, the local one bloated with merged setup logic |
| 12, 13 | searchCode("MISSING_SCOPE") and getCode(errors.ts) | Reference has a MISSING_SCOPE enum value the local code lacks |
| 14 | getCode(bridge/types.ts) | Reference PassBridge types include createDefaultVault, locally absent |
| 15 | getCode(GenericError.tsx) | Reference GenericErrorDisplay carries a customImage prop, locally absent |
After 15 calls, the agent had a precise inventory of what had been removed from or had never existed in the local checkout, all relative to the indexed reference.
The patch the architect evidence produced
Claude Opus 4.6 made 25 file operations across 8 source files, addressing each gap the Architect surfaced.
It created packages/components/components/drawer/views/SecurityCenter/PassAliases/PassAliasesProvider.helpers.ts and packages/components/components/drawer/views/SecurityCenter/constants.ts, both missing locally. It rewrote usePassAliasesProviderSetup.ts to extract the hook from the merged provider and rewrote PassAliasesProvider.tsx down to a lean context provider. It edited packages/pass/lib/api/errors.ts to add the MISSING_SCOPE enum value, updated interface.ts with the missing return-value interface, added the customImage prop to GenericError.tsx, and edited PassAliases.tsx to use the corrected dependencies.
Sonnet 4.5 had touched 3 of these 8 files. Opus 4.6 with Bito’s AI Architect touched all 8 because the comparison evidence had revealed that all 8 needed attention.
KEY ARCHITECTURAL INSIGHT
When local code drifts from a reference state, the bug surface and the architectural surface are not the same problem. Bito’s AI Architect lets the agent retrieve the indexed version of files it already has locally, turning debugging into a reference-diff exercise. That capability sits behind getCode, the tool Opus 4.6 reached for 11 times in this task and the one that turned a partial patch into a complete one.
Head-to-head comparison
| Claude Sonnet 4.5 (baseline agent) | Claude Opus 4.6 + Bito’s AI Architect | |
|---|---|---|
| Code Exploration | 30 local file reads, surgical bug hunt | 15 Architect calls, 11 of them getCode for reference comparison |
| What was found | 3 visible bugs in 3 files | 3 bugs plus 5 missing or drifted architectural pieces across 8 files |
| Files Modified | 3 files, 5 surgical edits | 8 files, 25 operations covering creates and rewrites |
| Architectural Gap | Not detected, the local-versus-reference drift was invisible | Fully mapped, every missing helper, enum, prop, and interface surfaced |
| Task Outcome | FAILED, surgical patch was orthogonal to verifier checks | PASSED, full architectural restoration matched the reference state |
Conclusion
The difference between a failed patch and a passed patch on this task came down to a single capability, the ability to ask the codebase what a file should look like, then diff against what is locally present.
Claude Sonnet 4.5 fixed every bug it could observe and stopped, because every tool it had access to operated on the local checkout. The drift between the local code and the reference architecture sat outside its visible surface area, so its patch addressed defects that were real but incomplete relative to the verifier.
Claude Opus 4.6 with Bito’s AI Architect operated in a different mode entirely. It treated the indexed knowledge graph as a working reference oracle, chaining searchSymbols to locate candidate files and getCode to retrieve their indexed content, then comparing each against the local version. That compare-against-reference workflow surfaced missing helpers, an absent enum value, a dropped prop, and an incomplete interface, all of which were structural and none of which were visible from local exploration alone.
For engineering teams, this pattern shows up everywhere code drifts. Partial refactors land halfway, bug fixes revert previous changes, intermediate states ship into branches and stay there. In those situations, the bottleneck is not generation, it is the agent’s access to a reliable reference state. Bito’s AI Architect closes that gap by giving coding agents grounding in your code and operational history, so the agent can tell the difference between a symptom and an architectural drift, and patch accordingly.
