AI Architect tops SWE-Bench Pro
Deep codebase context lifts coding agent task success and cuts token cost on large, real-world codebases.
Claude Opus 4.6
Without context
with codebase context
up to 68% ↓ on individual tasks
Even advanced coding agents resolve fewer than 52% of tasks when changes span large codebases and require coordinated multi-file updates. Most of the token cost on those tasks goes to navigation, with agents reading hundreds of files to find where the change belongs.
The Context Lab ran identical agent runs on SWE-Bench Pro with and without Bito’s AI Architect MCP enabled, measuring how deep codebase context shapes both task success and token cost.
Performance gains increase sharply with complexity
Large repositories, multi-file changes, and long-horizon tasks see the biggest lift—where agents must reason across dependencies, not just edit isolated code.
Large codebases
Biggest gains on repositories with 1.5M+ lines of code.
Multi-file changes
Tasks spanning 10+ files see sharply higher success.
Critical issues
Performance, security, and cross-component issues see the strongest lift.
Task resolution by file-change complexity
As tasks span more files, standalone models drop off sharply, while AI Architect continues to resolve complex changes.
Higher success. Half token cost.
AI Architect cuts token cost per task by 47% on substantial tasks, with reductions up to 68% on the heaviest ones.
Fewer reasoning steps
Average runs drop from ~75 to ~30 steps per task.
Fewer tool calls
Fewer searches, less navigation, less re-exploration.
Fewer file reads
Agent opens only the files relevant to the change.
How it plays out on real tasks
Two examples from the evaluation. A large refactor the baseline failed. A routine update where the baseline burned 80 steps in a search spiral.
On a complex refactoring task in the webClients repository (~720MB), the change required coordinating fragmented calendar logic across utilities, recurrence rules, alarms, encryption, and mail integrations—spanning 412 files.
With deep codebase context, the agent completed the refactor successfully, delivering 58K+ lines of code changes, passing all tests, while the baseline Claude Opus 4.6 agent failed to complete the task.
- Task completed successfully
- All tests passed
- Task not completed
- Agent failed to coordinate changes across the system
On the Flipt codebase, the task was to add audit-configuration reporting to the service’s anonymous telemetry, a routine engineering change.
With codebase context, the agent consulted the repository map and made its first code edit at step 14. The baseline agent took around 80 steps to ship the same fix, looping through repeated file reads and dead-end searches before landing the edit.
- First code edit at step ~14
- Same fix, same tests passing
- First code edit at step ~80
- Same fix after 60+ steps of search and re-reads
How AI Architect works
Models generate code. Systems require reasoning.
AI Architect builds a knowledge graph from your code, commits, issues, docs, and past decisions, then delivers deep system context across your engineering workflow, including grounded coding via MCP. Coding agents reason across dependencies and system impact, unlocking measurably higher success on complex engineering tasks.
Evaluated on SWE-Bench Pro.
In collaboration with The Context Lab*
No code storage or model training. End-to-end data encryption. Enterprise-ready.
Trusted by leading engineering teams at:
*Note: This evaluation was conducted by The Context Lab, an independent 3rd party that performs agent evaluations in a tightly controlled and measurement environment.