Developers evaluate AI coding tools on velocity. Architects evaluate them on drift over eighteen months. The two evaluations almost never agree, and the cost of getting yours wrong shows up in audit findings, postmortems that name the agent, and refactoring quarters you didn’t plan for. AI tools for software architects need to model the implicit architecture (decisions, intent, runtime), not just the explicit code, and most don’t.
The six concerns below are the places where the file-level view fails, the six questions to ask the vendor before the pilot, and the six signals to watch once the agent is running on your code. Codebase intelligence is the technical requirement underneath all of it.
Why the two evaluations diverge
A developer using an AI coding tool measures it on their own work: did the suggestion compile, did it pass tests, did it ship faster than writing it by hand. The cost of a bad suggestion is a few minutes of correction. The cost of a good one is a few minutes saved. The math is simple and it pencils out fast.
Your math runs on different inputs. You’re not evaluating the tool on individual edits. You’re evaluating it on what those edits look like at scale, across multiple teams, over several quarters. Architectural drift is the failure mode, and it doesn’t show up in any individual PR. It shows up at the system level, after hundreds of “reasonable” edits accumulate.
The implication is structural: the developer evaluation will almost always come back positive. The drift only becomes visible to you, six quarters later, when the codebase carries patterns nobody chose deliberately. At that point, the cost of the tool decision compounds across audit findings, postmortem patterns, refactoring quarters, and onboarding time.
So your evaluation matters. The criteria are different.
Six concerns developers won’t think to check
The six concerns below come back to the same structural gap: AI tools for software architects need to see the system that the files live inside, not just the files themselves. An AI coding tool that reads files and retrieves similar files by embedding similarity can’t do that. Each concern is a place where the file-level view fails.
Pattern adherence
The agent should reuse the team’s established patterns, not invent parallel ones. This fails for a straightforward reason: the agent’s training data contains thousands of implementations of common patterns, and without explicit knowledge of which one your team adopted, the agent picks the most common public pattern or the one closest to the local file context.
Each developer accepting a suggestion accepts a slightly different pattern. Six months later, you have three retry implementations in one service, four pagination patterns across services, two error-wrapping conventions in the same codebase.
The cost compounds in specific places: incidents take longer to debug because each pattern fails differently, new engineers can’t predict how any service behaves, architecture review time goes to reconciling patterns instead of building.
What to check during evaluation: when the agent generates code, does it reference the existing pattern in your codebase, or does it write a new one from training data.
Shared library reuse
Shared library discovery requires the agent to know the cross-repo or cross-service structure of your internal packages. File-level retrieval pulls in similar code chunks, not the actual exported functions from your shared/auth package. The agent doesn’t know your shared package exists unless something explicitly tells it.
So you end up with six implementations of the same auth utility, none of them maintained, all carrying the same security update lag. Each new CVE in the underlying library means six places to patch with different owners. Security audits start flagging duplicated security-sensitive code. Engineering capacity goes to reconciliation instead of features.
Check whether the agent surfaces your internal packages during code generation, or whether it writes standalone implementations.
Service boundary respect
The agent should route through canonical service contracts, not bypass them for convenience. Cross-service awareness requires knowing which services exist, which contracts they expose, and which boundaries the team has explicitly designed around. An agent that sees only the current file knows none of this. It sees a database table or a file path and uses the shortest path to the data, which often means crossing a boundary the team set up specifically to prevent that.
Failure looks like:
- Cross-service database reads
- Direct file-system access into other services
- Shared state that bypasses the API contracts the team designed around
The implication is operational: service teams lose the ability to independently update schemas, deploys become coordinated even when they shouldn’t be, future migrations get blocked because external readers depend on internal implementation details.
When the agent solves a cross-service problem, watch whether it respects the boundary or works around it.
Dependency hygiene
Most coding agents check whether a new package compiles, not whether the team already has functionality for that purpose. Each new dependency adds a supply chain risk and a maintenance liability your team inherits forever. The drift compounds across small additions that individually look reasonable.
A year in, the build pulls in twelve packages doing roughly the same thing. Each new CVE requires twelve evaluations. Build times degrade. Security audits flag the duplicate dependency footprint. Onboarding gets harder because new engineers don’t know which library to use.
What to ask the vendor: does the agent check against your existing dependency tree before suggesting new packages, and does the vendor have any opinion on dependency drift at all.
Convention enforcement
Conventions are implicit unless something encodes them. File-level retrieval picks up some conventions from neighbouring code, but the agent has to distinguish between patterns your team adopted deliberately and one-off choices in the local file. Without that distinction, the agent’s output gradually averages toward the generic patterns from its training data.
One service uses your conventions, another uses generic patterns from training data. The inconsistency surfaces when a new engineer takes six weeks to ramp instead of two. Code review takes longer because reviewers context-switch between conventions. Refactoring becomes high-risk because the conventions aren’t reliable. The codebase loses the navigability that took years to build.
Check whether new files generated by the agent match the structural conventions across your codebase, not just the file the agent is editing.
Decision rationale preservation
This is the hardest concern to evaluate, and the most important. The rationale doesn’t live in code at all. The defensive check the agent removes was there because of an incident two years ago, captured in a postmortem the agent never read. The abstraction the agent flattens was there because of a requirement captured in a Jira ticket the agent never saw. An agent that only reads code has no access to any of this.
The agent will remove things confidently:
- A defensive check, as “unnecessary”
- An abstraction layer, as “unnecessary complexity”
- A redundant pattern, as a simplification
Each one was load-bearing for a reason that lived in a Jira ticket, a design doc, or someone’s head. The cost shows up later, often badly: incidents trace back to checks the agent stripped months ago, compliance violations surface where regulatory logic got removed, postmortems start naming the agent as the proximate cause of failures the team didn’t think were possible.
The single question that matters here: does the agent have access to the why behind the code, or only to what the code does today.
All six come back to the same gap.
Why this comes down to codebase intelligence
The agent doesn’t understand your codebase as a system. That’s the whole problem.
A real codebase intelligence layer models four signal classes, not just code:
- Code is what exists today: repositories, services, modules, APIs, architectural patterns. Most AI coding tools have some version of this through file retrieval or embedding indexes.
- Business context is the reasoning behind decisions: Jira and Linear tickets, incident history, the record of what the team decided and why.
- Tribal knowledge is the intent behind the design: Confluence pages, design documents, the Slack threads where the real argument happened. This is where decision rationale actually lives.
- Runtime is what happens in production: observability data, deployment configuration, database schemas, message-queue topologies. The reality of what runs versus what was designed.
Strip any one and the agent fails in a specific way. Without business context, the agent reinvents conventions the team already settled. Without tribal knowledge, the agent ships code that conflicts with directions the team committed to in design docs. Without runtime, the agent reasons about how the system should behave, not how it does.
A tool that models all four keeps the codebase coherent at velocity. A tool that models one or two accelerates drift, because the agent works fast and confidently within an incomplete picture. The drift compounds faster than your review process can catch it, because the review process is also operating on the same incomplete information.
The full mechanism is in Why AI coding tools break architecture in large codebases. For chief architects, what matters is that all four inputs have to be present and indexed for the agent to operate above the file layer.
The tribal knowledge problem is particularly relevant because chief architects are usually the ones carrying it. AI compressed coding from weeks to hours but technical design still runs on tribal knowledge covers what happens when coding velocity goes up but design context doesn’t follow.
For enterprise codebases, the implicit architecture is where the architecture actually lives. AI tools for software architects need to model it, not just the explicit code.
How to evaluate, before and during the pilot
Two phases: questions you bring to the vendor, signals you watch once the agent is running.
Questions to ask the vendor
Each question targets the underlying architecture of the tool. The vendor’s answer tells you whether they’ve built a real codebase intelligence layer or wrapped a coding model in marketing.
- How does the tool model relationships across services? If the answer is “large context window,” that’s not modelling, that’s loading. You want a code graph or structural dependency model: typed relationships between services, files, and functions that the agent can traverse rather than retrieve as text chunks. Get this wrong and drift compounds at velocity with no structural model to catch it.
- How does it handle multi-repo setups? Polyrepo or monorepo, the agent has to reason across service boundaries either way. In a polyrepo, the dependency between two services is a Git commit referencing a contract in another repo. Single-repo indexing misses it. In a monorepo, the agent’s context window still can’t hold the full estate, so cross-cutting queries have to be modelled explicitly. Get this wrong and every cross-service task generates the same drift you’ve been trying to avoid.
- What signals beyond code does it use? Tickets, design docs, runtime data. “Code only” means the agent has no access to decision rationale or operational reality. Get this wrong and defensive code gets stripped, architectural decisions get undone, postmortems start naming the agent.
- How does it handle cross-service planning? This is where most coding agents fall apart, as covered in Why Claude Code plan mode falls apart on real codebases. Ask specifically. A real answer will mention dependency tracing, deployment ordering, and rollback paths. Get this wrong and cross-service changes become high-risk, your team starts routing the hard work around the agent.
- How does the model update? If indexing happens once at onboarding and never refreshes, the model goes stale fast. You want incremental updates as code merges, ticket changes, and deployments. A stale model drifts further than no model at all, because the agent works confidently from outdated assumptions.
- What’s the failure mode when wrong? Every coding agent will be wrong sometimes. Loud failure (asks for clarification) gives your team the chance to catch it. Silent failure (ships confidently incorrect code) doesn’t. For an architect, silent failure is the dangerous one because it accumulates as production incidents nobody can attribute to the tool decision.
Vague answers tell you the tool is selling itself rather than describing itself.
Signals to watch in the first week
Once the agent is running on your code, six observations distinguish a tool with system-level reasoning from one with file-layer fluency:
- Pattern reuse: Does the agent reference existing patterns in your codebase by name when generating new code, or does it produce a generic implementation. If generic, drift starts in week one and compounds across every team using the tool.
- Shared library imports: When the agent solves a problem your shared libraries already cover, does it import from your packages or write standalone code. If standalone, security exposure compounds with every new agent-generated implementation.
- Cross-service routing: Hand the agent a task that touches a service boundary. Watch whether it proposes a service call through the canonical contract, or whether it suggests a shortcut. If shortcut, operational coupling builds and future migrations get harder.
- Deployment awareness: When the agent proposes a cross-service change, does it surface deployment ordering and rollback considerations as part of the plan. If not, cross-service deploys become risk events your team has to mitigate manually.
- Sensitivity to defensive code: Hand the agent a task in an area with defensive checks or compliance-sensitive logic. Watch whether it preserves the defensive structure or simplifies it as “cleaner.” If it strips, compliance violations and incidents will trace back to agent edits.
- Reference to non-code context: Does the agent ever cite a ticket, a design doc, or an incident postmortem when explaining a suggestion. If no, the tool is operating in the architectural blind spot you can’t afford to scale across the org.
A week of these signals tells you what a year of developer evaluation never will.
Conclusion
The AI coding tool you approve will define what the codebase looks like in three years.
A tool that operates at the file layer authorises every engineer in the org to write code without architectural context. The drift compounds at the speed of your team’s velocity. The cost surfaces in specific places your leadership team already tracks: audit findings on duplicated security-sensitive code, postmortems that trace incidents to stripped defensive logic, compliance reviews that flag regulatory code that disappeared in a routine refactor, onboarding times that degrade from weeks to months, refactoring quarters that consume capacity you planned for features.
You’re the only one in the room who can see this coming. The developers will report on daily tasks. The vendor will demo features. You’re the one who can see what the codebase looks like when those tasks accumulate across all your teams over the next two product cycles.
The broader category of architectural reasoning is in How AI coding agents understand software architecture. The evaluation is yours. The cost of getting it wrong is everyone’s.
