Architecture as Capability Equalizer for Coding Agents

Arquimedes Canedo

cs.SEAug 22, 2026

Architecture as Capability Equalizer for Coding Agents

Authors: Arquimedes Canedo

Organizations: Siemens Digital Industries Software Princeton, NJ, USA

Abstract

LLM-based coding agents generate complete software systems from high-level descriptions, yet little is known about how the format of architecture specifications affects the quality of generated code or whether this effect depends on model capability. We present a controlled experiment comparing five informationally equivalent specification formats (informal prose, Mermaid diagrams with constraints and ADRs, OpenAPI, C4/Structurizr DSL, and TypeScript interface contracts with ArchUnit-style rules) across six models from three vendor families (Anthropic Claude, OpenAI GPT, Google Gemini). Across 90 multi-turn agent trials, specification format shows a strong format x model interaction. On the strongest models (Sonnet 4.6, GPT-5), format barely matters (quality spread 0.17-0.92). On weaker models, format produces spreads of 0.83-2.42 points, with code-proximate formats (OpenAPI, TypeScript contracts) recovering most of the capability gap. Mid-tier models can consume more tokens than frontier models for worse output when they enter compilation debugging loops that stronger models avoid. Self-validation rates collapse from 100% (Sonnet) to 0% (Gemini Flash) across the capability spectrum. TypeScript contracts triple API route coverage for the weakest model (33% to 100%). Structured architecture specifications serve as a capability equalizer, with value inversely proportional to model strength and the largest returns for cost-optimized deployments.

Figures & tables

Panel of Figure 1 — Fig. 1: Experimental pipeline. Five architecture specification formats are provided to 6 LLMs across 3 vendor families. Each multi-turn agent session produces a TypeScript codebase evaluated through three independent channels: an LLM judge (4 dimensions, 1–10 scale), automated verification (route coverage, constraint compliance), and process metrics (token cost, compilation, demo run). A separate no-architecture baseline validates the premise (§ VII ).

Format	Specification Excerpt
Prose	Services should not call each other directly. Instead, we use a simple in-memory Event Bus — a publish/subscribe system. When something notable happens, the originating service publishes an event to the Event Bus. Other services that care about those events subscribe to them and react accordingly.
Mermaid + Constr.	graph TD TaskSvc –¿—publish— EventBus[ Event Bus ] CommentSvc –¿—publish— EventBus EventBus –¿—subscribe— NotifSvc UserSvc — UserStore[( User Store )] % Constraint 1: Services MUST NOT import or call % other services directly.
OpenAPI 3.0	paths : /tasks/{id}/assign : put : summary : Assign task to user requestBody : schema : properties : assigneeId : { type : string } responses : ’200’ : { $ref : Task }
C4 / Structurizr	workspace { model { taskService = container ”Task Service” { taskStore = component ”Task Store” ”Map¡string,Task¿” taskOps = component ”Task Operations” } taskService -¿ eventBus ”Publishes task.assigned” commentService -¿ eventBus ”Publishes comment.added” eventBus -¿ notificationService ”Delivers events to” } }
TS Contracts	interface ITaskService { create(input: CreateTaskInput ): Task ; assign(taskId: string , assigneeId: string ): Task ; changeStatus(taskId: string , newStatus: TaskStatus ): Task ; } RULE 1 : NO_CROSS_SERVICE_IMPORTS Files matching ”services/-service.ts” MUST NOT import from other ”services/-service.ts”

TABLE I: The Same Architecture in Five Formats. Each excerpt shows how inter-service communication is specified. All encode identical constraints; only the representation differs.

Condition	Trial	Arch	Comp	Qual	Constr	Overall	Tokens (K)	Turns	LoC	Errors	Violations
		Judge Scores					Resource Consumption				Constr.
Prose	T1	8	9	9	8	8.50	367	23	1,823	2	0
	T2	8	9	9	8	8.50	602	28	1,810	3	0
	T3	7	9	9	8	8.25	332	24	1,424	2	0
Mermaid + Constr. + ADRs	T1	9	10	9	10	9.50	1,735	42	1,576	4	1
	T2	8	9	9	9	8.75	412	24	1,332	2	0

TABLE II: Sonnet 4.6 — Complete Trial Results: Judge Scores (1–10), Resource Consumption, and Constraint Violations for All 15 Trials. Best trial, Worst trial. Arch = Architectural Adherence, Comp = Completeness, Qual = Code Quality, Constr = Constraint Compliance, LoC = Lines of Code, Errors = Tool call errors during agent session.

Metric	Prose	Mermaid + Constr.	OpenAPI	C4	TS Contracts
Judge Scores (1–10 scale, higher is better)
Architectural Adherence	$7.67$	$7.67$	$\mathbf{8.00}$	$7.67$	$7.67$
Completeness	$9.00$	$\mathbf{9.33}$	$9.00$	$\mathbf{9.33}$	$9.00$
Code Quality	$\mathbf{9.00}$	$8.67$	$\mathbf{9.00}$	$8.33$	$\mathbf{9.00}$
Constraint Compliance	$8.00$	$\mathbf{8.33}$	$7.67$	$8.00$	$8.00$
Overall	$8.42$	$\mathbf{8.50}$	$8.42$	$8.33$	$8.42$

TABLE III: Sonnet 4.6 — Aggregated Results by Condition (mean

\pm

range across 3 trials; bold = best in row)

Condition	Trial	Arch	Comp	Qual	Constr	Overall	Tok. (K)	Turns	LoC	Viol.
		Judge Scores					Resources			Constr.
Prose	T1	4	6	7	4	5.25	558	30	1,319	7
	T2	5	6	6	4	5.25	506	26	1,334	1
	T3	6	7	7	5	6.25	867	38	1,350	2
Mermaid + Constr. + ADRs	T1	5	7	7	4	5.75	535	29	1,618	2
	T2	8	7	8	9	8.00	573	30	1,398	0

TABLE IV: Haiku 4.5 — Complete Trial Results (judged by Sonnet 4.6; GPT-5 re-judging confirmed consistent rankings,

r=0.60

, see §VII-E)

Format	Sonnet 4.6	Haiku 4.5	GPT-5	GPT-5-mini	Gem. Pro	Gem. Flash
	Anthropic		OpenAI		Google
Prose	$8.42$	$5.58$	$7.33$	$6.08$	$5.83$	$5.75$
Mermaid + Constr.	$\mathbf{8.50}$	$6.42$	$7.17$	$\mathbf{6.92}$	$6.42$	$6.00$
OpenAPI	$8.42$	$\mathbf{7.25}$	$6.58$	$6.83$	$\mathbf{6.92}$	$6.50$
C4 / Structurizr	$8.33$	$6.42$	$6.83$	$6.75$	$6.67$	$\mathbf{6.83}$
TS Contracts	$8.42$	$7.08$	$\mathbf{7.50}$	$6.83$	$4.50$	$6.67$

TABLE V: Format

\times

Model Interaction: Mean Overall Scores Across 6 Models and 3 Vendor Families (90 trials). Format spread = max

-

min overall score within each model. Larger spread = format matters more. Bold = best format per model.

Panel of Figure 7 — Fig. 2: Format $\times$ model interaction heatmap (scores rounded to 1 decimal). Darker cells indicate higher scores. The two frontier models (Sonnet, GPT-5) show uniform color regardless of format; non-frontier models show wide variation — the “capability equalizer” effect.

Model	TSC	TSC Pass	Demo Run	Demo
	Attempts	Rate	Rate	Fail
Sonnet 4.6	5.1	53%	100%	0.4
Haiku 4.5	8.6	61%	80%	0.5
GPT-5	4.1	13%	53%	0.7
GPT-5-mini	4.7	17%	40%	0.3
Gemini 2.5 Pro	3.7	22%	20%	0.3

TABLE VI: Agent Self-Validation Behavior by Model. TSC = TypeScript compiler invocations. Demo Run Rate = fraction of trials that executed the demo script at least once. All values are means across conditions.

Format	Sonnet	Haiku	GPT-mini	Gem. Fl.
Prose	15%	20%	11%	33%
Mermaid+Constr.	36%	9%	17%	20%
OpenAPI	15%	10%	8%	13%
C4	28%	16%	—	23%
TS Contracts	23%	17%	—	—

TABLE VII: File Rewrite Rate (% of writes that are overwrites). A dash indicates fewer than 3 file writes in that condition.

Format	Route	Compl	Route	Compl	Route	Compl	Route	Compl	Route	Compl	Route	Compl
	Anthropic				OpenAI				Google
	Sonnet		Haiku		GPT-5		GPT-mini		Gem. Pro		Gem. Flash
Prose	100	100	100	88	100	100	85	97	100	93	33	71
Mermaid+C.	100	97	100	93	100	100	100	98	100	95	33	73
OpenAPI	100	100	100	97	100	100	100	100	100	100	33	72
C4	100	97	100	97	100	100	100	98	67	85	67	85

TABLE VIII: Automated Architecture Verification by Model and Format. All values are percentages. Route = API route coverage (25 routes). Compl = weighted compliance (Components 20%, Communication 20%, Constraints 20%, Routes 20%, Data 10%, Structure 10%). Covers

\sim

80% of the architecture.

Model	Tokens (K)	Overall	Score/100K
Sonnet 4.6	640	8.42	1.32
GPT-5	248	7.08	2.85
Gemini Pro	329	6.07	1.85
Haiku 4.5	735	6.50	0.88
GPT-5-mini	225	6.72	2.99
Gemini Flash	223	6.35	2.85

TABLE IX: Resource Efficiency by Model

Metric	No Arch	Prose	$\Delta$
Judge Scores (1–10)
Arch. Adherence	$3.00$	$7.67$	$-4.67$
Completeness	$7.00$	$9.00$	$-2.00$
Code Quality	$7.33$	$9.00$	$-1.67$
Constraint Compliance	$3.00$	$8.00$	$-5.00$
Overall	$5.08$	$8.42$	$-3.34$

TABLE X: Architecture Guidance vs. Requirements Only (Sonnet 4.6, 3 trials each). No Arch = functional requirements only; Prose = lightest architecture specification. Both use identical tooling.

Panel of Figure 13 — Fig. 3: The capability equalizer effect. Under prose (light bars), quality drops steeply from frontier to weaker models. The best structured format per model (dark bars) lifts non-frontier scores, narrowing the gap from $2.8$ to $1.7$ points.

Dimension	Auto Metric	Sonnet ( $n$ =90)	GPT-5 ( $n$ =28)
Completeness	Route coverage	$0.63$	$0.85$
Constraint Compl.	Violations	$-0.33$	$-0.48$
Constraint Compl.	Compliance %	$0.21$	$-0.02$
Arch. Adherence	Compliance %	$0.15$	$0.15$

TABLE XI: Judge–Automation Correlation (Pearson

r

)

Dimension	Sonnet mean	GPT-5 mean	Bias	$r$
Arch. Adherence	$6.53$	$7.03$	$+0.50$	$0.51$
Completeness	$7.37$	$7.70$	$+0.33$	$0.60$
Code Quality	$7.60$	$7.17$	$-0.43$	$0.52$
Constraint Compl.	$6.50$	$7.77$	$+1.27$	$0.37$
Overall	$7.00$	$7.42$	$+0.42$	$0.47$

TABLE XII: Inter-Judge Agreement: Sonnet 4.6 vs GPT-5 (

n

=30). Bias = GPT-5 mean

-

Sonnet mean. Positive = GPT-5 more lenient.

Generator Family	Sonnet judge	GPT-5 judge	Bias
Anthropic ( $n$ =10)	$7.55$	$8.22$	$+0.67$
OpenAI ( $n$ =8)	$6.94$	$7.94$	$+1.00$
Google ( $n$ =10)	$6.60$	$7.67$	$+1.08$

TABLE XIII: Same-Family Bias Test: GPT-5

-

Sonnet Overall Score

Panel of Figure 17 — Fig. 4: Specification format decision guide. Frontier models (Sonnet, GPT-5) need no structured specification. Mid-tier models benefit from OpenAPI or TypeScript contracts. Small models require TypeScript contracts for complete route coverage.

Explore similar work

Formal Architecture Descriptors as Navigation Primitives for AI Coding Agents
Apr 11, 2026Ruoqi JinAI Coding Agents Software Engineering
Independent Researcher, China
How Generation Architecture Shapes Code Complexity in Multi-Agent LLM Systems: A Paired Study on HumanEval
May 29, 2026Nazmus AshrafiSoftware Engineering Code Generation Evaluation
Independent Researcher
Verus-SpecGym: An Agentic Environment for Evaluating Specification Autoformalization
May 26, 2026Anmol Agarwal, Natalie Neamtu, Pranjal Aggarwal +6LLM Agent Evaluation Formal Verification
CMU · Amazon · Work performed while at Amazon.

Apr 11, 2026J/K move · Enter open · S save

Formal Architecture Descriptors as Navigation Primitives for AI Coding Agents

Ruoqi Jin

Independent Researcher, China

AI Coding Agents Software Engineering

Open paper AskPDF arXiv

AI coding agents spend a substantial fraction of their tool calls on undirected codebase exploration. We investigate whether providing agents with formal architecture descriptors can reduce this navigational overhead. We present three complementary studies. First, a controlled experiment (24 code localization tasks x 4 conditions, Claude Sonnet 4.6, temperature=0) demonstrates that architecture context reduces navigation steps by 33-44% (Wilcoxon p=0.009, Cohen's d=0.92), with no significant format difference detected across S-expression, JSON, YAML, and Markdown. Second, an artifact-vs-process experiment (15 tasks x 3 conditions) demonstrates that an automatically generated descriptor achieves 100% accuracy versus 80% blind (p=0.002, d=1.04), proving direct navigational value independent of developer self-clarification. Third, an observational field study across 7,012 Claude Code sessions shows 52% reduction in agent behavioral variance. A writer-side experiment (96 generation runs, 96 error injections) reveals critical failure mode differences: JSON fails atomically, YAML silently corrupts 50% of errors, S-expressions detect all structural completeness errors. We propose intent.lisp, an S-expression architecture descriptor, and open-source the Forge toolkit.

Abstract

Explore similar work

Formal Architecture Descriptors as Navigation Primitives for AI Coding Agents

How Generation Architecture Shapes Code Complexity in Multi-Agent LLM Systems: A Paired Study on HumanEval

Verus-SpecGym: An Agentic Environment for Evaluating Specification Autoformalization