Theory of Code Space: Do Code Agents Understand Software Architecture?

Grigory Sapunov

cs.SEFeb 28, 2026

Theory of Code Space: Do Code Agents Understand Software Architecture?

Authors: Grigory Sapunov

Organizations: Intento

Abstract

AI code agents excel at isolated tasks yet struggle with multi-file software engineering requiring architectural understanding. We introduce Theory of Code Space (ToCS), a benchmark that evaluates whether agents can construct, maintain, and update coherent architectural beliefs during codebase exploration. Agents explore procedurally generated codebases under partial observability -- opening files under a budget -- and periodically externalize their belief state as structured JSON, producing a time-series of architectural understanding. Three findings emerge from experiments with four baselines and six frontier LLMs. First, the Active-Passive Gap is model-dependent: one model builds better maps through active exploration than from seeing all files at once, while another shows the opposite -- revealing that active exploration is itself a non-trivial capability absent from some models. Second, retaining structured belief maps in context acts as self-scaffolding for some models but not others, showing that the mechanism is model-dependent. Third, belief state maintenance varies dramatically: a smaller model maintains perfectly stable beliefs across probes while its larger sibling suffers catastrophic belief collapse -- forgetting previously-discovered components between probes. We release ToCS as open-source software. Code: https://github.com/che-shr-cat/tocs

Figures & tables

Metric	Seed 42	Seed 123	Seed 999
Modules	27	30	27
Total edges	70	84	70
imports	47 (67%)	56 (67%)	47 (67%)
calls_api	12 (17%)	15 (18%)	12 (17%)
data_flows_to	5 (7%)	6 (7%)	5 (7%)
registry_wires	6 (9%)	7 (8%)	6 (9%)

Table 1: Generated codebase characteristics (medium complexity, 3 seeds).

Method	Type	Dep F1	Precision	Recall	AUC	Inv F1	Files
Oracle	baseline	1.000	1.000	1.000	—	—	—
Config-Aware	baseline	0.577 $\pm$ .014	0.736	0.475	0.212 $\pm$ .007	0.000	13
Random	baseline	0.538 $\pm$ .028	1.000	0.368	0.142 $\pm$ .008	0.000	13
BFS-Import	baseline	0.293 $\pm$ .060	1.000	0.173	0.079 $\pm$ .003	0.000	13
GPT-5.3-Codex	LLM	0.676 $\pm$ .059	0.782	0.597	0.306 $\pm$ .018	0.739 $\pm$ .067	15
Claude Sonnet 4.6	LLM	0.664 $\pm$ .026	0.983	0.502	0.350 $\pm$ .005	0.778 $\pm$ .025	13

Table 2: Performance over 3 codebases (mean

\pm

half-range ). Baselines use AST parsing; LLM agents use prompted JSON externalization with improved probe prompts (per-type field semantics, pairwise invariant decomposition). Best non-Oracle in bold . Inv F1 uses relaxed matching (§ 3.3 ).

Method	imports	calls_api	data_flows	reg._wires
	( $n{=}150$ )	( $n{=}39$ )	( $n{=}16$ )	( $n{=}19$ )
Config-Aware	0.58	0.00	0.00	1.00
Random	0.55	0.00	0.00	0.00
BFS-Import	0.25	0.00	0.00	0.00
GPT-5.3-Codex	0.69	0.15	0.31	1.00
Claude Sonnet 4.6	0.56	0.15	0.00	1.00

Table 3: Edge type recall by method (aggregated over 3 codebases).

n

= total ground-truth edges of each type. Bold = best non-Oracle recall per type.

Panel of Figure 1 — Figure 1: Exploration analysis across 4 baselines and 6 LLM agents on medium-complexity codebases ( $n{=}3$ , budget $B{=}20$ ).

Figure 2: Belief trajectory: Dependency F1 at each probe step (mean over 3 codebases, shaded = std). GPT-5.3-Codex and Claude Sonnet 4.6 accumulate steadily; Gemini 2.5 Flash grows monotonically. Gemini 2.5 Pro peaks at step 6–9 then collapses; Gemini 3.1 Pro shows high variance (stable on one codebase, collapse on another); Gemini 3 Flash remains flat throughout (recency bias).

Model	Condition	Dep F1	Inv F1	Description
GPT-5.3	Passive-full	0.457 $\pm$ .147	0.757 $\pm$ .028	All files at once
	Passive-replay	0.665 $\pm$ .034	0.752 $\pm$ .036	Same trace, no decisions
	Active	0.676 $\pm$ .059	0.739 $\pm$ .067	Agent chooses actions
	Passive-oracle	0.737 $\pm$ .043	0.683 $\pm$ .091	Oracle-selected files
Gem. 2.5F	Passive-full	0.696 $\pm$ .125	0.696 $\pm$ .120	All files at once
	Passive-replay	0.561 $\pm$ .141	0.665 $\pm$ .037	Same trace, no decisions

Table 4: Active-Passive Gap decomposition (mean

\pm

half-range over 3 codebases,

B{=}20

). APG

{}_{\text{total}}

= passive-full

-

active; APG

{}_{\text{selection}}

= oracle

-

active; APG

{}_{\text{decision}}

= active

-

replay.

Model	Track	Dep F1	Inv F1
GPT-5.3-Codex	Probe-as-scratchpad	0.676 $\pm$ .059	0.739 $\pm$ .067
	No-probe	0.538 $\pm$ .127	0.570 $\pm$ .191
	Probe-only	0.464 $\pm$ .056	0.447 $\pm$ .028
Gemini 2.5 Flash	Probe-as-scratchpad	0.469 $\pm$ .099	0.516 $\pm$ .155
	No-probe	0.480 $\pm$ .161	0.273 $\pm$ .040
	Probe-only	0.401 $\pm$ .143	0.481 $\pm$ .222

Table 5: Probe condition ablation (mean

\pm

half-range over 3 codebases,

B{=}20

). Scratchpad effect = scratchpad

-

no-probe.

Edge	Gemini 2.5 Flash	Claude Sonnet 4.6
cli $\to$ config	IMPORTS	IMPORTS
cli $\to$ runner	IMPORTS	IMPORTS
cli $\to$ config::PipelineConfig	CALLS_API (FP)	—
cli $\to$ runner::run_pipeline	CALLS_API (FP)	—
cli $\to$ runner	—	CALLS_API (TP)

Table 6: Error analysis: edge type confusion on cli.py edges. Gemini reports both IMPORTS and CALLS_API for the same target (symbol-qualified), producing false positives; Claude merges into a single file-level edge.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Method	$B{=}10$	$B{=}15$	$B{=}20$	$B{=}25$
Oracle	1.000	1.000	1.000	1.000
Config-Aware	0.175	0.492	0.577	0.626
Random	0.056	0.317	0.538	0.632
BFS-Import	0.078	0.157	0.293	0.603

Appendix

Table 7: Dependency F1 vs. exploration budget (mean over 3 codebases).

Model	Old Prompt	New Prompt	$\boldsymbol{\Delta}$ F1
Gemini 2.5 Flash	0.328	0.469	+0.141
GPT-5.3-Codex	0.564	0.676	+0.113
Claude Sonnet 4.6	0.639	0.664	+0.025

Appendix

Table 8: Effect of improved probe prompt on Dependency F1 (same 3 codebases, same models, only probe prompt changed).

\Delta

= new

-

old.

Explore similar work

Apr 3, 2026

Inside the Scaffold: A Source-Code Taxonomy of Coding Agent Architectures

LLM-based coding agents can localize bugs, generate patches, and run tests with diminishing human oversight, yet the scaffolding code that surrounds the language model (the control loop, tool definitions, state management, and context strategy) remains poorly understood. Existing surveys classify agents by abstract capabilities (tool use, planning, reflection) that cannot distinguish between architecturally distinct systems, and trajectory studies observe what agents do without examining the scaffold code that determines why. This paper presents a source-code-level architectural taxonomy derived from analysis of 13 open-source coding agent scaffolds at pinned commit hashes. Each agent is characterized across 12 dimensions organized into three layers: control architecture, tool and environment interface, and resource management. The analysis reveals that scaffold architectures resist discrete classification: control strategies range from fixed pipelines to Monte Carlo Tree Search, tool counts range from 0 to 37, and context compaction spans seven distinct strategies. Five loop primitives (ReAct, generate-test-repair, plan-execute, multi-attempt retry, tree search) function as composable building blocks that agents layer in different combinations; 11 of 13 agents compose multiple primitives rather than relying on a single control structure. Dimensions converge where external constraints dominate (tool capability categories, edit formats, execution isolation) and diverge where open design questions remain (context compaction, state management, multi-model routing). All taxonomic claims are grounded in file paths and line numbers, providing a reusable reference for researchers studying agent behavior and practitioners designing new scaffolds.

Benjamin Rombaut

Coding Agents LLM Agent Orchestration LLM Agent Harnesses

Ask PDF

May 5, 2026

ProgramBench: Can Language Models Rebuild Programs From Scratch?

Turning ideas into full software projects from scratch has become a popular use case for language models. Agents are being deployed to seed, maintain, and grow codebases over extended periods with minimal human oversight. Such settings require models to make high-level software architecture decisions. However, existing benchmarks measure focused, limited tasks such as fixing a single bug or developing a single, specified feature. We therefore introduce ProgramBench to measure the ability of software engineering agents to develop software holisitically. In ProgramBench, given only a program and its documentation, agents must architect and implement a codebase that matches the reference executable's behavior. End-to-end behavioral tests are generated via agent-driven fuzzing, enabling evaluation without prescribing implementation structure. Our 200 tasks range from compact CLI tools to widely used software such as FFmpeg, SQLite, and the PHP interpreter. We evaluate 9 LMs and find that none fully resolve any task, with the best model passing 95% of tests on only 3% of tasks. Models favor monolithic, single-file implementations that diverge sharply from human-written code.

John Yang, Kilian Lieret, Jeffrey Ma +9

1Meta FAIR · 4Harvard University · 2Meta TBD +1

Software Engineering Agents Software Engineering Benchmarks LLM-Based Program Synthesis Program Synthesis

Ask PDF

Apr 11, 2026

Formal Architecture Descriptors as Navigation Primitives for AI Coding Agents

AI coding agents spend a substantial fraction of their tool calls on undirected codebase exploration. We investigate whether providing agents with formal architecture descriptors can reduce this navigational overhead. We present three complementary studies. First, a controlled experiment (24 code localization tasks x 4 conditions, Claude Sonnet 4.6, temperature=0) demonstrates that architecture context reduces navigation steps by 33-44% (Wilcoxon p=0.009, Cohen's d=0.92), with no significant format difference detected across S-expression, JSON, YAML, and Markdown. Second, an artifact-vs-process experiment (15 tasks x 3 conditions) demonstrates that an automatically generated descriptor achieves 100% accuracy versus 80% blind (p=0.002, d=1.04), proving direct navigational value independent of developer self-clarification. Third, an observational field study across 7,012 Claude Code sessions shows 52% reduction in agent behavioral variance. A writer-side experiment (96 generation runs, 96 error injections) reveals critical failure mode differences: JSON fails atomically, YAML silently corrupts 50% of errors, S-expressions detect all structural completeness errors. We propose intent.lisp, an S-expression architecture descriptor, and open-source the Forge toolkit.

Ruoqi Jin

Independent Researcher, China

AI Coding Agents Software Engineering

Ask PDF