Theory of Code Space: Do Code Agents Understand Software Architecture?

Grigory Sapunov

cs.SEFeb 28, 2026

Theory of Code Space: Do Code Agents Understand Software Architecture?

Authors: Grigory Sapunov

Organizations: Intento

Abstract

AI code agents excel at isolated tasks yet struggle with multi-file software engineering requiring architectural understanding. We introduce Theory of Code Space (ToCS), a benchmark that evaluates whether agents can construct, maintain, and update coherent architectural beliefs during codebase exploration. Agents explore procedurally generated codebases under partial observability -- opening files under a budget -- and periodically externalize their belief state as structured JSON, producing a time-series of architectural understanding. Three findings emerge from experiments with four baselines and six frontier LLMs. First, the Active-Passive Gap is model-dependent: one model builds better maps through active exploration than from seeing all files at once, while another shows the opposite -- revealing that active exploration is itself a non-trivial capability absent from some models. Second, retaining structured belief maps in context acts as self-scaffolding for some models but not others, showing that the mechanism is model-dependent. Third, belief state maintenance varies dramatically: a smaller model maintains perfectly stable beliefs across probes while its larger sibling suffers catastrophic belief collapse -- forgetting previously-discovered components between probes. We release ToCS as open-source software. Code: https://github.com/che-shr-cat/tocs

Figures & tables

Metric	Seed 42	Seed 123	Seed 999
Modules	27	30	27
Total edges	70	84	70
imports	47 (67%)	56 (67%)	47 (67%)
calls_api	12 (17%)	15 (18%)	12 (17%)
data_flows_to	5 (7%)	6 (7%)	5 (7%)
registry_wires	6 (9%)	7 (8%)	6 (9%)

Table 1: Generated codebase characteristics (medium complexity, 3 seeds).

Method	Type	Dep F1	Precision	Recall	AUC	Inv F1	Files
Oracle	baseline	1.000	1.000	1.000	—	—	—
Config-Aware	baseline	0.577 $\pm$ .014	0.736	0.475	0.212 $\pm$ .007	0.000	13
Random	baseline	0.538 $\pm$ .028	1.000	0.368	0.142 $\pm$ .008	0.000	13
BFS-Import	baseline	0.293 $\pm$ .060	1.000	0.173	0.079 $\pm$ .003	0.000	13
GPT-5.3-Codex	LLM	0.676 $\pm$ .059	0.782	0.597	0.306 $\pm$ .018	0.739 $\pm$ .067	15
Claude Sonnet 4.6	LLM	0.664 $\pm$ .026	0.983	0.502	0.350 $\pm$ .005	0.778 $\pm$ .025	13

Table 2: Performance over 3 codebases (mean

\pm

half-range ). Baselines use AST parsing; LLM agents use prompted JSON externalization with improved probe prompts (per-type field semantics, pairwise invariant decomposition). Best non-Oracle in bold . Inv F1 uses relaxed matching (§ 3.3 ).

Method	imports	calls_api	data_flows	reg._wires
	( $n{=}150$ )	( $n{=}39$ )	( $n{=}16$ )	( $n{=}19$ )
Config-Aware	0.58	0.00	0.00	1.00
Random	0.55	0.00	0.00	0.00
BFS-Import	0.25	0.00	0.00	0.00
GPT-5.3-Codex	0.69	0.15	0.31	1.00
Claude Sonnet 4.6	0.56	0.15	0.00	1.00

Table 3: Edge type recall by method (aggregated over 3 codebases).

n

= total ground-truth edges of each type. Bold = best non-Oracle recall per type.

Panel of Figure 1 — Figure 1: Exploration analysis across 4 baselines and 6 LLM agents on medium-complexity codebases ( $n{=}3$ , budget $B{=}20$ ).

Figure 2: Belief trajectory: Dependency F1 at each probe step (mean over 3 codebases, shaded = std). GPT-5.3-Codex and Claude Sonnet 4.6 accumulate steadily; Gemini 2.5 Flash grows monotonically. Gemini 2.5 Pro peaks at step 6–9 then collapses; Gemini 3.1 Pro shows high variance (stable on one codebase, collapse on another); Gemini 3 Flash remains flat throughout (recency bias).

Model	Condition	Dep F1	Inv F1	Description
GPT-5.3	Passive-full	0.457 $\pm$ .147	0.757 $\pm$ .028	All files at once
	Passive-replay	0.665 $\pm$ .034	0.752 $\pm$ .036	Same trace, no decisions
	Active	0.676 $\pm$ .059	0.739 $\pm$ .067	Agent chooses actions
	Passive-oracle	0.737 $\pm$ .043	0.683 $\pm$ .091	Oracle-selected files
Gem. 2.5F	Passive-full	0.696 $\pm$ .125	0.696 $\pm$ .120	All files at once
	Passive-replay	0.561 $\pm$ .141	0.665 $\pm$ .037	Same trace, no decisions

Table 4: Active-Passive Gap decomposition (mean

\pm

half-range over 3 codebases,

B{=}20

). APG

{}_{\text{total}}

= passive-full

-

active; APG

{}_{\text{selection}}

= oracle

-

active; APG

{}_{\text{decision}}

= active

-

replay.

Model	Track	Dep F1	Inv F1
GPT-5.3-Codex	Probe-as-scratchpad	0.676 $\pm$ .059	0.739 $\pm$ .067
	No-probe	0.538 $\pm$ .127	0.570 $\pm$ .191
	Probe-only	0.464 $\pm$ .056	0.447 $\pm$ .028
Gemini 2.5 Flash	Probe-as-scratchpad	0.469 $\pm$ .099	0.516 $\pm$ .155
	No-probe	0.480 $\pm$ .161	0.273 $\pm$ .040
	Probe-only	0.401 $\pm$ .143	0.481 $\pm$ .222

Table 5: Probe condition ablation (mean

\pm

half-range over 3 codebases,

B{=}20

). Scratchpad effect = scratchpad

-

no-probe.

Edge	Gemini 2.5 Flash	Claude Sonnet 4.6
cli $\to$ config	IMPORTS	IMPORTS
cli $\to$ runner	IMPORTS	IMPORTS
cli $\to$ config::PipelineConfig	CALLS_API (FP)	—
cli $\to$ runner::run_pipeline	CALLS_API (FP)	—
cli $\to$ runner	—	CALLS_API (TP)

Table 6: Error analysis: edge type confusion on cli.py edges. Gemini reports both IMPORTS and CALLS_API for the same target (symbol-qualified), producing false positives; Claude merges into a single file-level edge.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Method	$B{=}10$	$B{=}15$	$B{=}20$	$B{=}25$
Oracle	1.000	1.000	1.000	1.000
Config-Aware	0.175	0.492	0.577	0.626
Random	0.056	0.317	0.538	0.632
BFS-Import	0.078	0.157	0.293	0.603

Appendix

Table 7: Dependency F1 vs. exploration budget (mean over 3 codebases).

Model	Old Prompt	New Prompt	$\boldsymbol{\Delta}$ F1
Gemini 2.5 Flash	0.328	0.469	+0.141
GPT-5.3-Codex	0.564	0.676	+0.113
Claude Sonnet 4.6	0.639	0.664	+0.025

Appendix

Table 8: Effect of improved probe prompt on Dependency F1 (same 3 codebases, same models, only probe prompt changed).

\Delta

= new

-

old.

Explore similar work

Inside the Scaffold: A Source-Code Taxonomy of Coding Agent Architectures
Apr 3, 2026Benjamin RombautCoding Agents LLM Agent Orchestration
ProgramBench: Can Language Models Rebuild Programs From Scratch?
May 5, 2026John Yang, Kilian Lieret, Jeffrey Ma +9Software Engineering Agents Software Engineering Benchmarks
1Meta FAIR · 4Harvard University · 2Meta TBD +1
Formal Architecture Descriptors as Navigation Primitives for AI Coding Agents
Apr 11, 2026Ruoqi JinAI Coding Agents Software Engineering
Independent Researcher, China

Apr 3, 2026J/K move · Enter open · S save

Inside the Scaffold: A Source-Code Taxonomy of Coding Agent Architectures

Benjamin Rombaut

Coding Agents LLM Agent Orchestration LLM Agent Harnesses

Open paper AskPDF arXiv

LLM-based coding agents can localize bugs, generate patches, and run tests with diminishing human oversight, yet the scaffolding code that surrounds the language model (the control loop, tool definitions, state management, and context strategy) remains poorly understood. Existing surveys classify agents by abstract capabilities (tool use, planning, reflection) that cannot distinguish between architecturally distinct systems, and trajectory studies observe what agents do without examining the scaffold code that determines why. This paper presents a source-code-level architectural taxonomy derived from analysis of 13 open-source coding agent scaffolds at pinned commit hashes. Each agent is characterized across 12 dimensions organized into three layers: control architecture, tool and environment interface, and resource management. The analysis reveals that scaffold architectures resist discrete classification: control strategies range from fixed pipelines to Monte Carlo Tree Search, tool counts range from 0 to 37, and context compaction spans seven distinct strategies. Five loop primitives (ReAct, generate-test-repair, plan-execute, multi-attempt retry, tree search) function as composable building blocks that agents layer in different combinations; 11 of 13 agents compose multiple primitives rather than relying on a single control structure. Dimensions converge where external constraints dominate (tool capability categories, edit formats, execution isolation) and diverge where open design questions remain (context compaction, state management, multi-model routing). All taxonomic claims are grounded in file paths and line numbers, providing a reusable reference for researchers studying agent behavior and practitioners designing new scaffolds.

Abstract

Explore similar work

Inside the Scaffold: A Source-Code Taxonomy of Coding Agent Architectures

ProgramBench: Can Language Models Rebuild Programs From Scratch?

Formal Architecture Descriptors as Navigation Primitives for AI Coding Agents