When the Specification Emerges: Benchmarking Faithfulness Loss in Long-Horizon Coding Agents

Lu Yan; Xuan Chen; Xiangyu Zhang

cs.SEMar 17, 2026

When the Specification Emerges: Benchmarking Faithfulness Loss in Long-Horizon Coding Agents

Authors: Lu Yan, Xuan Chen, Xiangyu Zhang

Organizations: Purdue University

Abstract

Current coding-agent benchmarks usually pro- vide the full task specification upfront. Real research coding often does not: the intended system is progressively disclosed through in- teraction, requiring the agent to track durable design commitments across a long session. We introduce a benchmark for this setting and study faithfulne Ss Loss U nder eM ergent s Pecification (SLUMP), defined as the reduc- tion in final implementation faithfulness un- der emergent specification relative to a single- shot specification control. The benchmark con- tains 20 recent ML papers (10 ICML 2025, 10 NeurIPS 2025), 371 atomic verifiable compo- nents, and interaction scripts of approximately 60 coding requests that progressively disclose the target design without revealing the paper itself. Final repositories are scored with a five-level component-faithfulness rubric and accompanied by an exposure audit to verify that scored components are recoverable from the visible interaction. Evaluated on Claude Code and Codex, the single-shot specification control achieves higher overall implementation fidelity on 16/20 and 14/20 papers, respectively. Structural integration degrades under emergent specification on both platforms, while seman- tic faithfulness loss is substantial on Claude Code and small on Codex. As a mitigation case study, we introduce ProjectGuard, an exter- nal project-state layer for specification tracking. On Claude Code, ProjectGuard recovers 90% of the faithfulness gap, increases fully faith- ful components from 118 to 181, and reduces severe failures from 72 to 49. These results identify specification tracking as a distinct eval- uation target for long-horizon coding agents.

Figures & tables

Figure 1: Cumulative exposure over turns. Left: fraction of components explicitly specified (

R{=}4

). Right: fraction recoverable (

R{\geq}1

). Each line is one paper. The gap between panels reflects the ambiguous middle where components are inferable but not yet fully specified.

Score	Label	Definition
0	Absent	No code attempts this component.
1	Wrong	Code attempts the component but implements logic inconsistent with the specification.
2	Simplified	Code implements a degraded or approximate version of the intended component.
3	Equivalent	Code differs in form but preserves the same functional behavior and system role.
4	Faithful	Code closely matches the intended formulation and architectural realization.

Table 1: Component-level faithfulness rubric used to score final repositories against the frozen benchmark specification.

Metric	A1–Judge	A2–Judge	A1–A2
Weighted $\kappa$ (quadratic)	0.927	0.882	0.930
Exact-match accuracy	91.7%	86.7%	88.3%
Spearman $\rho$	0.938	0.899	0.932
Within $\pm 1$	97.5%	96.7%	99.2%
Boundary disagr. (2–3)	1.8%	4.9%	5.2%
Boundary disagr. (3–4)	3.3%	6.2%	7.9%

Table 2: Pairwise agreement among the automated component judge and two independent annotators (each blind to the other scores) on 120 stratified component-run pairs.

Figure 2: Per-paper IF50 under emergent specification and the single-shot specification control.

Platform / Condition	MCF	DIR	IF50
Claude Code / Emergent	2.718	0.149	0.414
Claude Code / Single-shot	3.031	0.303	0.530
Codex / Emergent	3.242	0.148	0.479
Codex / Single-shot	3.245	0.289	0.550

Table 3: Aggregate faithfulness under emergent specification and the single-shot specification control.

Figure 3: Test pass rate fails to detect SLUMP .

Case	Description
Committed design	Three-stream residual update committed in the visible interaction by turn 40/97: scalar state receives summed EU+SH deltas; coordinates receive an EU delta; SH features receive an SH delta.

Emergent	Final integrated model bypasses the earlier module, drops the coordinate update, and replaces the summed scalar residual with separate h_eu / h_sh paths. Score: Simplified (2/4).

Single-shot	Final model preserves all three residual paths in one layer module, including the coordinate update and summed scalar residual. Score: Faithful (4/4).

Table 4: Representative example of SLUMP on a clearly committed component. The emergent run implements the intended module earlier in the session but does not preserve it in the final integrated pipeline, whereas the single-shot specification control does.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

ID	Venue	Paper Title	# Comp.	# Req.
P1	NeurIPS 2025	ASGO: Adaptive Structured Gradient Optimization	14	60
P2	ICML 2025	AtlasD: Automatic Local Symmetry Discovery	12	49
P3	NeurIPS 2025	Capturing Polysemanticity with PRISM: A Multi-Concept Feature Description Framework	16	60
P4	ICML 2025	Curvature Enhanced Data Augmentation for Regression	12	61
P5	NeurIPS 2025	DualEqui: A Dual-Space Hierarchical Equivariant Network for Large Biomolecules	22	60
P6	ICML 2025	Hierarchical Equivariant Policy via Frame Transfer	14	57

Appendix

Table 5: Benchmark paper list. The sample comprises 10 ICML 2025 and 10 NeurIPS 2025 papers selected at random. Component counts range from 10 to 49 (mean 18.6); interaction scripts range from 48 to 61 requests (mean 57.4).

Figure 4: Default long-horizon coding workflow. User requests, tool outputs, and code edits accumulate in a single live conversation, with older context periodically condensed into a compact summary. This representation preserves recent interaction state but provides only a weak project-level view of earlier design commitments and repository structure, making specification tracking under emergent specification difficult.

		P1	P2	P3	P4	P5	P6	P7	P8	P9	P10	P11	P12	P13	P14	P15	P16	P17	P18	P19	P20	Mean
MCF Rec. (%)	CC	+22	$-$ 0	$-$ 100	+150	+70	+167	+250	+100	+33	+94	+5	+36	$-$ 200	$-$ 75	+0	$-$ 0	+111	+0	+200	+150	+51
MCF Rec. (%)	Codex	$-$ 600	—	$-$ 20	+33	$-$ 56	—	+29	$-$ 100	+75	$-$ 24	$-$ 8	+32	$-$ 45	$-$ 33	+11	+67	+80	+57	$-$ 20	+200	$-$ 18
DIR $\Delta$ (%)	CC	$-$ 47	$-$ 10	+48	+1125	+120	+955	+24	$-$ 58	+13	+1	+7	+24	$-$ 22	+19	$-$ 8	+7	+83	+12	$-$ 54	—	+118
DIR $\Delta$ (%)	Codex	+15	+67	+49	+54	+55	$-$ 71	$-$ 52	+61	—	+269	$-$ 23	+122	+58	+171	+21	+240	+292	+19	$-$ 9	+99	+76
IF50 Rec. (%)	CC	+118	$-$ 13	+5	+482	+27	+92	+142	$-$ 11	+80	+104	$-$ 8	+73	+355	$-$ 6	+19	+13	+41	$-$ 42	+15	+119	+80
IF50 Rec. (%)	Codex	$-$ 138	$-$ 109	+14	+60	$-$ 55	+66	$-$ 71	+102	+82	+92	+5	+77	+19	+29	$-$ 39	+141	+86	$-$ 9	$-$ 9	+68	+21

Appendix

Table 6: Per-paper ProjectGuard effectiveness across 20 benchmark papers. MCF Recovery and IF50 Recovery are expressed as percentages of the specification bottleneck gap (full-paper minus emergent) closed by ProjectGuard. DIR

\Delta

is the relative increase in dependency integration ratio. Memory is the external state size maintained by ProjectGuard.

\Delta

Compaction is the change in compaction events per session. Positive results (recovery or integration gain) are highlighted . CC = Claude Code; Papers are indexed P1–P20; — indicates a zero denominator (no specification gap or no baseline integration).

Figure 5: ProjectGuard overview. Before each coding turn, the system combines the request history with the current repository skeleton to maintain two external views of the project: a semantic state of committed but revisable project knowledge, and a structural state of files, symbols, and interface relations. A forecaster then prepares a project-aware brief for the coding agent, highlighting relevant design commitments, compatibility constraints, and candidate modules for revision or reuse. When the remaining live context is unlikely to support the upcoming request, ProjectGuard can trigger a proactive restart and inject the rendered project state into a fresh session. After the turn completes, both semantic and structural state are updated from the new interaction and repository changes.

AI Coding Agents

AI coding agents use language models and software tools to plan, edit, test, and maintain code with varying levels of autonomy.

See all 326 papers

SpecGuardBench: Can Coding Agents Construct Reliable Validation Harnesses from Natural Language Specifications?
Oct 12, 2026Zhiqing Zhong, Mingyu Chen, Aoyang Fang +5AI Coding Agents Specification-Based Testing
The Chinese University of Hong Kong, Shenzhen · Aalto University · Lero, the Research Ireland Centre for Software +2
LoLBench: Evaluating Coding Agents with Long-Horizon Proposals on Large Software Systems
Sep 29, 2026Yun Peng, Zihan Wu, Zeyang Zhuang +6Software Engineering AI Coding Agents
Fudan University, China · City University of Hong Kong, Hong Kong · Chinese University of Hong Kong, Hong Kong +5
EvoCode-Bench: Evaluating Coding Agents in Multi-Turn Iterative Interactions
May 22, 2026Haiyang Shen, Xuanzhong Chen, Wendong Xu +3AI Coding Agents Software Engineering Benchmarks
1UniPat AI · 2Peking University · 3Tsinghua University +1

Oct 12, 2026J/K move · Enter open · S save

SpecGuardBench: Can Coding Agents Construct Reliable Validation Harnesses from Natural Language Specifications?

Zhiqing Zhong, Mingyu Chen, Aoyang Fang, Yuzhong Zhang+4

The Chinese University of Hong Kong, Shenzhen · Aalto University · Lero, the Research Ireland Centre for Software+2

AI Coding Agents Specification-Based Testing Automated Test Generation Software Engineering Benchmarks

Open paper AskPDF arXiv

As coding agents become more capable, tests remain the primary evidence for judging whether their code is correct. Yet passing available tests does not establish that an implementation satisfies its natural language specification. Reliable validation requires checks that faithfully capture the specification and scenarios that thoroughly exercise the code, but current benchmarks provide limited insight into whether agents can construct such evidence themselves. We introduce SpecGuardBench, a benchmark of 291 tasks from 13 real-world repositories, covering 8 types of verification logic such as functional postconditions and interaction protocols. Given a natural language specification, repository context, and a reference implementation, an agent constructs a validation harness comprising executable checks and test scenarios. The checks should reject implementations that violate the specification while accepting those that satisfy it, and the scenarios should thoroughly exercise the code. Across four proprietary and open-weight coding agents, none constructs a harness scored as faithful on the benchmark's implementations for more than 63% of the tasks. The bottleneck lies less in exploring test scenarios than in constructing checks that faithfully reflect the specification. Specifically, the agents show a clear tendency to write checks that mirror the reference implementation rather than the specification, and therefore reject implementations that the specification permits.

Abstract

AI Coding Agents

Explore similar work

SpecGuardBench: Can Coding Agents Construct Reliable Validation Harnesses from Natural Language Specifications?

LoLBench: Evaluating Coding Agents with Long-Horizon Proposals on Large Software Systems

EvoCode-Bench: Evaluating Coding Agents in Multi-Turn Iterative Interactions