Current coding-agent benchmarks usually pro- vide the full task specification upfront. Real research coding often does not: the intended system is progressively disclosed through in- teraction, requiring the agent to track durable design commitments across a long session. We introduce a benchmark for this setting and study faithfulne Ss Loss U nder eM ergent s Pecification (SLUMP), defined as the reduc- tion in final implementation faithfulness un- der emergent specification relative to a single- shot specification control. The benchmark con- tains 20 recent ML papers (10 ICML 2025, 10 NeurIPS 2025), 371 atomic verifiable compo- nents, and interaction scripts of approximately 60 coding requests that progressively disclose the target design without revealing the paper itself. Final repositories are scored with a five-level component-faithfulness rubric and accompanied by an exposure audit to verify that scored components are recoverable from the visible interaction. Evaluated on Claude Code and Codex, the single-shot specification control achieves higher overall implementation fidelity on 16/20 and 14/20 papers, respectively. Structural integration degrades under emergent specification on both platforms, while seman- tic faithfulness loss is substantial on Claude Code and small on Codex. As a mitigation case study, we introduce ProjectGuard, an exter- nal project-state layer for specification tracking. On Claude Code, ProjectGuard recovers 90% of the faithfulness gap, increases fully faith- ful components from 118 to 181, and reduces severe failures from 72 to 49. These results identify specification tracking as a distinct eval- uation target for long-horizon coding agents.
Figures & tables
Figure 1: Cumulative exposure over turns. Left: fraction of components explicitly specified ( R=4 ). Right: fraction recoverable ( R≥1 ). Each line is one paper. The gap between panels reflects the ambiguous middle where components are inferable but not yet fully specified.
Score
Label
Definition
0
Absent
No code attempts this component.
1
Wrong
Code attempts the component but implements logic inconsistent with the specification.
2
Simplified
Code implements a degraded or approximate version of the intended component.
3
Equivalent
Code differs in form but preserves the same functional behavior and system role.
4
Faithful
Code closely matches the intended formulation and architectural realization.
Table 1: Component-level faithfulness rubric used to score final repositories against the frozen benchmark specification.
Metric
A1–Judge
A2–Judge
A1–A2
Weighted κ (quadratic)
0.927
0.882
0.930
Exact-match accuracy
91.7%
86.7%
88.3%
Spearman ρ
0.938
0.899
0.932
Within ±1
97.5%
96.7%
99.2%
Boundary disagr. (2–3)
1.8%
4.9%
5.2%
Boundary disagr. (3–4)
3.3%
6.2%
7.9%
Table 2: Pairwise agreement among the automated component judge and two independent annotators (each blind to the other scores) on 120 stratified component-run pairs.
Figure 2: Per-paper IF50 under emergent specification and the single-shot specification control.
Platform / Condition
MCF
DIR
IF50
Claude Code / Emergent
2.718
0.149
0.414
Claude Code / Single-shot
3.031
0.303
0.530
Codex / Emergent
3.242
0.148
0.479
Codex / Single-shot
3.245
0.289
0.550
Table 3: Aggregate faithfulness under emergent specification and the single-shot specification control.
Figure 3: Test pass rate fails to detect SLUMP .
Case
Description
Committed design
Three-stream residual update committed in the visible interaction by turn 40/97: scalar state receives summed EU+SH deltas; coordinates receive an EU delta; SH features receive an SH delta.
Emergent
Final integrated model bypasses the earlier module, drops the coordinate update, and replaces the summed scalar residual with separate h_eu / h_sh paths. Score: Simplified (2/4).
Single-shot
Final model preserves all three residual paths in one layer module, including the coordinate update and summed scalar residual. Score: Faithful (4/4).
Table 4: Representative example of SLUMP on a clearly committed component. The emergent run implements the intended module earlier in the session but does not preserve it in the final integrated pipeline, whereas the single-shot specification control does.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
ID
Venue
Paper Title
# Comp.
# Req.
P1
NeurIPS 2025
ASGO: Adaptive Structured Gradient Optimization
14
60
P2
ICML 2025
AtlasD: Automatic Local Symmetry Discovery
12
49
P3
NeurIPS 2025
Capturing Polysemanticity with PRISM: A Multi-Concept Feature Description Framework
16
60
P4
ICML 2025
Curvature Enhanced Data Augmentation for Regression
12
61
P5
NeurIPS 2025
DualEqui: A Dual-Space Hierarchical Equivariant Network for Large Biomolecules
22
60
P6
ICML 2025
Hierarchical Equivariant Policy via Frame Transfer
14
57
Appendix
Table 5: Benchmark paper list. The sample comprises 10 ICML 2025 and 10 NeurIPS 2025 papers selected at random. Component counts range from 10 to 49 (mean 18.6); interaction scripts range from 48 to 61 requests (mean 57.4).
Figure 4: Default long-horizon coding workflow. User requests, tool outputs, and code edits accumulate in a single live conversation, with older context periodically condensed into a compact summary. This representation preserves recent interaction state but provides only a weak project-level view of earlier design commitments and repository structure, making specification tracking under emergent specification difficult.
P1
P2
P3
P4
P5
P6
P7
P8
P9
P10
P11
P12
P13
P14
P15
P16
P17
P18
P19
P20
Mean
MCF Rec. (%)
CC
+22
− 0
− 100
+150
+70
+167
+250
+100
+33
+94
+5
+36
− 200
− 75
+0
− 0
+111
+0
+200
+150
+51
Codex
− 600
—
− 20
+33
− 56
—
+29
− 100
+75
− 24
− 8
+32
− 45
− 33
+11
+67
+80
+57
− 20
+200
− 18
DIR Δ (%)
CC
− 47
− 10
+48
+1125
+120
+955
+24
− 58
+13
+1
+7
+24
− 22
+19
− 8
+7
+83
+12
− 54
—
+118
Codex
+15
+67
+49
+54
+55
− 71
− 52
+61
—
+269
− 23
+122
+58
+171
+21
+240
+292
+19
− 9
+99
+76
IF50 Rec. (%)
CC
+118
− 13
+5
+482
+27
+92
+142
− 11
+80
+104
− 8
+73
+355
− 6
+19
+13
+41
− 42
+15
+119
+80
Codex
− 138
− 109
+14
+60
− 55
+66
− 71
+102
+82
+92
+5
+77
+19
+29
− 39
+141
+86
− 9
− 9
+68
+21
Appendix
Table 6: Per-paper ProjectGuard effectiveness across 20 benchmark papers. MCF Recovery and IF50 Recovery are expressed as percentages of the specification bottleneck gap (full-paper minus emergent) closed by ProjectGuard. DIR Δ is the relative increase in dependency integration ratio. Memory is the external state size maintained by ProjectGuard. Δ Compaction is the change in compaction events per session. Positive results (recovery or integration gain) are highlighted . CC = Claude Code; Papers are indexed P1–P20; — indicates a zero denominator (no specification gap or no baseline integration).
Figure 5: ProjectGuard overview. Before each coding turn, the system combines the request history with the current repository skeleton to maintain two external views of the project: a semantic state of committed but revisable project knowledge, and a structural state of files, symbols, and interface relations. A forecaster then prepares a project-aware brief for the coding agent, highlighting relevant design commitments, compatibility constraints, and candidate modules for revision or reuse. When the remaining live context is unlikely to support the upcoming request, ProjectGuard can trigger a proactive restart and inject the rendered project state into a fresh session. After the turn completes, both semantic and structural state are updated from the new interaction and repository changes.
As coding agents become more capable, tests remain the primary evidence for judging whether their code is correct. Yet passing available tests does not establish that an implementation satisfies its natural language specification. Reliable validation requires checks that faithfully capture the specification and scenarios that thoroughly exercise the code, but current benchmarks provide limited insight into whether agents can construct such evidence themselves. We introduce SpecGuardBench, a benchmark of 291 tasks from 13 real-world repositories, covering 8 types of verification logic such as functional postconditions and interaction protocols. Given a natural language specification, repository context, and a reference implementation, an agent constructs a validation harness comprising executable checks and test scenarios. The checks should reject implementations that violate the specification while accepting those that satisfy it, and the scenarios should thoroughly exercise the code. Across four proprietary and open-weight coding agents, none constructs a harness scored as faithful on the benchmark's implementations for more than 63% of the tasks. The bottleneck lies less in exploring test scenarios than in constructing checks that faithfully reflect the specification. Specifically, the agents show a clear tendency to write checks that mirror the reference implementation rather than the specification, and therefore reject implementations that the specification permits.
Zhiqing Zhong, Mingyu Chen, Aoyang Fang +5
The Chinese University of Hong Kong, Shenzhen · Aalto University · Lero, the Research Ireland Centre for Software +2
Modern coding agents can deliver increasingly large repository-level changes, and recent benchmarks reflect this by emphasizing long-horizon tasks with large reference implementations. Many benchmarks evaluate coding agents' implementation capability to produce correct code edits from detailed specifications. However, practical modular development tasks also require the perception capability of grounding user intent and high-level design to derive a specification. We introduce LoLBench to evaluate both capabilities through the entire proposal-to-implementation process on large software systems. It is a multilingual benchmark of 100 tasks across 29 software systems in five domains. Each task provides a human-written enhancement proposal with user intent and high-level design. On average, proposals contain about 5,000 words, software systems contain 2.4 million source lines of code (LoC), and implementation pull requests (PRs) change approximately 5,500 LoC. Across 28 agents we evaluated, the best agent resolves only 14% of tasks and achieves a 52.7% Fail-to-Pass (F2P) pass rate. Failure analysis identifies incomplete code localization as a major bottleneck, while providing reference-derived file trees alongside API specifications improves resolved rates by 16--22 percentage points (2.4--17×), reaching at most 34%. These results show that both perception and implementation remain central challenges for coding agents in practical modular development on large software systems. LoLBench is available at https://huggingface.co/datasets/lolbench26/LoLBench.
Yun Peng, Zihan Wu, Zeyang Zhuang +6
Fudan University, China · City University of Hong Kong, Hong Kong · Chinese University of Hong Kong, Hong Kong +5
Coding agents are increasingly used as iterative development partners, but most benchmarks still evaluate one specification followed by one final assessment. This leaves out a basic question: can an agent keep its own codebase working as requirements change? We introduce EvoCode-Bench, a benchmark of 26 stateful coding tasks and 227 evaluated rounds. Each task preserves the agent's workspace for 5-15 rounds, states requirements through observable behavior, and uses cumulative executable tests to check new requirements and still-active prior ones. We evaluate 13 coding agents with two metrics: MT@4, a four-attempt fail-stop multi-round score, and SR, a single-round score from a reference-completed prior state. For most agents, SR exceeds MT@4 by 22-40 points. The gap also changes rankings: the highest-SR agent (78.9) ranks only third in persistent execution (44.0 MT@4). Even the strongest agents achieve only about 50% success on multi-turn metrics, and aggregate pass rate drops below half of round-1 performance by round 5. Failure analysis shows tier-dependent behavior: weaker agents fail early, while stronger agents survive long enough to expose specification-tracking and regression failures. We release the benchmark data and Harbor multi-turn infrastructure.
Haiyang Shen, Xuanzhong Chen, Wendong Xu +3
1UniPat AI · 2Peking University · 3Tsinghua University +1