Coding-agent benchmarks often describe a task in natural language but grade it with tests taken from the upstream project. Passing those tests can require information the description never states: a constant's name, a module path, an exact message. We call this a specification-oracle gap: where it exists, a benchmark also measures whether an agent reproduces upstream implementation details. We formalise the gap, including the inconsistencies that chains of dependent tasks add, and build a static audit for five languages that classifies every name-, path- and literal-level dependency of every scored test (held-out precision 89.2%, recall 92.9%). On SWE-Milestone (98 graded milestones, 35 published runs), 25 milestones contain a test that depends on an unstated, non-inferable name or path. Build units with such a dependency fail in 80.6% of runs, against 48.4% for comparable units without one, and withholding a single hidden name from the reference stops its unit's tests from compiling in 126 of 128 cases. The leaderboard's ranking is robust to removing the affected tests, but they shift scores within repositories and favour agents that reproduce upstream names. Across four further benchmarks, prevalence appears to follow construction: a third or more of the tasks are affected where specifications are release notes or LLM summaries, almost none where tasks are screened or written for their tests; how much a gap costs depends on how the harness grades. We release the audit, corrected filter lists and specification addenda.
Figures & tables
Figure 1: Behaviour specified, names required. The broken bridge represents an unstated interface: go-zero M019 asks for SSE behaviour but its reference tests require valueSSE and headerAccept . Missing either name blocks the whole rest/handler package before tests execute; the dependency recurs in five other milestones. Test tiles and milestone links are schematic.
class
the dependency is
visible
defined in the trial base
specified
introduced by this milestone or an ancestor and named in its SRS
A1
introduced by this milestone’s gold change, not named in its SRS
A2
introduced by an ancestor’s gold change, not named in that SRS or this one
B1
introduced by a milestone that is not an ancestor
B2
introduced by upstream commits that belong to no milestone
Table 1: Classes of test dependencies.
Figure 2: Static audit: five language extractors recover test dependencies (TS: TypeScript), and the orange crescent is G(o)=D(o)∖Ft , dependencies that neither the base, the specification nor inferable conventions fix. Hard name/path gaps flag all required tests in blocked build units, whereas literals flag individual tests; outputs are reasoned audit records, filter lists and specification addenda. Gold diffs distinguish under-specification (A1: current-milestone gold change; A2: ancestor gold change) from chain inconsistency (B1: non-ancestor gold change; B2: unassigned upstream commits; B3: absent from own gold); B3 tests may already be ungraded as pass-to-none.
Repository
M
Required
Hidden
Tests at risk %
Hard M
go-zero (Go)
23
43,253
132
7.1
15/23
navidrome (Go+TS)
9
13,642
2
0.3
2/9
element-web (TS)
18
94,114
7
0.1
7/18
nushell (Rust)
13
67,504
0
0.0
0/13
dubbo (Java)
12
82,863
1
0.0
1/12
ripgrep (Rust)
11
11,964
0
0.0
0/11
Table 2: Hidden dependencies over the graded milestones (M), language in parentheses. Hidden: hard hidden names (unnamed by the SRS, absent from the trial base, not inferable), summed per milestone; Tests at risk: share of the required tests in units they block or with a hidden literal or test-level name; Hard M: milestones with at least one hard hidden dependency.
Var.
Runs
Δˉ S
Δˉ R
Δ R max
τS
τR
Mv.
Max
Full
35
+0.83
+0.51
+2.78
0.980
0.960
11
2
28
+0.84
+0.47
+2.78
0.984
0.942
5
2
Cons.
35
+0.37
+0.33
+2.04
0.987
0.979
8
1
28
+0.39
+0.30
+2.04
0.995
0.972
2
1
Table 3: Re-scoring without hidden-dependency tests. S: Score (macro F1), R: Resolve (%); mean changes in points against the same runs scored with the v1.0.2 lists, Δ R max the largest; τS , τR : Kendall τb with the original ranking, by Score and by Resolve; Mv.: runs whose Score rank changes; Max: largest rank change. Full waives (a) hidden-dependency units, (b) tests whose reference lacks a name (none in v1.0.2) and (c) explained all-fail tests; Cons. (conservative) only (a) and (b).
Benchmark
Spec.
Flagged
Prec.
Lift
SWE-Milestone
LLM
25/98
58/65 (89.2%)
1.67
SWE-EVO
notes
17/48
24/40 (60.0%)
–
SWE-Chain
LLM
64/155
28/40 (70.0%)
1.12
SWE-bench Verif.
issue
3/500
3/4
3.4
DeepSWE
written
7/113
1/23
–
Table 4: The audit on five benchmarks. Spec.: how the task text is made (LLM summary, release notes, screened issue, written for it). Flagged: tasks with a hidden name, path or literal (SWE-Milestone: hard names or paths, 31 with literals; SWE-Chain: A1 only); Prec.: reviewed flags, per held-out unit for SWE-Milestone (RQ4), per item elsewhere, not comparable across rows (Verified: 3 of 4 items, in 2 of 3 instances); Lift: failure-rate ratio—SWE-Milestone: blocked build units against comparable units (RQ2); SWE-Chain: flagged against unflagged steps; Verified: name-flagged against unflagged instances; no lift for SWE-EVO (no per-task results) or DeepSWE (one confirmed).
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Language
Extractor
Unit
Notes
Go
go/ast
package test binary
build constraints for linux/amd64; Ginkgo suites one unit
Python
ast
test module
module-level import failure fails collection; C extensions read by pattern
TypeScript
tree-sitter
test file
imports, jest.mock / vi.mock paths; path specified only if the SRS names the file
Rust
tree-sitter
cargo test target
macros and rstest cases mapped; workspace build stops at first failing target
Java
tree-sitter
Maven module
the multi-module build skips modules after a failed one
Appendix
Table 5: Build units and extractors per language.
Benchmark
Tasks
Flag.
Rev.
Conf.
Prec. (95% CI)
SWE-Milestone
98
89
65
58
89.2 (79.4–94.7)
SWE-EVO
48
79
40
24
60.0 (44.6–73.7)
SWE-Chain
155
357
40
28
70.0 (54.6–81.9)
SWE-bench Verif.
500
4
4
3
75.0 (30.1–95.4)
DeepSWE
113
23
23
1
4.3 (0.8–21.0)
Appendix
Table 6: Review protocol per benchmark. Tasks: graded milestones, release tasks, steps, instances, tasks. Flag.: flagged build units with a hidden dependency (SWE-Milestone) or flagged items, the frame each review sampled from (SWE-EVO counts an item once per test unit, 62 distinct). Rev.: reviewed; SWE-Milestone, the units of the two blind rounds drawn after the rules were frozen (RQ4); SWE-EVO and SWE-Chain, stratified random samples; Verified and DeepSWE, all flags. Conf.: confirmed. Prec.: confirmed among reviewed (%), Wilson interval.
As coding agents become more capable, tests remain the primary evidence for judging whether their code is correct. Yet passing available tests does not establish that an implementation satisfies its natural language specification. Reliable validation requires checks that faithfully capture the specification and scenarios that thoroughly exercise the code, but current benchmarks provide limited insight into whether agents can construct such evidence themselves. We introduce SpecGuardBench, a benchmark of 291 tasks from 13 real-world repositories, covering 8 types of verification logic such as functional postconditions and interaction protocols. Given a natural language specification, repository context, and a reference implementation, an agent constructs a validation harness comprising executable checks and test scenarios. The checks should reject implementations that violate the specification while accepting those that satisfy it, and the scenarios should thoroughly exercise the code. Across four proprietary and open-weight coding agents, none constructs a harness scored as faithful on the benchmark's implementations for more than 63% of the tasks. The bottleneck lies less in exploring test scenarios than in constructing checks that faithfully reflect the specification. Specifically, the agents show a clear tendency to write checks that mirror the reference implementation rather than the specification, and therefore reject implementations that the specification permits.
Zhiqing Zhong, Mingyu Chen, Aoyang Fang +5
The Chinese University of Hong Kong, Shenzhen · Aalto University · Lero, the Research Ireland Centre for Software +2
As long-horizon coding agents produce more code than any developer can review, oversight collapses onto a single surface: the automated test suite. Reward hacking naturally arises in this setup, as the agent optimizes for passing tests while deviating from the users true goal. We study this reward hacking phenomenon by decompose software engineering tasks into three parts: (i) a natural language description of the specification (ii) visible validation tests that exercise specified features in isolation, and (iii) held-out tests that compose those same features to simulate real-world usage. Based on the specification and the visible validation test suites, a genuine agent would be able to generate a solution that can also pass all of the held-out tests. Therefore we use the gap in pass rates on these two suites to quantify reward hacking. Based on this methodology, we introduce SpecBench, a benchmark comprising 30 systems-level programming tasks ranging from short horizon tasks like building a JSON parser to ultra long horizon tasks like building an entire OS kernel from scratch. Large-scale experiments reveal a consistent pattern: while every frontier agent saturates the visible suite, reward hacking persists, with smaller models exhibiting larger gaps on holdout suites. The gap also scales sharply with task length: it grows by 28 percentage points for every tenfold increase in code size. Failures range from subtle feature isolation to deliberate exploits, including a 2,900-line hash-table "compiler" that memorizes test inputs. SpecBench offers a principled testbed for measuring whether coding agents build genuine working systems or merely game the test suites developers hand them.
Benchmarks drive AI progress, but are hampered by task under-specification and over-specification. Task under-specification causes guesswork, where solutions have to make things up, or worse, resort to contamination or reward hacking. Task over-specification causes spoilers, where solutions can bypass some of the harder steps the benchmark was supposed to measure. Since future-looking benchmarks should be difficult, over-specification reduces a benchmark's shelflife of community adoption. Repository-level coding benchmarks have helped the field progress but experience unnecessary churn due to both over- and under-specification. For instance, SWE-bench Pro, FEA-bench, Doc2Feat-bench, and Multi-SWE-bench all tackle under-specification by augmenting task statements with hints derived from ground truth. We show that leaking ground-truth information into the task statement causes over-specification in some of these benchmarks. We experiment with various specification levels, trying to thread the needle between spoilers and guesswork. Based on these results, we recommend techniques for deriving parsimonious hints. We will release our hints so the community can obtain more reliable signals from popular benchmarks.