What the Specification Did Not Say: Specification-Oracle Gaps in Test-Graded Coding-Agent Benchmarks

Fei Ding

cs.CLOct 22, 2026

What the Specification Did Not Say: Specification-Oracle Gaps in Test-Graded Coding-Agent Benchmarks

Authors: Fei Ding

Organizations: Alibaba Group

Abstract

Coding-agent benchmarks often describe a task in natural language but grade it with tests taken from the upstream project. Passing those tests can require information the description never states: a constant's name, a module path, an exact message. We call this a specification-oracle gap: where it exists, a benchmark also measures whether an agent reproduces upstream implementation details. We formalise the gap, including the inconsistencies that chains of dependent tasks add, and build a static audit for five languages that classifies every name-, path- and literal-level dependency of every scored test (held-out precision 89.2%, recall 92.9%). On SWE-Milestone (98 graded milestones, 35 published runs), 25 milestones contain a test that depends on an unstated, non-inferable name or path. Build units with such a dependency fail in 80.6% of runs, against 48.4% for comparable units without one, and withholding a single hidden name from the reference stops its unit's tests from compiling in 126 of 128 cases. The leaderboard's ranking is robust to removing the affected tests, but they shift scores within repositories and favour agents that reproduce upstream names. Across four further benchmarks, prevalence appears to follow construction: a third or more of the tasks are affected where specifications are release notes or LLM summaries, almost none where tasks are screened or written for their tests; how much a gap costs depends on how the harness grades. We release the audit, corrected filter lists and specification addenda.

Figures & tables

Figure 1: Behaviour specified, names required. The broken bridge represents an unstated interface: go-zero M019 asks for SSE behaviour but its reference tests require valueSSE and headerAccept . Missing either name blocks the whole rest/handler package before tests execute; the dependency recurs in five other milestones. Test tiles and milestone links are schematic.

class	the dependency is
visible	defined in the trial base
specified	introduced by this milestone or an ancestor and named in its SRS
A1	introduced by this milestone’s gold change, not named in its SRS
A2	introduced by an ancestor’s gold change, not named in that SRS or this one
B1	introduced by a milestone that is not an ancestor
B2	introduced by upstream commits that belong to no milestone

Table 1: Classes of test dependencies.

Figure 2: Static audit: five language extractors recover test dependencies (TS: TypeScript), and the orange crescent is $G(o)=D(o)\setminus F_{t}$ , dependencies that neither the base, the specification nor inferable conventions fix. Hard name/path gaps flag all required tests in blocked build units, whereas literals flag individual tests; outputs are reasoned audit records, filter lists and specification addenda. Gold diffs distinguish under-specification (A1: current-milestone gold change; A2: ancestor gold change) from chain inconsistency (B1: non-ancestor gold change; B2: unassigned upstream commits; B3: absent from own gold); B3 tests may already be ungraded as pass-to-none.

Repository	M	Required	Hidden	Tests at risk %	Hard M
go-zero (Go)	23	43,253	132	7.1	15/23
navidrome (Go+TS)	9	13,642	2	0.3	2/9
element-web (TS)	18	94,114	7	0.1	7/18
nushell (Rust)	13	67,504	0	0.0	0/13
dubbo (Java)	12	82,863	1	0.0	1/12
ripgrep (Rust)	11	11,964	0	0.0	0/11

Table 2: Hidden dependencies over the graded milestones (M), language in parentheses. Hidden: hard hidden names (unnamed by the SRS, absent from the trial base, not inferable), summed per milestone; Tests at risk: share of the required tests in units they block or with a hidden literal or test-level name; Hard M: milestones with at least one hard hidden dependency.

Var.	Runs	$\bar{\Delta}$ S	$\bar{\Delta}$ R	$\Delta$ R max	$\tau_{S}$	$\tau_{R}$	Mv.	Max
Full	35	+0.83	+0.51	+2.78	0.980	0.960	11	2
	28	+0.84	+0.47	+2.78	0.984	0.942	5	2
Cons.	35	+0.37	+0.33	+2.04	0.987	0.979	8	1
	28	+0.39	+0.30	+2.04	0.995	0.972	2	1

Table 3: Re-scoring without hidden-dependency tests. S: Score (macro F1), R: Resolve (%); mean changes in points against the same runs scored with the v1.0.2 lists,

\Delta

R max the largest;

\tau_{S}

\tau_{R}

: Kendall

\tau_{b}

with the original ranking, by Score and by Resolve; Mv.: runs whose Score rank changes; Max: largest rank change. Full waives (a) hidden-dependency units, (b) tests whose reference lacks a name (none in v1.0.2) and (c) explained all-fail tests; Cons. (conservative) only (a) and (b).

Benchmark	Spec.	Flagged	Prec.	Lift
SWE-Milestone	LLM	25/98	58/65 (89.2%)	1.67
SWE-EVO	notes	17/48	24/40 (60.0%)	–
SWE-Chain	LLM	64/155	28/40 (70.0%)	1.12
SWE-bench Verif.	issue	3/500	3/4	3.4
DeepSWE	written	7/113	1/23	–

Table 4: The audit on five benchmarks. Spec.: how the task text is made (LLM summary, release notes, screened issue, written for it). Flagged: tasks with a hidden name, path or literal (SWE-Milestone: hard names or paths, 31 with literals; SWE-Chain: A1 only); Prec.: reviewed flags, per held-out unit for SWE-Milestone (RQ4), per item elsewhere, not comparable across rows (Verified: 3 of 4 items, in 2 of 3 instances); Lift: failure-rate ratio—SWE-Milestone: blocked build units against comparable units (RQ2); SWE-Chain: flagged against unflagged steps; Verified: name-flagged against unflagged instances; no lift for SWE-EVO (no per-task results) or DeepSWE (one confirmed).

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Language	Extractor	Unit	Notes
Go	go/ast	package test binary	build constraints for linux/amd64; Ginkgo suites one unit
Python	ast	test module	module-level import failure fails collection; C extensions read by pattern
TypeScript	tree-sitter	test file	imports, jest.mock / vi.mock paths; path specified only if the SRS names the file
Rust	tree-sitter	cargo test target	macros and rstest cases mapped; workspace build stops at first failing target
Java	tree-sitter	Maven module	the multi-module build skips modules after a failed one

Appendix

Table 5: Build units and extractors per language.

Benchmark	Tasks	Flag.	Rev.	Conf.	Prec. (95% CI)
SWE-Milestone	98	89	65	58	89.2 (79.4–94.7)
SWE-EVO	48	79	40	24	60.0 (44.6–73.7)
SWE-Chain	155	357	40	28	70.0 (54.6–81.9)
SWE-bench Verif.	500	4	4	3	75.0 (30.1–95.4)
DeepSWE	113	23	23	1	4.3 (0.8–21.0)

Appendix

Table 6: Review protocol per benchmark. Tasks: graded milestones, release tasks, steps, instances, tasks. Flag.: flagged build units with a hidden dependency (SWE-Milestone) or flagged items, the frame each review sampled from (SWE-EVO counts an item once per test unit, 62 distinct). Rev.: reviewed; SWE-Milestone, the units of the two blind rounds drawn after the rules were frozen (RQ4); SWE-EVO and SWE-Chain, stratified random samples; Verified and DeepSWE, all flags. Conf.: confirmed. Prec.: confirmed among reviewed (%), Wilson interval.

Software Engineering Benchmarks

Software engineering benchmarks evaluate how well AI systems understand, build, repair, test, and maintain software in realistic settings.

See all 156 papers

SpecGuardBench: Can Coding Agents Construct Reliable Validation Harnesses from Natural Language Specifications?
Oct 12, 2026Zhiqing Zhong, Mingyu Chen, Aoyang Fang +5AI Coding Agents Specification-Based Testing
The Chinese University of Hong Kong, Shenzhen · Aalto University · Lero, the Research Ireland Centre for Software +2
SpecBench: Measuring Reward Hacking in Long-Horizon Coding Agents
May 20, 2026Bingchen Zhao, Dhruv Srikanth, Yuxiang Wu +1Reward Hacking AI Coding Agents
Weco AI
Between Spoilers and Guesswork: Avoiding Over-Specification in Coding Benchmarks
Oct 16, 2026Yuanzhe Liu, Bastian Lipka, Avraham Shinnar +3Benchmark Design Software Engineering Benchmarks
Rensselaer Polytechnic Institute · IBM

Oct 12, 2026J/K move · Enter open · S save

SpecGuardBench: Can Coding Agents Construct Reliable Validation Harnesses from Natural Language Specifications?

Zhiqing Zhong, Mingyu Chen, Aoyang Fang, Yuzhong Zhang+4

The Chinese University of Hong Kong, Shenzhen · Aalto University · Lero, the Research Ireland Centre for Software+2

AI Coding Agents Specification-Based Testing Automated Test Generation Software Engineering Benchmarks

Open paper AskPDF arXiv

As coding agents become more capable, tests remain the primary evidence for judging whether their code is correct. Yet passing available tests does not establish that an implementation satisfies its natural language specification. Reliable validation requires checks that faithfully capture the specification and scenarios that thoroughly exercise the code, but current benchmarks provide limited insight into whether agents can construct such evidence themselves. We introduce SpecGuardBench, a benchmark of 291 tasks from 13 real-world repositories, covering 8 types of verification logic such as functional postconditions and interaction protocols. Given a natural language specification, repository context, and a reference implementation, an agent constructs a validation harness comprising executable checks and test scenarios. The checks should reject implementations that violate the specification while accepting those that satisfy it, and the scenarios should thoroughly exercise the code. Across four proprietary and open-weight coding agents, none constructs a harness scored as faithful on the benchmark's implementations for more than 63% of the tasks. The bottleneck lies less in exploring test scenarios than in constructing checks that faithfully reflect the specification. Specifically, the agents show a clear tendency to write checks that mirror the reference implementation rather than the specification, and therefore reject implementations that the specification permits.

Abstract

Software Engineering Benchmarks

Explore similar work

SpecGuardBench: Can Coding Agents Construct Reliable Validation Harnesses from Natural Language Specifications?

SpecBench: Measuring Reward Hacking in Long-Horizon Coding Agents

Between Spoilers and Guesswork: Avoiding Over-Specification in Coding Benchmarks