Built on Broken Assumptions (BOBA): Why Coding Agents Cannot Be Left Alone

Alejandro Cuadron; Arpit Gupta

cs.SENov 1, 2026

Built on Broken Assumptions (BOBA): Why Coding Agents Cannot Be Left Alone

Authors: Alejandro Cuadron, Arpit Gupta

Organizations: Amazon

Abstract

Coding agents are autonomous developers. Given a request, they change the code and report back when done. As such, they can handle not only independent requests but whole chains of them, each request built on the last. Agents are known to report wrong work as done, yet benchmarks still test them one request at a time, on human-written code, checked once. We build BOBA-Bench, 280 chains of 20 consecutive requests drawn from 140 real open-source projects, and evaluate ten coding agents on both. All ten do far worse on chains. On average, they complete 95% of requests taken one at a time, but when requests are chained together, they complete 51%. Worse code is not the cause. Once checked, agent code is nearly as good a base for later requests as human code. The cause is the mistakes agents miss. Coding agents declare a request done once their own tests pass, even when those tests miss a mistake. The next request then builds on broken code and fails about twice as often. The mistake is rarely found later. Nor can agents catch such mistakes by reviewing code. Asked whether an implementation is correct, they are right 53% of the time, barely better than a coin flip. Coding agents are therefore heavily dependent on access to verification. A single external ``that's wrong'' after a failed request and one repair round lift chain completion from 51% to 59%, and a repaired mistake no longer hurts the requests that follow. In simpler terms, coding agents are only as reliable as the checks they can reach: without verification, every request that follows is built on broken assumptions.

Figures & tables

Figure 1: An outside check lifts chain completion. Left: unchecked work collapses, verified work holds (schematic). Right: chains completed on the first attempt without feedback (hollow) and with one “verification failed” message and one repair after each failed request (filled), ten models; labels give the gains in percentage points.

				Graded patches		95% CI	Per task		Task
Benchmark	Granularity	Tasks	Repos	per task	total	( $\pm$ pp)	spec. words	patch lines	collection
Issue resolution
SWE-bench Verified ( OpenAI, 2024 )	Commit	500	12	1	500	4.4	189	14	Rule-based
SWE-bench Pro ( Deng et al., 2025 )	Commit	1.9k	41	1	1.9k	2.3	462	170	Rule-based
CodeThread ( Patel et al., 2026 )	Commit	1.4k	78	1	1.4k	2.6	n/r	n/r	Rule-based
Codebase generation

Table 1: BOBA-Bench next to representative coding benchmarks. Bold: best among the repository-evolution benchmarks; underlined: second.

Figure 2: Autonomous evaluation grades the repository an agent leaves after 20 requests on its own code, not one patch. (a) One specification, one submission, hidden F2P/P2P tests (SWE-bench shown). (b) BOBA-Bench : 20 specifications arrive one at a time on one persistent codebase, each in a fresh conversation; after each submission a fresh verifier runs the hidden tests of requests $1..k$ and records the verdict, and specification $k{+}1$ follows regardless.

Figure 3: Requests are specified from the developers’ tests, not their code, and certified against their patches. (1) Mine history : consecutive commits that change tests and source together become a request if the new tests fail before and pass after the human patch. (2) Specify : Claude Opus 4.8 writes each specification from commit messages, issues and the test diff. (3) Audit : three model families must accept identical text. (4) Certify : 20 consecutive requests form a chain, sealed with frozen dependencies, in which the developers’ patches must pass every hidden test.

Model	pass@1	[95% CI]	single attempt	strict pass@1	regression rate	pass@8	completed (%)	chain (median)
	Chains (%)						Requests	Calls per
Claude Opus 5.5	66.7	[61.3, 71.9]	66.8	55.8	22.9	83.9	84.0	216
Claude Opus 5	61.6	[56.2, 66.8]	60.7	50.5	28.1	82.9	80.8	401
GPT-6 Astra	53.2	[46.7, 59.5]	51.1	38.7	35.1	63.9	75.2	182
Claude Fable 5	51.5	[45.8, 57.1]	51.8	38.2	40.5	72.1	73.2	292
GLM-5.3	51.2	[45.4, 57.1]	46.4	39.8	30.1	76.1	73.0	560

Table 2: Chain completion of ten coding agents on the 280 chains of BOBA-Bench : each request in a fresh conversation on the repository the previous one left, all 20 requests run, eight independent attempts per chain. pass@1: the probability that an attempt leaves the repository after request 20 passing every hidden test of the chain, estimated from the eight attempts (single attempt: the first attempt alone); strict pass@1: in addition, every request passed its own tests on its own step and no passing test ever broke; regression rate: a passing test broke at some request, repaired later or not; pass@8: at least one of the eight attempts passes; request completion: share of all graded requests of the first attempt whose tests pass on the step they are issued. Repository-clustered 95% bootstrap intervals; an attempt without a final state counts as a failure.

Figure 4: Verified foundations hold whoever built them, unverified failures beneath a request erode its success, and one outside message raises their repair from 8–30% to 28–60%. Success rate (%) of request

k{+}1

by the state beneath it: who wrote the certified code (human or agent) and how many earlier requests were still failing (0, 1, 2, 3+); last two columns: newly failing tests repaired at the next request without and after one “verification failed” message ( § 5 ).

Figure 5: No model is a reliable reviewer: lenient reviewers accept most wrong code, strict ones reject about half of the correct code. (a) Wrong code rejected, (b) correct code accepted, (c) pairs whose wrong twin fails and correct twin passes the reviewer’s tests (%). Framed cells: self-review.

Appendix figures & tables49 assets

Supplementary material from the paper’s appendix.

Appendix

	p10	p25	median	p75	p90	max
GitHub stars	6	14	46	195	603	5,835
code LOC at first chain base	79	215	424	2,146	4,914	33,731
Python LOC	76	212	420	1,907	4,801	33,684
files	8	11	18	37	66	325
test LOC	9	29	97	538	1,317	11,574
chains per repository	1	2	2	2	3	5

Appendix

Table 3: Repositories (n=140). Size is measured at the base commit of the repository’s first chain; stars queried 2026-09-09 (packaging 2026-08-27..30). Domain labels: GPT-5.6 from public metadata (fixed 13-way list; a keyword heuristic agrees on 53/140). Test style: pytest is the runner for every chain; style is inferred from the chains’ own F2P test ids and test patches.

	p10	p25	median	p75	p90	max	mean [95% CI]
Reference (gold) patch
files changed	1	2	3	7	13	224	6.0 [5.3, 6.7]
hunks	2	4	7	15	30	1,654	15.5 [12.4, 19.3]
lines added	6	16	50	146	384	81,022	276.8 [179.8, 404.1]
lines removed	1	3	14	51	163	44,855	134.2 [83.8, 201.1]
lines changed (+/ $-$ )	9	24	73	215	596	94,155	411.1 [267.3, 579.1]

Appendix

Table 4: Per-segment statistics over the 5,600 segment occurrences (280 chains

\times

20). Mean is followed by its repository-clustered bootstrap 95% CI. Source commits: commits folded into the segment by the generation walk (see text). Contract nodes at

k

= F2P k

\cup

P2P k

\cup

still-valid earlier F2P .

	p10	p25	median	p75	p90	max	mean [95% CI]
gold lines added (20 segments)	781	1,246	2,177	4,274	7,013	121,416	5,536.6 [3,682.0, 8,007.4]
gold lines removed	278	442	775	1,836	3,396	56,611	2,684.6 [1,643.7, 4,033.9]
net gold lines	242	499	987	2,311	4,674	89,154	2,852.0 [1,552.4, 4,631.0]
distinct files touched by gold	11	16	25	45	74	297	38.0 [32.3, 44.7]
distinct test files touched	4	6	10	18	30	309	16.8 [13.2, 21.5]
new files created	2	4	10	24	46	201	19.1 [15.6, 23.4]

Appendix

Table 5: Per-chain statistics (n=280 chains of 20 segments). Dependency depth = longest path in the DAG whose edges

j\to k

(

j<k

) mark a segment whose gold or test patch references a symbol introduced, or imports a module created, by an earlier segment’s gold patch (symbol/import definition), or whose gold patch touches a file the earlier gold patch touched (file definition).

definition	any earlier $j<k$	$k{-}1$	nearest lag (med.)	chain depth med. (IQR)
gold patches share a file	98.1% [97.2, 98.8]	89.2% [86.8, 91.5]	1	18 (16–19)
references a function/class touched by $j$	79.3% [75.5, 82.8]	40.1% [36.7, 43.5]	1	8 (5–11)
references a function/class introduced by $j$	72.3% [68.3, 76.2]	28.2% [25.6, 31.0]	2	6 (4–8)
strict (names $\geq 4$ chars, non-generic)	68.2% [64.5, 71.9]	25.5% [23.1, 28.0]	2	5 (3–7)
gold-patch references only	64.6% [60.7, 68.6]	24.4% [21.8, 27.0]	2	5 (3–8)
test-patch references only	54.1% [49.8, 58.2]	15.0% [13.2, 16.9]	3	3 (2–5)

Appendix

Table 6: Segment dependency structure over the 5,320 segment occurrences at positions 2–20. “Any earlier”: some

j<k

in the same chain satisfies the definition; “

k{-}1

”: the immediately preceding segment does. CIs are repository-clustered bootstrap 95%.

	BOBA-Bench segment	BOBA-Bench chain (20 seg.)	SWE-bench Verified	SWE-bench Pro	SWE-bench Pro (Py)
	n=5,600	n=280	n=500	n=731	n=266
gold files changed	3 (2–7)	25 (16–45)	1 (1–1)	4 (2–7)	3 (2–4)
gold hunks	7 (4–15)	–	1 (1–2)	9 (5–16)	7 (4–12)
gold lines added	50 (16–146)	2,177 (1,246–4,274)	4 (2–10)	63 (33–135)	46 (30–108)
gold lines removed	14 (3–51)	775 (442–1,836)	2 (1–4)	20 (7–52)	15 (5–41)
gold lines changed	73 (24–215)	–	7 (3–13)	94 (50–207)	74 (45–142)

Appendix

Table 7: A BOBA-Bench segment next to a single-request task. Median (IQR) per instance; identical parser for all rows. SWE-bench Verified from princeton-nlp/SWE-bench_Verified (500); SWE-bench Pro public set from ScaleAI/SWE-bench_Pro (731; Go 280, Python 266, JS 165, TS 20). Specification = problem statement (Verified) / problem statement + requirements + interface (Pro) / frozen segment specification ( BOBA-Bench ).

stratum	bins	n
gold lines changed (terciles)	small $<36$ / medium $36$ – $144$ / large $\geq 145$	1,832 / 1,896 / 1,872
source commits per segment	1 / 2–4 / $\geq 5$	1,741 / 2,081 / 1,778
P2P protection	empty / non-empty	223 / 5,377
symbol/import dependency ( $k\geq 2$ )	none / on $k{-}1$ / only on $j<k{-}1$	1,421 / 1,533 / 2,366
position block	1–5 / 6–10 / 11–15 / 16–20	1,400 / 1,400 / 1,400 / 1,400

Appendix

Table 8: Stratification bins (segment occurrences per bin; assignments in data/segment_bins.jsonl ). Size terciles are computed over the 5,600 occurrences; position blocks are balanced by construction.

	Fable 5	GPT-6 Astra	Kimi K3	GPT-5.6	GLM-5.2
Boundaries (repositories)	311 (124)	298 (124)	332 (130)	283 (121)	286 (120)
Repositories never certified (of 140)	46	55	40	64	63
Success, agent prefix (%)	92.8 [89.0,96.2]	94.0 [90.5,97.0]	94.6 [92.1,96.9]	94.1 [91.0,96.8]	89.6 [85.6,93.2]
Success, human prefix (%)	95.1 [92.3,97.4]	93.2 [89.7,96.2]	95.1 [93.2,96.7]	93.7 [91.1,96.0]	92.2 [88.8,95.1]
Gap, agent $-$ human (pp)	$-$ 2.3 [ $-$ 5.2, $+$ 0.3]	$+$ 0.8 [ $-$ 1.9, $+$ 3.3]	$-$ 0.5 [ $-$ 2.7, $+$ 1.6]	$+$ 0.4 [ $-$ 2.2, $+$ 2.8]	$-$ 2.6 [ $-$ 5.2, $-$ 0.2]
Sign-flip $p$	0.11	0.59	0.67	0.76	0.04

Appendix

Table 9: Matched Foundations per model (140 repositories). Hidden-tests protocol; 95% repository-bootstrap CIs in brackets.

	Opus 5.5	Opus 5	Opus 4.8	GPT-6.1	GLM-5.3	GLM-5.3-Flash
Boundaries (repositories)	321 (123)	294 (118)	293 (121)	291 (122)	328 (131)	329 (127)
Repositories never certified (of 140)	41	52	57	61	43	40
Success, agent prefix (%)	95.9 [93.0,98.3]	91.2 [87.0,94.8]	91.3 [87.6,94.6]	94.0 [90.3,97.2]	92.8 [89.3,95.8]	95.0 [93.0,96.7]
Success, human prefix (%)	96.5 [94.3,98.2]	94.8 [91.7,97.4]	94.6 [91.5,97.3]	93.4 [90.0,96.4]	93.4 [90.7,95.6]	94.7 [92.9,96.3]
Gap, agent $-$ human (pp)	$-$ 0.6 [ $-$ 3.2, $+$ 1.7]	$-$ 3.6 [ $-$ 6.6, $-$ 1.1]	$-$ 3.3 [ $-$ 6.6, $-$ 0.1]	$+$ 0.6 [ $-$ 2.4, $+$ 3.5]	$-$ 0.6 [ $-$ 3.5, $+$ 1.9]	$+$ 0.3 [ $-$ 1.5, $+$ 2.2]
Sign-flip $p$	0.69	0.01	0.05	0.70	0.67	0.77

Appendix

Table 10: Matched Foundations per model, the six models added with the leaderboard (140 repositories). Hidden-tests protocol; 95% repository-bootstrap CIs in brackets.

	Fable 5	GPT-5.6	GLM-5.2	Kimi K3
Paired boundaries (repositories)	296 (121)	280 (120)	230 (104)	263 (108)
Agent arm, visible $-$ hidden	$+$ 5.4 [ $+$ 2.1, $+$ 9.3]	$+$ 4.0 [ $+$ 1.7, $+$ 6.9]	$-$ 4.1 [ $-$ 7.8, $-$ 0.2]	$-$ 0.4 [ $-$ 3.1, $+$ 2.4]
Human arm, visible $-$ hidden	$+$ 3.9 [ $+$ 1.5, $+$ 6.8]	$+$ 4.4 [ $+$ 2.6, $+$ 6.5]	$-$ 5.1 [ $-$ 8.1, $-$ 2.1]	$-$ 0.9 [ $-$ 3.0, $+$ 1.1]
Gap, tests visible	$-$ 0.4 [ $-$ 1.5, $+$ 0.6]	$+$ 0.2 [ $-$ 0.8, $+$ 1.2]	$-$ 1.3 [ $-$ 3.9, $+$ 1.0]	$+$ 0.4 [ $-$ 1.5, $+$ 2.6]
Gap, tests hidden	$-$ 1.9 [ $-$ 4.8, $+$ 0.7]	$+$ 0.5 [ $-$ 2.2, $+$ 2.9]	$-$ 2.3 [ $-$ 5.2, $+$ 0.3]	$-$ 0.0 [ $-$ 2.0, $+$ 1.8]
Gap difference, visible $-$ hidden	$+$ 1.5 [ $-$ 1.1, $+$ 4.5]	$-$ 0.4 [ $-$ 2.7, $+$ 2.2]	$+$ 0.9 [ $-$ 2.1, $+$ 4.2]	$+$ 0.5 [ $-$ 2.1, $+$ 3.2]

Appendix

Table 11: Ablation: human tests of the accepted requests visible in both prefixes versus hidden Paired on the boundaries complete under both conditions; pp with 95% repository-bootstrap CIs; gap = agent-minus-human next-request pass rate.

Reviewer	Wrong accepted (%)	Correct rejected (%)	Youden $J$	Pairs discr. (%)	Suites (%)	$n$
Opus 5.5	77.0 [73.7,80.0]	7.8 [5.7,10.1]	$+$ 0.15 [ $+$ 0.12, $+$ 0.19]	18.8	12.8	1090
Opus 5	88.9 [86.5,91.0]	2.8 [1.8,4.1]	$+$ 0.08 [ $+$ 0.06, $+$ 0.10]	9.7	6.7	1090
Fable 5	92.9 [90.9,94.7]	1.2 [0.5,2.0]	$+$ 0.05 [ $+$ 0.04, $+$ 0.07]	6.5	4.9	1086
GLM-5.3-Flash	90.5 [87.6,92.9]	1.2 [0.5,2.0]	$-$ 0.02 [ $-$ 0.05, $+$ 0.01]	3.7	1.8	1085
GPT-6 Astra	40.4 [36.2,44.3]	58.4 [54.0,62.7]	$+$ 0.01 [ $-$ 0.01, $+$ 0.04]	8.6	5.8	1090
GPT-6.1	41.6 [37.6,45.5]	57.1 [52.6,61.5]	$+$ 0.01 [ $-$ 0.01, $+$ 0.03]	8.3	6.3	1090

Appendix

Table 12: Cross-model review per reviewer (ten models; 109 matched pairs per author, 21,800 reviews). Equal weight over the ten authors; repository-clustered bootstrap 95% intervals in brackets. Youden

J

= wrong rejected

+

correct accepted

-

1. Pairs discriminated: the reviewer rejects the wrong implementation and accepts its correct twin. Suites: the reviewer’s own tests fail the wrong implementation and pass the twin, over the valid suites with a known counterpart (

n

	Fable 5	Kimi K3	GPT-5.6	GLM-5.2
Self-authored suites executable (%)	89.4	85.3	75.8	87.3
Suites passing the correct human implementation (%)	55.9 [47.9,63.7]	35.5 [28.7,42.5]	30.1 [23.4,37.0]	42.2 [35.0,49.6]
Suites failing wrong sibling implementations (%)	13.0 [1.0,28.9]	68.5 [42.5,92.7]	50.8 [23.3,78.3]	83.5 [67.0,97.5]
Repositories with a wrong sibling (sensitivity denominator)	7	8	10	20

Appendix

Table 13: Self-authored suites (four models of the earlier study; 95% repository-clustered bootstrap intervals ).

Step	Requests	Repositories
Verified changes from the history walk	25,519	1,491
in the primary pool	14,921	–
in continuity runs of at least 20	6,316	141
removed by retained-behavior eligibility	1,178	–
removed with runs below 20	240	22
remaining primary-pool requests	4,898	119

Appendix

Table 14: Attrition from verified changes to released requests. Counts are unique requests unless stated. The two pools are repository-disjoint.

Outcome	Candidates
Solved $\geq 1/8$ by Claude Opus 4.8	8,695 (58.3%)
Opus 0/8, then solved $\geq 1/8$ by GPT-5.6-sol	816 (5.5%)
0/8 by both	5,145 (34.5%)
Incomplete (51 GPT runs, 214 harness E2BIG)	265 (1.8%)

Appendix

Table 15: Blind solvability campaign on the 14,921 candidate specifications (August 2026).

Class	Specifications
Solved by Claude Opus 4.8	3,183 (72.3%)
Solved only by GPT-5.6-sol	263 (6.0%)
Never blind-solved by either	886 (20.1%)
Incomplete campaign runs	72 (1.6%)

Appendix

Table 16: Blind-solvability class of the benchmark’s 4,404 specifications (280 chains).

Status	Specifications
Hard but fair: never solved blind, no specification gap found	575
Rewritten, then solved in blind pass@4	45
Rewritten, blind validation not finished	68

Appendix

Table 17: Adequacy status of the never-solved and rewritten primary-pool specifications of the benchmark .

Step	Specifications
In the 439 candidate chains	6,740
After removing one abnormal request	6,716
After a provider content filter blocked two chains	6,688
Accepted unanimously in the first wave	4,989 (74.6%)
Contested after the first wave (8,539 missing-contract findings)	1,699
Rewrites by GPT-5.6: accepted / targets	1,389 / 1,459 (95.2%)

Appendix

Table 18: Specification audit, from candidate windows to the re-windowed chains.

Reason	Chains
Contested specifications, provider-filtered or malformed chains, and re-windowing	42
Gold-walk, packaging, apply and window-composition failures	17
Specification hygiene, gold patches on test paths, non-reproducible repaired environments	27
Environment manipulation ( § D.7 )	32
Nondeterminism on an identical digest	4
Failure at the first two-container gate	20

Appendix

Table 19: Chains dropped by the filtering rounds , from 439 candidate windows to 280 chains.

Figure 6: Three Claude models memorize these repositories, none recalls benchmark hunks more than other pre-cutoff code, and no model with ten or more memorized repositories passes them measurably more often. (a) Share of hunks a model continues near-verbatim from their first 40% (similarity

\geq

0.9): benchmark hunks, with 95% repository-bootstrap intervals, and same-repository hunks from before and after the model’s cutoff. (b) Handoff pass@1 on the repositories a model reproduces near-verbatim minus pass@1 on the rest, with repository-clustered 95% intervals; grey where fewer than ten repositories are memorized.

Model	Near-verbatim / sampled	Rate (%)	Pre-cutoff (%)	Contrast (pp)
Opus 5.5	78 / 407	19.2	13.8	$+4.0$ [ $-1.7$ , $+10.2$ ]
Opus 5	38 / 270	14.1	8.7	$+2.6$ [ $-3.9$ , $+9.4$ ]
Fable 5	19 / 126	15.1	14.5	$+0.6$ [ $-8.5$ , $+9.7$ ]
GLM-5.3-Flash	3 / 274	1.1	0.0	$+1.2$ [ $+0.0$ , $+3.2$ ]
GPT-6 Astra	7 / 407	1.7	1.7	$-0.1$ [ $-2.1$ , $+1.7$ ]
GPT-6.1	6 / 407	1.5	1.2	$+0.3$ [ $-1.3$ , $+1.8$ ]

Appendix

Table 20: Benchmark-hunk reproduction (280-chain set). These are probe samples, not whole-chain rates: benchmark hunks continued near-verbatim from their first 40%, the same rate for same-repository hunks from before the model’s cutoff, and their difference paired by repository with a 95% repository-bootstrap interval. Sampled = requests the model answered (Opus 5 refuses to reproduce source text verbatim in a third of them; GLM-5.3 and GLM-5.3-Flash ran to the token cap in the rest of theirs).

	Fable 5	GPT-6 Astra	Kimi K3	GPT-5.6	GLM-5.2
Starting points / control units	470 / 174	130 / 161	836 / 595	939 / 572	1451 / 1594
Control (verified state): $k{+}1$ own-work failure (%)	3.8	4.3	9.6	16.9	11.9
Arm D: after one untreated miss (%)	21.8 [18.3,25.8]	28.5 [21.4,36.7]	25.0 [22.2,28.1]	30.4 [27.5,33.4]	29.3 [27.0,31.7]
Multiplier, D / control	5.8 [2.2,38.0]	6.6 [2.2,24.1]	2.6 [1.6,4.8]	1.8 [1.2,2.9]	2.5 [1.8,3.4]
Difference, D $-$ control (pp)	18.0 [8.3,28.8]	24.2 [13.1,32.1]	15.4 [7.3,23.5]	13.4 [5.4,22.2]	17.3 [11.5,23.4]
Gold state: $k{+}1$ own-work failure, same requests (%)	13.1	13.5	5.6	—	—

Appendix

Table 21: Resume arms per model. D: silent continuation; E: one external “verification failed” message and one repair before request

k{+}1

; F: fresh conversation on the same broken state; gold state: the same request

k{+}1

attempted from the developers’ state at

k

	Fable 5	GPT-6 Astra	Kimi K3	GPT-5.6	GLM-5.2
No open defect beneath the request (%)	8.0	8.0	12.1	12.0	10.5
One (%)	12.7	16.2	18.1	21.5	21.7
Two (%)	36.1	29.5	38.8	28.3	32.2
Three (%)	33.7	40.0	48.6	43.8	49.0
Four (%)	66.1	74.1	80.1	57.3	55.0
Five or more (%)	92.5	84.3	90.0	70.2	67.1

Appendix

Table 22: Own-work failure at requests

k{+}1

to 20 by number of open defects beneath the request (silent continuations, 140 repositories).

Model	Usage / 280	Mean USD	Median USD
Fable 5	273	16.68	14.60
GPT-6 Astra	280	22.03	17.01
Kimi K3	280	9.88	8.12
GPT-5.6	279	12.34	10.19
GLM-5.2	266	9.03	6.43

Appendix

Table 23: Standardized first-attempt API cost (USD), single-conversation runs. Coverage gives the number of first attempts with recorded token usage out of the 280 chains. All models use the same caching assumption described above.

Figure 7: Occasional false alarms cost little; one in two gives back most of the supervisor’s gain, yet completion never drops below the unsupervised level. Completion at 20 against the supervisor’s false-alarm rate

p

( Table 37 ; GPT-5.6-sol and GPT-6 Astra, 100 chains per level), with repository-clustered 95% bootstrap bands. Dashed: perfect supervisor (

p{=}0

); dotted: autonomous (the same chains’ first attempt, no supervisor). Labels: false alarms that broke a correct state (%).

Label	First	Later	All	Opus 5.5	Opus 5	GLM-5.3-Flash	Fable 5	GPT-6 Astra	GPT-6.1	GLM-5.3	GPT-5.6	Kimi K3	Opus 4.8
Regression: pre-existing behavior broken	44.0	19.3	27.8	31	31	23	39	25	28	22	30	32	22
Missed requirement: part of the specification unimplemented	10.6	15.9	14.1	9	12	18	17	9	8	13	11	17	21
Wrong logic	10.6	10.6	10.6	9	13	12	6	11	10	18	10	10	9
Integration error: new code unreachable from the tested path	5.4	7.7	6.9	4	5	5	5	6	6	10	8	5	11
Other agent-side: wrong file, syntax, unverified assumption, knowledge gap	3.6	4.9	4.4	4	2	3	3	6	4	3	2	4	10
Benchmark-side, suspected: test stricter than the specification, broken or ambiguous	24.3	38.0	33.3	40	35	36	25	41	41	33	36	30	24

Appendix

Table 24: Labels of the 4,787 failed requests of the first attempts on the 140 repositories (primary judge; % of the column’s sites). First: the first failure of a chain; Later: every failure after it; All and the model columns: every failure. Benchmark-side tags are the primary judge’s suspicion from static evidence; their re-execution by the sandbox judge is reported in the text.

Figure 8: Breaking what already worked is the most common new failure for nine of ten models and nearly half of all first failures. Final root-cause label (static GPT-5.6 judge, Claude Fable 5.1 sandbox judge for the test-side and withheld tags, published merge rule; LLM labels, indicative) of the 4,787 requests at which a model’s own hidden tests failed or a protected test that passed before the request failed after it: first attempt of the ten models on the 280 chains evaluated to request 20 without a stopping rule (leaderboard order; bar labels in per cent). Regressions are 31% of all new failures (48% of the chains’ first failures but 23% of the 3,142 later ones); unverified assumptions (19%; 22% of later failures) and missed requirements (18%; 21%) follow.

Figure 9: Failures are seldom repaired: only 14–33% of the tests that fail during a chain ever pass again, and models carry an earlier failure at 14–28% of their requests. Left: outcome of each of the 55,545 graded requests (first attempt, ten models, no stopping rule): passed clean, a new failure on an otherwise clean tree, a new failure while earlier failures were still carried, or carrying only (a test from an earlier request failing before and after the request with no new failure; bar labels = carrying-only share in per cent). Right: share of the 12,401 tests that failed during a chain (misses, regressions and silent breaks; tests failing only under a later step’s test overlay excluded) repaired within

d

requests of their first failure; the repairs that happen come within 1–2 requests.

Model	pass@1 (attempt 1)	[95% CI]	strict pass@1	[95% CI]	regression rate	[95% CI]	healed	pass@8	[95% CI]
Claude Opus 5.5	187 (66.8)	[60.4, 73.0]	156 (55.7)	[49.1, 62.2]	59 (21.1)	[16.1, 26.4]	36	235 (83.9)	[79.2, 88.4]
Claude Opus 5	170 (60.7)	[54.3, 67.0]	143 (51.1)	[44.6, 57.4]	77 (27.5)	[22.0, 33.2]	30	232 (82.9)	[77.5, 88.0]
Claude Fable 5	145 (51.8)	[45.4, 58.0]	102 (36.4)	[29.8, 43.0]	118 (42.1)	[35.7, 48.9]	44	202 (72.1)	[66.3, 77.6]
GLM-5.3-Flash	143 (51.1)	[44.3, 57.8]	108 (38.6)	[31.8, 45.4]	87 (31.1)	[24.9, 37.4]	36	217 (77.5)	[71.6, 83.3]
GPT-6 Astra	143 (51.1)	[44.3, 57.8]	101 (36.1)	[29.6, 42.6]	98 (35.0)	[28.8, 41.5]	42	179 (63.9)	[57.2, 70.5]
GPT-6.1-sol	137 (48.9)	[42.3, 55.4]	102 (36.4)	[29.8, 43.2]	104 (37.1)	[30.9, 43.7]	36	170 (60.7)	[54.1, 67.2]

Appendix

Table 25: Complete-chain rates on the 280 chains (140 repositories), first attempt. Counts are chains; brackets are repository-clustered 95% bootstrap intervals in percent. Healed: chains counted by pass@1 but not by strict pass@1. pass@8: chains completed by at least one of the eight attempts. A chain without a request-20 state counts as a failure.

Model	Verifier error	Budget exit	Failed run	pass@1 with a final state
Claude Opus 5.5	2	0	0	187/278 (67.3)
Claude Opus 5	5	0	0	170/275 (61.8)
Claude Fable 5	1	0	3	145/276 (52.5)
GLM-5.3-Flash	2	5	0	143/273 (52.4)
GPT-6 Astra	2	0	0	143/278 (51.4)
GPT-6.1-sol	2	0	0	137/278 (49.3)

Appendix

Table 26: Chains without a request-20 state on the 280 chains, by cause, and pass@1 over the chains that have one. Verifier errors and budget exits include those of the no-stop continuation of a chain.

	Clean through request $k$ (%)				First-failure hazard by quarter (%)				Request
Model	$k{=}5$	$k{=}10$	$k{=}15$	$k{=}20$	1–5	6–10	11–15	16–20	completion (%)
Claude Opus 5.5	90.7	75.4	66.4	55.7	1.9	3.5	2.5	3.5	84.0
Claude Opus 5	88.2	73.9	62.1	51.1	2.5	3.2	3.4	3.8	80.8
Claude Fable 5	80.7	60.4	46.1	36.4	4.1	5.4	5.2	4.6	73.2
GLM-5.3-Flash	83.6	63.9	48.9	38.6	3.5	5.1	5.2	4.6	73.9
GPT-6 Astra	84.3	67.1	51.1	36.1	3.3	4.4	5.4	6.7	75.2

Appendix

Table 27: Clean share by request and first-failure hazard by quarter on the 280 chains. Clean through

k

: share of chains with no silent miss and no broken protected test at any request up to

k

(a chain without a verdict at some request

\leq k

counts as not clean). Hazard: first failures at requests 1–5, 6–10, 11–15 and 16–20 over the requests reached by chains that were still clean. Request completion: share of all graded requests that passed their tests on their own step with no protected test broken.

Figure 10: Completion erodes request by request, and the models that break the fewest passing tests complete the most chains. Left: for each model, the share of chains whose request

k

passed its own tests on its own step with no protected test broken (solid) and the share with no failure at any request up to

k

(dashed), first attempt, 280 chains. Right: pass@1 against the regression rate, both estimated from the eight attempts of each chain, with repository-clustered 95% intervals.

Study	Population and scope
Baseline	280 chains, 140 repositories; ten models; pass@1 estimated over eight attempts per chain (2,240 per model), single-attempt values alongside; pass@8 over the same attempts.
Whole-chain repair	280 chains, 140 repositories; five models, continuous-conversation first attempts; under the handoff protocol, ten models paired with their Autonomous first attempts ( Appendix K ).
Task-size and dependency statistics	280 chains, 140 repositories.
Verified-prefix comparison	Retained boundaries from the 140 repositories; eleven models, model-specific retention.
Failed-state continuation	Eligible silent misses from the continuous-conversation first attempts, with model-specific missing outcomes; gold-state control on 120 / 130 / 256 of those states (Fable 5 / GPT-6 Astra / Kimi K3), two attempts each.
Blind implementation review	109 wrong implementations per author from the Autonomous first attempts, each paired with a correct one: 2,180 items per reviewer, ten reviewers, 21,800 reviews.

Appendix

Table 28: Populations by study. The unit and eligible subset depend on the experiment.

Benchmark	Metric (source)	Unit	$n$	Count
SWE-Milestone ( Deng et al., 2026 )	Resolve Rate (Table 2), 15 agent–model configurations	milestone	98	round(rate $\times n$ ), $\pm 1$
SlopCodeBench ( Orlanski et al., 2026 )	strict and isolated solve (Table 1; best run per model)	checkpoint	196	round(rate $\times n$ )
SWE-Chain ( Lam et al., 2026 )	Build+Fix resolving (Table 3), single run	chain	12	not a count
BOBA-Bench	pass@1, attempt 1	chain	280	exact

Appendix

Table 29: Inputs. Unit: what one trial is. Count: how the per-model count was obtained.

Benchmark	Metric	Models	Spread (%)	All pairs	Adjacent	Top five	Leader vs. runner-up (Wilson 95%)
SWE-Milestone	Resolve Rate	15	13.4–1.3	9/105	0/14	0/10	13/98 [7.9, 21.4] vs. 12/98 [7.1, 20.2]
SlopCodeBench	strict solve	15	14.8–2.0	27/105	0/14	0/10	29/196 [10.5, 20.4] vs. 22/196 [7.5, 16.4]
SlopCodeBench	isolated solve	15	28.1–4.1	46/105	0/14	0/10	55/196 [22.2, 34.7] vs. 51/196 [20.4, 32.6]
SWE-Chain	resolving, Build+Fix	9	60.8–20.2	not testable: single run, means over 12 chains, no dispersion published; a binomial bound with 12 units separates 0/36

Appendix

Table 30: Pairs of models separated at

\alpha=0.05

(two-sided Fisher exact test on reported counts). Spread: best to worst reported rate.

Model	pass@1	Wilson 95%
Opus 5.5	187/280 (66.8%)	[61.1, 72.0]
Opus 5	170/280 (60.7%)	[54.9, 66.2]
Fable 5	145/280 (51.8%)	[46.0, 57.6]
GLM-5.3-Flash	143/280 (51.1%)	[45.2, 56.9]
GPT-6 Astra	143/280 (51.1%)	[45.2, 56.9]
GPT-6.1	137/280 (48.9%)	[43.1, 54.8]

Appendix

Table 31: BOBA-Bench , 280 chains, attempt 1: pass@1 with Wilson 95% intervals.

		Diff. (pp)	Discordant	McNemar $p$	Fisher $p$	Bootstrap 95%
Opus 5.5	Opus 5	$+6.1$	38 / 21	$0.036$	$0.16$	[ $+0.0$ , $+11.9$ ]
Opus 5.5	Fable 5	$+15.0$	69 / 27	$2.1\times 10^{-5}$	$4.1\times 10^{-4}$	[ $+8.1$ , $+21.8$ ]
Opus 5.5	GLM-5.3-Flash	$+15.7$	68 / 24	$4.9\times 10^{-6}$	$2.1\times 10^{-4}$	[ $+8.8$ , $+22.6$ ]
Opus 5.5	GPT-6 Astra	$+15.7$	59 / 15	$2.6\times 10^{-7}$	$2.1\times 10^{-4}$	[ $+9.8$ , $+21.8$ ]
Opus 5.5	GPT-6.1	$+17.9$	66 / 16	$2.3\times 10^{-8}$	$2.6\times 10^{-5}$	[ $+11.7$ , $+24.1$ ]
Opus 5.5	GLM-5.3	$+20.4$	75 / 18	$1.9\times 10^{-9}$	$1.7\times 10^{-6}$	[ $+13.2$ , $+27.4$ ]

Appendix

Table 32: BOBA-Bench pairwise comparisons on the same 280 chains. Discordant: chains passed only by the first / only by the second model. McNemar: exact two-sided

p

on the discordant chains. Fisher: unpaired two-sided

p

on the two counts. Bootstrap: repository-clustered 95% interval of the difference in percentage points.

	First failure						Later failures
	$n$	Own tests only	Earlier tests only	Both	Regression	Earlier request broken	$n$	Regression
Opus 5.5	116	56.0	37.9	6.0	44.0	37.9	119	17.6
Opus 5	130	53.8	41.5	4.6	46.2	41.5	180	18.3
GLM-5.3-Flash	165	61.2	31.5	7.3	38.8	33.3	309	17.5
Fable 5	170	47.6	45.9	6.5	52.4	47.6	295	31.2
GPT-6 Astra	170	55.3	39.4	5.3	44.7	37.1	300	19.3

Appendix

Table 33: The first failure of a first attempt, and the failures after it , from grader output alone (280 chains, ten models, first attempts). Each first failure is classed by its failing tests: those of the new request (own) and those of behavior that already existed (protected tests of the original repository or of an earlier request). A regression fails at least one protected test. Shares are percentages of a model’s first failures (

n

); the pooled row gives repository-clustered bootstrap 95% intervals. “Earlier request broken”: first failures that break a test added or changed by an earlier request of the same chain. The last two columns give the new failures after the first one and their regression share.

	Graded requests (%)					New failure rate (%)		Tests that fail			Chains
	$n$	Clean	New	Carrying	Both	clean repo.	carrying	$n$	Recovered (%)	Delay	Healed / broken
Opus 5.5	5574	82.9	3.1	12.9	1.1	3.6	7.7	437	33.2	2	36 / 123
Opus 5	5549	79.9	3.6	14.5	2.0	4.3	11.9	1112	14.4	1	30 / 134
GLM-5.3-Flash	5528	73.0	5.2	18.5	3.3	6.7	15.3	1212	22.7	1	36 / 168
Fable 5	5525	72.6	5.4	19.0	3.0	6.9	13.8	1333	28.5	1	44 / 174
GPT-6 Astra	5567	73.8	4.9	17.7	3.5	6.2	16.7	1012	30.7	1	42 / 177

Appendix

Table 34: Carried failures : every graded request of the first attempts (280 chains per model) by its outcome, and what happens to the tests that fail. Clean: every test of the universe passes; New: a new failure on a repository with no earlier failure; Carrying: an earlier failure is still failing and nothing new fails; Both. “New failure rate” is the share of requests with a new failure among those issued on a clean repository and among those issued on a repository carrying a failure (descriptive; the two groups differ in position and chain). Recovered: tests that fail at some request and pass again at a later one (median delay in requests). Healed: chains broken at some request whose repository passes every test at request 20. Tests that fail on the received tree only because the request’s test overlay rewrote them are excluded.

Model	Ran tests	Repo suite	Wrote test	Saw failure	Last run green	Claimed success	Green, then claimed
Fable 5	100.0 / 98.3	73.3 / 58.3	26.7 / 5.0	56.7 / 41.7	71.7 / 68.3	65.0 / 26.7	46.7 / 20.0
GPT-6 Astra	100.0 / 100.0	91.7 / 90.0	6.7 / 0.0	90.0 / 78.3	78.3 / 65.0	98.3 / 96.7	76.7 / 61.7
Kimi K3	98.3 / 98.3	85.0 / 83.3	25.0 / 15.0	75.0 / 65.0	66.7 / 65.0	96.7 / 98.3	66.7 / 65.0
GPT-5.6	100.0 / 100.0	86.7 / 88.3	13.3 / 15.0	80.0 / 61.7	95.0 / 91.7	40.0 / 35.0	40.0 / 31.7
GLM-5.2	100.0 / 96.7	88.3 / 80.0	36.7 / 15.0	81.7 / 65.0	78.3 / 75.0	96.7 / 91.7	76.7 / 70.0
All (300 / 300)	99.7 / 98.7	85.0 / 80.0	21.7 / 10.0	76.7 / 62.3	78.0 / 73.0	79.3 / 69.7	61.3 / 49.7

Appendix

Table 35: Self-checks before submission , first attempts. Each cell gives the percentage of sampled silent misses / accepted submissions (60 each per model) that show the behavior. “Green, then claimed”: the last test run passed and the closing message claimed success. “Wrote test” includes scratch files. Claim rates depend on whether a model writes closing text, and the success-claim odds ratio is driven by Fable 5 (without it: 1.35, 0.71–2.70). Pooled intervals and odds ratios (OR, Mantel–Haenszel across models, silent miss vs. accepted) use a repository-clustered bootstrap.

Request 8	Add @app.command , @app.argument and @app.option ; a lone @app.argument or @app.option registers a command under the function’s name. GPT-5.6’s request-8 code does this, and the request passes.
Request 9	Let the three decorators stack on one function. In a fresh conversation, the agent reads the history, finds the decorators “only just added in the latest commit” and concludes: “There was no earlier decorator contract to preserve”. It makes argument and option store definitions only, checks the stacked form (“all 41 application tests pass”) and submits.
Hidden tests	Request 9’s new test passes. Two request-8 tests fail: application.has(...) is False after a lone decorator. Opus 5.5, Opus 5, Fable 5, GLM-5.3 and GLM-5.3-Flash pass request 9 on their first attempts.

Appendix

Table 36: A regression that undoes accepted work ( python-poetry/cleo , GPT-5.6, first attempt; abridged, quotes verbatim). Request 9 is silent on a lone decorator, and the agent’s conversation holds nothing of request 8. A grader of request 9’s new test alone would accept this code. Chosen for clarity, not at random ( Appendix I.3 ).

	Reject (once, at the miss)				Gate (every request): completion (%)
	Repair	Next request fails (%) on a state that is			No	Perfect
Model	rate (%)	repaired	unrepaired	certified	supervisor	supervisor
Fable 5	44.6	5.7	22.7	3.8	65.7	77.5
GPT-6 Astra	45.4	5.1	28.2	4.3	52.1	68.2
Kimi K3	29.4	3.4	25.0	9.6	48.2	56.1 †
GPT-5.6	30.0	12.4	30.0	16.9	43.9	59.3

Appendix

Table 37: What one bit from outside recovers: summary of Section 5 (280 chains; one conversation per chain, continued after the miss; completion = every request passes on its own step). Reject : the message once, at a silent miss, with one repair round; Gate : the same after every failed submission of a chain. The certified column is the reference of a state that passed its contract. The lower block reports the separate 100-chain Noisy study:

p

is the false-alarm rate and

p{=}0

a perfect supervisor. † Lower bound ( Appendix K ).

	GPT-5.6	Kimi K3
Starting points, Reject / Name / Insist (repositories)	939 / 939 / 939 (100)	834 / 827 / 821 (96)
Reject, repair rate after one round (%)	30.0 [27.2, 33.0]	29.5 [26.5, 32.7]
Name, repair rate after one round (%)	46.4 [43.3, 49.6]	46.8 [43.4, 50.2]
Name $-$ Reject, paired (pp)	16.4 [11.0, 22.0]	17.1 [11.2, 22.8]
Insist, repair rate after round 1, 2, …, 8 (%)	30.0, 38.1, 44.3, 47.6, 49.8, 52.2, 53.0, 53.9	31.1, 42.3, 48.2, 52.6, 54.1, 56.0, 57.6, 58.7
Increment of rounds 2, …, 8 (pp)	8.1, 6.2, 3.3, 2.2, 2.3, 0.9, 0.9	11.2, 6.0, 4.4, 1.5, 2.0, 1.6, 1.1

Appendix

Table 38: Name and Insist (280 chains). Repair rate of the starting-point miss after

r

rounds; increments per round; calls until the miss passes among repaired misses and calls spent among misses never repaired (Insist arm). Wilson intervals for rates, repository-clustered bootstrap for the paired contrast.

	Fable 5	GPT-6 Astra	Kimi K3	GPT-5.6	GLM-5.2
No supervisor: first attempts completing 20 (of 280)	184	146	135	123	78
Failed first attempts audited	96	134	145	157	202
Rescued by verification and repair	33	45	22	43	38
Perfect supervisor: completion (%)	77.5 [72.3, 82.0]	68.2 [62.5, 73.4]	56.1 [50.2, 61.8]	59.3 [53.4, 64.9]	41.4 [35.8, 47.3]
Gain (pp; relative)	11.8 (18%)	16.1 (31%)	7.9 (16%)	15.4 (35%)	13.6 (49%)
Unresolved continuations (counted as failures)	0	0	18	0	0

Appendix

Table 39: Gate (280 chains). Completion with no supervisor (first attempt in the continuous conversation) and with a perfect supervisor (one repair round at every failed request); Wilson intervals over 280 chains.

GPT-5.6-sol — no supervisor 44 [35, 54]; perfect supervisor 60 [50, 70]
	$p{=}0.1$	$p{=}0.2$	$p{=}0.3$	$p{=}0.4$	$p{=}0.5$
Completion (%)	58 [49, 67]	54 [44, 64]	46 [37, 56]	49 [39, 59]	47 [37, 57]
Difference from perfect supervisor, paired (pp)	$-2$ [ $-9$ , 5]	$-6$ [ $-13$ , 1]	$-14$ [ $-23$ , $-5$ ]	$-11$ [ $-19$ , $-2$ ]	$-13$ [ $-21$ , $-5$ ]
Chains ended by a false alarm (%)	3	2	8	7	10
Phantom rounds ( $n$ ); broken (%); unchanged (%)	147; 2.0; 2.0	298; 0.7; 1.7	416; 1.9; 0.2	581; 1.2; 1.2	701; 1.4; 2.0
Calls per phantom round, median	8	8	7.5	8	8

Appendix

Table 40: Noisy supervisor (sealed run; 100 chains per level, 0 censored). Completion against the false-alarm rate

p

, its paired difference from the perfect supervisor on the same chains, and the fate of the phantom rounds. Brackets: repository-clustered bootstrap 95% intervals (2,000 replicates).

Model	A / V	$\Delta$ (pp)	A / V	$\Delta$ (pp)	A / V	A / V	A / V
	pass@1 (%)		Regression rate (%)		Strict (%)	Tests repaired (%)	Calls
Opus 5.5	66.8 / 73.9	$+7.1$ [2.5, 12.0]	21.1 / 25.7	$+4.6$ [0.7, 8.6]	55.7 / 57.9	30 / 60	217 / 234
Opus 5	60.7 / 69.6	$+8.9$ [3.6, 14.3]	27.5 / 32.5	$+5.0$ [ $-0.7$ , 10.6]	51.1 / 50.0	26 / 43	401 / 486
GPT-6 Astra	51.1 / 53.9	$+2.9$ [ $-2.4$ , 8.0]	35.0 / 42.5	$+7.5$ [3.0, 12.2]	36.1 / 36.4	12 / 39	182 / 209
Fable 5	51.8 / 63.9	$+12.1$ [6.8, 17.5]	42.1 / 46.1	$+3.9$ [ $-1.4$ , 9.4]	36.4 / 38.9	19 / 38	293 / 350
GLM-5.3	46.4 / 68.6	$+22.1$ [15.8, 28.4]	31.8 / 35.0	$+3.2$ [ $-2.9$ , 9.8]	38.2 / 40.7	16 / 49	560 / 684

Appendix

Table 41: Verified under the handoff protocol (ten models, 280 chains each, one attempt per arm; Wilson 95% intervals per arm, repository-clustered bootstrap 95% intervals for the paired differences Verified

-

Autonomous; A = Autonomous (attempt 1), V = Verified). Tests repaired is the share of tests that fail for the first time after a request and pass after the next one: after the repair round in the Verified arm, by the model’s own later work in the Autonomous arm. Calls per chain are medians and include the repair rounds.

Figure 11: One “verification failed” message raises pass@1 for every model, and raises the regression rate too. Single-attempt pass@1 (left) and regression rate (right) of the Autonomous run (hollow) and of the Verified arm (filled) for ten models on the same 280 chains; labels give the paired difference with its repository-clustered 95% interval, dark where the interval excludes zero (seven of ten for pass@1, four of ten for the regression rate).

	Slots	Name,	Reject once,	Name $-$ Reject once,	Insist,	Insist rounds	Round broke a test (%),
Model	(Name / Insist)	1 round (%)	1 round (%)	paired (pp)	4 rounds (%)	to match Name	Name / Reject once
Claude Opus 5.5	517 / 517	44.5	16.8	$+27.7$ [18.5, 37.3]	28.8	$>4$	12.0 / 7.2
Claude Opus 5	632 / 632	–	–	–	–	–	–
GPT-6 Astra	1090 / 1090	35.6	–	–	–	–	12.2 / –
Claude Fable 5	875 / 875	–	–	–	–	–	–
GLM-5.3	808 / 808	27.2	–	–	–	–	9.8 / –

Appendix

Table 42: Name and Insist under the handoff protocol. Repair rate of the starting-point miss after one round with the failing tests named (Name) and with the bare message (Reject once = Insist round 1), their paired difference on identical slots (repository-clustered bootstrap 95%), the bare message after four rounds, the Insist rounds needed to reach Name’s one round, and the share of rounds that broke a protected test. Wilson 95% intervals in the source; ‘–’: slots still running when the table was built.

Model	$p$	noisy / Auton. / $p{=}0$	(pp)	(pp)	noisy / Auton. / $p{=}0$	( $n$ ; broke, %)	of calls (%)
		pass@1 (%)	$\Delta$ vs Autonomous	$\Delta$ vs $p{=}0$	Regression (%)	Phantom rounds	Phantom share
Claude Opus 5.5	0.1	71 / 69 / 76	$+2$ [ $-6$ , 10]	$-5$ [ $-13$ , 2]	41 / 20 / 28	169; 9.5	11
	0.5	67 / 69 / 76	$-2$ [ $-11$ , 7]	$-9$ [ $-18$ , $-1$ ]	44 / 20 / 28	769; 4.5	33
Claude Opus 5	0.1	–	–	–	–	–	–
	0.5	–	–	–	–	–	–
GPT-6 Astra	0.1	55 / 52 / 53	$+3$ [ $-5$ , 11]	$+2$ [ $-6$ , 10]	48 / 41 / 47	143; 3.5	6

Appendix

Table 43: Noisy watcher under the handoff protocol (100 chains per cell, one attempt per arm; paired differences with repository-clustered bootstrap 95% intervals). Single-attempt pass@1 of the noisy arm next to the Autonomous run (attempt 1) and the Verified arm (

p{=}0

) on the same chains; regression rate of the three; the fate of the phantom rounds (share that broke a passing test) and their share of all model calls. ‘–’: chains still running when the table was built.

Model	Harness	pass@1 (of 58)	Paired $\Delta$ (pp)	Strict	Regressed	Same verdict
Claude Opus 5.5	mini-swe-agent	39 [54.4, 77.9]	—	29	15	—
	Codex CLI	31 [40.8, 65.7]	$-13.8$ [ $-26.7$ , $-1.7$ ]	27	16	44
	OpenHands SDK	34 [45.8, 70.4]	$-8.6$ [ $-21.3$ , 3.4]	30	17	47
	pi	35 [47.5, 71.9]	$-6.9$ [ $-17.5$ , 3.6]	29	12	48
GPT-5.6-sol	mini-swe-agent	32 [42.5, 67.3]	—	20	27	—
	Codex CLI	28 [35.9, 60.8]	$-6.9$ [ $-15.5$ , 0.0]	21	25	52

Appendix

Table 44: Scaffold sensitivity (58 chains, 52 repositories; handoff protocol; one attempt per arm). Single-attempt pass@1 of each model under three harnesses next to its mini-swe-agent run (attempt 1) on the same chains; Wilson 95% intervals per arm, repository-clustered bootstrap 95% intervals for the paired difference; strict pass@1 and regression rate as chain counts; chains with the same pass@1 verdict under the harness and under mini-swe-agent.

Figure 12: A different harness does not help: none raises pass@1 for any model. Single-attempt pass@1 of four models under Codex CLI, the OpenHands SDK and pi next to their mini-swe-agent first attempt on the same 58 chains (left), and the paired difference harness

-

mini-swe-agent with its repository-clustered 95% interval (right; dark labels where the interval excludes zero): five of the twelve intervals lie entirely below zero and none above it.

Coding Agents

Coding agents use language models, tools, and feedback to modify software and complete engineering tasks.

See all 147 papers

Benchmarking Coding Agents under Evolving Requirements
Oct 26, 2026Jia Feng, Alyssa Wu, Cuiyun Gao +3Coding Agent Benchmarks Code Generation
Hunyuan Team, Tencent · The Chinese University of Hong Kong
SpecGuardBench: Can Coding Agents Construct Reliable Validation Harnesses from Natural Language Specifications?
Oct 12, 2026Zhiqing Zhong, Mingyu Chen, Aoyang Fang +5AI Coding Agents Specification-Based Testing
The Chinese University of Hong Kong, Shenzhen · Aalto University · Lero, the Research Ireland Centre for Software +2
EvoCode-Bench: Evaluating Coding Agents in Multi-Turn Iterative Interactions
May 22, 2026Haiyang Shen, Xuanzhong Chen, Wendong Xu +3AI Coding Agents Software Engineering Benchmarks
1UniPat AI · 2Peking University · 3Tsinghua University +1

Oct 26, 2026J/K move · Enter open · S save

Benchmarking Coding Agents under Evolving Requirements

Jia Feng, Alyssa Wu, Cuiyun Gao, Yunlong Wang+2

Hunyuan Team, Tencent · The Chinese University of Hong Kong

Coding Agent Benchmarks Code Generation Process-Based Agent Evaluation LLM Agent Reliability

Open paper AskPDF arXiv

Software development with coding agents often begins with a rough request that users refine after trying the implementation. Final correctness alone does not capture the repeated guidance this process requires. We introduce FeatureLoop, a benchmark of 61 human-reviewed tasks from 34 repositories across five primary programming languages and seven software forms. Each task follows one development goal in a persistent workspace through evolution feedback, which adds or clarifies requirements, and completion feedback, which reports failures of stated requirements. Cumulative executable checks track new behavior, repair, and preservation of earlier functionality. We evaluate 12 frontier models and find that the strongest completes 72.1% of tasks with both feedback types. Across models, however, only 15.4% of observed failed stages recover within two corrective revisions. Further analysis reveals incomplete repairs, regressions, and model-specific differences in feedback use and token efficiency, identifying obstacles to reliable, low-intervention development.

Abstract

Coding Agents

Explore similar work

Benchmarking Coding Agents under Evolving Requirements

SpecGuardBench: Can Coding Agents Construct Reliable Validation Harnesses from Natural Language Specifications?

EvoCode-Bench: Evaluating Coding Agents in Multi-Turn Iterative Interactions