Coding Agents Have Converged: Why the SWE-bench Leaderboard Can No Longer Order Its Top Entries, and What to Measure Instead

Fengshuo Liu; Ying Liu; Ruize Sun; Lie Luo; Siyuan Guo

cs.SESep 15, 2026

Coding Agents Have Converged: Why the SWE-bench Leaderboard Can No Longer Order Its Top Entries, and What to Measure Instead

Authors: Fengshuo Liu, Ying Liu, Ruize Sun, Lie Luo, Siyuan Guo

Organizations: Imperial College London, London, United Kingdom · The Hong Kong Polytechnic University, Hong Kong SAR · Korea University, Seoul, Korea · Jinan University, Guangzhou, China

Abstract

Small differences on coding-agent leaderboards are often read as an ordering of systems. We audit whether the published verdicts support this reading, using 254 SWE-bench submissions across four splits without running models. On Verified, the leading two entries each resolve 396 of 500 instances. The top ten share 285 successes and 51 failures, leaving 164 instances that distinguish their outcomes. Frontier solution sets have median nesting 0.935 against a score-implied baseline of 0.774, indicating strongly shared successes. Scores also depend on the evaluated model-scaffold pair: observed within-model scaffold ranges reach 29.8 percentage points, compared with the 8.8-point spread of the top thirty. Six of nine cell-mean interaction tests remain significant after Holm correction, although this observational design does not identify causal scaffold effects. Exact paired McNemar tests separate none of the 29 adjacent Verified top-thirty pairs at alpha=0.05, while the larger Test split separates 14 of 23. A stated leader-based rule yields three descriptive tiers, or two after Holm correction; non-rejection does not establish equivalence. We release the partition and a five-step audit protocol that profiles shared outcomes, tests paired differences, reports grouping sensitivity, and estimates the instance budget needed for resolution. The results motivate reporting comparison-set-specific resolution and model-scaffold provenance instead of interpreting small aggregate gaps as established rank differences.

Figures & tables

Figure 1: The audit: $254$ public submissions and incomplete metadata feed analyses of shared outcomes, model–scaffold associations, and paired rank separability. Outputs are descriptive tiers, a Test-split counter-test, and benchmark-design guidance. Tiers use uncorrected leader comparisons; Holm gives $19/11$ groups. Attribution is observational, not causal. The $734{,}000$ instance figure is an independent-sample power reference, not a paired-test requirement (Section 5 ).

Split	Submissions	Instances	Verified overlap	Role
Verified	134	500	—	main analysis
Lite	84	299	93 (31%)	indep. replication
Test	24	2,294	—	counter-test
Multimodal	12 of 22	301	—	descriptive

Table 1: The analysis pool: the four public splits that publish per-instance verdicts, as retrieved on 30 July 2026. Sections 4 – 5 use Verified; Section 3 reports all four.

$K$	solved by all	solved by none	$n_{\mathit{eff}}/n$	KR-20
Verified ( $134$ submissions, $n=500$ ; top-10 mean $77.3\%$ )
2	378	86	0.07	—
5	334	60	0.21	—
10	285	51	0.33	0.475
20	250	41	0.42	0.722
50	146	35	0.64	0.940

Table 2: Degenerate and discriminating instances as the comparison set narrows. KR-20 is omitted for

K<10

, where it is unstable.

Figure 2: The $164$ instances that still separate the top ten, one row per system, with the two degenerate blocks compressed to side strips ( $285$ resolved by all ten, $51$ by none). The top strip counts how many of all $134$ submissions resolve each instance — a difficulty gradient the top ten themselves cannot see. The dashed line marks the tier-1/tier-2 boundary (Section 5 ).

band (resolved instances)	systems	median $\mathrm{cov}$	baseline	excess
frontier ( $\geq 370$ )	16	0.935	0.774	$+0.161$
strong ( $300$ – $369$ )	38	0.913	0.708	$+0.205$
mid ( $200$ – $299$ )	48	0.828	0.530	$+0.298$
weak ( $100$ – $199$ )	23	0.712	0.370	$+0.342$

Table 3: Nesting of solution sets within ability bands (SWE-bench Verified), against the baseline implied by the scores alone.

split	$n$	systems	$n_{\mathit{eff}}/n$	spread	adj. pairs	separable
Verified	0 500	134	0.33	0 8.8 pp	29	0 0
Lite	0 299	0 84	0.54	21.1 pp	29	0 0
Multimodal	0 301	0 12	0.44	18.3 pp	11	0 0
Test	2294	0 24	0.52	52.4 pp	23	14

Table 4: The same adjacent-pair test on every public split with per-instance results.

n_{\mathit{eff}}/n

and “spread” are computed over the top ten and the tested comparison set respectively. Separability tracks the spread and the instance count, exactly as the design arithmetic of Section 6 predicts.

Tier	resolved instances (% of $500$ )	systems
1	$382$ – $396$ ( $76.4$ – $79.2\%$ )	8
2	$362$ – $378$ ( $72.4$ – $75.6\%$ )	12
3	$352$ – $359$ ( $70.4$ – $71.8\%$ )	10

Table 5: Descriptive tiers of the SWE-bench Verified top thirty using uncorrected leader comparisons at

\alpha=0.05

. Membership does not establish equivalence or that every within-tier pair is non-significant. Boundaries depend on the rule; full membership is released with the analysis scripts.

Figure 3: The top thirty of SWE-bench Verified. Each dot is a submission’s score; the grey bar spans the scores of every other entry in the top thirty from which an exact McNemar test cannot separate it (

p\geq 0.05

). The bars are wide enough to cover much of the table. Bars are score spans, not confidence or rank intervals; non-significance need not hold for every intermediate score. Shaded bands mark the three uncorrected leader-based tiers.

Figure 4: Rescoring the top thirty on only the

209

instances non-degenerate for the top twenty. The scale widens and three adjacent pairs swap places, yet nothing is gained: the paired tests behind Section 5 ’s tiers already discard degenerate instances, so separable pairs stay at zero.

AI Agent Evaluation

AI agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently across interactive settings.

See all 516 papers

Frozen Judges, Moving Agents: Version-Dependent LLM-Judge Error and the Limits of Judge-Assisted Agent Evaluation
Sep 28, 2026Jiapeng LiLLM-as-a-Judge AI Agent Auditing
Are Performance-Optimization Benchmarks Reliably Measuring Coding Agents?
Jul 1, 2026Zhi Chen, Zhensu Sun, Yuling Shi +2Benchmark Auditing Benchmark Design
Singapore Management University · Singapore, Singapore · Shanghai Jiao Tong University +1
SWE-Bench Pro Verified: A Reliable Benchmark for Software Engineering Agents
Sep 8, 2026Pujun Zheng, Zixin Shang, Shufan Jiang +5Reward Hacking Software Engineering Agents
East China Normal University · Shanghai Artificial Intelligence Laboratory · Fudan University

Sep 28, 2026J/K move · Enter open · S save

Frozen Judges, Moving Agents: Version-Dependent LLM-Judge Error and the Limits of Judge-Assisted Agent Evaluation

Jiapeng Li

LLM-as-a-Judge AI Agent Auditing LLM Agent Evaluation Benchmark Validity

Open paper AskPDF arXiv

Language-model judges compare agent upgrades with their predecessors, but a fixed judge can make version-dependent mistakes. We analyze 35 public coding-agent submissions (20 prespecified version pairs on 250 SWE-bench Verified issues), two customer-service agents (155 tau-bench tasks), and 1,106 expert-labeled AgentRewardBench trajectories. An upstream outage left three judges for the primary SWE-bench analysis (8,743 aligned cells); the fourth is descriptive. All three coding-agent judges and all four tau-bench judges reject task-conditioned error invariance after multiplicity adjustment. On SWE-bench, 32 of 60 judge-by-pair units have a detectable differential comparison component; eight judge-only intervals declare upgrades that execution-based intervals cannot establish, despite rank correlations of 0.71-0.79. In tau-bench, one judge reverses a nine-point reference-reward gap by penalizing a procedural habit the reward ignores. False acceptance of failed coding patches rises with agent capability conditional on task and execution outcome, while a task-solvability prediction reverses sign across domains. A separately fixed post-submission OpenHands follow-up on the same 250 issues (eight configurations, 1,981 three-judge cells) reproduces the capability/false-acceptance association (mean Spearman +0.944, exact p=0.000099) and decreasing Youden contrast (mean -0.937, p=0.000397); this is observational, not a new-task replication. Transporting old-version calibration raises SWE-bench comparison error from 3.8 to 19.5 points, with 24.6% undefined bootstrap ratios. A paired audit saves only 5% in interval width at 80 labeled tasks. A randomized self-report test is negative (three adjusted p-values=1.0). These results favor paired audits of current outputs over judge-only release decisions or transported calibration; independent human patch review remains pending.

Abstract

AI Agent Evaluation

Explore similar work

Frozen Judges, Moving Agents: Version-Dependent LLM-Judge Error and the Limits of Judge-Assisted Agent Evaluation

Are Performance-Optimization Benchmarks Reliably Measuring Coding Agents?

SWE-Bench Pro Verified: A Reliable Benchmark for Software Engineering Agents