Organizations: Imperial College London, London, United Kingdom · The Hong Kong Polytechnic University, Hong Kong SAR · Korea University, Seoul, Korea · Jinan University, Guangzhou, China
Small differences on coding-agent leaderboards are often read as an ordering of systems. We audit whether the published verdicts support this reading, using 254 SWE-bench submissions across four splits without running models. On Verified, the leading two entries each resolve 396 of 500 instances. The top ten share 285 successes and 51 failures, leaving 164 instances that distinguish their outcomes. Frontier solution sets have median nesting 0.935 against a score-implied baseline of 0.774, indicating strongly shared successes. Scores also depend on the evaluated model-scaffold pair: observed within-model scaffold ranges reach 29.8 percentage points, compared with the 8.8-point spread of the top thirty. Six of nine cell-mean interaction tests remain significant after Holm correction, although this observational design does not identify causal scaffold effects. Exact paired McNemar tests separate none of the 29 adjacent Verified top-thirty pairs at alpha=0.05, while the larger Test split separates 14 of 23. A stated leader-based rule yields three descriptive tiers, or two after Holm correction; non-rejection does not establish equivalence. We release the partition and a five-step audit protocol that profiles shared outcomes, tests paired differences, reports grouping sensitivity, and estimates the instance budget needed for resolution. The results motivate reporting comparison-set-specific resolution and model-scaffold provenance instead of interpreting small aggregate gaps as established rank differences.
Figures & tables
Figure 1: The audit: 254 public submissions and incomplete metadata feed analyses of shared outcomes, model–scaffold associations, and paired rank separability. Outputs are descriptive tiers, a Test-split counter-test, and benchmark-design guidance. Tiers use uncorrected leader comparisons; Holm gives 19/11 groups. Attribution is observational, not causal. The 734,000 instance figure is an independent-sample power reference, not a paired-test requirement (Section 5 ).
Split
Submissions
Instances
Verified overlap
Role
Verified
134
500
—
main analysis
Lite
84
299
93 (31%)
indep. replication
Test
24
2,294
—
counter-test
Multimodal
12 of 22
301
—
descriptive
Table 1: The analysis pool: the four public splits that publish per-instance verdicts, as retrieved on 30 July 2026. Sections 4 – 5 use Verified; Section 3 reports all four.
K
solved by all
solved by none
neff/n
KR-20
Verified ( 134 submissions, n=500 ; top-10 mean 77.3% )
2
378
86
0.07
—
5
334
60
0.21
—
10
285
51
0.33
0.475
20
250
41
0.42
0.722
50
146
35
0.64
0.940
Table 2: Degenerate and discriminating instances as the comparison set narrows. KR-20 is omitted for K<10 , where it is unstable.
Figure 2: The 164 instances that still separate the top ten, one row per system, with the two degenerate blocks compressed to side strips ( 285 resolved by all ten, 51 by none). The top strip counts how many of all 134 submissions resolve each instance — a difficulty gradient the top ten themselves cannot see. The dashed line marks the tier-1/tier-2 boundary (Section 5 ).
band (resolved instances)
systems
median cov
baseline
excess
frontier ( ≥370 )
16
0.935
0.774
+0.161
strong ( 300 – 369 )
38
0.913
0.708
+0.205
mid ( 200 – 299 )
48
0.828
0.530
+0.298
weak ( 100 – 199 )
23
0.712
0.370
+0.342
Table 3: Nesting of solution sets within ability bands (SWE-bench Verified), against the baseline implied by the scores alone.
split
n
systems
neff/n
spread
adj. pairs
separable
Verified
0 500
134
0.33
0 8.8 pp
29
0 0
Lite
0 299
0 84
0.54
21.1 pp
29
0 0
Multimodal
0 301
0 12
0.44
18.3 pp
11
0 0
Test
2294
0 24
0.52
52.4 pp
23
14
Table 4: The same adjacent-pair test on every public split with per-instance results. neff/n and “spread” are computed over the top ten and the tested comparison set respectively. Separability tracks the spread and the instance count, exactly as the design arithmetic of Section 6 predicts.
Tier
resolved instances (% of 500 )
systems
1
382 – 396 ( 76.4 – 79.2% )
8
2
362 – 378 ( 72.4 – 75.6% )
12
3
352 – 359 ( 70.4 – 71.8% )
10
Table 5: Descriptive tiers of the SWE-bench Verified top thirty using uncorrected leader comparisons at α=0.05 . Membership does not establish equivalence or that every within-tier pair is non-significant. Boundaries depend on the rule; full membership is released with the analysis scripts.
Figure 3: The top thirty of SWE-bench Verified. Each dot is a submission’s score; the grey bar spans the scores of every other entry in the top thirty from which an exact McNemar test cannot separate it ( p≥0.05 ). The bars are wide enough to cover much of the table. Bars are score spans, not confidence or rank intervals; non-significance need not hold for every intermediate score. Shaded bands mark the three uncorrected leader-based tiers.
Figure 4: Rescoring the top thirty on only the 209 instances non-degenerate for the top twenty. The scale widens and three adjacent pairs swap places, yet nothing is gained: the paired tests behind Section 5 ’s tiers already discard degenerate instances, so separable pairs stay at zero.
Language-model judges compare agent upgrades with their predecessors, but a fixed judge can make version-dependent mistakes. We analyze 35 public coding-agent submissions (20 prespecified version pairs on 250 SWE-bench Verified issues), two customer-service agents (155 tau-bench tasks), and 1,106 expert-labeled AgentRewardBench trajectories. An upstream outage left three judges for the primary SWE-bench analysis (8,743 aligned cells); the fourth is descriptive. All three coding-agent judges and all four tau-bench judges reject task-conditioned error invariance after multiplicity adjustment. On SWE-bench, 32 of 60 judge-by-pair units have a detectable differential comparison component; eight judge-only intervals declare upgrades that execution-based intervals cannot establish, despite rank correlations of 0.71-0.79. In tau-bench, one judge reverses a nine-point reference-reward gap by penalizing a procedural habit the reward ignores. False acceptance of failed coding patches rises with agent capability conditional on task and execution outcome, while a task-solvability prediction reverses sign across domains. A separately fixed post-submission OpenHands follow-up on the same 250 issues (eight configurations, 1,981 three-judge cells) reproduces the capability/false-acceptance association (mean Spearman +0.944, exact p=0.000099) and decreasing Youden contrast (mean -0.937, p=0.000397); this is observational, not a new-task replication. Transporting old-version calibration raises SWE-bench comparison error from 3.8 to 19.5 points, with 24.6% undefined bootstrap ratios. A paired audit saves only 5% in interval width at 80 labeled tasks. A randomized self-report test is negative (three adjusted p-values=1.0). These results favor paired audits of current outputs over judge-only release decisions or transported calibration; independent human patch review remains pending.
Repository-level performance-optimization benchmarks such as GSO, SWE-Perf and SWE-fficiency evaluate coding agents by applying patches to real repositories and comparing runtime against unoptimized baselines and official reference patches. Their leaderboard scores are increasingly used as evidence of coding-agent progress, but those scores can conflate runtime instability, benchmark-specific scoring rules, and how many tasks are already solved by at least one public submission. We audit these issues across the three benchmarks. First, we replay the official reference patches for 740 code optimization tasks across four common types of Google Cloud machines. Most benchmark tasks can be replayed, but their reference patches satisfy the original benchmark validity rules in every cross-machine replay for only 39/102 GSO tasks, 11/140 SWE-Perf tasks, and 411/498 SWE-fficiency tasks; SWE-Perf is especially fragile because many reference patches produce close-to-zero runtime changes. Second, we show that public submission rankings depend strongly on the benchmark scoring rule. Among eight public submissions shared by GSO and SWE-fficiency, the official rankings disagree on 9 of 28 pairwise submission comparisons, and SWE-fficiency's leaderboard scoring rule assigns the worst ten tasks overly high score weights of 58.5%-82.8%. Third, looking across 10 public submissions for each task, we find that at least one submission matches or beats the reference patch on 85.3% (384/450) of replay-valid GSO and SWE-fficiency tasks, and beats the unoptimized base code on 99.8% (449/450). Our study complements leaderboard scores by identifying tasks with more reliable performance signals, quantifying per-task score contributions, and exposing the remaining performance gaps that are hidden by aggregate rankings.
Zhi Chen, Zhensu Sun, Yuling Shi +2
Singapore Management University · Singapore, Singapore · Shanghai Jiao Tong University +1
SWE-Bench Pro has emerged as a standard benchmark for evaluating software engineering agents on challenging repository-level tasks. However, our analysis work show that its evaluation is undermined by two sources of unreliability: reward hacking, enabled by leakage of gold solutions or hidden evaluation information, and task quality issues, including misleading problem statements and improperly scoped tests. These issues can inflate benchmark performance and obscure agents' true coding ability. We present SWE-Bench Pro Verified, a verified version of SWE-Bench Pro that addresses both problems. Our approach combines anti-hacking safeguards that eliminate major leakage channels without disrupting normal agent functionality, with task refinement that minimally corrects inconsistencies within flawed instances. Evaluations on SWE-Bench Pro Verified reveal that some models perform substantially worse than previously evaluated, suggesting that existing results on SWE-Bench Pro may overestimate real software engineering capability. SWE-Bench Pro Verified offers a more trustworthy benchmark for assessing software engineering agents.
Pujun Zheng, Zixin Shang, Shufan Jiang +5
East China Normal University · Shanghai Artificial Intelligence Laboratory · Fudan University