SOTA or Luck? The Winner's Curse in LLM Leaderboards
Authors: Sikun Xu, Raphael Thomadsen, Dennis J. Zhang, Heng Zhang
Organizations: Olin Business School, Washington University in St. Louis St. Louis, MO 63130 · W. P. Carey School of Business, Arizona State University Tempe, AZ 85287
Public LLM leaderboards guide model evaluation and adoption, yet the reported performance of top-ranked models can be systematically overstated. This leaderboard winner's curse arises because ranking noisy benchmark scores rewards sampling luck alongside genuine ability. The same selection effect can distort scores throughout the leaderboard, a phenomenon we call the rank-induced bias. We develop an m-out-of-N bootstrap to correct for this bias. It assumes a fixed model pool with independent, identically distributed bounded tasks while allowing arbitrary within-task correlation among models. Under exact population tiers separated by positive gaps, we prove that the bootstrap reduces Θ(N−1/2) ranking bias to expected residual bias o(N−1/2). Across 14 benchmarks from 5 benchmark families, the rank-1 adjustments range from approximately 0.09 to 3.86 percentage points. In our SWE-bench Verified snapshot, the tied leaders receive an adjustment of 0.90 percentage points. Such overstatement is large enough to make the apparent winner statistically fragile. Thus, we use the same bootstrap to further quantify the intervals of plausible rank positions for each model and group models into tiers. We establish sufficient conditions for consistent recovery of population tiers. On SWE-bench Verified, this procedure places 13 of 173 models in the leading tier. We combine the score adjustment and rank intervals in the Rank Uncertainty and Leaderboard Evaluation Report (RULER). We validate its finite-sample performance using sample splitting and calibrated simulations, and advocate publishing RULER alongside raw rankings so readers can assess both reported performance and the precision of the ordering. RULER also provides guidance for benchmark design that accounts for selection effects and ranking uncertainty.
Figures & tables
Figure 1: The winner on SWE-bench Verified remains statistically unresolved.
Magnitude of Mean Error
Rank
Before RULER
After RULER
Reduction (%)
1
2.38
0.73
69.5
2
1.43
0.74
48.0
3
1.34
0.71
46.9
4
1.10
0.50
55.0
5
1.23
0.58
52.7
Table 1: The correction reduces average selection error across 14 benchmarks.
Mean Signed Error (pp)
RMSE (pp)
Rank
No Correction
m -out-of- N bootstrap
No Correction
m -out-of- N bootstrap
1
+1.41±0.14
+0.21±0.15
1.97±0.13
1.55±0.10
2
+0.80±0.17
−0.06±0.17
1.85±0.18
1.74±0.14
3
+0.66±0.18
−0.12±0.18
1.87±0.16
1.78±0.14
4
+0.98±0.16
+0.23±0.16
1.85±0.11
1.64±0.13
5
+0.96±0.17
+0.24±0.18
1.97±0.15
1.79±0.13
Table 2: The m -out-of- N correction removes most of the winner’s bias and lowers its root mean squared error.
Raw Rank
Model
Raw Score (%)
Score Correction (pp)
80% Rank Interval
Tier
1–2
live-SWE + Claude 4.5 Opus med.
79.20
0.90
[1,6]
1
1–2
Sonar + Claude 4.5 Opus
79.20
0.90
[1,6]
1
3
TRAE + Doubao-Seed-Code
78.80
0.60
[1,7]
1
4
live-SWE + Gemini 3 Pro
77.40
0.69
[2,13]
1
5–7
Atlassian Rovo Dev
76.80
0.63
[3,15]
1
5–7
EPAM AI/Run v20250719 + Claude 4 Sonnet
76.80
0.63
[3,15]
1
Table 3: RULER reveals score overstatement and unresolved ranks among SWE-bench Verified’s leaders.
Benchmark
Number of Tasks
Number of Models
Position 1 Score (Raw)
Position 1 Score (Adjusted)
Number of Tiers
Number of Models in Tier 1
SWE Bash-only
500
41
76.80
75.71
10
6
SWE Verified
500
173
79.20
78.30
15
13
SWE Multimodal
510
12
36.47
35.84
3
5
SWE Multilingual
300
14
72.67
71.17
4
9
SWE Pro
395
14
44.30
43.43
6
4
Tau2 Airline
50
7
69.00
65.93
2
5
Table 4: Score overstatement and unresolved leaders occur across benchmark families.
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 2: Score correction reduces inflation while near-tied models remain hard to rank.
p=0.5
p=0.75
γ/J
2
5
10
20
50
100
2
5
10
20
50
100
N=500 tasks
0.1
1.87
1.35
1.05
0.87
0.81
0.83
1.64
1.23
1.05
0.94
0.91
0.94
0.2
1.87
1.35
1.03
0.79
0.64
0.63
1.64
1.20
0.97
0.82
0.75
0.76
0.3
1.88
1.35
1.03
0.77
0.53
0.45
1.64
1.18
0.92
0.70
0.55
0.51
0.4
1.88
1.35
1.03
0.77
0.51
0.39
1.65
1.18
0.91
0.68
0.47
0.39
Appendix
Table 5: When all models are tied, exponents below the default minimize error, and their advantage over the default grows with the number of models.
p=0.5
p=0.75
γ/J
2
5
10
20
50
100
2
5
10
20
50
100
N=500 tasks
0.1
2.30
1.78
1.54
1.42
1.39
1.40
1.98
1.56
1.39
1.33
1.31
1.32
0.2
2.31
1.79
1.54
1.38
1.31
1.30
1.99
1.56
1.36
1.27
1.22
1.22
0.3
2.32
1.80
1.54
1.38
1.27
1.23
2.01
1.56
1.34
1.22
1.13
1.11
0.4
2.33
1.80
1.54
1.38
1.26
1.21
2.01
1.56
1.34
1.21
1.10
1.06
Appendix
Table 6: With closely spaced models, error is nearly flat across a wide range of exponents.
p=0.5
p=0.75
γ/J
2
5
10
20
50
100
2
5
10
20
50
100
N=500 tasks
0.1
2.47
2.81
2.81
2.64
2.67
2.69
2.11
2.44
2.38
2.23
2.23
2.26
0.2
2.43
2.80
2.80
2.61
2.61
2.62
2.08
2.44
2.39
2.22
2.22
2.24
0.3
2.32
2.73
2.74
2.57
2.55
2.55
1.97
2.39
2.35
2.20
2.18
2.19
0.4
2.20
2.62
2.63
2.49
2.47
2.46
1.88
2.29
2.25
2.13
2.11
2.12
Appendix
Table 7: With well-separated models, large exponents minimize error.
Family
Score and sampling unit
Target population and interpretation
SWE-bench (5 benchmarks)
Binary issue resolution. Unit: issue.
Software issues under each variant’s task and scaffold rules, evaluated by the retained model-plus-agent systems. The analysis conditions on submitted trajectories. Model-plus-agent systems can differ across entries.
Tau2-Bench (3 domains)
Released task reward in [0,1] . Unit: customer-service scenario.
Airline, retail, or telecom scenarios under the released agents and environment. Frozen outcomes retain the realized agent randomness and environment.
AlpacaEval 2
Raw weighted rate: soft preference minus 1. Unit: instruction.
Instructions compared against a common GPT-4-Turbo reference, conditional on released responses and judge annotations. Instructions are equally weighted. The raw score differs from the length-controlled leaderboard metric.
WildBench v2
Checklist-guided judge score divided by 10, in [0.1,1] . Unit: prompt/session.
Real-user-style prompts under the frozen scoring protocol. Uncertainty concerns sampling prompts, conditional on the released judge scores.
LiveBench (4 benchmarks)
Released task score in [0,1] . Unit: task/question.
Tasks in the retained model-coverage subset. We report Coding, Language, Instruction Following, and their task-weighted pooled mean. This subset and pooled score differ from the full live leaderboard.
Appendix
Table 8: Each RULER report refers to a specified score and task population.
Benchmark
Snapshot date
Frozen source
SWE Verified
2026-02-27
Official leaderboard, 7c4289f , and task outcomes, 82be2b5 (2026-01-26).
SWE Bash-only
2026-02-27
Official leaderboard, 7c4289f , and task outcomes, 2f95aad (2026-02-27).
SWE Multimodal
2026-02-27
Official leaderboard, 7c4289f , and task outcomes, cff22d0 (2025-11-21).
SWE Multilingual
2026-02-27
Official leaderboard, 7c4289f , and task outcomes, 6083d31 (2026-02-24).
Department of Computer Science, University of Kaiserslautern–Landau (RPTU), Germany · Data Science and its Applications, German Research Centre for Artificial Intelligence (DFKI), Germany