SOTA or Luck? The Winner's Curse in LLM Leaderboards
Authors: Sikun Xu, Raphael Thomadsen, Dennis J. Zhang, Heng Zhang
Organizations: Olin Business School, Washington University in St. Louis St. Louis, MO 63130 · W. P. Carey School of Business, Arizona State University Tempe, AZ 85287
Public LLM leaderboards guide model evaluation and adoption, yet the reported performance of top-ranked models can be systematically overstated. This leaderboard winner's curse arises because ranking noisy benchmark scores rewards sampling luck alongside genuine ability. The same selection effect can distort scores throughout the leaderboard, a phenomenon we call the rank-induced bias. We develop an m-out-of-N bootstrap to correct for this bias. It assumes a fixed model pool with independent, identically distributed bounded tasks while allowing arbitrary within-task correlation among models. Under exact population tiers separated by positive gaps, we prove that the bootstrap reduces Θ(N−1/2) ranking bias to expected residual bias o(N−1/2). Across 14 benchmarks from 5 benchmark families, the rank-1 adjustments range from approximately 0.09 to 3.86 percentage points. In our SWE-bench Verified snapshot, the tied leaders receive an adjustment of 0.90 percentage points. Such overstatement is large enough to make the apparent winner statistically fragile. Thus, we use the same bootstrap to further quantify the intervals of plausible rank positions for each model and group models into tiers. We establish sufficient conditions for consistent recovery of population tiers. On SWE-bench Verified, this procedure places 13 of 173 models in the leading tier. We combine the score adjustment and rank intervals in the Rank Uncertainty and Leaderboard Evaluation Report (RULER). We validate its finite-sample performance using sample splitting and calibrated simulations, and advocate publishing RULER alongside raw rankings so readers can assess both reported performance and the precision of the ordering. RULER also provides guidance for benchmark design that accounts for selection effects and ranking uncertainty.
Figures & tables
Figure 1: The winner on SWE-bench Verified remains statistically unresolved.
Magnitude of Mean Error
Rank
Before RULER
After RULER
Reduction (%)
1
2.38
0.73
69.5
2
1.43
0.74
48.0
3
1.34
0.71
46.9
4
1.10
0.50
55.0
5
1.23
0.58
52.7
Table 1: The correction reduces average selection error across 14 benchmarks.
Mean Signed Error (pp)
RMSE (pp)
Rank
No Correction
m -out-of- N bootstrap
No Correction
m -out-of- N bootstrap
1
+1.41±0.14
+0.21±0.15
1.97±0.13
1.55±0.10
2
+0.80±0.17
−0.06±0.17
1.85±0.18
1.74±0.14
3
+0.66±0.18
−0.12±0.18
1.87±0.16
1.78±0.14
4
+0.98±0.16
+0.23±0.16
1.85±0.11
1.64±0.13
5
+0.96±0.17
+0.24±0.18
1.97±0.15
1.79±0.13
Table 2: The m -out-of- N correction removes most of the winner’s bias and lowers its root mean squared error.
Raw Rank
Model
Raw Score (%)
Score Correction (pp)
80% Rank Interval
Tier
1–2
live-SWE + Claude 4.5 Opus med.
79.20
0.90
[1,6]
1
1–2
Sonar + Claude 4.5 Opus
79.20
0.90
[1,6]
1
3
TRAE + Doubao-Seed-Code
78.80
0.60
[1,7]
1
4
live-SWE + Gemini 3 Pro
77.40
0.69
[2,13]
1
5–7
Atlassian Rovo Dev
76.80
0.63
[3,15]
1
5–7
EPAM AI/Run v20250719 + Claude 4 Sonnet
76.80
0.63
[3,15]
1
Table 3: RULER reveals score overstatement and unresolved ranks among SWE-bench Verified’s leaders.
Benchmark
Number of Tasks
Number of Models
Position 1 Score (Raw)
Position 1 Score (Adjusted)
Number of Tiers
Number of Models in Tier 1
SWE Bash-only
500
41
76.80
75.71
10
6
SWE Verified
500
173
79.20
78.30
15
13
SWE Multimodal
510
12
36.47
35.84
3
5
SWE Multilingual
300
14
72.67
71.17
4
9
SWE Pro
395
14
44.30
43.43
6
4
Tau2 Airline
50
7
69.00
65.93
2
5
Table 4: Score overstatement and unresolved leaders occur across benchmark families.
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 2: Score correction reduces inflation while near-tied models remain hard to rank.
p=0.5
p=0.75
γ/J
2
5
10
20
50
100
2
5
10
20
50
100
N=500 tasks
0.1
1.87
1.35
1.05
0.87
0.81
0.83
1.64
1.23
1.05
0.94
0.91
0.94
0.2
1.87
1.35
1.03
0.79
0.64
0.63
1.64
1.20
0.97
0.82
0.75
0.76
0.3
1.88
1.35
1.03
0.77
0.53
0.45
1.64
1.18
0.92
0.70
0.55
0.51
0.4
1.88
1.35
1.03
0.77
0.51
0.39
1.65
1.18
0.91
0.68
0.47
0.39
Appendix
Table 5: When all models are tied, exponents below the default minimize error, and their advantage over the default grows with the number of models.
p=0.5
p=0.75
γ/J
2
5
10
20
50
100
2
5
10
20
50
100
N=500 tasks
0.1
2.30
1.78
1.54
1.42
1.39
1.40
1.98
1.56
1.39
1.33
1.31
1.32
0.2
2.31
1.79
1.54
1.38
1.31
1.30
1.99
1.56
1.36
1.27
1.22
1.22
0.3
2.32
1.80
1.54
1.38
1.27
1.23
2.01
1.56
1.34
1.22
1.13
1.11
0.4
2.33
1.80
1.54
1.38
1.26
1.21
2.01
1.56
1.34
1.21
1.10
1.06
Appendix
Table 6: With closely spaced models, error is nearly flat across a wide range of exponents.
p=0.5
p=0.75
γ/J
2
5
10
20
50
100
2
5
10
20
50
100
N=500 tasks
0.1
2.47
2.81
2.81
2.64
2.67
2.69
2.11
2.44
2.38
2.23
2.23
2.26
0.2
2.43
2.80
2.80
2.61
2.61
2.62
2.08
2.44
2.39
2.22
2.22
2.24
0.3
2.32
2.73
2.74
2.57
2.55
2.55
1.97
2.39
2.35
2.20
2.18
2.19
0.4
2.20
2.62
2.63
2.49
2.47
2.46
1.88
2.29
2.25
2.13
2.11
2.12
Appendix
Table 7: With well-separated models, large exponents minimize error.
Family
Score and sampling unit
Target population and interpretation
SWE-bench (5 benchmarks)
Binary issue resolution. Unit: issue.
Software issues under each variant’s task and scaffold rules, evaluated by the retained model-plus-agent systems. The analysis conditions on submitted trajectories. Model-plus-agent systems can differ across entries.
Tau2-Bench (3 domains)
Released task reward in [0,1] . Unit: customer-service scenario.
Airline, retail, or telecom scenarios under the released agents and environment. Frozen outcomes retain the realized agent randomness and environment.
AlpacaEval 2
Raw weighted rate: soft preference minus 1. Unit: instruction.
Instructions compared against a common GPT-4-Turbo reference, conditional on released responses and judge annotations. Instructions are equally weighted. The raw score differs from the length-controlled leaderboard metric.
WildBench v2
Checklist-guided judge score divided by 10, in [0.1,1] . Unit: prompt/session.
Real-user-style prompts under the frozen scoring protocol. Uncertainty concerns sampling prompts, conditional on the released judge scores.
LiveBench (4 benchmarks)
Released task score in [0,1] . Unit: task/question.
Tasks in the retained model-coverage subset. We report Coding, Language, Instruction Following, and their task-weighted pooled mean. This subset and pooled score differ from the full live leaderboard.
Appendix
Table 8: Each RULER report refers to a specified score and task population.
Benchmark
Snapshot date
Frozen source
SWE Verified
2026-02-27
Official leaderboard, 7c4289f , and task outcomes, 82be2b5 (2026-01-26).
SWE Bash-only
2026-02-27
Official leaderboard, 7c4289f , and task outcomes, 2f95aad (2026-02-27).
SWE Multimodal
2026-02-27
Official leaderboard, 7c4289f , and task outcomes, cff22d0 (2025-11-21).
SWE Multilingual
2026-02-27
Official leaderboard, 7c4289f , and task outcomes, 6083d31 (2026-02-24).
Leaderboards rank models by their average scores on benchmark items, and they are consulted repeatedly while the evaluation is still running. Existing confidence intervals for a model's rank control their error rate only if they are computed once, after a number of items chosen in advance. If they are recomputed as results arrive, and the evaluation stops once they look decisive, their error rate exceeds its nominal level. Anytime-valid methods keep their guarantees at all sample sizes simultaneously and hence under any stopping rule. They exist for the accuracy of one model, for one pair of models and for the set of models that may be best. For pairwise battles they also give ranks. None gives ranks when all models are scored on the same items, which makes their scores dependent. We construct rank confidence sequences: for every model, a set of ranks that contains its true rank, simultaneously for all models and at all times, at a chosen error level α, in finite samples. The construction combines betting e-processes, one for each ordered pair of models, with closed testing over the possible orderings of the models. It allows any dependence between the models' scores on an item. The method has two advantages. A leaderboard can be inspected after every item without inflating its error rate. The evaluation of each model can stop as soon as the question asked about it is answered, which saves compute. When results are examined only once, halfway through or later, little power is lost relative to fixed-sample methods. The paper quantifies these advantages in simulations and on public leaderboard data.
Hamed Khosravi, Xiaoming Huo
H. Milton Stewart School of Industrial and Systems Engineering Georgia Institute of Technology, Atlanta, GA 30332, USA
LLM-as-a-judge has become the de facto standard for scalable, subjective evaluation, yet current leaderboards compensate for systematic measurement bias by running ever more comparisons, an approach that is both statistically unsound and computationally wasteful. The root cause is an incomplete measurement model, treating LLM judges as neutral, interchangeable instruments ignores documented biases like position bias, verbosity bias, judge severity, and self-enhancement, that no volume of additional data can eliminate. We propose a unified latent variable framework that jointly models pairwise and ordinal data while explicitly correcting for these confounders, recovering reliable rankings from substantially fewer comparisons. Because fitting this model costs negligible compute relative to a single round of LLM inference, bias correction is not only more statistically rigorous but also a more sustainable approach to trustworthy evaluation.
Harshita Katoch, David Antony Selby, Gerrit Großmann +1
Department of Computer Science, University of Kaiserslautern–Landau (RPTU), Germany · Data Science and its Applications, German Research Centre for Artificial Intelligence (DFKI), Germany
Evaluation leaderboards such as LMArena play a central role in benchmarking large language models by aggregating pairwise human preferences into model rankings, yet the robustness of these rankings remains poorly understood. We present a unified perturbation framework for analyzing Bradley-Terry leaderboards under structured data modifications using influence-based approximations. Our framework studies three match-level perturbations -- Drop, Add, and Flip -- together with player removal, and evaluates their effects on top-k membership, global ranking consistency via Kendall's tau, and confidence-interval-based uncertainty. Across Chatbot Arena and six additional pairwise-comparison datasets, we show that modern leaderboards are non-robust across all three objectives: sub-1% targeted perturbations can change the top-ranked model, degrade Kendall's tau, and alter confidence intervals. Beyond robustness auditing, we show that the same influence scores enable efficient targeted perturbations, promoting or demoting specific models and reducing target-model uncertainty with fewer actions than previous manipulation and active-sampling baselines. By summarizing these effects with normalized dataset-level robustness scores, our framework provides a practical and helpful tool for auditing leaderboard stability and motivating more robust evaluation protocols.
Hosna Oyarhoseini, Jimmy Lin, Amir-Hossein Karimi
University of Waterloo, Waterloo, Ontario, Canada.