Rank Confidence Sequences:Anytime-valid Leaderboards

Hamed Khosravi; Xiaoming Huo

stat.MESep 26, 2026

Rank Confidence Sequences:Anytime-valid Leaderboards

Authors: Hamed Khosravi, Xiaoming Huo

Organizations: H. Milton Stewart School of Industrial and Systems Engineering Georgia Institute of Technology, Atlanta, GA 30332, USA

Abstract

Leaderboards rank models by their average scores on benchmark items, and they are consulted repeatedly while the evaluation is still running. Existing confidence intervals for a model's rank control their error rate only if they are computed once, after a number of items chosen in advance. If they are recomputed as results arrive, and the evaluation stops once they look decisive, their error rate exceeds its nominal level. Anytime-valid methods keep their guarantees at all sample sizes simultaneously and hence under any stopping rule. They exist for the accuracy of one model, for one pair of models and for the set of models that may be best. For pairwise battles they also give ranks. None gives ranks when all models are scored on the same items, which makes their scores dependent. We construct rank confidence sequences: for every model, a set of ranks that contains its true rank, simultaneously for all models and at all times, at a chosen error level $α$ , in finite samples. The construction combines betting e-processes, one for each ordered pair of models, with closed testing over the possible orderings of the models. It allows any dependence between the models' scores on an item. The method has two advantages. A leaderboard can be inspected after every item without inflating its error rate. The evaluation of each model can stop as soon as the question asked about it is answered, which saves compute. When results are examined only once, halfway through or later, little power is lost relative to fixed-sample methods. The paper quantifies these advantages in simulations and on public leaderboard data.

Figures & tables

Figure 1: Probability that some report is wrong against the number of looks,

\alpha=0.05

. Most fixed-sample curves (dashed) exceed

\alpha

as looks accumulate. E-process curves (solid) stay below

\alpha

. Both axes use a log scale: tick labels are actual values a factor of ten apart, except 0.

	Top twenty ( $179$ – $190$ dominances)			All $395$ models ( $\approx 77{,}700$ )
Benchmark	half	end	tiers	half	end	tiers
MMLU	$50\%$	$99\%$	$3.0$	$86\%$	$99.6\%$	$21.2$
HellaSwag	$49\%$	$96\%$	$3.2$	$77\%$	$99.2\%$	$20.4$
GSM8K	$22\%$	$96\%$	$2.9$	$78\%$	$98.9\%$	$10.8$
WinoGrande	$2\%$	$85\%$	$2.0$	$26\%$	$95.5\%$	$4.2$
ARC-Challenge	$0\%$	$82\%$	$1.1$	$48\%$	$97.0\%$	$6.8$

Table 1: True dominances certified, as a percentage of those that exist, and mean tiers at halfway. Ours checked at every

1\%

50

item orders. No report was wrong in any of these runs.

Figure 2: Twenty highest-scoring models,

50

item orders: fraction of true dominances certified against fraction evaluated. Ours (Algorithm 2 ) checked at every

1\%

, Holm and Romano–Wolf at one look. Each panel gives the mean tiers at halfway, ours against Romano–Wolf.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

		$0.25$		$0.50$		$1.00$		tiers
Benchmark	items	ours	Holm	ours	Holm	ours	Holm	ours / Holm
MMLU	$14{,}042$	$76$	$77$	$86$	$83$	$99.6$	$88$	$225$ / $26$
HellaSwag	$10{,}042$	$62$	$63$	$77$	$74$	$99.2$	$81$	$165$ / $24$
GSM8K	$1319$	$63$	$66$	$78$	$75$	$98.9$	$83$	$124$ / $14$
WinoGrande	$1267$	$5$	$6$	$26$	$17$	$95.5$	$34$	$53$ / $5$
ARC-Challenge	$1172$	$18$	$22$	$48$	$39$	$97.0$	$56$	$68$ / $8$

Appendix

Table 2: All

395

released models, no further selection,

50

item orders per benchmark. True dominances certified, as a percentage of those that hold on the whole benchmark; tiers at the end. Holm uses the paired

z

-test.

Benchmark	cost	settled
MMLU	$6.2\%$	$100.0\%$
HellaSwag	$15.2\%$	$100.0\%$
GSM8K	$16.7\%$	$99.2\%$
WinoGrande	$47.3\%$	$99.5\%$
ARC-Challenge	$36.9\%$	$99.5\%$
TruthfulQA	$27.3\%$	$97.7\%$

Appendix

Table 3: Certifying every model’s top-

3

status, all

395

models,

50

item orders per benchmark. Cost is model-item evaluations divided by

395N

; settled is the mean fraction of models whose top-

3

status is certified at the end.

True dominances certified at one half of the benchmark (%)
	MMLU	HellaSwag	GSM8K	WinoGrande	ARC-Challenge	TruthfulQA a
Ours, monitored	50	49	22	2	0	17
Holm, McNemar	47	47	19	1	0	61 b
Holm, paired $z$ -test	48	48	20	1	0	16
Romano–Wolf	49	49	22	1	0	21
Probability of a false statement at some look

Appendix

Table 4: Fixed-sample constructions against ours, twenty highest-scoring models,

50

item orders. Top: true dominances certified at one half, as a percentage of those that exist. Bottom: probability that some report, at any look, was false.

Dependence	Anytime error	True dominances	Mean rank width	Tiers	Top- $3$ cost (goal)
Independent	2.0%	70%	1.49	3.3	41% (49%)
Positive, $\gamma=0.5$	1.5%	76%	1.18	3.7	32% (79%)
Positive, $\gamma=0.9$	1.6%	96%	0.22	5.3	16% (100%)
Mixed sign, $\gamma=0.8$	1.6%	80%	1.01	3.9	46% (26%)

Appendix

Table 5: Four dependence regimes with identical marginal accuracies. “Anytime error” uses six tied models; the other columns use evenly spread logistic abilities after half the items, with the probability of reaching the top-

3

goal in parentheses.

	fraction of the benchmark						looks vs. e-Bonferroni
	$0.05$	$0.10$	$0.25$	$0.50$	$0.75$	$1.00$	ahead	behind
Eight highest-scoring models, $27.5$ true dominances, $31{,}600$ looks
e-Bonferroni with closure (reference)	2.15	3.11	4.86	7.51	12.62	26.04	reference
Shortcut (Algorithm 2 )	2.15	3.11	4.86	7.51	12.63	26.04	118	0
Exact (Algorithm 1 )	2.19	3.21	4.99	7.90	13.02	26.11	6,320	0
Exact, adjacent-level weights	2.04	2.35	3.90	4.97	7.87	26.20	831	15,052

Appendix

Table 6: True dominances certified from identical wealths, so that only the correction differs; means over

50

item orders and the six benchmarks, and for twenty models also over the top and spread subsets. The last two columns count the looks at which the row beat or trailed the reference.

Benchmark, subset	true	exact	shortcut	exact	shortcut	exact	shortcut	exact	shortcut	programs	s
		$0.25$		$0.50$		$1.00$		width at $0.50$		cost
MMLU, top	190	84.1	84.1	93.6	92.4	187.3	187.1	9.64	9.76	244	6
HellaSwag, top	190	77.4	76.3	94.9	94.1	181.7	181.3	9.51	9.59	292	8
GSM8K, top	190	14.4	13.4	45.1	43.1	183.2	182.8	14.49	14.69	331	9
WinoGrande, top	180	0.1	0.1	3.3	3.1	153.9	153.2	18.67	18.69	238	6
ARC-Challenge, top	179	0.0	0.0	0.3	0.2	148.2	147.1	18.97	18.98	213	6

Appendix

Table 7: The exact test by integer programming against the shortcut, at

M=20

with

50

item orders per row, drawn independently of those behind Table 1 , so the shortcut columns differ slightly from it. True dominances certified, rank width at one half, and what the exact test cost to run.

Fixed-sample rank inference
	Rank sets, simultaneous	Monitoring	Finite-sample	Structure	Early stop of evaluation
Mogstad et al. (2024)	yes	no	no	no	no
Al Mohamad et al. (2022)	yes	no	yes	no	no
Al Mohamad et al. (2021)	yes	no	yes	yes	no
Neuhof & Benjamini (2024)	yes	no	yes a	no	no
Neuhof & Benjamini (2026a)	yes	no	yes a	no	no

Appendix

Table 8: Closest existing methods. Monitoring: valid under continuous looks and any stopping rule. Finite-sample: no asymptotics. Structure: the correction uses the relations among pairs.

Anytime-Valid Inference

Anytime-valid inference provides statistical tests and confidence bounds that remain valid under repeated monitoring, adaptive sampling, and data-dependent stopping.

See all 33 papers

Jun 7, 2026

Rank Intervals for Leaderboards: A Hierarchical Framework for Model Evaluation

Pretrained models are often evaluated on multi-task leaderboards to measure their applicability in diverse contexts. However, current methods for aggregating performance across tasks into leaderboard-level rankings do not address the uncertainty and variability at the task level. While recent works have proposed interval-based model rankings, the principled aggregation of uncertainty from individual tasks to leaderboard-level rankings remains unaddressed, and variation in models' performance across tasks is frequently obscured. In this work, we introduce a hierarchical framework that constructs model rank intervals with statistical guarantees at both levels: task-level rank confidence intervals from pairwise comparisons, and leaderboard-level rank prediction intervals using a conformal approach. This enables reliable quantification of model rank for each observed task and for new potential tasks. Experiments on simulated data and the TabArena and PromptEval (MMLU) benchmarks show that our method yields statistically valid and informative intervals, enabling reliable, uncertainty-aware model ranking on leaderboards.

Bitya Neuhof, Yuval Benjamini

Department of Statistics and Data Science The Hebrew University of Jerusalem

Uncertainty Quantification

Ask PDF

Jun 22, 2026

You Don't Need to Run Every Eval

A modern model release reports scores on 40+ benchmarks and the same evaluations were run many more times before it: to track training progress, compare design choices, and select the checkpoint for the release. But do we need to run every eval? We compile a public score matrix of 84 frontier models on 133 benchmarks (2,604 cells, 23.3% filled) and find it is approximately rank-2: a model's scores across all 133 benchmarks are largely determined by just two numbers. We confirm this in two ways: scores hidden from the matrix are best recovered using two factors, and two factors already explain over 90% of the variation among models on the benchmarks they share. Building on this, we design BenchPress: a logit-space rank-2 matrix completion method that recovers held-out scores to within 4.6 points, and a confidence layer that says when each prediction can be trusted. Using BenchPress, we find a subset of five benchmarks {GPQA-D, HLE, Codeforces, MMLU-Pro, ARC-AGI-1} that can recover the rest of a model's public scorecard to within 3.93 points. For a tighter inference budget, a cheaper set {GPQA-D, MMLU-Pro, Aider Polyglot, MATH-500, AIME 2026} can predict a model's evals to within 4.55. We release the score matrix, the BenchPress code, and an interactive tool that predicts any model's score on any benchmark.

Yuchen Zeng, Dimitris Papailiopoulos

Microsoft Research, AI Frontiers

LLM Evaluation Robust Matrix Completion Matrix Completion

Ask PDF

May 15, 2026

A Unified Perturbation Framework for Analyzing Leaderboard Stability and Manipulation

Evaluation leaderboards such as LMArena play a central role in benchmarking large language models by aggregating pairwise human preferences into model rankings, yet the robustness of these rankings remains poorly understood. We present a unified perturbation framework for analyzing Bradley-Terry leaderboards under structured data modifications using influence-based approximations. Our framework studies three match-level perturbations -- Drop, Add, and Flip -- together with player removal, and evaluates their effects on top-k membership, global ranking consistency via Kendall's tau, and confidence-interval-based uncertainty. Across Chatbot Arena and six additional pairwise-comparison datasets, we show that modern leaderboards are non-robust across all three objectives: sub-1% targeted perturbations can change the top-ranked model, degrade Kendall's tau, and alter confidence intervals. Beyond robustness auditing, we show that the same influence scores enable efficient targeted perturbations, promoting or demoting specific models and reducing target-model uncertainty with fewer actions than previous manipulation and active-sampling baselines. By summarizing these effects with normalized dataset-level robustness scores, our framework provides a practical and helpful tool for auditing leaderboard stability and motivating more robust evaluation protocols.

Hosna Oyarhoseini, Jimmy Lin, Amir-Hossein Karimi

University of Waterloo, Waterloo, Ontario, Canada.

LLM Evaluation Learning to Rank LLM Auditing Pairwise Ranking Bradley–Terry Model

Ask PDF

Abstract

Anytime-Valid Inference

Explore similar work

Rank Intervals for Leaderboards: A Hierarchical Framework for Model Evaluation

You Don't Need to Run Every Eval

A Unified Perturbation Framework for Analyzing Leaderboard Stability and Manipulation