Leaderboards rank models by their average scores on benchmark items, and they are consulted repeatedly while the evaluation is still running. Existing confidence intervals for a model's rank control their error rate only if they are computed once, after a number of items chosen in advance. If they are recomputed as results arrive, and the evaluation stops once they look decisive, their error rate exceeds its nominal level. Anytime-valid methods keep their guarantees at all sample sizes simultaneously and hence under any stopping rule. They exist for the accuracy of one model, for one pair of models and for the set of models that may be best. For pairwise battles they also give ranks. None gives ranks when all models are scored on the same items, which makes their scores dependent. We construct rank confidence sequences: for every model, a set of ranks that contains its true rank, simultaneously for all models and at all times, at a chosen error level α, in finite samples. The construction combines betting e-processes, one for each ordered pair of models, with closed testing over the possible orderings of the models. It allows any dependence between the models' scores on an item. The method has two advantages. A leaderboard can be inspected after every item without inflating its error rate. The evaluation of each model can stop as soon as the question asked about it is answered, which saves compute. When results are examined only once, halfway through or later, little power is lost relative to fixed-sample methods. The paper quantifies these advantages in simulations and on public leaderboard data.
Figures & tables
Figure 1: Probability that some report is wrong against the number of looks, α=0.05 . Most fixed-sample curves (dashed) exceed α as looks accumulate. E-process curves (solid) stay below α . Both axes use a log scale: tick labels are actual values a factor of ten apart, except 0.
Top twenty ( 179 – 190 dominances)
All 395 models ( ≈77,700 )
Benchmark
half
end
tiers
half
end
tiers
MMLU
50%
99%
3.0
86%
99.6%
21.2
HellaSwag
49%
96%
3.2
77%
99.2%
20.4
GSM8K
22%
96%
2.9
78%
98.9%
10.8
WinoGrande
2%
85%
2.0
26%
95.5%
4.2
ARC-Challenge
0%
82%
1.1
48%
97.0%
6.8
Table 1: True dominances certified, as a percentage of those that exist, and mean tiers at halfway. Ours checked at every 1% , 50 item orders. No report was wrong in any of these runs.
Figure 2: Twenty highest-scoring models, 50 item orders: fraction of true dominances certified against fraction evaluated. Ours (Algorithm 2 ) checked at every 1% , Holm and Romano–Wolf at one look. Each panel gives the mean tiers at halfway, ours against Romano–Wolf.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
0.25
0.50
1.00
tiers
Benchmark
items
ours
Holm
ours
Holm
ours
Holm
ours / Holm
MMLU
14,042
76
77
86
83
99.6
88
225 / 26
HellaSwag
10,042
62
63
77
74
99.2
81
165 / 24
GSM8K
1319
63
66
78
75
98.9
83
124 / 14
WinoGrande
1267
5
6
26
17
95.5
34
53 / 5
ARC-Challenge
1172
18
22
48
39
97.0
56
68 / 8
Appendix
Table 2: All 395 released models, no further selection, 50 item orders per benchmark. True dominances certified, as a percentage of those that hold on the whole benchmark; tiers at the end. Holm uses the paired z -test.
Benchmark
cost
settled
MMLU
6.2%
100.0%
HellaSwag
15.2%
100.0%
GSM8K
16.7%
99.2%
WinoGrande
47.3%
99.5%
ARC-Challenge
36.9%
99.5%
TruthfulQA
27.3%
97.7%
Appendix
Table 3: Certifying every model’s top- 3 status, all 395 models, 50 item orders per benchmark. Cost is model-item evaluations divided by 395N ; settled is the mean fraction of models whose top- 3 status is certified at the end.
MMLU
HellaSwag
GSM8K
WinoGrande
ARC-Challenge
TruthfulQA a
True dominances certified at one half of the benchmark (%)
Ours, monitored
50
49
22
2
0
17
Holm, McNemar
47
47
19
1
0
61 b
Holm, paired z -test
48
48
20
1
0
16
Romano–Wolf
49
49
22
1
0
21
Probability of a false statement at some look
Appendix
Table 4: Fixed-sample constructions against ours, twenty highest-scoring models, 50 item orders. Top: true dominances certified at one half, as a percentage of those that exist. Bottom: probability that some report, at any look, was false.
Dependence
Anytime error
True dominances
Mean rank width
Tiers
Top- 3 cost (goal)
Independent
2.0%
70%
1.49
3.3
41% (49%)
Positive, γ=0.5
1.5%
76%
1.18
3.7
32% (79%)
Positive, γ=0.9
1.6%
96%
0.22
5.3
16% (100%)
Mixed sign, γ=0.8
1.6%
80%
1.01
3.9
46% (26%)
Appendix
Table 5: Four dependence regimes with identical marginal accuracies. “Anytime error” uses six tied models; the other columns use evenly spread logistic abilities after half the items, with the probability of reaching the top- 3 goal in parentheses.
Table 6: True dominances certified from identical wealths, so that only the correction differs; means over 50 item orders and the six benchmarks, and for twenty models also over the top and spread subsets. The last two columns count the looks at which the row beat or trailed the reference.
0.25
0.50
1.00
width at 0.50
cost
Benchmark, subset
true
exact
shortcut
exact
shortcut
exact
shortcut
exact
shortcut
programs
s
MMLU, top
190
84.1
84.1
93.6
92.4
187.3
187.1
9.64
9.76
244
6
HellaSwag, top
190
77.4
76.3
94.9
94.1
181.7
181.3
9.51
9.59
292
8
GSM8K, top
190
14.4
13.4
45.1
43.1
183.2
182.8
14.49
14.69
331
9
WinoGrande, top
180
0.1
0.1
3.3
3.1
153.9
153.2
18.67
18.69
238
6
ARC-Challenge, top
179
0.0
0.0
0.3
0.2
148.2
147.1
18.97
18.98
213
6
Appendix
Table 7: The exact test by integer programming against the shortcut, at M=20 with 50 item orders per row, drawn independently of those behind Table 1 , so the shortcut columns differ slightly from it. True dominances certified, rank width at one half, and what the exact test cost to run.
Rank sets, simultaneous
Monitoring
Finite-sample
Structure
Early stop of evaluation
Fixed-sample rank inference
Mogstad et al. (2024)
yes
no
no
no
no
Al Mohamad et al. (2022)
yes
no
yes
no
no
Al Mohamad et al. (2021)
yes
no
yes
yes
no
Neuhof & Benjamini (2024)
yes
no
yes a
no
no
Neuhof & Benjamini (2026a)
yes
no
yes a
no
no
Appendix
Table 8: Closest existing methods. Monitoring: valid under continuous looks and any stopping rule. Finite-sample: no asymptotics. Structure: the correction uses the relations among pairs.
Anytime-valid inference provides statistical tests and confidence bounds that remain valid under repeated monitoring, adaptive sampling, and data-dependent stopping.