Ranking Reasoning LLMs under Test-Time Scaling

Mohsen Hariri; Michael Hinczewski; Jing Ma; Vipin Chaudhary

cs.LGMar 11, 2026

Ranking Reasoning LLMs under Test-Time Scaling

Authors: Mohsen Hariri, Michael Hinczewski, Jing Ma, Vipin Chaudhary

Organizations: Department of Computer and Data Sciences, Case Western Reserve University, Cleveland, OH, USA · Department of Physics, Case Western Reserve University, Cleveland, OH, USA

Abstract

Test-time scaling evaluates reasoning LLMs by sampling multiple outputs per prompt, but ranking models in this regime remains underexplored. We formalize dense benchmark ranking under test-time scaling and introduce Scorio, a library that implements statistical ranking methods such as paired-comparison models, item response theory (IRT) models, voting rules, and graph- and spectral-based methods. Across $20$ reasoning models on four Olympiad-style math benchmarks (AIME'24, AIME'25, HMMT'25, and BrUMO'25; up to $N=80$ trials), most full-trial rankings agree closely with the Bayesian gold standard $\mathrm{Bayes}_{\mathcal{U}}@80$ (mean Kendall's $τ_b = 0.93$ -- $0.95$ ), and $19$ -- $34$ methods recover exactly the same ordering. In the single-trial regime, the best methods reach $τ_b \approx 0.86$ . Using greedy decoding as an empirical prior ( $\mathrm{Bayes}_{\mathbf{R}_0}@N$ ) reduces variance at $N=1$ by $16$ -- $52\%$ , but can bias rankings when greedy and stochastic sampling disagree. These results identify reliable ranking methods for both high- and low-budget test-time scaling. We release Scorio as an open-source library at https://github.com/mohsenhariri/scorio.

Figures & tables

Figure 1: Agreement between each method’s full-trial ranking and the gold standard. Kendall’s $\tau_{b}$ is computed between each method’s ranking (at $N=80$ trials) and $\mathrm{Bayes}_{\mathcal{U}}@80$ on an easier benchmark (BrUMO’25, left) and the hardest benchmark (HMMT’25, right). On BrUMO’25, multiple methods achieve near-perfect or perfect agreement: $\mathrm{Bayes}_{\mathbf{R}_{0}}@N$ and HodgeRank reach $\tau_{b}=1.0$ , while Rasch MML achieves $0.997$ . On HMMT’25, Bradley–Terry and HodgeRank maintain perfect agreement ( $\tau_{b}=1.0$ ), but $\mathrm{Bayes}_{\mathbf{R}_{0}}@N$ drops to $0.989$ and Pass@ $\!2$ falls to $0.937$ . This divergence is consistent with the lower greedy–sampling alignment observed on harder benchmarks ( Section 3.4 ).

Benchmark	Mean	Median	Min	#( $\tau_{b}=1$ )	#( $\tau_{b}\geq 0.95$ )
AIME’24	0.941	0.989	0.682	20	40
AIME’25	0.934	0.947	0.771	19	29
HMMT’25	0.950	0.989	0.758	34	44
BrUMO’25	0.954	0.968	0.789	26	49
Combined	0.962	0.989	0.748	22	53

Table 1: Agreement between the gold-standard ranking (

\mathrm{Bayes}_{\mathcal{U}}@80

) and each other ranking method, measured by Kendall’s

\tau_{b}

, when all methods are computed from the full

N=80

trials. Statistics are computed over the other

71

methods; “Combined” pools all benchmarks.

Benchmark	Best vs. gold standard	$\tau_{b}$	Best self-consistency (vs. method@80 )	$\tau_{b}$
AIME’24	$\mathrm{Bayes}_{\mathbf{R}_{0}}@1$	$0.779\pm 0.034$	Rasch MML LCB ( rasch_mml_credible )	$0.804\pm 0.051$
AIME’25	$\mathrm{Bayes}_{\mathbf{R}_{0}}@1$	$0.798\pm 0.045$	Rasch MML LCB ( rasch_mml_credible )	$0.834\pm 0.054$
HMMT’25	Bayes@ $\!1$ $\dagger$	$0.790\pm 0.053$	Rasch MML LCB ( rasch_mml_credible )	$0.810\pm 0.056$
BrUMO’25	$\mathrm{Bayes}_{\mathbf{R}_{0}}@1$	$0.858\pm 0.028$	$\mathrm{Bayes}_{\mathbf{R}_{0}}@1$	$0.858\pm 0.028$
Combined	Bayes@ $\!1$ $\dagger$	$0.865\pm 0.049$	Nanson avg ties ( nanson_rank_ties_average )	$0.892\pm 0.050$

Table 2: Best-performing ranking methods in the low-budget regime (

N=1

) under two targets: (i) agreement with the gold standard (

\mathrm{Bayes}_{\mathcal{U}}@80

) and (ii) self-consistency with each method’s own full-trial ranking (method@

80

). Kendall’s

\tau_{b}

is averaged over

80

single-trial draws;

\dagger

denotes a 21-way tie for best gold-standard agreement (see Table 18 ). Pass@

\!k

variants are excluded at

N=1

because they require

N\geq 2

. Method identifiers correspond to the APIs listed in Section I.2 .

Benchmark	Pool	Best Method	$\tau_{b}$ vs Target
Benchmark	Pool	Best Method	avg@80	method@80
AIME’24	5	$\mathrm{Bayes}_{\mathbf{R}_{0}}@1$	$0.769\pm{\scriptstyle 0.209}$	$0.773\pm{\scriptstyle 0.207}$
	10		$0.776\pm{\scriptstyle 0.107}$	$0.781\pm{\scriptstyle 0.105}$
	15		$0.780\pm{\scriptstyle 0.057}$	$0.785\pm{\scriptstyle 0.057}$
AIME’25	5	$\mathrm{Bayes}_{\mathbf{R}_{0}}@1$	$0.802\pm{\scriptstyle 0.144}$	$0.809\pm{\scriptstyle 0.144}$
	10		$0.797\pm{\scriptstyle 0.071}$	$0.803\pm{\scriptstyle 0.073}$

Table 3: Bootstrapped model-pool results in the low-budget regime (

N=1

). For each model-pool subset, we compute Kendall’s

\tau_{b}

over the

80

single-trial rankings against two targets: the subset-specific gold standard avg@80 and each method’s own subset-specific full-trial ranking ( method@80 ). The table reports the mean and standard deviation of the subset-level mean score across bootstrap model pools for each subset size.

Figure 2: Gold-standard agreement of

\mathrm{Bayes}_{\mathcal{U}}@N

(blue) and

\mathrm{Bayes}_{\mathbf{R}_{0}}@N

(red) as a function of

N

across benchmarks. Shaded regions show

\pm 1

standard deviation over

50

resampled datasets.

Benchmark	Difficulty	$\tau_{\text{G-S}}$	$\Delta\tau$	Std. Red.
AIME’24	0.620	0.739	$+0.020$	42%
AIME’25	0.533	0.660	$+0.008$	17%
HMMT’25	0.333	0.635	$\mathbf{-0.022}$	16%
BrUMO’25	0.588	0.768	$\mathbf{+0.049}$	52%

Table 4: Dataset difficulty (mean accuracy), greedy–sampling alignment (

\tau_{\text{G-S}}

), and the effect of the greedy empirical prior at

N=1

\Delta\tau

is the difference in gold-standard agreement (greedy minus uniform), and Std. Red. is the relative reduction in the standard deviation of

\tau_{b}

Figure 3: Model-level ranks under greedy decoding versus stochastic sampling ( $N=80$ ) for each benchmark. Points on the diagonal indicate perfect alignment; color shows rank displacement ( $\Delta$ ).

Scheme	$\tau_{\text{GS}}$	$\tau_{\text{Self}}$	$\tau_{\text{Greedy}}$
Conservative	$0.856\pm{\scriptstyle 0.076}$	$0.861\pm{\scriptstyle 0.066}$	$0.858\pm{\scriptstyle 0.074}$
Efficiency-adj.	$0.850\pm{\scriptstyle 0.070}$	$0.875\pm{\scriptstyle 0.057}$	$0.859\pm{\scriptstyle 0.071}$
Format-aware	$0.849\pm{\scriptstyle 0.071}$	$0.881\pm{\scriptstyle 0.064}$	$0.869\pm{\scriptstyle 0.069}$
Balanced comp.	$0.843\pm{\scriptstyle 0.075}$	$0.877\pm{\scriptstyle 0.067}$	$0.862\pm{\scriptstyle 0.073}$
OOD-robust	$0.840\pm{\scriptstyle 0.071}$	$0.892\pm{\scriptstyle 0.063}$	$0.870\pm{\scriptstyle 0.066}$
Rare-event	$0.838\pm{\scriptstyle 0.073}$	$0.888\pm{\scriptstyle 0.065}$	$0.867\pm{\scriptstyle 0.069}$

Table 5: Categorical ranking at

N\!=\!1

on the combined benchmark (

L\!=\!11

, the first 11 models from Table 23 ,

M\!=\!120

). Eight representative schemes are ordered by agreement with the gold standard (

\tau_{\text{GS}}

, vs.

\mathrm{Bayes}_{\mathcal{U}}@80

). Self:

\tau_{b}

vs. Scheme@

80

; Greedy:

\tau_{b}

vs.

\mathrm{Bayes}_{\mathbf{R}_{0}}@80

. Values are mean

\pm

std over

80

draws.

Figure 4: Gold-standard agreement vs. self-consistency for

25

categorical schemes at

N=1

on the Combined benchmark. Blue markers indicate the

8

representative schemes; gray markers show the remaining

17

. Schemes in the upper-left are self-consistent but deviate from

\mathrm{Bayes}_{\mathcal{U}}@80

; those in the lower-right closely track the gold standard but are less stable across single-trial draws.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Table 10

Figure 5: Overview of model accuracies across all four benchmarks. Each panel shows each model’s mean accuracy under stochastic sampling (over

N=80

trials), together with greedy accuracy (markers). Error bars denote one standard deviation across trials and illustrate the variability introduced by test-time scaling. Models are color-coded consistently across benchmarks for ease of comparison. The figure shows substantial heterogeneity in both absolute performance and sampling variance, with HMMT’25 notably harder than the other three benchmarks.

Benchmark	Mean rank	$\mathrm{Bayes}_{\mathcal{U}}@80$ avg.	Best method	Best avg.	Gap
AIME’24	2	0.9414	rasch_mml	0.9417	0.0003
AIME’25	1	0.9344	avg (tie)	0.9344	0.0000
HMMT’25	1	0.9499	avg (tie)	0.9499	0.0000
BrUMO’25	2	0.9542	rasch_mml	0.9547	0.0005
Combined	1	0.9616	avg (tie)	0.9616	0.0000

Appendix

Table 10:

\mathrm{Bayes}_{\mathcal{U}}@80

as a consensus ranking. “Consensus rank” sorts methods by their average Kendall’s

\tau_{b}

agreement with all other methods (computed at

N=80

; ties broken by lower std).

Method	$\tau_{b}$
AIME’24
minimax_variant_margin_tie_ignore	0.682
minimax_variant_margin_tie_half	0.682
minimax_variant_winning_votes_tie_half	0.682
minimax_variant_winning_votes_tie_ignore	0.693
nanson_rank_ties_average	0.798

Appendix

Table 11: Low-agreement tail: methods whose full-trial rankings have Kendall’s

\tau_{b}<0.85

relative to

\mathrm{Bayes}_{\mathcal{U}}@80

(computed at

N=80

Rank	Method(s)	Avg.	Std.
1	rasch_mml	0.9417	0.0799
2	alpharank, bayes, bayes_ci, bradley_terry_davidson, bradley_terry_davidson_map, dynamic_irt_linear, glicko_tie_draw, hodge_rank_binary_decisive, hodge_rank_binary_total, hodge_rank_binary_uniform, avg, nash_advantage_vs_equilibrium, nash_vs_equilibrium, pagerank, rank_centrality_tie_half, rasch, rasch_map, serial_rank_prob_diff, serial_rank_sign, spectral, thompson	0.9414	0.0815
3	bayes_greedy	0.9407	0.0817
4	bayesian_mcmc, bradley_terry, bradley_terry_map, elo_tie_skip, glicko_tie_correct_draw_only, glicko_tie_skip, mg_pass_at_k_2, pass_hat_k_2, plackett_luce, plackett_luce_map, trueskill	0.9403	0.0817
5	hodge_rank_log_odds_decisive, hodge_rank_log_odds_total, hodge_rank_log_odds_uniform, rank_centrality_tie_ignore, rao_kupper, rao_kupper_map	0.9349	0.0816
6	borda	0.9179	0.0729

Appendix

Table 12: Consensus ranking on AIME’24 by average Kendall’s

\tau_{b}

agreement with all other methods at

N=80

(higher is better). Method variants with identical (Avg., Std.) are collapsed; we show the top 10 and bottom 5 groups.

Rank	Method(s)	Avg.	Std.
1	alpharank, bayes, bayes_ci, bradley_terry, bradley_terry_davidson, bradley_terry_davidson_map, bradley_terry_map, dynamic_irt_linear, elo_tie_correct_draw_only, elo_tie_skip, glicko_tie_correct_draw_only, glicko_tie_draw, glicko_tie_skip, hodge_rank_binary_decisive, hodge_rank_binary_total, hodge_rank_binary_uniform, hodge_rank_log_odds_decisive, hodge_rank_log_odds_total, hodge_rank_log_odds_uniform, avg, nash_advantage_vs_equilibrium, nash_vs_equilibrium, pagerank, plackett_luce, plackett_luce_map, rank_centrality_tie_half, rao_kupper, rao_kupper_map, rasch, rasch_map, serial_rank_prob_diff, serial_rank_sign, spectral, thompson, trueskill	0.9499	0.0631
2	rasch_mml	0.9494	0.0442
3	bayes_greedy	0.9468	0.0556
4	rasch_3pl, rasch_3pl_map	0.9420	0.0624
5	rank_centrality_tie_ignore	0.9415	0.0511
6	elo_tie_draw	0.9356	0.0561

Appendix

Table 14: Consensus ranking on HMMT’25 by average Kendall’s

\tau_{b}

agreement with all other methods at

N=80

(higher is better). Method variants with identical (Avg., Std.) are collapsed; we show the top 10 and bottom 5 groups.

Rank	Method(s)	Avg.	Std.
1	rasch_mml	0.9547	0.0581
2	alpharank, bayes, bayes_ci, bayes_greedy, bradley_terry_davidson, bradley_terry_davidson_map, dynamic_irt_linear, glicko_tie_draw, hodge_rank_binary_decisive, hodge_rank_binary_total, hodge_rank_binary_uniform, hodge_rank_log_odds_decisive, hodge_rank_log_odds_total, hodge_rank_log_odds_uniform, avg, nash_advantage_vs_equilibrium, nash_vs_equilibrium, pagerank, rank_centrality_tie_half, rao_kupper, rao_kupper_map, rasch, rasch_map, serial_rank_prob_diff, serial_rank_sign, spectral, thompson	0.9542	0.0588
3	glicko_tie_correct_draw_only	0.9501	0.0575
4	mg_pass_at_k_2, pass_hat_k_2	0.9490	0.0582
5	elo_tie_draw	0.9423	0.0589
6	borda, win_rate	0.9399	0.0470

Appendix

Table 15: Consensus ranking on BrUMO’25 by average Kendall’s

\tau_{b}

agreement with all other methods at

N=80

(higher is better). Method variants with identical (Avg., Std.) are collapsed; we show the top 10 and bottom 5 groups.

Rank	Method(s)	Avg.	Std.
1	alpharank, bayes, bayes_ci, bradley_terry_davidson, bradley_terry_davidson_map, dynamic_irt_linear, glicko_tie_draw, hodge_rank_binary_decisive, hodge_rank_binary_total, hodge_rank_binary_uniform, kemeny_young_tie_half, kemeny_young_tie_ignore, avg, nash_advantage_vs_equilibrium, nash_vs_equilibrium, pagerank, rank_centrality_tie_half, rasch, rasch_map, serial_rank_prob_diff, serial_rank_sign, spectral, thompson	0.9616	0.0559
2	copeland, ranked_pairs_strength_margin_tie_half, ranked_pairs_strength_margin_tie_ignore, ranked_pairs_strength_winning_votes_tie_half, ranked_pairs_strength_winning_votes_tie_ignore, schulze_tie_half, schulze_tie_ignore	0.9602	0.0546
3	hodge_rank_log_odds_decisive, hodge_rank_log_odds_total, hodge_rank_log_odds_uniform, rao_kupper, rao_kupper_map	0.9567	0.0516
4	glicko_tie_correct_draw_only	0.9566	0.0522
5	baldwin_rank_ties_average	0.9558	0.0495
6	borda	0.9557	0.0516

Appendix

Table 16: Consensus ranking on the combined benchmark by average Kendall’s

\tau_{b}

agreement with all other methods at

N=80

(higher is better). Method variants with identical (Avg., Std.) are collapsed; we show the top 10 and bottom 5 groups.

Benchmark	Count	Methods
AIME’24	20	alpharank, bayes, bayes_ci, bradley_terry_davidson, bradley_terry_davidson_map, dynamic_irt_linear, glicko_tie_draw, hodge_rank_binary_decisive, hodge_rank_binary_total, hodge_rank_binary_uniform, nash_advantage_vs_equilibrium, nash_vs_equilibrium, pagerank, rank_centrality_tie_half, rasch, rasch_map, serial_rank_prob_diff, serial_rank_sign, spectral, thompson
AIME’25	19	alpharank, bayes, bayes_ci, bradley_terry_davidson, bradley_terry_davidson_map, dynamic_irt_linear, hodge_rank_binary_decisive, hodge_rank_binary_total, hodge_rank_binary_uniform, nash_advantage_vs_equilibrium, nash_vs_equilibrium, pagerank, rank_centrality_tie_half, rasch, rasch_map, serial_rank_prob_diff, serial_rank_sign, spectral, thompson
HMMT’25	34	alpharank, bayes, bayes_ci, bradley_terry, bradley_terry_davidson, bradley_terry_davidson_map, bradley_terry_map, dynamic_irt_linear, elo_tie_correct_draw_only, elo_tie_skip, glicko_tie_correct_draw_only, glicko_tie_draw, glicko_tie_skip, hodge_rank_binary_decisive, hodge_rank_binary_total, hodge_rank_binary_uniform, hodge_rank_log_odds_decisive, hodge_rank_log_odds_total, hodge_rank_log_odds_uniform, nash_advantage_vs_equilibrium, nash_vs_equilibrium, pagerank, plackett_luce, plackett_luce_map, rank_centrality_tie_half, rao_kupper, rao_kupper_map, rasch, rasch_map, serial_rank_prob_diff, serial_rank_sign, spectral, thompson, trueskill
BrUMO’25	26	alpharank, bayes, bayes_ci, bayes_greedy, bradley_terry_davidson, bradley_terry_davidson_map, dynamic_irt_linear, glicko_tie_draw, hodge_rank_binary_decisive, hodge_rank_binary_total, hodge_rank_binary_uniform, hodge_rank_log_odds_decisive, hodge_rank_log_odds_total, hodge_rank_log_odds_uniform, nash_advantage_vs_equilibrium, nash_vs_equilibrium, pagerank, rank_centrality_tie_half, rao_kupper, rao_kupper_map, rasch, rasch_map, serial_rank_prob_diff, serial_rank_sign, spectral, thompson
Combined	22	alpharank, bayes, bayes_ci, bradley_terry_davidson, bradley_terry_davidson_map, dynamic_irt_linear, glicko_tie_draw, hodge_rank_binary_decisive, hodge_rank_binary_total, hodge_rank_binary_uniform, kemeny_young_tie_half, kemeny_young_tie_ignore, nash_advantage_vs_equilibrium, nash_vs_equilibrium, pagerank, rank_centrality_tie_half, rasch, rasch_map, serial_rank_prob_diff, serial_rank_sign, spectral, thompson

Appendix

Table 17: Methods that induce exactly the same ranking as

\mathrm{Bayes}_{\mathcal{U}}@80

(

\tau_{b}=1

) when computed on the full

N=80

trials (excluding avg itself).

Rank	Method(s)	Mean	Std.
1	baldwin_rank_ties_average, bayes, bayes_ci, borda, copeland, majority_judgment, avg, minimax_variant_margin_tie_half, minimax_variant_margin_tie_ignore, minimax_variant_winning_votes_tie_half, nash_advantage_vs_equilibrium, nash_vs_equilibrium, pagerank, rank_centrality_tie_half, ranked_pairs_strength_margin_tie_half, ranked_pairs_strength_margin_tie_ignore, ranked_pairs_strength_winning_votes_tie_half, ranked_pairs_strength_winning_votes_tie_ignore, schulze_tie_half, schulze_tie_ignore, spectral	0.8647	0.0486
2	alpharank	0.8646	0.0486
3	rasch_mml_credible	0.8642	0.0351
4	hodge_rank_binary_uniform	0.8623	0.0491
5	hodge_rank_binary_decisive	0.8623	0.0484
6	hodge_rank_binary_total	0.8616	0.0493

Appendix

Table 18: Gold-standard agreement at

N=1

on the combined benchmark, measured as Kendall’s

\tau_{b}

between each method’s single-trial ranking and the gold standard (

\mathrm{Bayes}_{\mathcal{U}}@80

). Statistics are computed over

80

single-trial draws. Methods with identical mean/std. values are collapsed; we show the top 10 and bottom 5 groups.

Rank	Method(s)	Mean	Std.
1	nanson_rank_ties_average	0.8925	0.0497
2	rasch_mml_credible	0.8831	0.0370
3	nanson_rank_ties_max	0.8669	0.0589
4	baldwin_rank_ties_average	0.8664	0.0492
5	copeland, ranked_pairs_strength_margin_tie_half, ranked_pairs_strength_margin_tie_ignore, ranked_pairs_strength_winning_votes_tie_half, ranked_pairs_strength_winning_votes_tie_ignore, schulze_tie_half, schulze_tie_ignore	0.8654	0.0489
6	rasch_mml	0.8648	0.0417

Appendix

Table 19: Self-consistency at

N=1

on the combined benchmark, measured as Kendall’s

\tau_{b}

between each method’s single-trial ranking and its own full-trial ranking (method@80). Statistics are computed over 80 single-trial draws. Methods with identical (Mean, Std.) are collapsed; we show the top 10 and bottom 5 groups.

Figure 6: Across our four benchmarks, the prior advantage is not monotonically related to difficulty (a), but it is associated with greedy–sampling alignment (b). The sampling–greedy accuracy gap (c) shows no clear relationship.

Figure 7: Bootstrap distributions of Kendall’s

\tau_{b}

N=1

(

50

samples). Violin plots show the full distribution; the greedy prior (red) yields narrower distributions but can shift the mean negatively (HMMT’25) or positively (BrUMO’25).

#	Signal	Description
1	has_box	Boxed final answer present (0/1)
2	is_correct	Exact-match correctness (0/1)
3	token_ratio	Completion tokens / 32768
4	repeated_pattern	Non- stop finish reason (0/1)
5	prompt_bpt	Prompt bits-per-token
6	completion_bpt	Completion bits-per-token

Appendix

Table 20: Nine base signals extracted per completion for the categorical ranking experiments. Each model–question–trial entry produces a vector in

\mathbb{R}^{9}

Scheme	Intent	Categories ( $k$ )	Weights $\mathbf{w}$
Conservative	Penalize confidently-wrong	1: Wrong $\wedge$ HighConf 2: Wrong $\wedge$ LowConf 3: Correct	$(0,\,{-}0.10,\,0.05,\,1.00)$
Efficiency-adj.	Discount verbose correct	1–3: Wrong $\times$ {Econ., Mod., Verb.} 4–6: Correct $\times$ {Econ., Mod., Verb.}	$(0,\,0.10,\,0.07,\,0.03,\,1,\,0.92,\,0.85)$
Format-aware	Reward boxed correct	1: Wrong $\wedge$ Unboxed; 2: Wrong $\wedge$ Boxed 3: Correct; 4: Correct $\wedge$ Boxed	$(0,\,0.10,\,0.05,\,0.90,\,1)$
Balanced comp.	Format $\times$ confidence	1: Wrong $\wedge$ Unboxed; 2–3: Wrong $\wedge$ Boxed $\times$ Conf 4–7: Correct $\times$ {Un/Boxed} $\times$ Conf	$(0,\,0.10,\,0.06,\,{-}0.02,\,0.90,\,0.95,\,0.97,\,1)$
OOD-robust	Reward in-distribution	1: OOD $\wedge$ Wrong; 2: InDist $\wedge$ Wrong 3: OOD $\wedge$ Correct; 4: InDist $\wedge$ Correct	$(0,\,0.05,\,0.10,\,0.95,\,1)$
Rare-event	OOD + abstention	1: OOD $\wedge$ Wrong; 2: OOD $\wedge$ Correct 3: InDist $\wedge$ Wrong; 4: InDist $\wedge$ Correct; 5: Abstain	$(0,\,0.05,\,1,\,0.08,\,0.95,\,0.20)$

Appendix

Table 21: Eight representative categorical schemes used in Section 3.5 . Each scheme maps a completion to one of

C+1

categories using the base signals in Table 20 and scores the result with a utility weight vector

\mathbf{w}

. Category

0

is always Invalid (

w_{0}=0

) unless otherwise noted.

	Gold-standard agreement ( $\tau_{\text{GS}}$ )				Self-consistency ( $\tau_{\text{Self}}$ )
Scheme	AIME’24	AIME’25	HMMT’25	BrUMO’25	AIME’24	AIME’25	HMMT’25	BrUMO’25
Conservative	0.814	0.813	0.801	0.815	0.814	0.813	0.801	0.820
Efficiency-adj.	0.814	0.821	0.812	0.817	0.814	0.814	0.814	0.828
Format-aware	0.820	0.808	0.812	0.819	0.820	0.811	0.813	0.830
Balanced comp.	0.816	0.810	0.804	0.806	0.816	0.813	0.805	0.816
OOD-robust	0.819	0.806	0.788	0.810	0.819	0.803	0.802	0.824

Appendix

Table 22: Per-dataset categorical ranking at

N\!=\!1

. Gold-standard agreement (

\tau_{\text{GS}}

: vs.

\mathrm{Bayes}_{\mathcal{U}}@80

) and self-consistency (

\tau_{\text{Self}}

: vs. Scheme@

80

) for the

8

representative schemes. Values are mean Kendall’s

\tau_{b}

over

80

single-trial draws.

ID	Model	Short name
1	DeepSeek-R1-Distill-Qwen-1.5B	DS-R1-Qwen
2	LIMO-v2	LIMO-v2
3	OpenThinker2-32B	OpenThinker2
4	OpenThinker3-1.5B	OpenThinker3
5	Qwen3-30B-A3B-Thinking-2507	Qwen3-Thinking
6	Sky-T1-32B-Flash	Sky-T1-Flash

Appendix

Table 23: Mapping between model IDs, full model names, and the shortened names used in figures and legends.

Task	Inference Time (hours)	Completion Tokens (M)
AIME’24	1,699.4	680.0
AIME’25	1,878.4	728.3
HMMT’25	2,216.5	851.2
BrUMO’25	1,650.9	666.9
TOTAL	7,445.2	2,926.4

Appendix

Table 24: Task-level computational cost aggregated over 20 models, 80 trials, four tasks, and 30 questions per task. Token counts correspond to completion tokens only.

	$\displaystyle\mathcal{L}(\theta,b,a,c)={}$	$\displaystyle-\sum_{l,m}\ell(k_{lm};n,p_{lm})$
		$\displaystyle-\log p(\theta).$

Appendix

Algorithm 6 Binomial xPL IRT (JMLE/MAP) for ranking

-\sum_{l,q}w_{lq}\,\ell\!\left(k_{lm};N,\sigma(\theta_{q}-b_{m})\right).

Appendix

Algorithm 7 Rasch MML (EM + quadrature) with EAP and posterior-quantile scoring

Explore similar work

Apr 29, 2026

When to Vote, When to Rewrite: Disagreement-Guided Strategy Routing for Test-Time Scaling

Large Reasoning Models (LRMs) achieve strong performance on mathematical reasoning tasks but remain unreliable on challenging instances. Existing test-time scaling methods, such as repeated sampling, self-correction, and tree search, improve performance at the cost of increased computation, yet often exhibit diminishing returns on hard problems. We observe that output disagreement is strongly correlated with instance difficulty and prediction correctness, providing a useful signal for guiding instance-level strategy selection at test time. Based on this insight, we propose a training-free framework that formulates test-time scaling as an instance-level routing problem, rather than allocating more computation within a single strategy, dynamically selecting among different scaling strategies based on output disagreement. The framework applies lightweight resolution for consistent cases, majority voting for moderate disagreement, and rewriting-based reformulation for highly ambiguous instances. Experiments on seven mathematical benchmarks and three models show that our method improves accuracy by 3% - 7% while reducing sampling cost compared to existing approaches.

Zhimin Lin, Yixin Ji, Jinpeng Li +5

School of Computer Science and Technology, Soochow University · Department of Foundation Model, 2012 Labs, Huawei · 3Harbin Institute of Technology, Shenzhen (HITSZ)

Test-Time Scaling LLM Routing LLM Mathematical Reasoning Large Reasoning Models

Ask PDF

Aug 4, 2026

Test-Time Scaling in Reasoning LLMs: Inference Regimes, Evaluation, and Reproducibility

Large language models can solve harder reasoning problems with more inference-time compute. The term "test-time scaling," however, covers several inference algorithms: extending deliberation along one trajectory, sampling completed candidates and aggregating them by voting or verification, and searching over partial states. These algorithms differ in statistical structure, compute requirements, and failure modes. Treating them as interchangeable under a scalar "budget," or reporting accuracy without specifying the inference protocol, makes results difficult to compare across studies. We study test-time scaling along three axes. First, we formalize it as budgeted inference over the implicit prefix tree of an autoregressive model and distinguish single-trajectory sequential scaling, leaf-level scaling with terminal reduction, and prefix-level scaling. Second, we treat the full inference system as the evaluated object and separate end-to-end performance from candidate-bank diagnostics. We introduce an evaluation profile whose coordinates and simple functionals recover or bound common repeated-sampling metrics, and require compute accounting and uncertainty estimates that match the protocol. Third, we distinguish exact replay from distributional reproducibility and state the requirements for each. We also organize open-weight reasoning models by model-side and interface mechanisms. Our empirical study covers broad knowledge, symbolic reasoning, and competition mathematics, and we publicly release 1,403,520 sampled model attempts. The project website is available at https://mohsenhariri.github.io/scorio/tts. The released datasets are Trace (https://huggingface.co/datasets/harimo/scorio-trace), Lite (https://huggingface.co/datasets/harimo/scorio-lite), Math (https://huggingface.co/buckets/harimo/scorio-math), and SuperGPQA (https://huggingface.co/buckets/harimo/scorio-gpqa).

Mohsen Hariri, Weicong Chen, Nahal Shahini +11

Case Western Reserve University

LLM Evaluation Test-Time Scaling ML Reproducibility LLM Reasoning

Ask PDF

Mar 25, 2025

Scaling Evaluation-time Compute with Reasoning Models as Evaluators

As language model (LM) outputs get more and more natural, it is becoming more difficult than ever to evaluate their quality. Simultaneously, increasing LMs' "thinking" time through scaling test-time compute has proven an effective technique to solve challenging problems in domains such as math and code. This raises a natural question: can an LM's evaluation capability also be improved by spending more test-time compute? To answer this, we investigate employing reasoning models-LMs that natively generate long chain-of-thought reasoning-as evaluators. Specifically, we examine methods to leverage more test-time compute by (1) using reasoning models, and (2) prompting these models to evaluate not only the response as a whole (i.e., outcome evaluation) but also assess each step in the response separately (i.e., process evaluation). In experiments, we observe that the evaluator's performance improves monotonically when generating more reasoning tokens, similar to the trends observed in LM-based generation. Furthermore, we use these more accurate evaluators to rerank multiple generations, and demonstrate that spending more compute at evaluation time can be as effective as using more compute at generation time in improving an LM's problem-solving capability.

Seungone Kim, Ian Wu, Jinu Lee +8

CMU · UIUC2 · KAIST AI3 +3

LLM Evaluation Language Model Generation Evaluation Inference-Time Scaling Large Reasoning Models

Ask PDF