Compare to Correct: Pairwise-Guided Logit Adjustment for LLM Test-Time Scaling

Cong Hua; Qianqian Xu; Zitai Wang; Zhiyong Yang; Shilong Bao; Linchao Zhang; Qingming Huang

cs.AIOct 13, 2026

Compare to Correct: Pairwise-Guided Logit Adjustment for LLM Test-Time Scaling

Authors: Cong Hua, Qianqian Xu, Zitai Wang, Zhiyong Yang, Shilong Bao, Linchao Zhang, Qingming Huang

Organizations: State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences. · School of Computer Science and Technology, University of Chinese Academy of Sciences. · Beijing Academy of Artificial Intelligence. · Artificial Intelligence Institute of China Electronics Technology Group Corporation.

Abstract

Sampling-based test-time scaling enhances LLM reasoning by aggregating multiple reasoning paths for the same query. Existing methods use answer frequencies or path confidence to compute candidate scores, which do not explicitly characterize comparisons between competing answers. Inspired by the Condorcet principle in social choice theory, we introduce pairwise confidence comparisons between paths supporting different answers to capture answer-level comparative advantage. We combine this comparative evidence with voting support through a KL-regularized objective. Theoretically, we show that this comparative-advantage is equivalent to the adjustment to the traditional voting logits. In this way, our proposed Pairwise-Guided Logit Adjustment (PGLA) framework allows comparative evidence to compensate for weaker voting support and correct voting-based ranking errors. Experiments on 10 challenging benchmarks including mathematical reasoning, knowledge reasoning, and code generation with 7 models ranging from 1.7B to 32B parameters speak to the effectiveness of this plug-and-play term. Additional evaluations with external reward scores and online sampling further demonstrate its applicability.

Figures & tables

Figure 1: From voting support to pairwise comparative evidence. (a) Plurality voting selects $A$ , whereas pairwise preference comparisons identify $B$ as the Condorcet winner. (b) Sampled reasoning paths provide answers and confidence scores. Pairwise confidence comparisons complement voting support, enabling the proposed aggregation method to select $B$ .

Figure 4: Disagreement between voting and pairwise confidence comparisons: (a) an illustrative example; (b) percentage of recoverable SC errors.

Method	AIME24	AIME25	HMMT25	GPQA-D	HE	MBPP	LCB
Qwen3-1.7B Reasoning mode
PPL Best-of- $N$	$36.00_{\pm 4.42}$	$34.00_{\pm 2.91}$	$23.00_{\pm 3.48}$	$39.90_{\pm 2.02}$	$81.10_{\pm 2.06}$	$75.13_{\pm 0.68}$	$30.67_{\pm 1.47}$
DeepConf Best-of- $N$	$62.00_{\pm 4.52}$	$42.67_{\pm 2.49}$	$28.33_{\pm 3.73}$	$42.93_{\pm 1.52}$	$84.15_{\pm 2.69}$	$76.72_{\pm 1.18}$	$34.29_{\pm 1.62}$
Self-Certainty Best-of- $N$	$60.67_{\pm 4.16}$	$46.67_{\pm 3.33}$	$30.67_{\pm 4.42}$	$42.17_{\pm 0.25}$	$85.06_{\pm 2.70}$	$81.75_{\pm 1.04}$	$33.33_{\pm 1.64}$
SC	$69.50_{\pm 1.98}$	$47.33_{\pm 3.67}$	$27.00_{\pm 1.00}$	$44.44_{\pm 0.51}$	$83.84_{\pm 1.32}$	$83.07_{\pm 0.89}$	$29.90_{\pm 1.33}$
+ Ours (DeepConf)	$\mathbf{75.00}_{\pm 1.67}$	$\mathbf{51.33}_{\pm 2.67}$	$\mathbf{30.00}_{\pm 2.98}$	$\mathbf{46.72}_{\pm 0.25}$	$\mathbf{88.11}_{\pm 2.24}$	$\mathbf{85.45}_{\pm 0.80}$	$\mathbf{34.29}_{\pm 1.36}$

Table 1: Performance of reasoning models at different scales on three mathematical reasoning benchmarks, GPQA, and three open-ended code-generation benchmarks, using

K=32

sampled responses per problem. Each voting-based baseline is followed by its

\mathsf{PGLA}

-enhanced variant, denoted by +Ours . The enhanced HumanEval+ and MBPP+ tests for the same sampled pools are reported in Tab. 6 .

Method	MATH		MathOdyssey		AIME		Average
	Acc. $\uparrow$	ECE $\downarrow$	Acc. $\uparrow$	ECE $\downarrow$	Acc. $\uparrow$	ECE $\downarrow$	Acc. $\uparrow$	ECE $\downarrow$
InternLM2-Math-Plus-1.8B
PPL	$33.23_{\pm 0.25}$	$61.58_{\pm 0.25}$	$16.63_{\pm 0.83}$	$77.20_{\pm 0.82}$	$1.66_{\pm 0.15}$	$92.28_{\pm 0.16}$	$17.17$	$77.02$
SC	$36.48_{\pm 0.14}$	$6.66_{\pm 0.14}$	$14.57_{\pm 0.45}$	$18.51_{\pm 0.41}$	$2.66_{\pm 0.20}$	$16.84_{\pm 0.16}$	$17.90$	$14.00$
+ Ours	$\mathbf{39.77}_{\pm 0.17}$	$\mathbf{5.63}_{\pm 0.13}$	$\mathbf{18.28}_{\pm 0.56}$	$\mathbf{13.97}_{\pm 0.59}$	$\mathbf{3.37}_{\pm 0.17}$	$\mathbf{15.51}_{\pm 0.15}$	$\mathbf{20.47}$	$\mathbf{11.70}$
Weighted SC	$36.72_{\pm 0.12}$	$6.52_{\pm 0.14}$	$14.86_{\pm 0.47}$	$18.05_{\pm 0.50}$	$2.75_{\pm 0.23}$	$16.65_{\pm 0.19}$	$18.11$	$13.74$

Table 2: Performance comparison of different mathematics-specialized models in terms of accuracy and expected calibration error (ECE). Following the RPC setting, perplexity-based confidence (PPL) is used by default. Each voting-based baseline is followed by its

\mathsf{PGLA}

-enhanced variant, denoted by +Ours .

Method	MATH		MathOdyssey		AIME		Average
	$K/2$ $\uparrow$	$K$ $\uparrow$	$K/2$ $\uparrow$	$K$ $\uparrow$	$K/2$ $\uparrow$	$K$ $\uparrow$	$K/2$ $\uparrow$	$K$ $\uparrow$
InternLM2-Math-Plus-7B
PRM-Weighted SC	$57.56_{\pm 0.31}$	$58.67_{\pm 0.10}$	$33.57_{\pm 1.13}$	$34.34_{\pm 0.58}$	$15.49_{\pm 0.36}$	$16.23_{\pm 0.48}$	$35.54$	$36.42$
+ Ours	$\mathbf{58.86}_{\pm 0.29}$	$\mathbf{60.40}_{\pm 0.24}$	$\mathbf{37.63}_{\pm 1.18}$	$\mathbf{38.79}_{\pm 0.62}$	$\mathbf{15.55}_{\pm 0.42}$	$\mathbf{16.57}_{\pm 0.35}$	$\mathbf{37.35}$	$\mathbf{38.59}$
PRM-RPC	$57.74_{\pm 0.28}$	$58.90_{\pm 0.13}$	$34.83_{\pm 1.18}$	$35.48_{\pm 0.40}$	$\mathbf{15.57}_{\pm 0.42}$	$16.31_{\pm 0.44}$	$36.05$	$36.90$
+ Ours	$\mathbf{58.88}_{\pm 0.24}$	$\mathbf{60.41}_{\pm 0.25}$	$\mathbf{37.63}_{\pm 1.24}$	$\mathbf{38.87}_{\pm 0.60}$	$15.53_{\pm 0.37}$	$\mathbf{16.50}_{\pm 0.28}$	$\mathbf{37.35}$	$\mathbf{38.59}$

Table 3: Accuracy (%) using Qwen2.5-Math-PRM-7B scores at sampling budgets

K/2

and

K

(

K=64

for MATH;

128

otherwise). + Ours denotes the

\mathsf{PGLA}

-enhanced voting rule.

Panel of Figure 5 — Figure 5: Tradeoff between Case I and Case II on AIME24 and AIME25 using Qwen3-1.7B and Qwen3-8B. DWSC denotes the DeepConf-Weighted SC method.

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Model	MATH	MathOdyssey	AIME
NoCorrect / all SC errors
InternLM2-Math-Plus-1.8B	$50.16$	$46.94$	$75.83$
InternLM2-Math-Plus-7B	$48.58$	$53.10$	$63.70$
DeepSeek-Math-RL-7B	$51.83$	$59.42$	$62.18$
Case II / recoverable SC errors
InternLM2-Math-Plus-1.8B	$82.22$	$83.81$	$91.62$

Appendix

Table 5: Breakdown of SC errors (%). NoCorrect is measured over all SC errors; recoverable Case II is measured after excluding NoCorrect pools. Results use PPL confidence.

Figure 9: Illustration of $\mathsf{PGLA}$ .

	Qwen3-1.7B Reasoning		Qwen3-4B Reasoning
Methods	HumanEval+	MBPP+	HumanEval+	MBPP+
PPL Best-of- $N$	$73.48_{\pm 1.80}$	$64.55_{\pm 0.99}$	$86.99_{\pm 0.55}$	$77.59_{\pm 0.74}$
DeepConf Best-of- $N$	$77.44_{\pm 2.36}$	$66.40_{\pm 0.76}$	$90.04_{\pm 0.99}$	$78.89_{\pm 0.76}$
Self-Certainty Best-of- $N$	$78.05_{\pm 2.67}$	$67.46_{\pm 0.97}$	$90.24_{\pm 1.06}$	$78.99_{\pm 0.62}$
SC	$77.29_{\pm 1.14}$	$71.16_{\pm 0.83}$	$88.01_{\pm 1.16}$	$80.58_{\pm 0.62}$
SC + Ours (DeepConf)	$81.10_{\pm 2.46}$	$71.43_{\pm 0.53}$	$90.04_{\pm 1.14}$	$80.79_{\pm 0.46}$

Appendix

Table 6: Enhanced code-test accuracy (%) in reasoning mode (

K=32

). Here + Ours denotes a

\mathsf{PGLA}

-enhanced variant.

Qwen3-1.7B
Methods	Non-thinking mode			Reasoning mode
Methods	AIME 2024	AIME 2025	HMMT 2025	AIME 2024	AIME 2025	HMMT 2025
PPL Best-of- $N$	$12.00_{\pm 2.21}$	$11.33_{\pm 3.40}$	$5.00_{\pm 3.42}$	$36.00_{\pm 4.42}$	$34.00_{\pm 2.91}$	$23.00_{\pm 3.48}$
DeepConf Best-of- $N$	$12.33_{\pm 2.13}$	$14.33_{\pm 2.60}$	$8.33_{\pm 3.73}$	$62.00_{\pm 4.52}$	$42.67_{\pm 2.49}$	$28.33_{\pm 3.73}$
Self-Certainty Best-of- $N$	$12.67_{\pm 2.49}$	$15.00_{\pm 3.42}$	$9.67_{\pm 2.33}$	$60.67_{\pm 4.16}$	$46.67_{\pm 3.33}$	$30.67_{\pm 4.42}$
SC	$15.17_{\pm 1.55}$	$14.83_{\pm 1.57}$	$6.94_{\pm 0.57}$	$69.50_{\pm 1.98}$	$47.33_{\pm 3.67}$	$27.00_{\pm 1.00}$

Appendix

Table 7: Comparison of non-thinking and reasoning modes for Qwen3-1.7B on three mathematical reasoning benchmarks (

K=32

Qwen3-4B
Methods	Non-thinking mode			Reasoning mode
Methods	AIME 2024	AIME 2025	HMMT 2025	AIME 2024	AIME 2025	HMMT 2025
PPL Best-of- $N$	$30.00_{\pm 3.94}$	$24.14_{\pm 2.82}$	$13.00_{\pm 2.77}$	$57.00_{\pm 2.33}$	$64.33_{\pm 3.96}$	$40.33_{\pm 4.33}$
DeepConf Best-of- $N$	$31.67_{\pm 4.77}$	$25.29_{\pm 1.63}$	$15.33_{\pm 3.06}$	$77.00_{\pm 2.33}$	$76.00_{\pm 3.89}$	$50.00_{\pm 4.94}$
Self-Certainty Best-of- $N$	$32.33_{\pm 3.96}$	$26.44_{\pm 3.25}$	$14.33_{\pm 2.60}$	$81.67_{\pm 2.24}$	$71.67_{\pm 3.07}$	$51.67_{\pm 4.01}$
SC	$39.56_{\pm 4.24}$	$26.25_{\pm 1.51}$	$15.67_{\pm 1.53}$	$80.00_{\pm 0.00}$	$76.33_{\pm 0.67}$	$51.50_{\pm 1.57}$

Appendix

Table 8: Comparison of non-thinking and reasoning modes for Qwen3-4B on three mathematical reasoning benchmarks (

K=32

Method	MATH		MathOdyssey		AIME		Average
	$K/2$ $\uparrow$	$K$ $\uparrow$	$K/2$ $\uparrow$	$K$ $\uparrow$	$K/2$ $\uparrow$	$K$ $\uparrow$	$K/2$ $\uparrow$	$K$ $\uparrow$
InternLM2-Math-Plus-1.8B
PRM Best-of- $N$	$46.50_{\pm 0.33}$	$48.48_{\pm 0.21}$	$24.75_{\pm 1.44}$	$25.47_{\pm 1.24}$	$4.43_{\pm 0.24}$	$4.93_{\pm 0.34}$	$25.23$	$26.30$
PRM-Weighted SC	$45.64_{\pm 0.26}$	$47.39_{\pm 0.23}$	$23.24_{\pm 1.04}$	$24.91_{\pm 1.07}$	$4.63_{\pm 0.25}$	$5.02_{\pm 0.23}$	$24.50$	$25.77$
+ Ours	$\mathbf{46.88}_{\pm 0.29}$	$\mathbf{49.09}_{\pm 0.20}$	$\mathbf{26.99}_{\pm 1.16}$	$\mathbf{28.95}_{\pm 1.26}$	$\mathbf{4.73}_{\pm 0.33}$	$\mathbf{5.17}_{\pm 0.27}$	$\mathbf{26.20}$	$\mathbf{27.73}$
PRM-RPC	$45.77_{\pm 0.25}$	$47.61_{\pm 0.23}$	$23.60_{\pm 1.19}$	$25.32_{\pm 1.04}$	$4.66_{\pm 0.24}$	$5.07_{\pm 0.21}$	$24.68$	$26.00$

Appendix

Table 9: Accuracy comparison under the sampling budgets of

\frac{K}{2}

and

K

, using product-aggregated rewards from an external process reward model Qwen2.5-Math-PRM-7B ( Lightman et al., 2024 ; Zhang et al., 2025 ) . + Ours denotes the

\mathsf{PGLA}

-enhanced variant of each voting baseline.

Panel of Figure 10 — Figure 10: Tradeoff between Case I and Case II on AIME24, AIME25, and HMMT25 using Qwen3-1.7B, Qwen3-4B, and Qwen3-8B. Ours denotes $\mathsf{PGLA}$ . DeepConf, PPL, and Self-Certainty denote DeepConf Best-of- $N$ , PPL Best-of- $N$ , and Self-Certainty Best-of- $N$ , respectively. SWSC and DWSC denote Self-Certainty-Weighted SC and DeepConf-Weighted SC, respectively.

Table 17

Method	MathOdyssey				AIME
	$T=1.1$		$T=1.3$		$T=1.1$		$T=1.3$
	Acc. $\uparrow$	ECE $\downarrow$	Acc. $\uparrow$	ECE $\downarrow$	Acc. $\uparrow$	ECE $\downarrow$	Acc. $\uparrow$	ECE $\downarrow$
InternLM2-Math-Plus-1.8B
PPL	$16.43_{\pm 0.91}$	$75.99_{\pm 0.89}$	$16.38_{\pm 1.42}$	$71.72_{\pm 1.43}$	$1.97_{\pm 0.29}$	$90.29_{\pm 0.28}$	$2.08_{\pm 0.27}$	$84.85_{\pm 0.27}$
SC	$13.75_{\pm 0.51}$	$18.02_{\pm 0.57}$	$14.95_{\pm 0.87}$	$5.15_{\pm 1.12}$	$2.12_{\pm 0.10}$	$16.52_{\pm 0.13}$	$1.26_{\pm 0.08}$	$8.80_{\pm 0.12}$
SC + Ours	$\mathbf{18.59_{\pm 0.78}}$	$\mathbf{12.75_{\pm 0.75}}$	$\mathbf{19.23_{\pm 0.76}}$	$\mathbf{6.86_{\pm 1.04}}$	$\mathbf{3.11_{\pm 0.17}}$	$\mathbf{14.78_{\pm 0.19}}$	$\mathbf{1.82_{\pm 0.17}}$	$\mathbf{10.64_{\pm 0.16}}$

Appendix

Table 16: Temperature sensitivity under the same internal-confidence protocol as Tab. 2 . We vary only the decoding temperature; + Ours denotes the

\mathsf{PGLA}

-enhanced variant of each voting baseline.

Method	MATH		MathOdyssey		AIME
	Acc.	$K$	Acc.	$K$	Acc.	$K$
Best of SC	36.48 $\pm$ 0.14	64	14.91 $\pm$ 0.95	48	2.67 $\pm$ 0.21	112
SC + Ours	37.76 $\pm$ 0.34	16	16.53 $\pm$ 0.92	8	2.76 $\pm$ 0.34	32
$\Delta$ vs. SC	+1.27	-75.0%	+1.62	-83.3%	+0.09	-71.4%
RPC	37.09 $\pm$ 0.20	32	15.37 $\pm$ 1.45	8	2.71 $\pm$ 0.39	32
$\Delta$ vs. SC	+0.61	-50.0%	+0.46	-83.3%	+0.04	-71.4%

Appendix

Table 17: Post-hoc sampling-budget comparison for InternLM2-Math-Plus-1.8B . Best of SC gives its highest grid mean and smallest attaining

K

. Alternatives use the minimum

K

matching or exceeding this unrounded target.

\Delta

gives absolute accuracy change (%) and sample-count change (%) relative to SC.

Method	MATH		MathOdyssey		AIME
	Acc.	$K$	Acc.	$K$	Acc.	$K$
Best of SC	53.33 $\pm$ 0.08	64	36.57 $\pm$ 0.56	128	9.56 $\pm$ 0.24	96
SC + Ours	53.52 $\pm$ 0.10	64	36.76 $\pm$ 1.00	40	9.80 $\pm$ 0.37	64
$\Delta$ vs. SC	+0.19	0.0%	+0.19	-68.8%	+0.23	-33.3%
RPC	53.36 $\pm$ 0.11	64	36.86 $\pm$ 0.57	112	-	-
$\Delta$ vs. SC	+0.03	0.0%	+0.29	-12.5%	-	-

Appendix

Table 18: Post-hoc sampling-budget comparison for DeepSeek-Math-RL-7B . Best of SC gives its highest grid mean and smallest attaining

K

. Alternatives use the minimum

K

matching or exceeding this unrounded target.

\Delta

gives absolute accuracy change (%) and sample-count change (%) relative to SC.

MATH
$K$	PPL	SC	SC + Ours	RPC	RPC + Ours
1	26.94 $\pm$ 0.49	26.94 $\pm$ 0.49	26.94 $\pm$ 0.49	26.94 $\pm$ 0.49	26.94 $\pm$ 0.49
2	29.62 $\pm$ 0.47	27.13 $\pm$ 0.44	29.62 $\pm$ 0.47	29.62 $\pm$ 0.47	29.62 $\pm$ 0.47
4	31.32 $\pm$ 0.44	31.01 $\pm$ 0.36	31.32 $\pm$ 0.42	32.36 $\pm$ 0.41	31.30 $\pm$ 0.42
8	32.53 $\pm$ 0.42	33.93 $\pm$ 0.26	34.81 $\pm$ 0.46	34.78 $\pm$ 0.32	34.34 $\pm$ 0.47
16	33.15 $\pm$ 0.42	35.49 $\pm$ 0.28	37.76 $\pm$ 0.34	36.08 $\pm$ 0.30	37.67 $\pm$ 0.36

Appendix

Table 19: Accuracy across sampling budgets for InternLM2-Math-Plus-1.8B on MATH, MathOdyssey, and AIME.

MATH
$K$	PPL	SC	SC + Ours	RPC	RPC + Ours
1	41.69 $\pm$ 0.42	41.69 $\pm$ 0.42	41.69 $\pm$ 0.42	41.69 $\pm$ 0.42	41.69 $\pm$ 0.42
2	43.93 $\pm$ 0.39	41.73 $\pm$ 0.22	43.93 $\pm$ 0.39	43.93 $\pm$ 0.39	43.93 $\pm$ 0.39
4	45.57 $\pm$ 0.38	45.92 $\pm$ 0.26	45.51 $\pm$ 0.37	47.09 $\pm$ 0.33	45.48 $\pm$ 0.38
8	46.49 $\pm$ 0.42	48.45 $\pm$ 0.21	49.06 $\pm$ 0.51	49.10 $\pm$ 0.18	48.54 $\pm$ 0.46
16	47.14 $\pm$ 0.22	49.76 $\pm$ 0.16	52.07 $\pm$ 0.19	50.24 $\pm$ 0.13	51.80 $\pm$ 0.17

Appendix

Table 20: Accuracy across sampling budgets for InternLM2-Math-Plus-7B on MATH, MathOdyssey, and AIME.

Dataset	Method	Accuracy		Sampling Cost
Dataset	Method	Acc.	Gain over Online SC	$\overline{K}$	Saving
InternLM2-Math-Plus-1.8B
MATH	SC (Full Budget)	$36.48_{\pm 0.14}$	–	64.0	–
	Adaptive SC	$36.49_{\pm 0.16}$	–	36.3	43.3%
	Online SC	$36.38_{\pm 0.19}$	–	33.4	47.7%
	Online SC + $\mathsf{PGLA}$	$\mathbf{39.32}_{\pm 0.31}$	$\mathbf{+2.95}$	31.5	50.8%

Appendix

Table 21: Online aggregation across the two InternLM2-Math-Plus backbones.

\overline{K}

is the average number of sampled paths. Gain is the absolute accuracy change (%) against Online SC under the same stopping rule, and saving is relative to

K_{\max}

, which is 64 for MATH and 128 otherwise.

Method	AIME24	AIME25	HMMT25
Qwen3-32B Reasoning mode
PPL Best-of- $N$	$75.67_{\pm 3.96}$	$67.33_{\pm 6.29}$	$41.00_{\pm 3.96}$
DeepConf Best-of- $N$	$84.33_{\pm 3.96}$	$79.33_{\pm 2.00}$	$55.33_{\pm 2.67}$
Self-Certainty Best-of- $N$	$88.33_{\pm 2.69}$	$77.00_{\pm 2.33}$	$62.33_{\pm 3.00}$
SC	$90.17_{\pm 1.57}$	$81.33_{\pm 1.63}$	$59.50_{\pm 2.67}$
+ Ours (DeepConf)	$\mathbf{90.67}_{\pm 2.00}$	$\mathbf{81.67}_{\pm 1.67}$	$63.67_{\pm 2.77}$

Appendix

Table 22: Accuracy (%) of Qwen3-32B in reasoning mode at

K=32

Method	MATH		MathOdyssey		AIME		Average
	Acc. $\uparrow$	ECE $\downarrow$	Acc. $\uparrow$	ECE $\downarrow$	Acc. $\uparrow$	ECE $\downarrow$	Acc. $\uparrow$	ECE $\downarrow$
InternLM2-Math-Plus-1.8B
PPL	$33.23_{\pm 0.25}$	$61.58_{\pm 0.25}$	$16.63_{\pm 0.83}$	$77.20_{\pm 0.82}$	$1.66_{\pm 0.15}$	$92.28_{\pm 0.16}$	$17.17$	$77.02$
SC	$36.48_{\pm 0.14}$	$6.66_{\pm 0.14}$	$14.57_{\pm 0.45}$	$18.51_{\pm 0.41}$	$2.66_{\pm 0.20}$	$16.84_{\pm 0.16}$	$17.90$	$14.00$
+ Ours (voting)	$\mathbf{39.77}_{\pm 0.17}$	$5.63_{\pm 0.13}$	$18.28_{\pm 0.56}$	$13.97_{\pm 0.59}$	$3.37_{\pm 0.17}$	$15.51_{\pm 0.15}$	$20.47$	$11.70$
+ Ours (Uniform)	$39.24_{\pm 0.14}$	$\mathbf{4.94}_{\pm 0.13}$	$17.46_{\pm 0.61}$	$14.39_{\pm 0.69}$	$3.25_{\pm 0.14}$	$\mathbf{15.34}_{\pm 0.11}$	$19.98$	$11.55$

Appendix

Table 23: Comparative-advantage functions on InternLM2-Math-Plus-1.8B . We expand Tab. 2 using the same coefficients ( Sec. G.11 ). Voting denotes the default voting-weighted comparative advantage and + Ours denotes the scaled

\mathsf{PGLA}

variants described in Sec. G.11 .

Method	MATH		MathOdyssey		AIME		Average
	Acc. $\uparrow$	ECE $\downarrow$	Acc. $\uparrow$	ECE $\downarrow$	Acc. $\uparrow$	ECE $\downarrow$	Acc. $\uparrow$	ECE $\downarrow$
InternLM2-Math-Plus-7B
PPL	$47.00_{\pm 0.20}$	$48.98_{\pm 0.20}$	$27.51_{\pm 1.18}$	$67.54_{\pm 1.18}$	$5.96_{\pm 0.48}$	$88.98_{\pm 0.49}$	$26.82$	$68.50$
SC	$50.57_{\pm 0.18}$	$6.72_{\pm 0.19}$	$28.30_{\pm 0.61}$	$12.18_{\pm 0.57}$	$9.40_{\pm 0.21}$	$14.35_{\pm 0.23}$	$29.42$	$11.08$
+ Ours (voting)	$\mathbf{53.99}_{\pm 0.18}$	$6.10_{\pm 0.20}$	$\mathbf{35.30}_{\pm 0.55}$	$9.69_{\pm 1.02}$	$9.95_{\pm 0.24}$	$13.90_{\pm 0.23}$	$\mathbf{33.08}$	$9.90$
+ Ours (Uniform)	$53.72_{\pm 0.13}$	$\mathbf{5.13}_{\pm 0.23}$	$33.62_{\pm 0.72}$	$8.68_{\pm 0.75}$	$10.04_{\pm 0.23}$	$\mathbf{13.26}_{\pm 0.25}$	$32.46$	$\mathbf{9.02}$

Appendix

Table 24: Comparative-advantage functions on InternLM2-Math-Plus-7B . We expand Tab. 2 using the same coefficients ( Sec. G.11 ). Voting denotes the default voting-weighted comparative advantage and + Ours denotes the scaled

\mathsf{PGLA}

variants described in Sec. G.11 .

Method	MATH		MathOdyssey		AIME		Average
	Acc. $\uparrow$	ECE $\downarrow$	Acc. $\uparrow$	ECE $\downarrow$	Acc. $\uparrow$	ECE $\downarrow$	Acc. $\uparrow$	ECE $\downarrow$
DeepSeek-Math-RL-7B
PPL	$42.50_{\pm 0.23}$	$53.84_{\pm 0.23}$	$22.34_{\pm 1.00}$	$73.85_{\pm 0.99}$	$3.37_{\pm 0.46}$	$93.38_{\pm 0.46}$	$22.74$	$73.69$
SC	$53.33_{\pm 0.08}$	$\mathbf{6.43}_{\pm 0.11}$	$36.57_{\pm 0.56}$	$9.40_{\pm 1.21}$	$9.42_{\pm 0.23}$	$12.12_{\pm 0.26}$	$33.11$	$9.32$
+ Ours (voting)	$\mathbf{53.52}_{\pm 0.10}$	$6.86_{\pm 0.09}$	$37.87_{\pm 0.55}$	$9.31_{\pm 0.95}$	$\mathbf{10.02}_{\pm 0.23}$	$10.88_{\pm 0.25}$	$\mathbf{33.80}$	$9.02$
+ Ours (Uniform)	$53.49_{\pm 0.11}$	$6.57_{\pm 0.09}$	$37.51_{\pm 0.64}$	$9.24_{\pm 0.76}$	$9.99_{\pm 0.22}$	$\mathbf{10.78}_{\pm 0.20}$	$33.66$	$8.87$

Appendix

Table 25: Comparative-advantage functions on DeepSeek-Math-RL-7B . We expand Tab. 2 using the same coefficients ( Sec. G.11 ). Voting denotes the default voting-weighted comparative advantage and + Ours denotes the scaled

\mathsf{PGLA}

variants described in Sec. G.11 .

Implementation	$K=32$	$K=64$	$K=128$
	AIME 2024	MATH	MathOdyssey
Direct pairwise evaluation	29.7	48.9	108.3
Exact sorting + prefix sums	53.4	57.4	60.3

Appendix

Table 26: CPU aggregation latency of two exact

\mathsf{PGLA}

implementations (

\mu

s per query), measured on real cached candidate pools. Both use identical inputs and confidence-weighted voting support.

	Qwen3-1.7B		Qwen3-4B		Avg. results
Method	AIME24	HMMT25	AIME24	HMMT25	Avg. results
DeepConf
SC	$69.50_{\pm 1.98}$	$27.00_{\pm 1.00}$	$80.00_{\pm 0.00}$	$51.50_{\pm 1.57}$	$57.00$
SC- $\mathsf{PGLA}$	$\mathbf{75.00}_{\pm 1.67}$	$30.00_{\pm 2.98}$	$83.00_{\pm 1.00}$	$\mathbf{54.00}_{\pm 2.91}$	$60.50$
Borda	$72.67_{\pm 2.49}$	$29.00_{\pm 3.35}$	$82.00_{\pm 1.63}$	$52.33_{\pm 2.60}$	$59.00$
Borda+ $\mathsf{PGLA}$	$74.00_{\pm 2.49}$	$\mathbf{33.17}_{\pm 3.83}$	$\mathbf{83.67}_{\pm 2.33}$	$\mathbf{54.00}_{\pm 2.91}$	$\mathbf{61.21}$

Appendix

Table 27: Qwen3 reasoning-mode accuracy on AIME24 and HMMT25 (%;

K=32

	InternLM2-Math-Plus-1.8B			InternLM2-Math-Plus-7B			Avg. results
Method	MATH	MathOdyssey	AIME	MATH	MathOdyssey	AIME	Avg. results
SC	$36.48_{\pm 0.14}$	$14.57_{\pm 0.45}$	$2.66_{\pm 0.20}$	$50.57_{\pm 0.18}$	$28.30_{\pm 0.61}$	$9.40_{\pm 0.21}$	$23.66$
SC- $\mathsf{PGLA}$	$39.77_{\pm 0.17}$	$18.28_{\pm 0.56}$	$3.37_{\pm 0.17}$	$\mathbf{53.99}_{\pm 0.18}$	$35.30_{\pm 0.55}$	$9.95_{\pm 0.24}$	$26.77$
Borda	$38.83_{\pm 0.08}$	$17.12_{\pm 0.62}$	$3.23_{\pm 0.18}$	$53.07_{\pm 0.16}$	$33.14_{\pm 0.59}$	$9.97_{\pm 0.12}$	$25.89$
Borda+ $\mathsf{PGLA}$	$\mathbf{40.58}_{\pm 0.21}$	$\mathbf{21.13}_{\pm 0.57}$	$\mathbf{3.56}_{\pm 0.20}$	$53.60_{\pm 0.16}$	$\mathbf{37.48}_{\pm 0.58}$	$\mathbf{10.20}_{\pm 0.34}$	$\mathbf{27.76}$

Appendix

Table 28: InternLM2-Math-Plus accuracy (%; PPL ordering), with

K=64

for MATH and

K=128

otherwise.

Explore similar work

Ranking Reasoning LLMs under Test-Time Scaling
Mar 11, 2026Mohsen Hariri, Michael Hinczewski, Jing Ma +1LLM Evaluation Test-Time Scaling
Department of Computer and Data Sciences, Case Western Reserve University, Cleveland, OH, USA · Department of Physics, Case Western Reserve University, Cleveland, OH, USA
Refining Over Resampling: Test-Time Self-Correction for LLM Reasoning
Aug 6, 2026Ahsan Bilal, Muhammad Ahmed Mohsin, Muhammad Umer +4Numerical Reasoning in Language Models LLM Self-Correction
University of Oklahoma · Stanford University · Universitat Pompeu Fabra +1
Interpretable Adaptive Sampling for LLM Test-Time Scaling
Aug 4, 2026Mobina Kashaniyan, Ali JannesariLLM Inference Efficiency Test-Time Scaling
Iowa State University Ames, IA, USA

Mar 11, 2026J/K move · Enter open · S save

Ranking Reasoning LLMs under Test-Time Scaling

Mohsen Hariri, Michael Hinczewski, Jing Ma, Vipin Chaudhary

Department of Computer and Data Sciences, Case Western Reserve University, Cleveland, OH, USA · Department of Physics, Case Western Reserve University, Cleveland, OH, USA

LLM Evaluation Test-Time Scaling

Open paper AskPDF arXiv

Test-time scaling evaluates reasoning LLMs by sampling multiple outputs per prompt, but ranking models in this regime remains underexplored. We formalize dense benchmark ranking under test-time scaling and introduce Scorio, a library that implements statistical ranking methods such as paired-comparison models, item response theory (IRT) models, voting rules, and graph- and spectral-based methods. Across

20

reasoning models on four Olympiad-style math benchmarks (AIME'24, AIME'25, HMMT'25, and BrUMO'25; up to

N=80

trials), most full-trial rankings agree closely with the Bayesian gold standard

\mathrm{Bayes}_{\mathcal{U}}@80

(mean Kendall's

τ_b = 0.93

0.95

), and

19

34

methods recover exactly the same ordering. In the single-trial regime, the best methods reach

τ_b \approx 0.86

. Using greedy decoding as an empirical prior (

\mathrm{Bayes}_{\mathbf{R}_0}@N

) reduces variance at

N=1

16

52\%

, but can bias rankings when greedy and stochastic sampling disagree. These results identify reliable ranking methods for both high- and low-budget test-time scaling. We release Scorio as an open-source library at https://github.com/mohsenhariri/scorio.

Abstract

Explore similar work

Ranking Reasoning LLMs under Test-Time Scaling

Refining Over Resampling: Test-Time Self-Correction for LLM Reasoning

Interpretable Adaptive Sampling for LLM Test-Time Scaling