cs.LGOct 30, 2026
SaveOn the Limits of Test-Time Compute for Self-Evaluation
Organizations: Mila · Université de Montréal · Nous Research · Google ·
Abstract
Self-verification and aggregation have emerged as popular approaches for bridging the gap between pass@1 and pass@N. However, increasing the number of candidates does not consistently narrow the gap between final-answer accuracy and pass@N. To understand this limitation, we use ranking error to measure how reliably models distinguish their correct generations from incorrect ones. We show that ranking error admits a bias-variance decomposition, with both components shaped by algorithmic design choices and inference compute. Across our experiments, additional compute reduces variance, while bias initially decreases before approaching a plateau. Guided by this analysis, we introduce Scorpio, a simple and general method that jointly scores randomly grouped candidates. Across math, code, and puzzle benchmarks, Scorpio improves verification, tree search, and aggregation. For verification, Scorpio improves accuracy over the strongest baseline by up to 7 percentage points while matching or exceeding that baseline's peak accuracy with 17-42% lower inference cost. For tree search, using Scorpio to score partial solutions and select the final answer improves accuracy by up to 10 percentage points over prior methods. Finally, Scorpio improves the scaling of aggregation-based methods by 11 percentage points on coding tasks.
Figures & tables
Figure 1: Sampling more candidates improves the chance of a correct answer, but selecting it does not scale well with . (a) The gap between and exists even for frontier models on agentic tasks. 2 2 2 Accuracies from the Terminal-Bench 4 leaderboard . (b) As increases, the gap between and the selection accuracy of V 1 ( Singh et al., 2026 ) , a recent self-verification method for scaling test-time compute, widens. (c) A similar trend holds for Recursive Self-Aggregation (RSA) ( Venkatraman et al., 2025 ) . For (b) and (c), we run each method until accuracy plateaus; refer Sec. E.1 for further details.
Figure 2: Scorpio scales better with inference compute than prior verification methods across math (HMMT), puzzle (RG Games), and code benchmarks (LCBv6). We compare Scorpio ’s scoring with V1 ( Singh et al., 2026 ) , a recent test-time scaling algorithm that scores candidates through pairwise comparisons between similar candidates, and pointwise scoring, where LLMs evaluate candidates in isolation. Scorpio jointly scores candidates in groups of four. Inference compute is measured as the total dollar cost of all input and output tokens, and is scaled by repeating candidate evaluations. Results are averaged over 20 seeds, with shaded regions indicating confidence intervals.
Figure 3: Decomposition of ranking error into bias and variance. Results for GPT-OSS-20B on HMMT (math) and LCBv6 (code). As inference compute increases, variance generally decreases, while bias decreases before approaching a plateau (see Sec. C.3 for why bias can decrease with inference compute). Despite its simpler design, Scorpio reduces variance faster than pointwise scoring and V1’s ( Singh et al., 2026 ) pairwise scoring, while achieving similar or lower ranking error. Similar trends are observed for Qwen3.5-4B (see Fig. 17 ).
Figure 4: Larger groups reduce variance faster but can increase bias. Results for GPT-OSS-20B on HMMT (math) and LCBv6 (code). Increasing the group size allows more evaluations within the same cost budget, reducing variance faster. However, these gains can come at the expense of increased bias: on LCBv6, yields higher bias than smaller groups. Similar trends are observed for Qwen3.5-4B (see Fig. 18 ).
Figure 5: Residual bias persists at scale with Gemini-3.1-Pro on Terminal-Bench-2.1. We generate candidates and score them with Scorpio . Top-1 accuracy for Scorpio ( ) plateaus about percentage points below . Scorpio ’s bias plateaus with compute, while its variance continues to decrease. Results are averaged across seeds.
Figure 6: Test-time search using Scorpio matches or outperforms prior methods without any training. Results for GPT-OSS-20B on AIME 2024 and LCBv6. We compare Scorpio , which jointly scores partial solutions during search, with ThinkPRM-1.5B ( Khalifa et al., 2025 ) , which uses a trained process reward model, and DeepConf, which uses confidence scores derived from token probabilities. We vary the number of initial solutions: for Scorpio and ThinkPRM, and for DeepConf to keep total costs comparable. Top-1 accuracy is plotted against total inference cost. Similar trends are observed for Qwen3.5-4B (see Fig. 20 ).
Figure 7: Pruning candidates with Scorpio allows RSA to scale with pool size . Using GPT-OSS-20B on LCBv6, we plot (a) accuracy across aggregation rounds for different pool sizes . Increasing degrades RSA performance, whereas RSA + Scorpio improves with larger pools. One failure mode contributing to this degradation is shown in (b) : RSA amplifies rare incorrect responses over successive rounds on problem 120. LCBv6 requires a complete Python solution in a single turn and does not permit tool calls. However, RSA amplifies rare incorrect responses containing tool calls over successive rounds, with the effect becoming more severe as increases (see Fig. 22 ). After each round, Scorpio scores all candidates, and only the top are used to construct aggregation sets for the next round, limiting this bad-mode amplification. Results are averaged over 8 seeds, with CI.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Increasing compute does not change the mean of the score distribution, but can reduce the bias in rankings. At low compute (solid curves), overlapping score distributions allow the incorrect candidate to receive a higher score than the correct one. Averaging more evaluations (dotted curves) concentrates the scores around their respective means. Since the correct candidate has a higher mean score in this example, this makes an incorrect ordering less likely, reducing the pair’s squared bias .
Figure 9: Residual bias dominates ranking error as compute grows. Variance and squared bias of the ranking error of Scorpio ( ) versus the number of scoring rounds for Qwen3.5-4B (top) and GPT-OSS-20B (bottom). Black circles show the squared bias, which flattens as grows, and teal squares the variance; curves show least-squares fits of to the variance. The box gives the fitted , , and for the variance, and the share of the ranking error due to squared bias at . Using the asymptotic variance together with the squared bias at , we estimate that bias would account for to of the ranking error as grows.
Figure 10: A judge from a different model can reduce ranking bias. Candidates generated by Qwen3.5-4B are scored with Scorpio ( ) either by Qwen3.5-4B itself (the self judge) or by one of four other judges, on HMMT Feb 2025 (top) and LCBv6 (bottom). We plot bias 2 , variance, ranking error, and top-1 accuracy as the number of scoring rounds increases, averaged over repetitions; shaded regions show confidence intervals for top-1 accuracy. At , replacing the self judge with GPT-OSS-20B reduces the ranking error from to on HMMT and from to on LCBv6, and improves top-1 accuracy from to and from to , respectively.
Figure 11: Different models have different biases. For every pair of a correct and an incorrect candidate generated by Qwen3.5-4B, we record whether the self judge and another judge rank the correct candidate higher after scoring rounds of Scorpio ( ), on HMMT Feb 2025 (top) and LCBv6 (bottom). Fractions are averaged over questions and repetitions.
Figure 12: ( Singh et al., 2026 ) accuracy plateaus well below , and the gap widens as increases. Results for GPT-OSS-20B on LCBv6. (a) Increasing the candidate pool from to yields modest gains in selection accuracy, while the gap to grows from roughly to percentage points. Points show means over seeds, with error bars indicating confidence intervals. (b) Accuracy plateaus as inference compute increases for each pool size, remaining below the corresponding ceiling (dashed lines). Inference compute is measured as the total cost of the input and output tokens of each LLM call; see Sec. D.2 for further details.
Figure 13: RSA’s ( Venkatraman et al., 2025 ) final accuracy decreases as the candidate pool grows. Results for GPT-OSS-20B on LCBv6. (a) Increasing from to lowers final accuracy, widening the gap to from roughly to percentage points. Points show means over seeds, with error bars indicating confidence intervals. (b) Each curve tracks accuracy over successive aggregation rounds for a fixed . Larger pools plateau at lower accuracy despite higher compute costs. Inference compute is measured as the total cost of the input and output tokens of each LLM call; see Sec. D.2 for further details.
Figure 14: ( Singh et al., 2026 ) also exhibits a widening selection gap on HMMT Feb 2025. Results for GPT-OSS-20B. (a) Increasing from to yields modest gains in selection accuracy, while the gap to grows from roughly to percentage points. (b) Selection accuracy plateaus as inference compute increases, remaining below the corresponding ceiling (dashed lines). Inference compute is measured as the total cost of the input and output tokens of each LLM call; see Sec. D.2 for further details.
Figure 15: Scorpio achieves higher accuracy on math and similar accuracy at lower cost on code compared with PPT ( Kwok et al., 2026 ) . Selection accuracy versus total inference cost for GPT-OSS-20B with candidates. On HMMT Feb 2025, Scorpio ( ) improves peak accuracy by 8 percentage points over PPT. On LCBv6, it reaches PPT’s accuracy at 1.5 lower total cost. PPT’s strategy proceeds in two phases: an initial ring phase ensures that every candidate is evaluated, followed by a tournament phase. Points labeled “ring” show accuracy after the first phase, while and indicate whether one or two evaluations are used per pair. Curves show mean accuracy, with shaded regions indicating confidence intervals. Inference compute is measured as the total dollar cost of all input and output tokens across LLM calls; see Sec. D.2 for details.
Figure 16: Random pairing matches or outperforms tournament-based pairing. Results for GPT-OSS-20B on HMMT and LCBv6. Bias-variance decomposition of ranking error for ( Singh et al., 2026 ) and Scorpio ( ) using GPT-OSS-20B. On both datasets, variance decreases quicker with Scorpio as compute increases. attains lower bias on LCBv6, while Scorpio attains lower bias on HMMT. At higher budgets, Scorpio achieves similar or lower ranking error, demonstrating that a simpler pairing strategy can be equally or more effective than tournament based strategies. Inference compute is measured as the total dollar cost of all input and output tokens across LLM calls; see Sec. D.2 for details.
Figure 17: Decomposition of the ranking error as a bias term and a variance term. Results for Qwen3.5-4B on HMMT, Reasoning-Gym and LCBv6. While the variance term generally vanishes with increased compute, the bias term converges to a plateau (see: Sec. C.3 ). Scorpio ’s groupwise scoring leads to a higher variance reduction rate than pointwise or V1’s ( Singh et al., 2026 ) pairwise scoring. Compute is measured as the total dollar cost of all input and output tokens across LLM calls; see Sec. D.2 for further details.
Figure 18: Larger groups reduce variance faster but can increase bias. Results for Qwen3.5-4B on HMMT and LCBv6. Increasing group size allows more evaluations within the same cost budget, reducing variance faster. However, these gains can come at the expense of bias: on LCBv6, yields higher bias than smaller groups. Compute is measured as the total dollar cost of all input and output tokens across LLM calls; see Sec. D.2 for further details.
Figure 19: DeepConf’s ( Fu et al., 2025 ) pruning threshold depends on the model. Top-1 Accuracy versus total inference cost for DeepConf with aggressive ( ) and conservative ( ) pruning. Aggressive pruning generally works better for Qwen3.5-4B, while conservative pruning works better for GPT-OSS-20B. Compute is measured as the total dollar cost of all input and output tokens across LLM calls; see Sec. D.2 for further details.
Figure 20: Test-time search using Scorpio matches or outperforms prior methods. Results for Qwen3.5-4B on AIME 2024 and LCBv6. Top-1 accuracy of test-time search versus total inference cost on AIME 2024 and LCBv6 for Qwen3.5-4B. We vary the number of solutions we start with: for Scorpio and ThinkPRM, and for DeepConf to keep total costs comparable. Inference compute is measured as the total dollar cost of all input and output tokens across LLM calls; see Sec. D.2 for details.
Figure 21: Repeated evaluation of partial solutions improves search accuracy. Top-1 accuracy of test-time search using Scorpio versus total inference cost on AIME 2024, HMMT Feb 2025, and LCBv6. We fix the number of solutions we start with at and vary the number of times the random grouping and scoring process is repeated, . Repeated evaluations improve accuracy over across all three datasets. Inference compute is measured as the total dollar cost of all input and output tokens across LLM calls; see Sec. D.2 for details.
Figure 22: Larger pools amplify tool-call responses in RSA, while Scorpio limits their amplification. Proportion of responses containing tool calls across aggregation rounds for a single LCBv6 problem using GPT-OSS-20B. This is a single-turn task that requires a solution in a Python code block. Some responses attempt tool calls to test their logic, which we identify using the pattern to=[A-Za-z_] . The harness does not support tool calls, and of these responses receive zero reward. Tool-call responses remain uncommon with but become increasingly common with and as aggregation rounds progress. In RSA + Scorpio , candidates are scored using Scorpio after each round, and only the top half of candidates are used to build aggregation sets for the next round. This selection reduces the proportion of tool-call responses to nearly zero across all three pool sizes.
More in this topic
Language Model Self-Assessment
Language model self-assessment studies how models judge their own confidence, capabilities, errors, and behavior.
See all 96 papers