Compute scaling for LLM reasoning trades off exploring solution approaches (breadth) against refining promising ones (depth), yet why a given trade-off works, and why it often fails to transfer across models, remains unclear. We argue that the optimal strategy depends on the model's diversity profile, the spread of probability mass across solution approaches, and that this must be characterized before any exploration strategy is adopted. We formalize this with a framework decomposing reasoning uncertainty, deriving when depth-based refinement outperforms parallel sampling, and validate it across three model families at both inference and training. Our central finding is that the diversity regime dictates the strategy: low-diversity aligned models benefit from depth-based refinement with lightweight intrinsic signals, whereas high-diversity base models are often harmed by it, and instead need breadth or stronger signals to compensate.
Figures & tables
Model
Baseline
ENT
SR
ENT Lift
SR Lift
Qwen-3 4B base
0.9004
0.6377
0.6406
-29.18%
-28.85%
Qwen-3 4B instruct
0.9121
0.9180
0.9170
0.65%
0.54%
Qwen-3 4B RLVR
0.6885
0.7471
0.7373
9.34%
7.90%
OLMo-3 7B base
0.8506
0.9072
0.8828
6.65%
3.79%
OLMo-3 7B instruct
0.6797
0.8154
0.8086
19.96%
18.96%
OLMo-3 7B RLVR
0.7201
0.8300
0.7813
15.26%
8.50%
Table 1: Pass@ 16 for Baseline , ENT and SR . The Lift showcases the gain over the Baseline .
Figure 1: Question-level Vendi diversity with cosine similarity. Base models produce the largest effective response diversity, with Nemotron Base showing the strongest contrast against its Instruct counterpart.
Figure 2: Correctness-conditioned Vendi score. Each bar splits responses within a question and model cell into correct and incorrect subsets, computes Vendi separately for each, and averages over questions.
Model
Baseline
ENT
Lift
Qwen-3 1.7B base ( 0.10 )
0.27
0.16
-40.7%
Qwen-3 1.7B instruct ( 0.37 )
0.61
0.59
-3.2%
Table 2: Best validation accuracy during training for the GSPO baseline and the MCTS-entropy (ENT) variant, on Qwen-3 1.7B base and post-trained checkpoints. Starting validation accuracy is shown in brackets. Lift is the relative gain over the corresponding baseline.
Symbol
Meaning
Prefix-selection setting
N,K,L
Total rollout budget, number of selected rollouts, and prefix length.
x1:L(i)
Length- L prefix of rollout i .
ℓt(i),mt(i)
Chosen-token log-probability and top-1/top-2 log-probability gap at position t .
q(i),q^(i)
Prefix score and its normalized version.
Ut,disagree(t)
Number of distinct tokens and disagreement score at position t .
Table 3: Notation
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Quality signal
Diversity signal
Uses top- k ?
Random@ K
none
none
No
MMR
prefix logprob q^(i)
Jaccard over prefix tokens
No
Adaptive Dispersion
prefix logprob q^(i)
token-disagreement anchors + hybrid distance
Yes
Appendix
Table 4: Taxonomy of prefix-selection methods used in the main text. “Top- k ” indicates whether the method requires per-position top- k logprob vectors (beyond chosen-token logprobs).
Figure 3: Assumed-cluster spread. Base cells form wider response clusters than Instruct cells, showing that instruction tuning compresses both mode diversity and geometric spread.
Figure 4: Correctness-conditioned clustering. Incorrect responses are typically more spatially dispersed than correct responses, especially for Base models.
Figure 5: Critic rewards offer a direct signal on quality of exploration; the plot verifies the expected trend on Qwen family i.e. the performance of Entropy-based MCTS closely matches standard sampling for the instruct model, while significantly degrades on the base model.
Figure 6: The RBF kernel compresses the Vendi scale but preserves the main ordering: Base cells remain the most diverse.
Figure 7: Under the RBF kernel, incorrect traces still account for the largest correctness-conditioned diversity gaps.
Figure 8: Qwen shows the clearest Base–Instruct geometric separation, OLMo the weakest, and Nemotron an intermediate but still visible shift.
Uncertainty estimation is critical for deploying reasoning language models, yet remains poorly understood under extended chain-of-thought reasoning. We study parallel sampling as a fully black-box approach using verbalized confidence and self-consistency. Across three reasoning models and 17 tasks spanning mathematics, STEM, and humanities, we characterize how these signals scale. Both self-consistency and verbalized confidence scale in reasoning models, but self-consistency exhibits lower initial discrimination and lags behind verbalized confidence under moderate sampling. Most uncertainty gains, however, arise from signal combination: with just two samples, a hybrid estimator improves AUROC by up to +12 on average and already outperforms either signal alone even when scaled to much larger budgets, after which returns diminish. These effects are domain-dependent: in mathematics, the native domain of RLVR-style post-training, reasoning models achieve higher uncertainty quality and exhibit both stronger complementarity and faster scaling than in STEM or humanities.
Maksym Del, Markus Kängsepp, Marharyta Domnich +4
Institute of Computer Science, University of Tartu
Test-time scaling improves LLM reasoning by using additional inference compute, but wider sampling alone can suffer from diminishing returns: new rollouts often repeat existing answer patterns instead of adding useful reasoning diversity. Verifier-based selection offers an alternative, but its performance depends on the calibration of an external reward model. We propose a verifier-free breadth--depth refinement framework that uses test-time compute to both explore and improve candidate solutions. The method samples multiple independent reasoning rollouts, refines each rollout through iterative self-critique and self-correction, and aggregates the refined answers by majority voting. Breadth preserves diverse initial attempts, while depth repairs local reasoning errors before aggregation. Across AIME24, AIME25, AMC, OlympiadBench, and MATH500, our method consistently improves over greedy decoding, majority voting, verifier-based best-of-N, beam search, and lookahead decoding across multiple open-weight models. For instance, with Qwen2.5-1.5B, accuracy increases from the strongest verifier-based baseline to 58.0% on MATH500, and from 25.0% to 32.5% on AMC. These results show that test-time compute can be more effective when used to refine sampled trajectories rather than only to sample more candidates or rely on verifier-guided selection.
Ahsan Bilal, Muhammad Ahmed Mohsin, Muhammad Umer +4
University of Oklahoma · Stanford University · Universitat Pompeu Fabra +1
Recent studies on reinforcement learning (RL) report seemingly conflicting evidence about large language model (LLM) reasoning. Training on mathematics can improve performance in other domains, yet gains in Pass@1 can coincide with lower Pass@N than the base model. This raises a fundamental question: does RL expand an LLM's reasoning boundary, or merely reweight its existing reasoning space? We revisit these phenomena across Qwen and Gemma model families, showing both cross-domain gains and forgetting, while coverage at large sampling budgets increases on some tasks and decreases on others. Detailed analysis of solution traces before and after RL indicates a shift in the reasoning strategies the model employs, motivating a two-stage autoregressive policy model that separates strategy selection from problem-specific execution. Within this framework, we prove how RL's implicit bias reshapes strategy preferences, allowing gains on some tasks while suppressing strategies required by others. This mechanism can also broaden or narrow coverage at a given sampling budget even without expanding strategy support. We further provide theoretical justifications for log-sigmoid and log-linear scaling laws in RL compute, and evaluate their predictive power. Together, these results connect changes in strategy selection to cross-domain transfer, reasoning coverage, and compute scaling.