Reinforcement learning with verifiable rewards (RLVR) has substantially improved LLM reasoning, yet growing evidence suggests it mainly performs distribution sharpening: concentrating probability mass on reasoning paths the base model already produces. Recent work therefore seeks RLVR's gains without needing reward signals, by sampling from the power distribution πα∝πbaseα via Markov chain Monte Carlo or sequential Monte Carlo. We show that these samplers fail to converge to πα under realistic compute budgets, and that their gains are largely recreated via simple Best-of-N with appropriate grading . This suggests they succeed by selecting high-probability sequences, not by approximating πα, raising a natural question: is the power distribution itself a worthwhile target? We test this hypothesis directly by fine-tuning the base model to match πα, sidestepping sampler convergence issues entirely. Despite approximating the target far more faithfully than existing samplers, this strategy yields gains that are noisy and specific to certain models and sampling tasks. Together, our results suggest that the value of power sampling lies not in faithfully targeting πα, but in the simpler goal of extracting sequences the base model already deems likely.
Figures & tables
Figure 1: Base-model log-likelihoods of samples drawn from low-temperature decoding ( τ=1/α ), Power-SMC, and our Power-RLOO on Qwen2.5-7B-Instruct (MATH500, Question 1), for α∈{1.5,2.0,4.0} . Power-SMC’s samples collapse into a narrow, near-identical likelihood band that is largely insensitive to α , consistent with particle degeneracy. The narrow band indicates fewer than 10 effective samples. Power-RLOO, in contrast, shifts its mass toward higher likelihood as α grows, matching the behavior expected under πα .
Figure 2: Diagnostics for the MCMC sampler. As α increases, the chains progressively fail to explore: the trace plots (c–d) show increasing stagnation, and the acceptance behavior (a–b) reveals the sampler collapsing into a greedy hill-climber that accepts almost exclusively on the basis of higher likelihood.
Figure 3: Ancestral diversity and effective sample size over the SMC trajectory. Resampling occurs whenever diversity drops too low, which, when compounded over the sequence, leaves the final population with few distinct ancestors and limited diversity.
Model
Method
MATH500 (%)
GPQA (%)
Qwen2.5-7B-Base
Baseline decoding
39.0
28.6
Low-temperature decoding ( τ=1/α )
63.1
31.4
Power-SMC
70.6
31.9
Best-of- N -mean
67.2
32.2
Best-of- N -sum
62.6
32.2
Best-of- N w/ Resampling
70.0
29.7
Table 1: Pass@1 accuracy on MATH500 and GPQA at α=4 . Power-RLOO and Power-SMC yield only modest, model-dependent gains over simpler mode-seeking baselines such as Best-of-N. N/A denotes a lack of convergence due to an undesirable length degeneracy pathology arising from more precisely targeting the power distribution (see Section 5.2 )
Figure 4: Pass@ k accuracy for Power-RLOO and Power-SMC ( α=4 ) on Qwen2.5-7B-Instruct (MATH500). As expected, pass@ k decreases with α ; however, mild sharpening ( α=1.5 ) actually improves pass@ k , suggesting it elicits reasoning paths that exist in the base model but are rarely sampled at α=1
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Progression of KL divergence to the target distribution over the course of training (lower is better). By maintaining a valid probability distribution, we are able to compute estimates of KL to track training stability. We observe faster and more stable convergence when using low-temperature sampling.