Reinforcement learning with verifiable rewards (RLVR) has substantially improved LLM reasoning, yet growing evidence suggests it mainly performs distribution sharpening: concentrating probability mass on reasoning paths the base model already produces. Recent work therefore seeks RLVR's gains without needing reward signals, by sampling from the power distribution πα∝πbaseα via Markov chain Monte Carlo or sequential Monte Carlo. We show that these samplers fail to converge to πα under realistic compute budgets, and that their gains are largely recreated via simple Best-of-N with appropriate grading . This suggests they succeed by selecting high-probability sequences, not by approximating πα, raising a natural question: is the power distribution itself a worthwhile target? We test this hypothesis directly by fine-tuning the base model to match πα, sidestepping sampler convergence issues entirely. Despite approximating the target far more faithfully than existing samplers, this strategy yields gains that are noisy and specific to certain models and sampling tasks. Together, our results suggest that the value of power sampling lies not in faithfully targeting πα, but in the simpler goal of extracting sequences the base model already deems likely.
Figures & tables
Figure 1: Base-model log-likelihoods of samples drawn from low-temperature decoding ( τ=1/α ), Power-SMC, and our Power-RLOO on Qwen2.5-7B-Instruct (MATH500, Question 1), for α∈{1.5,2.0,4.0} . Power-SMC’s samples collapse into a narrow, near-identical likelihood band that is largely insensitive to α , consistent with particle degeneracy. The narrow band indicates fewer than 10 effective samples. Power-RLOO, in contrast, shifts its mass toward higher likelihood as α grows, matching the behavior expected under πα .
Figure 2: Diagnostics for the MCMC sampler. As α increases, the chains progressively fail to explore: the trace plots (c–d) show increasing stagnation, and the acceptance behavior (a–b) reveals the sampler collapsing into a greedy hill-climber that accepts almost exclusively on the basis of higher likelihood.
Figure 3: Ancestral diversity and effective sample size over the SMC trajectory. Resampling occurs whenever diversity drops too low, which, when compounded over the sequence, leaves the final population with few distinct ancestors and limited diversity.
Model
Method
MATH500 (%)
GPQA (%)
Qwen2.5-7B-Base
Baseline decoding
39.0
28.6
Low-temperature decoding ( τ=1/α )
63.1
31.4
Power-SMC
70.6
31.9
Best-of- N -mean
67.2
32.2
Best-of- N -sum
62.6
32.2
Best-of- N w/ Resampling
70.0
29.7
Table 1: Pass@1 accuracy on MATH500 and GPQA at α=4 . Power-RLOO and Power-SMC yield only modest, model-dependent gains over simpler mode-seeking baselines such as Best-of-N. N/A denotes a lack of convergence due to an undesirable length degeneracy pathology arising from more precisely targeting the power distribution (see Section 5.2 )
Figure 4: Pass@ k accuracy for Power-RLOO and Power-SMC ( α=4 ) on Qwen2.5-7B-Instruct (MATH500). As expected, pass@ k decreases with α ; however, mild sharpening ( α=1.5 ) actually improves pass@ k , suggesting it elicits reasoning paths that exist in the base model but are rarely sampled at α=1
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Progression of KL divergence to the target distribution over the course of training (lower is better). By maintaining a valid probability distribution, we are able to compute estimates of KL to track training stability. We observe faster and more stable convergence when using low-temperature sampling.
Reinforcement learning (RL) post-training is a dominant approach for improving the reasoning performance of large language models (LLMs), yet growing evidence suggests that its gains arise primarily from distribution sharpening rather than the acquisition of new capabilities. Recent work has shown that sampling from the power distribution of LLMs using Markov chain Monte Carlo (MCMC) can recover performance comparable to RL post-training without relying on external rewards; however, the high computational cost of MCMC makes such approaches impractical for widespread adoption. In this work, we propose a theoretically grounded alternative that eliminates the need for iterative MCMC. We derive a token-level decomposition of the autoregressive power distribution, showing that the global power distribution can be approximated by a token-level scaled low-temperature one, where the scaling factor captures future trajectory quality. Leveraging this insight, we introduce a training-free and verifier-free algorithm that sharpens the base model's generative distribution autoregressively. Empirically, we evaluate our method on math, QA, and code tasks, and show that our method matches or surpasses one-shot GRPO without relying on any external rewards, while reducing inference latency by over 10× compared to MCMC-based sampling.
Xiaotong Ji, Rasul Tutunov, Matthieu Zimmer +1
Huawei Noah’s Ark Lab · UCL Centre for Artificial Intelligence
Power-sharpened sampling is an inference-time alternative to reinforcement-learning (RL) post-training for enhancing reasoning in large language models (LLMs). High-probability sequences are amplified under the base model without parameter updates or external rewards, avoiding the costly optimization and jagged generalization of RL. However, this approach faces a fundamental exploration--exploitation trade-off, as % strong sharpening restricts exploration, trapping samplers in plausible but incorrect reasoning trajectories, whereas weak sharpening leaves the answer distribution diffuse. To resolve this trade-off, we introduce Parallel Power Tempering (PPT), instantiating power-sharpened LLM sampling via parallel tempering. Running multiple interacting replicas in parallel at different sharpening levels allows lower-power replicas to explore diverse reasoning trajectories and higher-power chains to further exploit higher-likelihood responses favored by the sharpened target. Specifically, we tailor \method{} to inference-time sampling by mitigating a truncation bias, identified in prior power samplers, and investigate effective swap strategies under finite memory and compute budgets. Extensive experimentation shows that \method{} substantially improves single-chain power-sharpened sampling and outperforms RL-post-trained models, producing higher-quality reasoning traces and even achieving performance comparable to frontier models.
Panagiotis Theodoropoulos, Nan Jiang, Xintong Duan +4
Georgia Institute of Technology · Morgan Stanley · University of Texas at El Paso +1
Recent analyses question whether reinforcement learning (RL) is responsible for strong reasoning in large language models (LLMs). At the same time, distillation and inference-time sampling, including power sampling, have emerged as effective ways to improve LLM performance. However, the relationship among RL, distillation, and sampling remains unclear. In this study, we focus on the power distribution, the target distribution of power sampling, and show that the power distribution bridges sampling, self-reward KL-regularized RL, and self-distillation. From the sampling perspective, we show that inexpensive local approximations cannot reproduce sequence-level power without information about possible suffixes. From the RL perspective, the power distribution is the closed-form optimizer of KL-regularized RL when the model's sequence-level log-probabilities are used as the reward. This identification leads to power self-distillation, an offline distillation surrogate that shares the same target distribution and amortizes the cost of power sampling into supervised training on teacher samples. We further show that power self-distillation can achieve self-reward sharpening, while improvement in a downstream true reward is governed by the covariance between true reward and self-reward under the power distribution. Experiments on reasoning tasks support our analysis: power sampling raises self-reward, true-reward gains depend on alignment with self-reward, and power self-distillation can match or exceed the performance of power sampling at much lower inference cost.
Akiyoshi Tomihari, Issei Sato
Department of Computer Science, The University of Tokyo