We study how to improve LLM reasoning at inference time without updating model weights or using external reward models. Recent reward-free power-sampling methods define the sequence-level power target πα(y∣x)∝p0(y∣x)α, where α>1, which increases the relative probability of completions that are already likely under the base model. Existing methods use Metropolis--Hastings or Sequential Monte Carlo (SMC) to approximately sample from this target. However, two challenges arise at different levels. At the target level, full-sequence sharpening can concentrate probability on a few exact reasoning--answer trajectories, suppressing alternative paths that support the same answer. At the sampler level, finite-particle SMC can suffer from path collapse when resampling repeatedly duplicates a few high-weight prefixes. To address these challenges, we propose Decoupled Marginal Sharpening (DMS), which separately controls how strongly the model favors particular reasoning paths and how strongly it favors final answers after aggregating support across those paths. At the target level, DMS first samples from a sharpened reasoning distribution, then marginalizes the model's answer probabilities over the resulting weighted reasoning population and sharpens the induced answer marginal. Separate exponents allow mild reasoning sharpening to encourage exploration of diverse paths, followed by stronger answer-marginal sharpening to concentrate probability on answers with greater aggregate support across those paths. At the sampler level, we develop Fully Adapted Annealed SMC, which combines an annealed sequence of power targets with locally fully adapted proposals and predictive ancestor resampling. Experiments on MATH500, GPQA-Diamond, HumanEval, and AIME24/25 across three base models show that DMS achieves the highest mean pass@1 among the compared methods in all tasks.
Figures & tables
Figure 1: Accuracy and inference latency across three models and four reasoning benchmarks. Our method DMS achieves the highest accuracy in all settings while requiring less inference time than the strong baseline Marginal Sharpening. Vertical error bars indicate ±1 standard deviation in pass@1. In each subplot, higher accuracy and lower latency are better (toward the upper left).
Figure 2: Overview of DMS. FAA-SMC constructs K weighted reasoning paths that define an empirical answer marginal. We sample M candidates from this marginal and select the final answer according to their normalized answer-sharpening weights.
Model
Method
MATH500
GPQA-Diamond
HumanEval
AIME24/25
Qwen3-4B
Standard Decoding
76.00±0.53
50.00±1.75
84.96±0.35
59.44±3.47
Low-Temperature Decoding
74.07±0.81
50.84±2.59
84.55±1.27
53.33±7.64
Power-SMC
77.67±0.95
49.83±2.92
84.15±1.61
48.89±6.31
Marginal Sharpening
75.67±0.23
54.21±1.77
90.04±0.93
71.67±1.67
DMS (Ours)
85.67±0.58
60.77±0.77
95.53±0.70
75.00±1.67
Qwen3-8B
Standard Decoding
71.73±0.58
50.67±3.36
83.74±0.35
55.56±2.55
Table 1: Pass@1 (%) across models and benchmarks. Shaded rows denote DMS, and boldface indicates the highest mean within each model–benchmark setting. The DMS results on AIME24/25 use the Island-DMS variant.
Variant
αZ
αA
Expected
Pass@1
Oracle pass@32
Structured decoding only
1
1
66.48
67.78
82.78
Answer sharpening only
1
4
76.89
77.22
82.78
Reasoning sharpening only
1.005
1
71.77
72.78
78.33
Standard DMS
1.005
4
74.55
74.44
78.33
Island-DMS
1.005
4
76.83
77.22
81.67
Table 2: Component analysis on AIME24/25 using Qwen3-8B. Expected accuracy is the probability of selecting a correct answer under the normalized answer weights, averaged over problems. Pass@1 evaluates one sampled answer per problem, while oracle pass@32 is the fraction of problems with at least one correct answer among the 32 candidates. All accuracy metrics are percentages.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Model
MATH500
GPQA-Diamond
HumanEval
AIME24/25
Qwen3-4B
(1.25,3)
(1.025,2)
(1.25,1.5)
(1.025,3.5)
Qwen3-8B
(1.025,2.5)
(1.025,2)
(1.25,1.5)
(1.005,4)
DeepSeek-R1-Distill-Llama-8B
(1.001,6)
(1.001,5)
(1.25,1.5)
(1.001,8)
Appendix
Table 3: DMS reasoning and answer exponents (αZ,αA) used in the main results ( Table 1 ).
Figure 3: Expected accuracy (%) of standard DMS with Qwen3-8B on AIME24/25 across reasoning and answer exponents, averaged over three seeds.
Variant
Expected
Oracle pass@32
Final ESS /K
Resamples
Unique roots
standard DMS
74.55
78.33
0.765
1.39
8.17
No annealing
71.72
77.78
0.773
2.26
7.11
No resampling
72.09
80.00
0.252
0
16.00
Appendix
Table 4: Sampler ablations for non-island DMS with Qwen3-8B on AIME24/25, averaged over three seeds. Expected accuracy and oracle pass@32 are percentages. Final ESS is computed from the final reasoning weights and divided by K=16 . Resamples and unique roots report the mean number of reasoning-stage resampling events and surviving initial-particle ancestors, respectively.
αZ
Expected accuracy ( αA=4 )
Oracle pass@32
Fine ladder
1
95.06
98.17
1.025
96.17
97.97
1.05
96.59
97.97
1.1
96.39
97.97
Extended ladder
Appendix
Table 5: Reasoning-exponent sensitivity with a higher-budget replica-exchange sampler using Qwen3-4B on HumanEval. Expected accuracy and oracle pass@32 are reported as percentages averaged over three seeds. The fine and extended ladders are run independently.
Model
Method
MATH500
GPQA-Diamond
HumanEval
AIME24/25
Qwen3-4B
Standard Decoding
50.57
124.41
43.06
249.27
Low-Temperature Decoding
49.36
123.26
43.09
259.47
Power-SMC
46.01
163.14
43.33
376.40
Marginal Sharpening
140.51
710.42
103.05
1957.74
DMS (Ours)
62.09
241.09
83.49
617.09
Qwen3-8B
Standard Decoding
57.09
133.99
43.92
261.37
Appendix
Table 6: Mean wall-clock inference time in seconds per problem, averaged over seeds.
Power-sharpened sampling is an inference-time alternative to reinforcement-learning (RL) post-training for enhancing reasoning in large language models (LLMs). High-probability sequences are amplified under the base model without parameter updates or external rewards, avoiding the costly optimization and jagged generalization of RL. However, this approach faces a fundamental exploration--exploitation trade-off, as % strong sharpening restricts exploration, trapping samplers in plausible but incorrect reasoning trajectories, whereas weak sharpening leaves the answer distribution diffuse. To resolve this trade-off, we introduce Parallel Power Tempering (PPT), instantiating power-sharpened LLM sampling via parallel tempering. Running multiple interacting replicas in parallel at different sharpening levels allows lower-power replicas to explore diverse reasoning trajectories and higher-power chains to further exploit higher-likelihood responses favored by the sharpened target. Specifically, we tailor \method{} to inference-time sampling by mitigating a truncation bias, identified in prior power samplers, and investigate effective swap strategies under finite memory and compute budgets. Extensive experimentation shows that \method{} substantially improves single-chain power-sharpened sampling and outperforms RL-post-trained models, producing higher-quality reasoning traces and even achieving performance comparable to frontier models.
Panagiotis Theodoropoulos, Nan Jiang, Xintong Duan +4
Georgia Institute of Technology · Morgan Stanley · University of Texas at El Paso +1
Reinforcement learning (RL) post-training is a dominant approach for improving the reasoning performance of large language models (LLMs), yet growing evidence suggests that its gains arise primarily from distribution sharpening rather than the acquisition of new capabilities. Recent work has shown that sampling from the power distribution of LLMs using Markov chain Monte Carlo (MCMC) can recover performance comparable to RL post-training without relying on external rewards; however, the high computational cost of MCMC makes such approaches impractical for widespread adoption. In this work, we propose a theoretically grounded alternative that eliminates the need for iterative MCMC. We derive a token-level decomposition of the autoregressive power distribution, showing that the global power distribution can be approximated by a token-level scaled low-temperature one, where the scaling factor captures future trajectory quality. Leveraging this insight, we introduce a training-free and verifier-free algorithm that sharpens the base model's generative distribution autoregressively. Empirically, we evaluate our method on math, QA, and code tasks, and show that our method matches or surpasses one-shot GRPO without relying on any external rewards, while reducing inference latency by over 10× compared to MCMC-based sampling.
Xiaotong Ji, Rasul Tutunov, Matthieu Zimmer +1
Huawei Noah’s Ark Lab · UCL Centre for Artificial Intelligence
Inference-time sampling can elicit strong reasoning abilities from language models without additional training. Existing power-sampling methods do so by sharpening the distribution over full generated outputs, favoring completions that are individually likely under the model. We argue that this is the wrong object to target for reasoning: a completion entangles a reasoning trace with a final answer, whereas what matters is whether an answer is supported by many plausible reasoning paths. We therefore shift the target from the full-output distribution to the sharpened answer marginal, making self-consistency an inference-time objective rather than a post-hoc voting criterion. Surprisingly, this marginal target admits an efficient approximation: we propose a simple, purely autoregressive parallel sampling algorithm that approximately samples from the sharpened answer marginal, eliciting stronger performance than standard power sampling on mathematics and coding benchmarks while being orders of magnitude faster.
Aleksei Arzhantsev, Otmane Sakhi, Nicolas Chopin
Criteo AI Lab, CREST IP Paris · Criteo AI Lab · ENSAE, Institut Polytechnique de Paris