This work presents self-rewarding sequential Monte Carlo (SMC), an inference-time scaling algorithm enabling effective sampling of masked diffusion language models (MDLMs). Our algorithm stems from the observation that most existing MDLMs rely on a confidence-based sampling strategy, where only tokens with the highest prediction confidence are preserved at each step. This restricts the generation to a noise-sensitive, greedy decoding paradigm, resulting in an inevitable collapse in the diversity of possible paths. We address this problem by launching multiple interacting diffusion processes in parallel, referred to as particles, for trajectory exploration. Importantly, we introduce the trajectory-level confidence as a self-rewarding signal for assigning particle importance weights. During sampling, particles are iteratively weighted and resampled to systematically steer generation towards globally confident, high-quality samples. Our self-rewarding SMC is verified on various masked diffusion language models and benchmarks, achieving significant improvement without extra training or reward guidance, while effectively converting parallel inference capacity into improved sampling quality. Our code is available at https://github.com/Algolzw/self-rewarding-smc.
Figures & tables
Figure 1: Illustrative example of text generation using (a) masked diffusion models and (b) our self-rewarding SMC framework. Here, ‘M’ represents [MASK] tokens and ‘ Resa. ’ denotes resampling. Our method maintains multiple diffusion processes, called particles , to explore trajectories in parallel.
Benchmark
Length
LLaDA-1.5
w/ SR-SMC
Dream-7B
w/ SR-SMC
GSM8K
256
79.8
80.7 (+0.9)
76.2
78.0 (+1.8)
(5-shot)
512
80.4
82.0 (+1.6)
78.0
78.0 (+0.0)
MATH
256
38.2
41.8 (+3.6)
41.6
45.2 (+3.6)
(4-shot)
512
41.4
45.4 (+4.0)
44.6
47.6 (+3.0)
HumanEval
256
38.7
41.5 (+2.8)
47.9
53.7 (+5.8)
(0-shot)
512
35.4
41.5 (+6.1)
45.1
53.7 (+8.6)
Table 1: Performance of Self-Rewarding SMC (SR-SMC) on Diffusion Large Language Models. Results compare baselines versus SR-SMC variants using block decoding with block size = 32 with KV cache and parallel decoding enabled [ 50 ] .
Model
L=1024
L=2048
AR
14.1
13.2
Diffusion
SEDD
52.0
–
MDLM
46.8
41.3
MDLM w/ SR-SMC
25.8
25.9
Block Diffusion (with block size L′ )
Table 2: Generative perplexity (Gen. PPL; ↓ ) comparison of 300 samples with sequence lengths L=1024,2048 . We do not perform hyperparameter tuning.
Figure 2: A qualitative comparison of reasoning trajectories. Greedy decoding leads to a hallucinated identity (3 Blinkets = 7 Blinkets) that persists through the chain. SR-SMC utilizes trajectory-level confidence to explore multiple trajectories and successfully recovers the correct multi-step conversion.
Table 5
Benchmark
# Overtake
Percentage
GSM8K
160 / 640
25.0%
MBPP
154 / 640
24.1%
MATH
197 / 640
30.8%
HumanEval
196 / 640
30.6%
Average
-
27.6%
Table 5: Analysis of particle “Overtake”.
Figure 3: Comparison results of LLaDA 1.5 and Dream-7B using different numbers of particles on four tasks. Each marker denotes the empirical result while the dashed curves indicate first-order polynomial fits used solely to illustrate the overall trends as N increases.
Figure 4: The accuracy of LLaDA-1.5 and Dream-7B on MBPP and MATH datasets as the temperature varies uniformly from 0.0 to 1.0. The blue circles represent the baseline with standard parallel decoding, while the red stars denote the results using our SR-SMC with N=4 particles. Our method consistently demonstrates better robustness across benchmarks and the entire temperature range.
Table 9
Figure 5: Confidence–correctness analysis on LLaDA-1.5 with N=4 . Rank 1 denotes the highest-confidence particle, and dashed lines denote random selection.
Model
Benchmark
Baseline
w/ SR-SMC
LLaDA-1.5
GSM8K (0-shot)
75.4
76.3
MATH (0-shot)
35.2
36.6
Dream-7B
GSM8K (0-shot)
77.9
81.1
MATH (0-shot)
44.2
49.2
Table 8: Zero-shot performance of LLaDA-1.5 and Dream-7B variants on GSM8K and MATH benchmarks with length L=256 .
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Generative perplexity ( ↓ ) results on 300 samples of lengths L=1024,2048 using our self-rewarding SMC and the corresponding baselines.
L=1024
L=2048
Model
Gen. PPL
Entropy
Gen. PPL
Entropy
AR
14.1
5.29
13.2
5.54
MDLM
46.8
5.33
41.3
5.64
MDLM w/ Best-of-N
38.6
5.31
40.3
5.61
MDLM w/ ReMDM
35.5
5.35
45.1
5.27
MDLM w/ SR-SMC
25.8
5.15
25.9
5.41
Appendix
Table 9: Generative perplexity (Gen. PPL; ↓ ) and entropy results of 300 samples of lengths L=1024,2048 . All models are trained on OWT dataset. For reference, the Gen. PPL and entropy of the original data are 14.8 and 5.44, respectively, as reported in Wang et al. [48] .
Benchmark
Method
τ=0.0
τ=0.1
τ=0.2
τ=0.3
τ=0.4
τ=0.5
τ=0.6
τ=0.7
τ=0.8
τ=0.9
τ=1.0
MBPP
LLaDA-1.5
42.6
42.2
42.8
42.8
41.0
42.0
40.2
42.6
40.6
41.6
40.4
LLaDA-1.5 w/ SR-SMC
-
43.2
43.0
42.6
43.4
45.2
43.2
44.0
43.6
43.4
44.2
Dream-7B
48.2
1.80
5.60
14.2
28.0
34.0
39.0
41.2
41.4
41.8
42.0
Dream-7B w/ SR-SMC
-
43.4
46.8
47.4
48.4
48.6
47.4
48.4
48.6
48.0
48.6
MATH
LLaDA-1.5
39.8
39.8
38.0
37.6
40.0
38.4
38.8
37.2
38.8
38.2
38.2
LLaDA-1.5 w/ SR-SMC
-
39.2
41.0
42.0
40.0
39.4
39.2
41.4
39.6
40.6
41.8
Appendix
Table 10: Effect of Gumbel noise temperature τ on MBPP and MATH.
Method
GSM8K
MATH
HumanEval
MBPP
Average
Dream-7B (baseline)
77.3
42.2
50.3
43.2
53.2
Best-of-N
77.4
44.6
54.3
45.8
55.5
MCMC
76.9
44.2
54.3
40.2
53.9
Majority vote
81.9
37.6
50.0
47.4
54.2
SR-SMC
78.9
45.4
60.7
45.2
57.5
Appendix
Table 11: Compute-matched comparison on diffusion large language models Dream-7B.
Method
GSM8K
MATH
HumanEval
MBPP
Average
CFG (1.5)
75.4
40.4
50.3
43.0
52.3
SR-SMC (2 particles)
76.3
43.6
54.3
41.0
53.8
Appendix
Table 12: Classifier-free guidance on Dream-7B.
GSM8K
HumanEval
Method
Length
Latency (s) ↓
Tok/s ↑
Acc. ↑
Latency (s) ↓
Tok/s ↑
Acc. ↑
Baseline
256
1.96
107.81
79.5
2.81
31.33
37.7
Best-of- N
256
6.18
34.46
81.8
5.32
15.52
40.2
SR-SMC
256
6.31
32.84
82.2
5.45
14.46
41.7
Baseline
512
2.97
69.31
80.6
4.42
20.32
33.8
Best-of- N
512
10.50
18.01
80.7
13.08
10.83
39.0
Appendix
Table 13: Quality-efficiency comparison of LLaDA-1.5. Best-of- N and SR-SMC use the same four-particle budget. Latency is measured under identical hardware and decoding settings.
Benchmark
Length
LLaDA-1.5
w/ SR-SMC
Dream-7B
w/ SR-SMC
GSM8K
256
75.4
76.3
77.9
81.1
(0-shot)
512
80.4
82.0
78.0
78.0
MATH
256
35.2
36.6
44.2
49.2
(0-shot)
512
39.0
44.8
48.8
51.6
Appendix
Table 14: Zero-shot performance of LLaDA-1.5 and Dream-7B variants on GSM8K and MATH benchmarks across different generation lengths.
Method
GSM8K
MATH
HumanEval
MBPP
Average
KLASS
77.48
34.2
40.24
47.47
49.85
KLASS + SR-SMC
78.92
35.0
42.68
47.47
51.02
Appendix
Table 15: Applying SR-SMC to KLASS framework.
Task
GSM8K
MATH
HumanEval
MBPP
ESS value
2.79
3.52
2.65
3.05
Appendix
Table 16: The final ESS values using SR-SMC (LLaDA-1.5 with 4 particles) across tasks.
Figure 7: Additional qualitative comparisons of reasoning trajectories. Greedy decoding can commit to locally confident but incorrect intermediate steps, while SR-SMC maintains multiple candidate trajectories and recovers globally coherent solutions.
Inference-time sequential Monte Carlo (SMC) decoding can improve the reasoning accuracy of masked diffusion language models (diffusion LLMs) without post-training by resampling a population of partial generations according to a reward. Its effectiveness therefore depends on the resampling step, which must concentrate computation on promising candidates while retaining enough diversity to explore alternative reasoning paths. The reward can be internal, the decoder's own confidence on the committed prefix, or external, a process reward model scoring that prefix. Under either reward, multinomial resampling can disrupt this balance by duplicating high-weight particles and discarding lineages that may still reach a correct solution. The effective sample size (ESS) does not register this loss: multinomial resampling resets it to its maximum even as distinct lineages disappear. To retain such lineages, we introduce lineage-preserving SMC (LP-SMC), which combines Chopthin resampling with an input weight floor. Instead of equalizing weights, Chopthin bounds the ratio between the largest and smallest weights it returns and carries the unequal weights forward. The floor limits how far below the maximum a weight may fall before resampling so that lineages with low early weight keep a chance to recover. Across mathematical-reasoning and code-generation benchmarks on two base models, LP-SMC raises oracle coverage, the fraction of problems with at least one correct particle, in every setting, by 1.8 to 25.6 points, and raises end-to-end accuracy under a common reward-model selector in eleven of twelve settings, by up to 7.9 points on LLaDA-1.5 and 18.3 on Dream-7B. The coverage gain persists under the external reward model, which suggests that diversity-preserving resampling and the guidance signal contribute separately to inference-time search.
Minoo Ahmadi, Seyedarmin Azizi, Mehdi Kamal +1
University of Southern California, Los Angeles, USA
Masked Diffusion Models (MDMs) enable flexible decoding orders, yet existing samplers remain largely greedy, selecting locally certain tokens without accounting for their downstream effects. We show that this myopia can increase cumulative uncertainty and lead to suboptimal generation. To address this, we propose the Info-Gain Sampler, a training-free decoding method that uses the bidirectional structure of MDMs to balance immediate uncertainty with the information gained over remaining masked positions. Across reasoning, coding, creative writing, and image generation tasks, Info-Gain Sampler consistently outperforms existing MDM samplers, improving average reasoning accuracy by 2.9--11.6 percentage points and achieving a 62.8% average win rate in creative writing. The code is available at https://github.com/yks23/Information-Gain-Sampler.
Kaisen Yang, Jayden Teoh, Kaicheng Yang +2
Department of Computer Science and Technology, Tsinghua University · Massachusetts Institute of Technology · Shanghai Jiao Tong University +2
Masked diffusion language models (MDLMs) re-predict every position at each denoising step, but standard samplers commit tokens once revealed, leaving this revision capability unused. Existing approaches either add heuristic or learned mechanisms to revise committed tokens, or remask them back to [MASK] before re-predicting; a principled sampler that directly revises visible tokens without auxiliary modules remains underexplored. We introduce D3IM, a parameter-free sampler derived as a corrector-style reverse update that permits direct visible-to-visible revision without additional modules or auxiliary passes. D3IM also reveals a model-side obstacle we term preservation bias: the model tends to reproduce its own wrong committed tokens rather than correct them. We address this with SCOPE (Self-Conditioned On Prediction Errors), a lightweight post-training procedure that simulates D3IM's sampling process. On LLaDA-8B at 64 denoising steps, SCOPE+D3IM improves over the original LLaDA-8B with standard unmasking by +13.0 on GSM8K (68.3%), +4.8 on MATH-500 (23.6%), +15.3 on HumanEval (29.3%), and +10.4 on MBPP (30.8%), with gains that increase as more denoising steps are used on math and HumanEval.