Policy gradient methods are widely used to align large language model outputs with human preferences, but the log-likelihoods they require cannot be directly computed in Masked Diffusion Language Models (MDLMs). MDLM post-training therefore relies on a surrogate objective based on the Evidence Lower Bound (ELBO), and recent work estimates a blockwise variant of this objective via Monte Carlo methods to support blockwise generation. However, existing estimation schemes randomly sample the block index, so they may evaluate the same block more than once and leave some blocks unevaluated. This introduces variance arising from block selection into the estimates. In this work, we propose a full-block estimator that evaluates every block exactly once and averages the results. We theoretically show that the proposed estimator remains unbiased while eliminating the between-group variance arising from block selection, thereby achieving lower variance than the baseline for the same sample size. In experiments on five benchmarks spanning mathematics, planning, and coding, our method achieves the highest accuracy on four benchmarks and outperforms the random block sampling baseline by an average of 12.79 points.
Figures & tables
Figure 1: Comparison of block selection in Monte Carlo estimation of the blockwise surrogate objective. For each Monte Carlo sample index k , a block b is sampled, and the corresponding set of token positions is denoted by Bb . For each block b , the masking ratio tb and the masked positions Mtb are sampled to compute the blockwise surrogate objective in ( 1 ). Top: In our method (full-block evaluation), each sample k corresponds to a distinct block, so that every block is evaluated exactly once. Bottom: In the existing method (random block sampling), each sample independently selects a block, which may cause the same block to be evaluated multiple times ( B2 is selected for both k=1,2 ) or some blocks to remain unevaluated ( B1 ).
Figure 2: Mask state during the generation of block b=1 for a response o of length L=12 ( C=3 , B=4 ). Block B0 is considered already generated, so its original tokens are retained. In this example, positions 5 and 7 within block B1 are masked ( Mt1={5,7} ), and all positions in block B2 are masked. The full set of masked positions is S1(Mt1)={5,7,8,9,10,11} .
Figure 3: Instability of random block sampling with a small number of blocks. (a) Some blocks receive zero selections after K=4 independent draws. (b) The miss probability increases rapidly with C (see Appendix C.7 for a formal derivation).
Benchmark
Method
Blockwise
GSM8K
MATH-500
Countdown
Sudoku
HumanEval
Avg.
LLaDA
×
76.27
31.80
19.14
26.56
27.44
36.24
LLaDA-1.5
×
78.70
33.00
18.36
23.64
32.93
37.33
D1
×
81.43 †
36.00
36.72
13.21
32.93
40.06
WD1
×
81.20
34.60
44.14
62.25
28.05
50.05
SPG
✓
80.29
36.20 †
70.31 ↑ 3.90
72.73 †
29.88
57.88 †
Table 1: Performance comparison across benchmarks. Higher scores indicate better performance. LLaDA denotes the base model (LLaDA-8B-Instruct) before fine-tuning. The Blockwise column indicates methods that use the blockwise surrogate objective ( 3 ). Bold denotes the highest accuracy in each column, and † denotes the second highest. ▲ indicates the improvement of L^Full over L^Rand , and ↑ indicates the improvement of the highest accuracy over the † value.
Figure 4: Comparison of estimation quality for LBlock in ( 3 ) between L^Full in ( 6 ) and L^Rand in ( 5 ) on GSM8K ( C=4 ). L^Rand is computed with varying sample size K . The estimate at K=1024 is used as the high-precision reference estimate Lref for computing Spearman ρ and RMSE. Higher ρ and lower RMSE indicate better estimation quality. The star indicates the result of L^Full at K=C=4 , which achieves estimation quality roughly comparable to L^Rand at K≈32 .
GSM8K
MATH
Countdown
Sudoku
Code
0.122
0.119
0.095
0.076
0.107
Table 2: Variance ratio Var[L^Full]/Var[L^Rand] in ( 12 ), evaluated numerically for each sample and averaged within each dataset ( K=C=4 ). Smaller values indicate a greater variance reduction effect of L^Full .
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Value
Optimization
Optimizer
AdamW
Learning rate
1.5×10−6
LR scheduler
constant_with_warmup
Weight decay
0.1
Max gradient norm
0.2
Appendix
Table 3: Common training hyperparameters.
Parameter
Default
Code
Temperature
0.9 *
0.9
Block length
64
128
Diffusion steps
128
256
Generation length
256
512
Remasking strategy
low_confidence
Appendix
Table 4: Diffusion generation settings during training. Default applies to GSM8K, MATH, Countdown and Sudoku. * Sudoku uses 0.3.
Table 5: Datasets used for training and evaluation.
Parameter
Value
Temperature
0.0
Remasking strategy
low_confidence
Generation length ( L )
256, 512
Diffusion steps
L/2
Appendix
Table 6: Inference hyperparameters.
Benchmark
Samples
Metric
GSM8K
1,319
Accuracy
MATH-500
500
Accuracy
Countdown
256
Accuracy
Sudoku
256
Accuracy
HumanEval
164
Pass@1
Appendix
Table 7: Evaluation benchmarks.
Setting
Variance ratio vs L^Rand
General ( K=qC+r )
1−λ+λ⋅K(C−1)r(C−r)
K=1
1
K<C ( q=0 , r=K )
1−λ⋅C−1K−1
K=mC ( m≥1 , r=0 )
1−λ
Appendix
Table 8: Variance ratio of L^Full(K) relative to L^Rand . Here K=qC+r with q=⌊K/C⌋ and r=KmodC∈{0,…,C−1} , and λ is defined in ( 19 ). The case K=C corresponds to K=mC with m=1 .
Method
GSM8K
MATH-500
Countdown
Sudoku
HumanEval
Avg.
SPG
81.88
35.20
64.84
89.76
30.49
60.43
L^Rand
81.58
33.40
54.30
80.96
31.10
56.27
L^Full (ours)
82.26
38.00
66.41
91.39
36.59
62.93
Appendix
Table 9: Empirical verification for K<C . Baselines (SPG and L^Rand ) are re-trained with K=2 , C=4 , I=8,000 , and our method L^Full uses K=C=4 , I=4,000 (reused from Table 1 ). Both settings have the same total number of block evaluations K⋅I=16,000 .
K=4
K=6
K=8
Method
GSM8K
MATH-500
GSM8K
MATH-500
GSM8K
MATH-500
L^Rand
76.20
34.40
80.89
34.00
80.97
34.00
L^Full(K) (ours)
82.26
38.00
80.89
36.20
81.43
34.20
Appendix
Table 10: Empirical verification for K>C ( C=4 fixed, I=4,000 ). The K=4 column is reused from Table 1 . The K=6 and K=8 columns are new experiments.
C=K=4
C=K=8
Method
GSM8K
MATH-500
GSM8K
MATH-500
L^Rand
76.20
34.40
80.52
31.80
L^Full (ours)
82.26
38.00
81.35
36.80
Appendix
Table 11: Empirical verification of doubling C . Comparison between the main experiment ( C=K=4 ) and the re-trained models with C=K=8 . The C=K=4 columns are reused from Table 1 . All other hyperparameters are identical to those of the main experiment (LLaDA-8B-Instruct, I=4,000 ). Code, Countdown, and Sudoku are excluded due to limited computation time and resources. Bold indicates the higher value within each C=K setting.
Mean
Std
Median
GSM8K
L^Rand
109.32
85.14
90.81
L^Full
36.25
31.82
28.85
MATH
L^Rand
118.14
90.89
99.45
L^Full
36.64
33.32
29.18
Countdown
L^Rand
131.66
98.14
113.46
L^Full
37.82
32.09
30.90
Appendix
Table 12: Statistics of the estimation error ∣L^−Lref∣ ( K=4 , 50 independent repetitions). The reference value is computed with K=1024 .
Full < Rand (%)
Var. ratio
GSM8K
99.5
7.35×
MATH
98.9
7.47×
Countdown
99.7
9.71×
Sudoku
100.0
11.28×
Code
99.6
7.36×
Appendix
Table 13: Comparison of per-sample variances of the absolute error ∣L^−Lref∣ ( K=4 , computed from 50 independent repetitions). Full < Rand denotes the proportion of samples for which L^Full has lower variance than L^Rand . Var. ratio denotes the ratio of the mean variances ( L^Rand/L^Full ). Wilcoxon signed-rank test p<10−100 for all datasets.
Figure 5: Detailed estimation quality analysis across five datasets. Rows correspond to datasets (GSM8K, MATH-500, Countdown, Sudoku, and Code); columns correspond to the five analysis types described in the introduction of this section.
Training step
Task
1,000
2,000
3,000
4,000
GSM8K
0.811
0.915
0.889
0.961
MATH-500
0.791
0.926
0.825
0.915
Appendix
Table 14: Sample variance ratio of the policy gradient Var[G^Full]/Var[G^Rand] . Computed for the four training checkpoints and the two tasks of the main experiment ( C=K=4 ), aggregating over the remaining prompts after excluding those with Aj=0∀j from the N=100 samples. Each prompt contributes S=50 gradient samples. A ratio below 1 indicates that the variance of Full is smaller than that of Rand.
Training step
Task
1,000
2,000
3,000
4,000
GSM8K
0.115
0.113
0.138
0.134
MATH-500
0.095
0.155
0.104
0.132
Appendix
Table 15: Estimator variance ratio 1−λ=σˉwithin2/(σbetween2+σˉwithin2) ( K=C=4 ) at the mid-training checkpoints of the main experiment. Computed from the variance decomposition ( 12 ) of the per-block log-likelihood ℓb,tb,Mtb and aggregated over the prompts excluding those with Aj=0∀j . Directly comparable to Table 2 on the base model. Smaller values indicate larger variance reduction.
Reinforcement learning has proven effective for improving reasoning in large language models, but extending it to Masked Diffusion Language Models (MDLMs) remains challenging due to the intractability of the log-likelihood estimation. Existing approaches approximate this log-likelihood by modeling only the token predictions, ignoring the order in which positions are unmasked during generation. We observe that MDLM generation involves two decisions at each step: what tokens to place at each masked position and which positions to remask. We formalize this as a two-stage action MDP, showing that the policy gradient naturally decomposes into a token term and a masking term. Combining optimization of both terms leads to state-of-the-art outcomes on mathematical reasoning and coding benchmarks, with scores of 87.1% on GSM8K and 53.4% on MBPP.
Haran Raajesh, Kulin Shah, Adam Klivans +1
Department of Computer Science The University of Texas at Austin
Masked diffusion language models (MDMs) have recently emerged as a promising alternative to standard autoregressive large language models (AR-LLMs), yet their optimization can be substantially less stable. We study blockwise MDMs and compare them with AR-LLMs on three controlled tasks that stress different aspects of structured generation: in-context linear regression, graph path-finding, and Sudoku solving. We find that standard random-masking MDMs fail to reliably learn linear regression, exhibit high variance training dynamics on graph path-finding, while outperforming AR-LLMs on Sudoku. To mitigate these instabilities, we propose two locality aware blockwise models, namely Jigsaw and Scatter, that inject left-to-right inductive bias by enforcing autoregressive locality within blocks while preserving iterative refinement at the block level. Empirically, Jigsaw matches AR-LLM stability on linear regression and remains strong on Sudoku, while Scatter retains diffusion's planning advantage on path-finding. Our results indicate that standard random-masking MDMs, even with blockwise variants, may be a suboptimal instantiation of diffusion LMs for ordered generation, motivating models beyond random masking.
Yuxiang Wang, Yu Xiang, Baojian Zhou +4
1Fudan University, Shanghai, China · 2Alibaba Group, China.
We introduce a post-training method for diffusion language models (DLMs) that minimizes Maximum Mean Discrepancy (MMD) between generated and reference distributions in the feature space of a frozen pretrained DLM. To estimate MMD, we retain contextual features at individual token positions, obtaining multiple observations per sequence from a single extractor pass. We optimize this objective using policy gradients for discrete models and direct differentiation through generated latents for continuous models. In both cases, computing the loss directly from these features enables efficient post-training without full sampling trajectories or jointly trained auxiliary models. Experiments show lower generative perplexity at comparable entropy on OpenWebText and better accuracy-computation trade-offs on GSM8K. On 16B DMax-LLaDA2.0 models with hybrid masked-uniform diffusion, we increase decoding parallelism with similar or higher accuracy on math and code benchmarks.
Ilya Drobyshevskiy, Ilia Sudakov, Maksim Semenov +7