Policy gradient methods are widely used to align large language model outputs with human preferences, but the log-likelihoods they require cannot be directly computed in Masked Diffusion Language Models (MDLMs). MDLM post-training therefore relies on a surrogate objective based on the Evidence Lower Bound (ELBO), and recent work estimates a blockwise variant of this objective via Monte Carlo methods to support blockwise generation. However, existing estimation schemes randomly sample the block index, so they may evaluate the same block more than once and leave some blocks unevaluated. This introduces variance arising from block selection into the estimates. In this work, we propose a full-block estimator that evaluates every block exactly once and averages the results. We theoretically show that the proposed estimator remains unbiased while eliminating the between-group variance arising from block selection, thereby achieving lower variance than the baseline for the same sample size. In experiments on five benchmarks spanning mathematics, planning, and coding, our method achieves the highest accuracy on four benchmarks and outperforms the random block sampling baseline by an average of 12.79 points.
Figures & tables
Figure 1: Comparison of block selection in Monte Carlo estimation of the blockwise surrogate objective. For each Monte Carlo sample index k , a block b is sampled, and the corresponding set of token positions is denoted by Bb . For each block b , the masking ratio tb and the masked positions Mtb are sampled to compute the blockwise surrogate objective in ( 1 ). Top: In our method (full-block evaluation), each sample k corresponds to a distinct block, so that every block is evaluated exactly once. Bottom: In the existing method (random block sampling), each sample independently selects a block, which may cause the same block to be evaluated multiple times ( B2 is selected for both k=1,2 ) or some blocks to remain unevaluated ( B1 ).
Figure 2: Mask state during the generation of block b=1 for a response o of length L=12 ( C=3 , B=4 ). Block B0 is considered already generated, so its original tokens are retained. In this example, positions 5 and 7 within block B1 are masked ( Mt1={5,7} ), and all positions in block B2 are masked. The full set of masked positions is S1(Mt1)={5,7,8,9,10,11} .
Figure 3: Instability of random block sampling with a small number of blocks. (a) Some blocks receive zero selections after K=4 independent draws. (b) The miss probability increases rapidly with C (see Appendix C.7 for a formal derivation).
Benchmark
Method
Blockwise
GSM8K
MATH-500
Countdown
Sudoku
HumanEval
Avg.
LLaDA
×
76.27
31.80
19.14
26.56
27.44
36.24
LLaDA-1.5
×
78.70
33.00
18.36
23.64
32.93
37.33
D1
×
81.43 †
36.00
36.72
13.21
32.93
40.06
WD1
×
81.20
34.60
44.14
62.25
28.05
50.05
SPG
✓
80.29
36.20 †
70.31 ↑ 3.90
72.73 †
29.88
57.88 †
Table 1: Performance comparison across benchmarks. Higher scores indicate better performance. LLaDA denotes the base model (LLaDA-8B-Instruct) before fine-tuning. The Blockwise column indicates methods that use the blockwise surrogate objective ( 3 ). Bold denotes the highest accuracy in each column, and † denotes the second highest. ▲ indicates the improvement of L^Full over L^Rand , and ↑ indicates the improvement of the highest accuracy over the † value.
Figure 4: Comparison of estimation quality for LBlock in ( 3 ) between L^Full in ( 6 ) and L^Rand in ( 5 ) on GSM8K ( C=4 ). L^Rand is computed with varying sample size K . The estimate at K=1024 is used as the high-precision reference estimate Lref for computing Spearman ρ and RMSE. Higher ρ and lower RMSE indicate better estimation quality. The star indicates the result of L^Full at K=C=4 , which achieves estimation quality roughly comparable to L^Rand at K≈32 .
GSM8K
MATH
Countdown
Sudoku
Code
0.122
0.119
0.095
0.076
0.107
Table 2: Variance ratio Var[L^Full]/Var[L^Rand] in ( 12 ), evaluated numerically for each sample and averaged within each dataset ( K=C=4 ). Smaller values indicate a greater variance reduction effect of L^Full .
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Value
Optimization
Optimizer
AdamW
Learning rate
1.5×10−6
LR scheduler
constant_with_warmup
Weight decay
0.1
Max gradient norm
0.2
Appendix
Table 3: Common training hyperparameters.
Parameter
Default
Code
Temperature
0.9 *
0.9
Block length
64
128
Diffusion steps
128
256
Generation length
256
512
Remasking strategy
low_confidence
Appendix
Table 4: Diffusion generation settings during training. Default applies to GSM8K, MATH, Countdown and Sudoku. * Sudoku uses 0.3.
Table 5: Datasets used for training and evaluation.
Parameter
Value
Temperature
0.0
Remasking strategy
low_confidence
Generation length ( L )
256, 512
Diffusion steps
L/2
Appendix
Table 6: Inference hyperparameters.
Benchmark
Samples
Metric
GSM8K
1,319
Accuracy
MATH-500
500
Accuracy
Countdown
256
Accuracy
Sudoku
256
Accuracy
HumanEval
164
Pass@1
Appendix
Table 7: Evaluation benchmarks.
Setting
Variance ratio vs L^Rand
General ( K=qC+r )
1−λ+λ⋅K(C−1)r(C−r)
K=1
1
K<C ( q=0 , r=K )
1−λ⋅C−1K−1
K=mC ( m≥1 , r=0 )
1−λ
Appendix
Table 8: Variance ratio of L^Full(K) relative to L^Rand . Here K=qC+r with q=⌊K/C⌋ and r=KmodC∈{0,…,C−1} , and λ is defined in ( 19 ). The case K=C corresponds to K=mC with m=1 .
Method
GSM8K
MATH-500
Countdown
Sudoku
HumanEval
Avg.
SPG
81.88
35.20
64.84
89.76
30.49
60.43
L^Rand
81.58
33.40
54.30
80.96
31.10
56.27
L^Full (ours)
82.26
38.00
66.41
91.39
36.59
62.93
Appendix
Table 9: Empirical verification for K<C . Baselines (SPG and L^Rand ) are re-trained with K=2 , C=4 , I=8,000 , and our method L^Full uses K=C=4 , I=4,000 (reused from Table 1 ). Both settings have the same total number of block evaluations K⋅I=16,000 .
K=4
K=6
K=8
Method
GSM8K
MATH-500
GSM8K
MATH-500
GSM8K
MATH-500
L^Rand
76.20
34.40
80.89
34.00
80.97
34.00
L^Full(K) (ours)
82.26
38.00
80.89
36.20
81.43
34.20
Appendix
Table 10: Empirical verification for K>C ( C=4 fixed, I=4,000 ). The K=4 column is reused from Table 1 . The K=6 and K=8 columns are new experiments.
C=K=4
C=K=8
Method
GSM8K
MATH-500
GSM8K
MATH-500
L^Rand
76.20
34.40
80.52
31.80
L^Full (ours)
82.26
38.00
81.35
36.80
Appendix
Table 11: Empirical verification of doubling C . Comparison between the main experiment ( C=K=4 ) and the re-trained models with C=K=8 . The C=K=4 columns are reused from Table 1 . All other hyperparameters are identical to those of the main experiment (LLaDA-8B-Instruct, I=4,000 ). Code, Countdown, and Sudoku are excluded due to limited computation time and resources. Bold indicates the higher value within each C=K setting.
Mean
Std
Median
GSM8K
L^Rand
109.32
85.14
90.81
L^Full
36.25
31.82
28.85
MATH
L^Rand
118.14
90.89
99.45
L^Full
36.64
33.32
29.18
Countdown
L^Rand
131.66
98.14
113.46
L^Full
37.82
32.09
30.90
Appendix
Table 12: Statistics of the estimation error ∣L^−Lref∣ ( K=4 , 50 independent repetitions). The reference value is computed with K=1024 .
Full < Rand (%)
Var. ratio
GSM8K
99.5
7.35×
MATH
98.9
7.47×
Countdown
99.7
9.71×
Sudoku
100.0
11.28×
Code
99.6
7.36×
Appendix
Table 13: Comparison of per-sample variances of the absolute error ∣L^−Lref∣ ( K=4 , computed from 50 independent repetitions). Full < Rand denotes the proportion of samples for which L^Full has lower variance than L^Rand . Var. ratio denotes the ratio of the mean variances ( L^Rand/L^Full ). Wilcoxon signed-rank test p<10−100 for all datasets.
Figure 5: Detailed estimation quality analysis across five datasets. Rows correspond to datasets (GSM8K, MATH-500, Countdown, Sudoku, and Code); columns correspond to the five analysis types described in the introduction of this section.
Training step
Task
1,000
2,000
3,000
4,000
GSM8K
0.811
0.915
0.889
0.961
MATH-500
0.791
0.926
0.825
0.915
Appendix
Table 14: Sample variance ratio of the policy gradient Var[G^Full]/Var[G^Rand] . Computed for the four training checkpoints and the two tasks of the main experiment ( C=K=4 ), aggregating over the remaining prompts after excluding those with Aj=0∀j from the N=100 samples. Each prompt contributes S=50 gradient samples. A ratio below 1 indicates that the variance of Full is smaller than that of Rand.
Training step
Task
1,000
2,000
3,000
4,000
GSM8K
0.115
0.113
0.138
0.134
MATH-500
0.095
0.155
0.104
0.132
Appendix
Table 15: Estimator variance ratio 1−λ=σˉwithin2/(σbetween2+σˉwithin2) ( K=C=4 ) at the mid-training checkpoints of the main experiment. Computed from the variance decomposition ( 12 ) of the per-block log-likelihood ℓb,tb,Mtb and aggregated over the prompts excluding those with Aj=0∀j . Directly comparable to Table 2 on the base model. Smaller values indicate larger variance reduction.