Diffusion Large Language Models (dLLMs) introduce a new paradigm for language generation, which in turn presents new challenges for aligning them with human preferences. In this work, we aim to improve the policy optimization for dLLMs by reducing the cost of the trajectory probability calculation, thereby enabling scaled-up offline policy training. We prove that: (i) under reference policy regularization, the probability ratio of the newly unmasked tokens is an unbiased estimate of that of intermediate diffusion states, and (ii) the probability of the full trajectory can be effectively estimated with a single forward pass of a re-masked final state. By integrating these two trajectory reduction strategies into a policy optimization objective, we propose Trajectory Reduction Policy Optimization (dTRPO). We evaluate dTRPO on 7B dLLMs across instruction-following and reasoning benchmarks. Results show that it substantially improves the core performance of state-of-the-art dLLMs, achieving gains of up to 9.6% on STEM tasks, up to 4.3% on coding tasks, and up to 3.0% on instruction-following tasks. Moreover, dTRPO exhibits strong training efficiency due to its offline, single-forward nature, and achieves improved generation efficiency through high-quality outputs.
Figures & tables
Figure 1: Performance gains on MATH dataset v.s. normalized online (left) and offline (right) training cost. Online training requires hundrads more ( N× ) computation for the rollout stage, while our offline method requires only 4 forward passes per training example and achieves comparable performance.
Figure 2: (a): Generation processes in ARMs and dLLMs. ARMs generate tokens via causal conditioning, whereas dLLMs generate sequences via a multi-step diffusion process. (b): dTRPO samples masked tokens for each block and estimates trajectory probability ratios using only the probabilities of newly unmasked tokens under πθ .
Objective
GSM8K
LCBv6
IFEval
maxθΠyi=[M]πθ(y~)(i)
66.26
2.37
25.69
minθ ELBO( Πyi=[M]πθ(y~)(i) )
79.98
11.56
51.02
dTRPO
85.97
15.17
65.06
Table 1: Directly optimizing masked token probability is not effective in dLLM post training.
GPQA (cot,diamond)
GSM8K (cot)
MATH
LCB (v6)
MBPP+ (extra)
HEval+ (extra)
IFEval (prompt)
ArenaHard (V2.0)
MTBench
dLLM from Scratch
LLaDA Instruct
19.19
78.47
42.48
6.07
38.1
34.1
53.23
-
-
LLaDA 1.5
19.19
79.45
43.64
6.54
37
39
59.52
-
-
Qwen 2.5 7B Instruct v.s. dLLM from Qwen2.5 7B Instruct
Qwen2.5 Instruct
36.36
87.87
73.06
24.42
67.5
74.4
71.38
10.43
8.08
Dream Instruct
28.79
75.36
50.22
12.61
54.5
53
50.65
6.79
3.88
Table 2: Performance of dLLMs under zero-shot evaluation. dTRPO achieves overall best performance and close the gap to strong ARMs.
Figure 3: Ablation study of algorithm design and implementation choices; Inference speed comparison for ours and baseline models.
Model
GSM8K
Arena-Hard
TPS
Accuracy
TPS
Score
Qwen2.5-7B-Instruct
38.9
87.87
16.20
10.43
Fast-dLLM-v2
38.84
82.34
19.55
6.79
dTRPO
38.80
85.97
29.87
7.41
Table 3: Token per second (TPS) and average end to end inference time (seconds) comparison on GSM8K and Arena-Hard between dTRPO, Fast-dLLM-v2, and Qwen2.5-7B-Instruct.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Causal attention and block attention in the training time. In (b), each row represents the tokens that an input token attends to. For a masked token, it attends to the masked input of the same block and clean tokens in the previous blocks.
μθ(⋅∣τ~(y)),μref(⋅∣τ~(y))
Appendix
Algorithm 1 dTRPO: Block-wise Ratio Reduction for Preference Optimization.
Figure 5: The post-training pipeline comparison between ARMs and dLLMs.