Diffusion Large Language Models (dLLMs) have emerged as a promising paradigm for parallel token generation, with block-wise variants garnering significant research interest. Despite their potential, existing dLLMs typically suffer from a rigid accuracy-parallelism trade-off: increasing the number of tokens per forward (TPF) via aggressive parallel decoding often leads to performance degradation and increased generation instability. We identify that this limitation stems from the model's inability to navigate high-parallelism regimes where approximation errors and local corruptions accumulate, ultimately undermining the reliability of parallel generation. To address this, we propose LightningRL, a post-training framework designed to directly optimize the speed-quality Pareto frontier of pre-trained dLLMs. Instead of forcing uniform parallelization, our approach leverages reinforcement learning to identify and reinforce high-parallelism trajectories that maintain generation accuracy. Built upon the Group Relative Policy Optimization (GRPO) framework, LightningRL introduces several enhancements tailored for dLLMs: (1) stabilized training via per-reward decoupled normalization; (2) token-level negative log-likelihood (NLL) regularization on correct trajectories to anchor model performance; and (3) a dynamic sampling strategy with TPF-aware filtering to enhance training efficiency. Experimental results across mathematical and coding benchmarks demonstrate that LightningRL consistently advances the Pareto frontier, achieving competitive task accuracy while significantly increasing parallelism, reaching an average TPF of 7.32 (with a peak of 11.10 on the MBPP dataset). Our code is available at https://github.com/SJTU-DENG-Lab/LightningRL.
Figures & tables
Figure 2: Overview of LightningRL. LightningRL samples a group of decoding trajectories per prompt, applies per-reward decoupled normalization to preserve within-group ranking under heterogeneous scales. The policy is optimized with a GRPO-style objective plus a token-level NLL anchor. The bottom panel shows the resulting shift toward the fastest correct trajectory, improving TPF without degrading accuracy.
Figure 3: Per-reward decoupled normalization improves training stability. It reduces signal collapse (a) and yields more stable reward optimization (b) under the same training setup.
Figure 3
Model
GSM8K
MATH500
MBPP
HumanEval
Acc (%)
TPF
AUP
Acc (%)
TPF
AUP
Acc (%)
TPF
AUP
Acc (%)
TPF
AUP
SDAR ( Cheng et al., 2025 )
88.9
2.85
252.5
63.6
4.81
318.9
58.0
2.44
108.4
73.5
2.39
165.0
Coupled-GRPO ( Gong et al., 2025 )
75.3
4.22
254.7
59.1
4.93
283.1
54.0
2.61
139.9
68.2
2.42
164.6
TraceRL ( Wang et al., 2025b )
76.9
5.04
378.6
60.6
4.82
284.6
57.8
2.50
144.2
75.0
2.29
171.6
DiRL ( Zhu et al., 2026 )
86.6
4.87
414.1
61.6
5.04
301.2
56.8
2.70
152.7
76.0
2.33
176.7
LightningRL
90.3
5.58
507.5
63.0
6.28
409.2
58.3
11.10
412.9
72.6
6.30
421.5
Table 1: Evaluation results of LightningRL and methods on math and code benchmarks. All methods are trained on the SDAR 8B model with block size 32. Compared with the other three algorithms and the SDAR base model, LightningRL delivers substantial improvements across both math and code benchmarks.
Figure 5
Model
GSM8K
MATH500
MBPP
HumanEval
Acc (%)
TPF
AUP
Acc (%)
TPF
AUP
Acc (%)
TPF
AUP
Acc (%)
TPF
AUP
dLLMs
Dream ( Ye et al., 2025 )
83.9
1.00
83.9
39.6
1.00
39.6
57.2
1.00
57.2
55.2
1.00
55.2
Fast-dLLM-Dream ( Wu et al., 2025b )
79.0
1.44
110.4
38.3
1.78
49.0
53.2
1.20
63.6
54.3
1.33
63.2
dParallel-Dream ( Chen et al., 2025 )
82.1
3.02
215.2
38.7
2.94
62.6
55.4
2.24
108.4
54.3
2.57
97.1
d3LLM-Dream ( Qian et al., 2026 )
81.4
4.94
333.6
38.2
3.92
74.6
55.6
2.96
142.1
57.1
3.20
126.6
Table 2: Evaluation results of LightningRL and the baselines on math and code benchmarks. The notation LightningRL-8B-b32 denotes the 8B LightningRL model with a block size of 32. LightningRL consistently advances the speed–quality frontier over its SDAR baseline and prior diffusion and autoregressive baselines, achieving substantially higher AUP at comparable accuracy.
Table 7
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Acc (%)
TPF
AUP
w/ Value Model
80.7
5.14
408.1
LightningRL
90.3
5.58
492.4
Appendix
Table 5: Comparison of Performance with and without Value Model Incorporation on GSM8K. The AUP in this table is computed using each model’s own accuracy at TPF ≈ 1 as ymax .
Scale
BS
Model
GSM8K
MATH500
MBPP
HumanEval
Acc (%)
TPF
AUP
Acc (%)
TPF
AUP
Acc (%)
TPF
AUP
Acc (%)
TPF
AUP
1.7B
32
SDAR
71.5
2.48
176.4
41.2
5.62
226.3
39.0
3.86
149.8
48.8
2.81
136.0
LightningRL
71.7
3.40
251.4
41.2
6.01
241.2
37.3
5.07
185.8
48.2
3.43
165.2
4B
32
SDAR
86.6
3.10
243.8
53.6
5.09
263.9
54.0
1.96
105.9
59.8
3.67
216.8
LightningRL
85.4
4.37
374.7
56.4
6.55
358.7
52.2
3.05
158.3
57.3
4.35
242.3
8B
32
SDAR
88.9
2.85
252.5
63.6
4.81
299.5
58.0
2.44
81.1
73.5
2.39
123.8
Appendix
Table 6: Comparison of SDAR and LightningRL under identical settings. The results are grouped by Model Scale and Block Size (BS) to facilitate direct comparison across four datasets. The AUP in this table is computed using each model’s own accuracy at TPF ≈ 1 as ymax .
Model
H100 TPS
TPF
Acc (%)
Qwen-2.5-7B-it
57.3
1.00
74.1
Fast-dLLM-v2
150.0
2.21
77.5
dParallel-LLaDA
172.2
5.14
72.6
d3LLM-LLaDA
288.9
9.11
73.1
SDAR
105.6
2.85
88.9
LightningRL
336.0
5.58
90.3
Appendix
Table 7: Tokens Per Second (TPS) performance comparison on H100 GPUs. We report on GSM8K datasets. LightningRL achieves superior throughput.
Figure 8: Training dynamics and Accuracy-Parallelism Trade-off on GSM8K. LightningRL avoids the objective drift and reward collapse observed in TraceRL, while sustaining higher decoding throughput and more synchronized termination.
*Equal contribution. The work is done during Tianyi’s internship at CBG Celia DeviceAI Team, Huawei. · 1Harbin Institute of Technology, Shenzhen · 2Huawei +1