We present DMax, a new paradigm for efficient diffusion language models (dLLMs). It mitigates error accumulation in parallel decoding, enabling aggressive decoding parallelism while preserving generation quality. Unlike conventional masked dLLMs that decode through a binary mask-to-token transition, DMax reformulates decoding as a progressive self-refinement from mask embeddings to token embeddings. At the core of our approach is On-Policy Uniform Training, a novel training strategy that efficiently unifies masked and uniform dLLMs, equipping the model to recover clean tokens from both masked inputs and its own erroneous predictions. Building on this foundation, we further propose Soft Parallel Decoding. We represent each intermediate decoding state as an interpolation between the predicted token embedding and the mask embedding, enabling iterative self-revising in embedding space. Extensive experiments across a variety of benchmarks demonstrate the effectiveness of DMax. Compared with the original LLaDA-2.0-mini, our method improves TPF on GSM8K from 2.04 to 5.47 while preserving accuracy. On MBPP, it increases TPF from 2.71 to 5.86 while maintaining comparable performance. On two H200 GPUs, our model achieves an average of 1,338 TPS at batch size 1. Code is available at: https://github.com/czg1225/DMax
Figures & tables
Figure 1: Comparison between the original LLaDA-2.0-mini and our proposed DMax. Unlike the original binary mask-to-token decoding process, DMax introduces a self-revising mask-to-hybrid-embedding-to-token process, enabling highly parallel decoding without accuracy dropping.
Figure 2: Overview of the proposed On-Policy Uniform Training.
Figure 3: Overview of the Soft Parallel Decoding procedure in DMax.
Benchmark
Method
TPF ↑
TPS ↑
Acc. ↑
AUP Score ↑
Math & Reasoning Benchmarks
GSM8K
LLaDA-2.0-mini
2.04
512
92.6%
340
Hierarchical Decoding
2.44
577
91.6%
357
dParallel SFT
2.79
721
92.3%
395
Uniform Diffusion Training
2.26
493
68.7%
0
DMax-Math
5.48
1258
92.1%
557
Table 1: Comparison with the original model and different baselines. For our DMax-Math model, we set the decoding threshold to 0.5; for the DMax-Coder model, we set it to 0.65. In addition to TPF, TPS, and accuracy, we also report the AUP score to provide a more comprehensive evaluation of parallel decoding performance. All evaluations are under zero-shot and a batch size of 1.
Figure 4: Comparison of accuracy-TPF trade-off curves between original LLaDA-2.0-mini model and our method. We present curves on GSM8K, MATH500, HumanEval and MBPP benchmarks.
Benchmarks
LLaDA-2.0-mini
DMax
TPF ↑
Acc. ↑
TPF ↑
Acc. ↑
GSM8K
2.04
92.6%
3.54 (+1.50)
93.4% (+0.8%)
MATH500
2.58
75.8%
3.45 (+0.87)
78.0% (+2.2%)
Minerva-Algebra
3.01
91.4%
4.96 (+1.95)
93.6% (+2.2%)
ASDIV
2.03
92.8%
3.10 (+1.07)
93.5% (+0.7%)
HumanEval-Instruct
4.38
84.2%
4.58 (+0.20)
87.2% (+3.0%)
Table 2: Our proposed new paradigm also improves the model’s accuracy at low parallelism.
Train
Inference
τdec=0.95
τdec=0.50
τdec=0.0
On-Policy Rollout
Contiguous Prefix
Hybrid Embedding
TPF ↑
Acc. ↑
TPF ↑
Acc. ↑
TPF ↑
Acc. ↑
2.04
92.6%
4.47
78.0%
7.86
0.9%
✓
✓
1.04
0.0%
1.73
0.0%
5.39
0.0%
✓
2.95
92.6%
5.14
90.1%
5.89
68.2%
✓
✓
2.85
93.0%
5.28
91.3%
5.89
68.2%
✓
✓
3.25
92.8%
5.64
91.4%
6.01
90.4%
Table 3: Ablation on different training and inference strategies with different decoding parallelism.
Convergence
GSM8K
MBPP
Consistency
Confidence
TPF ↑
Acc. ↑
TPF ↑
Acc. ↑
✓
5.13
92.1%
5.16
79.9%
✓
2.28
92.2%
3.36
80.1%
✓
✓
5.48
92.1%
5.86
79.2%
Table 4: Ablation study on block-level convergence criteria. The decoding threshold is set to 0.5.
Parallel decoding for Diffusion LLMs (dLLMs) is difficult because each denoising step provides only token-wise marginal distributions, while unmasking multiple tokens simultaneously requires accounting for inter-token dependencies. We propose Dependency-Aware Parallel Decoding (DAPD), a simple, training-free decoding method that uses self-attention to induce a conditional dependency graph over masked tokens. At each iteration, edges in this graph capture strong token interactions, while non-edges indicate weak dependence. Parallel decoding is then reduced to selecting an independent set on the graph and unmasking the selected tokens in parallel. This avoids co-updating strongly coupled tokens without auxiliary models or retraining. Experiments on LLaDA and Dream show that DAPD improves the accuracy-steps trade-off over existing methods and enables more globally distributed parallel updates that better exploit the any-order generation capability of dLLMs. The project is available at https://ai-isl.github.io/dapd
Bumjun Kim, Dongjae Jeon, Moongyu Jeon +1
Department of Artificial Intelligence, Yonsei University, Seoul, Korea · Department of Computer Science, Yonsei University, Seoul, Korea
Masked diffusion language models enable parallel token decoding, providing a promising alternative to the sequential nature of autoregressive generation. However, their iterative denoising process remains computationally expensive because it repeatedly processes the entire sequence at every step. We observe that across these diffusion steps, most token representations remain stable; only a small subset, which we term salient tokens, contributes meaningfully to the next update. Leveraging this temporal sparsity, we present DyLLM, a training-free inference framework that accelerates decoding by selectively computing only these salient tokens. DyLLM identifies saliency by measuring the cosine similarity of attention contexts between adjacent denoising steps. It recomputes feed-forward and attention operations only for salient tokens while reusing cached activations for the remainder. Across diverse reasoning and code-generation benchmarks, DyLLM achieves up to 9.6x higher throughput while largely preserving the baseline accuracy of representative open-source diffusion LLMs, LLaDA, and Dream.
Younjoo Lee, Seungkyun Dan, Junghoo Lee +2
Seoul National University, Seoul, Republic of Korea
Diffusion large language models (dLLMs) generate text by iteratively denoising masked token sequences. Although dLLMs can predict all masked positions in parallel within each step, the large number of denoising iterations still makes inference expensive. This cost can be reduced spatially by unmasking multiple tokens per step, or temporally by collapsing multiple denoising steps into one verification call. We propose Parallel Speculative Decoding (PSD), a training-free framework that jointly improves inference along both axes. Using the confidence scores from a single forward pass, PSD selects positions to unmask via a configurable, adaptive unmasking policy and constructs multi-depth speculative drafts without extra model calls. A final batched verification pass then applies hierarchical acceptance, keeping the deepest draft that remains consistent with the updated predictions. Experiments on three dLLMs across reasoning and code generation tasks show that PSD achieves favorable trade-offs between inference efficiency and generation quality, reaching up to 5.5× tokens per forward pass with accuracy comparable to greedy decoding.
Shengyin Sun, Yiming Li, Renxi Liu +7
Department of Computer Science City University of Hong Kong · Huawei Technologies · City University of Hong Kong