Masked diffusion models (MDMs) generate text by iteratively selecting positions to unmask and then predicting tokens at those positions. Yet MDMs lack proper likelihood evaluation: the evidence lower bound (ELBO) is not only a loose bound on log-likelihood, but, as we show, is also computed under the training distribution rather than the test-time distribution. We resolve this within our DUEL framework, which unifies leading MDM sampling strategies that employ deterministic position selection. We prove that DUEL samplers admit exact likelihood computation under the test-time distribution -- giving MDMs proper likelihood, and hence proper perplexity, for the first time. This proper perplexity is the natural analogue of autoregressive perplexity and lets us revisit key questions about MDMs. MDMs are substantially better than previously thought: the MDM-autoregressive perplexity gap shrinks by up to 32% on in-domain data and 82% on zero-shot benchmarks. DUEL enables the first principled comparison of fast,parallel samplers across compute budgets -- an analysis impossible with the ELBO and unreliable with generative perplexity -- identifying a strong default method. Finally, oracle search over position orderings reveals MDMs can far surpass autoregressive models -- achieving 36.47 vs. 52.11 perplexity on AG News -- demonstrating the ceiling of MDM performance has not yet been reached.
Figures & tables
Figure 1: One step of MDM generation. Given a partially masked sequence, the unmasking policy π performs position selection (choosing position 2), and the denoising network pθ performs token prediction at that position (sampling “big”). This repeats until no masked positions remain.
Unmask Rule F
Selected Positions F(z)
Left-to-Right
{k smallest ℓ:zℓ=m}
Greedy Conf.
argmax∣I∣=k∑ℓ∈IPℓ(1)
Prob. Margin
argmax∣I∣=k∑ℓ∈I(Pℓ(1)−Pℓ(2))
Conf. Thresh.
{ℓ∈M:Pℓ(1)≥μ}
KLASS
{ℓ∈M:Pℓ(1)≥μ,KL(P^ℓ∥Pℓ)≤ν}
Table 1: Unmasking rules defining deterministic policies. M=M(z) denotes masked positions; P=softmax(xθ(z)) is the token probability matrix.
Figure 2: Unmasking trajectory for σ=({1,3},{2},{4}) . Starting from a fully masked sequence ( t=0 ), positions are progressively revealed according to the ordered partition. At t=1 , positions 1 and 3 are unmasked in parallel; subsequent steps unmask one position each.
Figure 3: DUEL sampling and exact likelihood computation. Highlights mark the only differences. Both algorithms iterate with a denoising network xθ and unmasking rule F : compute token probabilities P=softmax(xθ(z)) , select positions via F(z) , then either sample tokens or accumulate log-probabilities. Section 4 proves that Algorithm 2 computes the exact likelihood under the distribution induced by F .
Model
ELBO
DUEL
Gap Closed
SEDD
≤ 24.10
22.58
23.2%
MDLM
≤ 22.98
21.86
20.6%
BD3-LM L′ =4
≤ 20.73
19.73
31.3%
BD3-LM L′ =8
≤ 21.68
20.37
31.6%
BD3-LM L′ =16
≤ 22.27
20.76
31.9%
Table 2: In-domain test perplexity ( ↓ ) on OWT. ARM baseline: 17.54. Bold : best MDM. Gap closed defined in Equation 7 .
Model
ELBO
DUEL
Gap Closed
SEDD
≤ 32.68
31.51
11.9%
MDLM
≤ 31.78
29.03
30.7%
BD3-LM L′ =4
≤ 28.23
26.97
23.3%
BD3-LM L′ =8
≤ 29.83
27.85
28.3%
BD3-LM L′ =16
≤ 30.60
28.39
28.4%
Table 3: In-domain test perplexity ( ↓ ) on LM1B. ARM baseline: 22.83. Bold : best MDM. Gap closed defined in Equation 7 .
Dataset
SEDD
MDLM
BD3-LM
PTB
31.3%
34.3%
81.8%
Wikitext
40.8%
28.5%
31.4%
LM1B
22.1%
29.0%
29.8%
AG News
25.7%
27.8%
51.7%
Average
30.0%
29.9%
48.7%
Table 4: Gap closed (%) by DUEL on zero-shot evaluation ( Equation 7 ). Models trained on OWT; BD3-LM uses L′=4 . Full perplexity values in Table 8 .
Model
Method
Wikitext
Lambada
AG News
Llama3
Exact
7.94
32.40
41.29
LLaDA
ELBO
≤ 15.29
≤ 39.04
≤ 85.17
DUEL
14.50
36.00
78.91
Table 5: Zero-shot perplexity ( ↓ ) for 8B models. Bold : better of ELBO/ DUEL .
Unmask Rule F
128
256
512
1024
Left-to-Right
240.27
109.71
45.99
21.46
Greedy Conf.
164.94
66.64
34.74
22.03
Prob. Margin
140.38
57.48
32.24
22.05
Conf. Thresh. †
226.97
116.83
43.48
22.05
Table 6: DUEL perplexity ( ↓ ) by unmasking rule on OWT. Model: BD3-LM ( L′ =16). ELBO: ≤ 23.52. Bold : best per NFE.
Model
Method
Unmask Rule F
Perplexity
ARM
Exact
—
52.11
BD3-LM
ELBO
—
≤ 61.67
BD3-LM
DUEL
Left-to-Right
54.94
DUEL
Greedy Conf.
56.73
DUEL
Prob. Margin
57.80
DUEL
Conf. Thresh. †
55.68
Table 7: Oracle perplexity ( ↓ ) on AG News for BD3-LM ( L′=4 ). All DUEL evaluations use block-restricted unmasking with L′=4 . The oracle searches all 4!=24 block permutations.
Figure 4: Comparing fast samplers. Top: DUEL perplexity. Bottom: Generative perplexity. Dashed line: ELBO ( ≤ 23.52). DUEL yields consistent rankings across NFE (probability margin best at low NFE, convergence at high NFE). Generative perplexity rankings cross repeatedly, making it unreliable.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Method
PTB
Wikitext
LM1B
Lambada
AG News
PubMed
ArXiv
ARM
Exact
81.07
25.32
51.14
52.13
52.11
48.59
41.22
SEDD
ELBO
≤ 96.33
≤ 35.98
≤ 68.14
≤ 48.93
≤ 67.82
≤ 43.13
≤ 37.89
DUEL
91.56
31.63
64.39
46.01
63.78
39.31
35.48
MDLM
ELBO
≤ 90.96
≤ 33.22
≤ 64.94
≤ 48.29
≤ 62.78
≤ 43.13
≤ 37.89
DUEL
87.57
30.97
60.94
48.42
59.81
41.96
37.57
BD3-LM L′ =4
ELBO
≤ 96.81
≤ 31.31
≤ 60.88
≤ 50.03
≤ 61.67
≤ 42.52
≤ 39.20
Appendix
Table 8: Zero-shot perplexity ( ↓ ) for models trained on OWT, evaluated on unseen datasets. ARM baseline in first row. Bold : better of ELBO/ DUEL per model-dataset pair.
Figure 5: Comparing fast samplers across four metrics. Dashed line in DUEL perplexity: ELBO ( ≤ 23.52). DUEL perplexity yields consistent rankings across NFE (probability margin best at low NFE), while generative perplexity rankings cross repeatedly. Entropy reveals that left-to-right produces degenerate low-entropy text at low NFE despite achieving favorable generative perplexity. MAUVE saturates near zero at low NFE for all rules, providing little discriminative signal.
Masked diffusion models (MDMs) have emerged as a promising alternative to autoregressive models (ARMs) for language modeling. However, MDMs are known to learn substantially more slowly than ARMs, which may become problematic when scaling MDMs to larger models. Therefore, we ask the following question: how can we accelerate standard MDM training while maintaining its final performance? To this end, we first provide a detailed analysis of why MDM training is slow. We find that the main factor is the locality bias of language: the predictive information for a token is concentrated in nearby positions. We further investigate how this bias slows learning and suggest a simple yet effective remedy: bell-shaped time sampling as a training strategy. Notably, MDMs trained with our training recipe reach the same validation negative log-likelihood (NLL) up to ∼4× faster than standard training on One Billion Word Benchmark (LM1B). We also show faster improvements in generative perplexity, zero-shot perplexity, and downstream task performance on various benchmarks.
Masked Diffusion Models (MDMs) enable flexible decoding orders, yet existing samplers remain largely greedy, selecting locally certain tokens without accounting for their downstream effects. We show that this myopia can increase cumulative uncertainty and lead to suboptimal generation. To address this, we propose the Info-Gain Sampler, a training-free decoding method that uses the bidirectional structure of MDMs to balance immediate uncertainty with the information gained over remaining masked positions. Across reasoning, coding, creative writing, and image generation tasks, Info-Gain Sampler consistently outperforms existing MDM samplers, improving average reasoning accuracy by 2.9--11.6 percentage points and achieving a 62.8% average win rate in creative writing. The code is available at https://github.com/yks23/Information-Gain-Sampler.
Kaisen Yang, Jayden Teoh, Kaicheng Yang +2
Department of Computer Science and Technology, Tsinghua University · Massachusetts Institute of Technology · Shanghai Jiao Tong University +2
Diffusion language models are a promising alternative to autoregressive models due to their potential for faster generation. Among discrete diffusion approaches, Masked diffusion currently dominates, largely driven by strong perplexity on language modeling benchmarks. In this work, we present the first scaling law study of uniform-state and interpolating discrete diffusion methods. We also show that Masked diffusion models can be made approximately 12% more FLOPs-efficient when trained with a simple cross-entropy objective. We find that perplexity is informative within a diffusion family but can be misleading across families, where models with worse likelihood scaling may be preferable due to faster and more practical sampling, as reflected by the speed-quality Pareto frontier. These results challenge the view that Masked diffusion is categorically the future of diffusion language modeling and that perplexity alone suffices for cross-algorithm comparison. Scaling all methods to 1.7B parameters, we show that uniform-state diffusion remains competitive on likelihood-based benchmarks and outperforms autoregressive and Masked diffusion models on GSM8K, despite worse validation perplexity. We provide the code, model checkpoints, and video tutorials on the project page: http://s-sahoo.github.io/scaling-dllms
Subham Sekhar Sahoo, Jean-Marie Lemercier, Zhihan Yang +4
NVIDIA, Santa Clara, USA · Department of Computer Science, Cornell Tech, NYC, USA · Department of Computer Science, Cornell University, Ithaca, USA +2