Scaling Beyond Masked Diffusion Language Models

Subham Sekhar Sahoo; Jean-Marie Lemercier; Zhihan Yang; Justin Deschenaux; Jingyu Liu; John Thickstun; Ante Jukic

cs.LGFeb 16, 2026

Scaling Beyond Masked Diffusion Language Models

Authors: Subham Sekhar Sahoo, Jean-Marie Lemercier, Zhihan Yang, Justin Deschenaux, Jingyu Liu, John Thickstun, Ante Jukic

Organizations: NVIDIA, Santa Clara, USA · Department of Computer Science, Cornell Tech, NYC, USA · Department of Computer Science, Cornell University, Ithaca, USA · School of Computer and Communication Sciences, EPFL Lausanne, Switzerland · Department of Computer Science, University of Chicago, Illinois

Abstract

Diffusion language models are a promising alternative to autoregressive models due to their potential for faster generation. Among discrete diffusion approaches, Masked diffusion currently dominates, largely driven by strong perplexity on language modeling benchmarks. In this work, we present the first scaling law study of uniform-state and interpolating discrete diffusion methods. We also show that Masked diffusion models can be made approximately 12% more FLOPs-efficient when trained with a simple cross-entropy objective. We find that perplexity is informative within a diffusion family but can be misleading across families, where models with worse likelihood scaling may be preferable due to faster and more practical sampling, as reflected by the speed-quality Pareto frontier. These results challenge the view that Masked diffusion is categorically the future of diffusion language modeling and that perplexity alone suffices for cross-algorithm comparison. Scaling all methods to 1.7B parameters, we show that uniform-state diffusion remains competitive on likelihood-based benchmarks and outperforms autoregressive and Masked diffusion models on GSM8K, despite worse validation perplexity. We provide the code, model checkpoints, and video tutorials on the project page: http://s-sahoo.github.io/scaling-dllms

Figures & tables

Figure 1: Speed-Quality Pareto Frontier . We report the highest throughput achieved by compute-optimal models across a range of training FLOPs budgets. AR produces the highest-quality samples but is slow. Sample diversity (measured by entropy) remains broadly similar across algorithms, with Duo exhibiting slightly reduced diversity; see Fig. 5 . Duo dominates in the throughput ranges

[200,400]\cup[600,\infty]

, while Eso-LM dominates in the range

[400,600]

Panel of Figure 2 — Figure 2: IsoFLOP Analysis under fixed computation budgets.

Panel of Figure 3 — Figure 3: Scaling Laws. Diffusion models exhibit similar scaling behavior wrt AR models.

		ARC-e	BoolQ	OBQA	PIQA	RACE	SIQA
	Chance	24.7	50.4	26.6	51.6	24.2	32.2
Prior Work
	SMDM-1B ∗ ( Nie et al., 2025a )	37.4	61.5	27.0	60.3	29.3	37.9
	LLaDa-8B-Base ( Nie et al., 2025b )	-	-	-	74.4	-	-
Ours
	AR-1.7B † (Autoregressive)	72.7	71.9	40.4	78.1	36.2	41.9

Table 1: Likelihood Evaluation. SMDM-1B was trained on slim-pajama , while LLaDa-8B-Base was trained on proprietary data comprising 2.3T tokens. † Models were trained for approximately 2.1T tokens on Nemotron-Pre-Training-Dataset. Because prior work uses models of different sizes, training data, and token budgets, the results are not directly comparable to ours and are included only for reference.

\underline{\text{Underline}}

denotes the best accuracy (

\uparrow

) across all models and the bolded numbers denote the best diffusion accuracy.

		Acc. ( $\uparrow$ )	Throughput ( $\uparrow$ )
Prior work
	SMDM-1B ∗ †	58.5	-
	LLaDa-8B	70.7	-
Ours
	AR-1.7B †	62.9	25.6
	MDLM-1.7B †	58.8	24.6

Table 2: GSM8K benchmark. SMDM-1B is trained on slim-pajama while LLaDa-8B-Base on proprietary data. † Models are trained for 2.1T tokens on the Nemotron-Pre-Training-Dataset. Throughput (tokens/sec) is measured using a batch size of 1. In this memory-bound setting, AR models are expected to match the latency of diffusion models despite using KV caching.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Throughput (toks / sec;

\uparrow

) vs time discretization

T

for various diffusion models.

Panel of Figure 5 — Figure 5: Gen. PPL (sample quality; $\downarrow$ ) and entropy (sample diversity; $\uparrow$ ) vs time discretization ( $T$ ) for (a) MDLM w/ ancestral sampler, (b) Eso-LM w/ block sampler, and (c) Duo w/ ancestral sampler.

Panel of Figure 6 — Figure 6: Scaling law comparison between MDLM trained with the low-variance training loss ( 8 ) and MDLM trained with the ELBO ( 7 ). MDLM trained with the low variance training loss yields compute optimal models with fewer parameters.

Hyperparams		Accuracy ( $\uparrow$ )
$\eta_{\max}$	$\eta_{\min}$	5 ep	10 ep	20 ep
AR
1e-5	2e-6	62.9	–	–
2e-5	5e-8	59.0	–	–
2e-5	5e-7	61.6	–	–
2e-5	2e-6	59.1	–	–

Appendix

Table 3: Accuracy after SFT on GSM8K with a cosine schedule, with maximal and minimial learning rates

\eta_{\max}

and

\eta_{\min}

Non-Embedding Parameters ( $M$ )	n_embed	n_layers	n_heads
14	256	6	4
29	384	8	6
44	512	8	8
58	576	9	9
74	640	10	10
91	640	13	10

Appendix

Table 4: Transformer configurations of AR, MDLM, Duo, and Eso-LM used in the scaling law study.

Explore similar work

Understanding and Accelerating the Training of Masked Diffusion Language Models
May 13, 2026Chunsan Hong, Sanghyun Lee, Chieh-Hsin Lai +5Masked Diffusion Models Diffusion Model Acceleration
1KAIST · 2Sony AI · University of Tokyo +1
DiLaDiff: Distilled Latent-Augmented Diffusion for Language Modeling
May 22, 2026Jean-Marie Lemercier, Tomas Geffner, Karsten Kreis +3Masked Diffusion Models Diffusion Language Model Inference
NVIDIA
Looped Diffusion Language Models
May 25, 2026Sanghyun Lee, Chunsan Hong, Seungryong Kim +3Masked Diffusion Models Diffusion Language Model Inference
1KAIST · 2KRAFTON · University of California, Berkeley

May 13, 2026J/K move · Enter open · S save

Understanding and Accelerating the Training of Masked Diffusion Language Models

Chunsan Hong, Sanghyun Lee, Chieh-Hsin Lai, Satoshi Hayakawa+4

1KAIST · 2Sony AI · University of Tokyo+1

Masked Diffusion Models Diffusion Model Acceleration Diffusion Language Models

Open paper AskPDF arXiv

Masked diffusion models (MDMs) have emerged as a promising alternative to autoregressive models (ARMs) for language modeling. However, MDMs are known to learn substantially more slowly than ARMs, which may become problematic when scaling MDMs to larger models. Therefore, we ask the following question: how can we accelerate standard MDM training while maintaining its final performance? To this end, we first provide a detailed analysis of why MDM training is slow. We find that the main factor is the locality bias of language: the predictive information for a token is concentrated in nearby positions. We further investigate how this bias slows learning and suggest a simple yet effective remedy: bell-shaped time sampling as a training strategy. Notably, MDMs trained with our training recipe reach the same validation negative log-likelihood (NLL) up to

\sim4\times

faster than standard training on One Billion Word Benchmark (LM1B). We also show faster improvements in generative perplexity, zero-shot perplexity, and downstream task performance on various benchmarks.

Abstract

Explore similar work

Understanding and Accelerating the Training of Masked Diffusion Language Models

DiLaDiff: Distilled Latent-Augmented Diffusion for Language Modeling

Looped Diffusion Language Models