Simple Diffusion Language Models Are More Effective Few-Step Generators Than Reported

Hasan Amin; Ming Yin; Rajiv Khanna

cs.CLSep 27, 2026

Simple Diffusion Language Models Are More Effective Few-Step Generators Than Reported

Authors: Hasan Amin, Ming Yin, Rajiv Khanna

Organizations: Purdue University

Abstract

Diffusion language models (DLMs) promise fast parallel generation, yet high-quality samples often require large number of refinement steps, which diminishes their advantage in practice. This has led to massive interest in and rapid development of new methods for effective few-step generation. We show that much of the supposed quality gap at few steps can instead arise from a suboptimally configured sampler. Modest sampler sharpening, without any model retraining, enables a couple years old masked DLM to rival supposedly far improved successors. This differently sampled DLM in fact achieves lower generative perplexity in just 16 steps than what its standard sampler obtains with 1024, while improving both judged quality and semantic diversity. We further show that conventional per-output metrics can fundamentally obscure these gains, since any optimal trade-off between two such metrics can be attained by a generator supported on at most two outputs. We subsequently introduce GroupEval, which separately evaluates quality and across-output semantic diversity, and offers fresh insights including uncovering how 1.5-4.7x perplexity gains of a distilled model yield no corresponding quality gain. Finally, we explain why sharpening helps: parallel unmasking destroys dependencies among simultaneously generated tokens, creating a gap between prediction and generation. We prove that pervasive temperature choice of one is generically suboptimal under parallel sampling even for an exact denoiser, and that worse predictions can yield better samples. Through these results, we argue for a broader evaluation principle of treating the deployed generator as the object of comparison, benchmarking it against tuned baselines, and assessing quality and diversity jointly and with more human-aligned measures.

Figures & tables

Figure 1: Same weights, sharper sampler result in a better generator from a worse predictor. (a) The unchanged public MDLM, sampled as MDLM ♯

(\tau{=}.9,\mathrm{fp32})

instead of the standard

(\tau{=}1,\mathrm{fp64})

, rivals distilled successors evaluated under one protocol. (b) Under the default GroupEval judge, quality and semantic diversity both rise despite fall in token entropy. (c) Across three checkpoints and five budgets, sharpening to

\tau=.9

worsens held-out posterior cross-entropy in all 15 cases (paired 95% intervals above zero) while cutting gPPL

2.6

–

3.8\times

. Section 4 explains the prediction–generation gap.

Steps $M$	Standard $(1,\mathrm{fp64})$	$(1,\mathrm{fp32})$	$(.9,\mathrm{fp64})$	MDLM ♯ $(.9,\mathrm{fp32})$
16	357.6 [5.79]	338.7 [5.78]	109.8 [5.37]	102.2 [5.35]
64	154.9 [5.71]	107.6 [5.62]	53.1 [5.36]	41.3 [5.27]
256	115.5 [5.66]	61.5 [5.45]	40.9 [5.31]	28.1 [5.13]
1024	107.1 [5.63]	42.5 [5.33]	36.6 [5.26]	21.8 [5.01]

Table 1: Two sampling knobs compound: at 64 steps, MDLM ♯ decisively outperforms the standard sampler at 1,024 (bold). GPT-2 gPPL (lower is better) with unigram entropy

H

in brackets, 128 samples per cell. Temperature and precision each help on their own.

Figure 2: At the standard sampler, more training and more parameters make few-step generation worse. Ratio of gPPL for the larger or longer-trained model to the smaller or shorter-trained one, at matched steps and sampler. Values above one (shaded) mean that less is more. (a) MDLM-Small after 915k versus 20.8k training steps. (b) MDLM-Small versus MDLM-Tiny. Less wins only at 2–8 steps, and these inversions shrink under the sharpened sampler

(.9,\mathrm{fp32})

and vanish with stronger sharpening. GPT-2 gPPL; 128 (Small) and 256 (Tiny) samples per cell.

(A) Public MDLM-Small: standard $\to$ MDLM ♯
Steps	gPPL $\downarrow$	Token entropy $H$	GEQ $\uparrow$	GED $\uparrow$
16	$357.6\to 102.2$	$5.79\to 5.35$	$3.85\to 4.36$	$6.00\to 7.57$
64	$154.9\to\phantom{0}41.3$	$5.71\to 5.27$	$4.69\to 5.00$	$8.28\to 8.79$
256	$115.5\to\phantom{0}28.1$	$5.66\to 5.13$	$4.92\to 5.99$	$8.67\to 9.02$
1024	$107.1\to\phantom{0}21.8$	$5.63\to 5.01$	$5.15\to 5.47$	$8.07\to 8.75$

Table 2: GEQ and GED reveal what gPPL and token entropy miss. Default judge; groups of eight. (A) Public MDLM, standard

\to

MDLM ♯ , 128 outputs per cell; GPT-2 gPPL. (B) MDLM minus DiDi-Instruct in a separate, matched, interleaved study, 40 outputs per model and cell; the same saved outputs receive GroupEval and GPT-2-Large scores. gPPL ratios above one favor DiDi.

Figure 3: Training and generation want different temperatures, even for a perfect model. (a) The sampler as a mixture over reveal paths, for the pair family ( 6 ) with $s=.7$ and two equal rounds. Cells give the joint law of $(x_{1},x_{2})$ . At $\lambda=1$ , the independent same-round branch dilutes the exact sequential branch and the output is too random. At $\lambda_{\star}\approx 2.59$ , the sequential branch over-agrees just enough to offset it. (b) The Bayes posterior minimizes posterior loss, but output error vanishes only at $\lambda_{\star}$ . (c) Best inverse temperature for the exact Bayes denoiser on 280 random full-support targets ( $L\in\{2,3\}$ , $|\mathcal{V}|\leq 4$ , random grids), by enumeration: never $1$ , and above $1$ in all but one case.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

(a) Public MDLM-Small ( Sahoo et al., 2024 ) (the canonical baseline).
$M$	$(1,\mathrm{fp64})$	$(1,\mathrm{fp32})$	$(.9,\mathrm{fp64})$	$(.9,\mathrm{fp32})$	$(.85,\mathrm{fp32})$
$1$	$\underline{5043.3}$ [5.83]	$5001.7$ [5.83]	$\mathbf{2091.2}$ [5.27]	$2148.8$ [5.26]	$1306.6^{\dagger}$ [4.91]
$2$	$\underline{3893.6}$ [5.93]	$3725.3$ [5.93]	$1293.3$ [5.29]	$\mathbf{1224.3}$ [5.27]	$675.8^{\dagger}$ [4.84]
$8$	$\underline{945.1}$ [5.89]	$853.9$ [5.87]	$247.2$ [5.34]	$\mathbf{235.5}$ [5.31]	$131.6^{\dagger}$ [4.98]
$32$	$\underline{196.8}$ [5.73]	$158.7$ [5.70]	$66.0$ [5.35]	$\mathbf{58.0}$ [5.34]	$40.0$ [5.13]
$128$	$\underline{120.2}$ [5.67]	$80.2$ [5.55]	$43.7$ [5.33]	$\mathbf{32.3}$ [5.21]	$22.9$ [5.01]

Appendix

Table 3: A stronger unchanged baseline across checkpoints and budgets. Three MDLM-Small checkpoints, 128 unconditional OpenWebText samples per cell; entries are GPT-2 gPPL with unigram entropy

H

in brackets. The first four columns of settings form the four-corner audit; the last shows additional sharpening. Underlined : standard sampler. Bold : best of the four corners based on both gPPL and

H

\dagger

H<5

. The standard sampler has the largest gPPL in every listed checkpoint–budget cell;

(\tau=.9,\mathrm{fp32})

is best among the four corners for every

M\geq 2

. At

M=1024

it improves public-checkpoint gPPL by

4.9\times

while passing the guard.

M

counts configured updates, not NFE.

Figure 4: MDLM-Tiny shows the same sampler gap at smaller scale. Late MDLM-Tiny checkpoint; standard

(\tau{=}1,\mathrm{fp64})

, dashed; sharpened

(\tau{=}.9,\mathrm{fp32})

, solid. The gap mirrors the MDLM-Small headline ( Table 1 ).

fp64 categorical sampling.
$M$	$\tau{=}0.8$	$\tau{=}0.85$	$\tau{=}0.9$	$\tau{=}0.95$	$\tau{=}1.0$	$\tau{=}1.05$	$\tau{=}1.10$
$1$	$786.0$ $\dagger$ [4.55]	$1276$ $\dagger$ [4.92]	$2050$ [5.27]	$3246$ [5.58]	$\mathbf{4961}$ [5.84]	$7231$ [6.05]	$10037$ [6.23]
$2$	$428.7$ $\dagger$ [4.57]	$732.8$ $\dagger$ [4.95]	$1263$ [5.32]	$2132$ [5.64]	$\mathbf{3496}$ [5.91]	$5464$ [6.13]	$8034$ [6.31]
$4$	$199.9$ $\dagger$ [4.63]	$323.9$ $\dagger$ [4.99]	$561.9$ [5.33]	$975.9$ [5.62]	$\mathbf{1661}$ [5.87]	$2807$ [6.10]	$4607$ [6.28]
$8$	$99.0$ $\dagger$ [4.76]	$154.0$ [5.07]	$262.9$ [5.36]	$440.5$ [5.61]	$\mathbf{734.1}$ [5.81]	$1283$ [6.01]	$2261$ [6.19]
$16$	$59.9$ $\dagger$ [4.88]	$93.1$ [5.14]	$151.3$ [5.37]	$253.7$ [5.59]	$\mathbf{435.1}$ [5.78]	$748.9$ [5.95]	$1371$ [6.13]

Appendix

Table 4: MDLM-Tiny late checkpoint, full operating-point grid. The standard

(\tau{=}1,\mathrm{fp64})

and sharpened

(\tau{=}0.9,\mathrm{fp32})

cells are in bold ;

\dagger

marks

H<5

fp64 categorical sampling.
$M$	$\tau{=}0.8$	$\tau{=}0.85$	$\tau{=}0.9$	$\tau{=}0.95$	$\tau{=}1.0$	$\tau{=}1.05$	$\tau{=}1.10$
$1$	$759.3$ $\dagger$ [4.52]	$1244$ $\dagger$ [4.90]	$1992$ [5.25]	$3135$ [5.55]	$\mathbf{4780}$ [5.82]	$6957$ [6.04]	$9722$ [6.21]
$2$	$426.1$ $\dagger$ [4.51]	$734.8$ $\dagger$ [4.92]	$1280$ [5.31]	$2181$ [5.64]	$\mathbf{3634}$ [5.93]	$5780$ [6.16]	$8385$ [6.34]
$4$	$179.8$ $\dagger$ [4.54]	$333.0$ $\dagger$ [4.96]	$600.2$ [5.34]	$1075$ [5.66]	$\mathbf{1931}$ [5.95]	$3344$ [6.18]	$5612$ [6.38]
$8$	$86.3$ $\dagger$ [4.67]	$141.0$ [5.03]	$236.4$ [5.35]	$421.5$ [5.62]	$\mathbf{795.6}$ [5.88]	$1561$ [6.10]	$2938$ [6.30]
$16$	$45.3$ $\dagger$ [4.82]	$66.8$ [5.12]	$107.6$ [5.38]	$186.6$ [5.59]	$\mathbf{333.9}$ [5.78]	$615.9$ [5.97]	$1262$ [6.17]

Appendix

Table 5: MDLM-Small late checkpoint, full operating-point grid. The standard

(\tau{=}1,\mathrm{fp64})

and sharpened

(\tau{=}0.9,\mathrm{fp32})

cells are in bold ;

\dagger

marks

H<5

fp64 categorical sampling.
$M$	$\tau{=}0.8$	$\tau{=}0.85$	$\tau{=}0.9$	$\tau{=}0.95$	$\tau{=}1.0$	$\tau{=}1.05$	$\tau{=}1.10$
$1$	$782.9$ $\dagger$ [4.48]	$1251$ $\dagger$ [4.87]	$2007$ [5.22]	$3162$ [5.54]	$\mathbf{4941}$ [5.80]	$7116$ [6.03]	$9854$ [6.20]
$2$	$431.3$ $\dagger$ [4.55]	$703.7$ $\dagger$ [4.94]	$1212$ [5.31]	$2022$ [5.62]	$\mathbf{3278}$ [5.89]	$5088$ [6.11]	$7647$ [6.29]
$4$	$189.0$ $\dagger$ [4.72]	$305.5$ [5.07]	$507.2$ [5.38]	$835.1$ [5.63]	$\mathbf{1376}$ [5.86]	$2313$ [6.07]	$3891$ [6.25]
$8$	$96.1$ $\dagger$ [4.90]	$144.2$ [5.17]	$235.2$ [5.41]	$365.8$ [5.61]	$\mathbf{605.9}$ [5.80]	$1024$ [5.99]	$1801$ [6.15]
$16$	$55.3$ $\dagger$ [4.94]	$85.3$ [5.19]	$131.1$ [5.38]	$211.5$ [5.57]	$\mathbf{343.1}$ [5.73]	$581.9$ [5.91]	$1052$ [6.07]

Appendix

Table 6: MDLM-Small early checkpoint, full operating-point grid. The standard

(\tau{=}1,\mathrm{fp64})

and sharpened

(\tau{=}0.9,\mathrm{fp32})

cells are in bold ;

\dagger

marks

H<5

fp64 categorical sampling.
$M$	$\tau{=}0.8$	$\tau{=}0.85$	$\tau{=}0.9$	$\tau{=}0.95$	$\tau{=}1.0$	$\tau{=}1.05$	$\tau{=}1.10$
$1$	$506.6$ $\dagger$ [4.44]	$832.6$ $\dagger$ [4.79]	$1393$ [5.15]	$2268$ [5.46]	$\mathbf{3371}$ [5.72]	$4958$ [5.95]	$7007$ [6.13]
$2$	$288.3$ $\dagger$ [4.26]	$483.8$ $\dagger$ [4.70]	$866.1$ [5.12]	$1538$ [5.50]	$\mathbf{2487}$ [5.79]	$3958$ [6.05]	$6114$ [6.25]
$4$	$110.4$ $\dagger$ [4.38]	$160.8$ $\dagger$ [4.80]	$241.1$ [5.10]	$367.1$ [5.36]	$\mathbf{532.5}$ [5.56]	$757.3$ [5.72]	$1168$ [5.90]
$8$	$46.07$ $\dagger$ [4.72]	$57.75$ $\dagger$ [4.94]	$77.23$ [5.15]	$106.9$ [5.31]	$\mathbf{148.2}$ [5.46]	$192.7$ [5.58]	$281.9$ [5.72]
$16$	$25.82$ $\dagger$ [4.87]	$32.36$ [5.02]	$40.25$ [5.17]	$52.63$ [5.30]	$\mathbf{69.50}$ [5.41]	$91.69$ [5.53]	$124.4$ [5.63]

Appendix

Table 7: DiDi-Instruct, unguided ancestral sampling. GPT-2-Large gPPL [

H

], 40 samples per cell. Bold: standard

(1,\mathrm{fp64})

and sharpened

(.9,\mathrm{fp32})

;

\dagger

H<5

Panel of Figure 5 — Figure 5: The sampler exposes an empirical quality–diversity–cost frontier. Left: gPPL and generated-sequence $H$ show why neither metric alone suffices. Right: cost is measured on the same H100 protocol for the two displayed operating points.

Model	$M{=}4$	$M{=}16$	$M{=}64$	$M{=}128$	$M{=}256$	$M{=}1024$
MDLM-Small	1948.8 [5.95]	344.2 [5.79]	145.4 [5.71]	117.7 [5.66]	109.9 [5.64]	103.8 [5.64]
MDLM ♯ (same checkpoint)	519.4 [5.29]	103.6 [5.36]	44.5 [5.30]	33.3 [5.23]	30.1 [5.15]	21.4 $\dagger$ [4.98]
DUO	518.3 [5.55]	126.2 [5.59]	86.2 [5.60]	86.2 [5.57]	70.1 [5.54]	85.6 [5.57]
DUO-DCD	314.9 [5.51]	83.1 [5.55]	66.0 [5.52]	62.0 [5.50]	51.2 [5.46]	65.7 [5.47]
SEDD	1863.3 [5.91]	385.7 [5.81]	146.5 [5.69]	131.9 [5.66]	100.1 [5.62]	114.0 [5.65]
DiDi-Instruct	523.6 [5.56]	71.6 [5.43]	37.3 [5.37]	34.5 [5.35]	30.9 [5.33]	30.1 [5.31]

Appendix

Table 9: GPT-2-Large gPPL [within-sequence entropy] in one consolidated 40-sample stress test.

M

is configured reverse updates, not NFE; each official sampler uses its terminal-noise-removal convention. Sharpened rows fix

(\tau{=}.9,\mathrm{fp32})

; all other rows use the standard

(\tau{=}1,\mathrm{fp64})

sampler. DUO-DCD is the distilled DUO model.

\dagger

H<5

. MDLM uses the public checkpoint; DiDi uses the EMA student and reward-guided sampler.

Figure 6: Sharpened samplers stay near the

H{=}5

guard. Within-sequence entropy versus configured updates in the cross-model stress test. The dashed line marks

H=5

and the dotted line the autoregressive GPT-2 generator; MDLM ♯ and sharpened DiDi-Instruct cross the guard slightly at the largest budgets.

Checkpoint	$M{=}2$	$M{=}8$	$M{=}32$	$M{=}128$	$M{=}1024$
Public	$+.038\ [.011,.064]$	$+.052\ [.029,.076]$	$+.045\ [.021,.068]$	$+.051\ [.028,.073]$	$+.036\ [.014,.058]$
Early	$+.047\ [.021,.075]$	$+.055\ [.030,.081]$	$+.041\ [.016,.068]$	$+.068\ [.042,.094]$	$+.037\ [.012,.061]$
Late	$+.041\ [.014,.067]$	$+.056\ [.033,.081]$	$+.048\ [.024,.072]$	$+.055\ [.031,.078]$	$+.043\ [.020,.066]$

Appendix

Table 10: Sharpening worsens the schedule-matched posterior loss. Entries are

\{\mathcal{J}_{M+1}(\tau{=}.9)-\mathcal{J}_{M+1}(\tau{=}1)\}/L

in per-position nats with paired 95% intervals. Positive values mean worse teacher-forced cross-entropy. Every interval excludes zero; the temperature-grid optimum is

1

1.05

Figure 7: For large diffusion LMs, sharpened sampler with lower temperature improves downstream performance. LLaDA-8B-Instruct and Dream-v0-Base-7B on (a) GSM8K and (b) HumanEval. Greedy (temperature=0) is the default setting in official codebases.

Figure 8: Autoregressive control. Llama-3.1-8B-Instruct, Ministral-3-8B-Instruct, Qwen3.5-9B, and Gemma-4-12B-it on (a) GSM8K, through

\tau=2

, and (b) HumanEval, through

\tau=1

, after complete-program extraction. Greedy runs once, while stochastic markers show three-seed means

\pm 1

seed SEM.

GPT-OSS (default)
Steps $M$	GEQ $\uparrow$	$\Delta$ GEQ [95% interval]	GED $\uparrow$
4	$1.66\to 1.53$	$-0.12\ [-0.61,0.35]$	$0.79\to 1.11$
16	$3.85\to 4.36$	$+0.50\ [-0.08,1.08]$	$6.00\to 7.57$
64	$4.69\to 5.00$	$+0.31\ [-0.21,0.82]$	$8.28\to 8.79$
256	$4.92\to 5.99$	$\mathbf{+1.07\ [0.42,1.73]}$	$8.67\to 9.02$
1024	$5.15\to 5.47$	$+0.32\ [-0.20,0.85]$	$8.07\to 8.75$

Appendix

Table 11: MDLM improves in quality under all five judges at every primary budget. GEQ and GED are standard

\to

MDLM ♯ on the same 128 outputs per cell.

\Delta

GEQ is sharpened minus standard, with conditional 95% intervals; bold marks intervals excluding zero before rounding. All budgets, including the

M=4

floor case, are retained.

GPT-OSS (default)
Steps $M$	GEQ $\uparrow$	$\Delta$ GEQ [95% interval]	GED $\uparrow$
4	$1.59\to 1.75$	$+0.16\ [-0.68,1.00]$	$1.81\to 1.73$
16	$3.53\to 3.72$	$+0.19\ [-0.65,0.96]$	$2.45\to 4.20$
64	$4.48\to 4.48$	$+0.00\ [-0.75,0.70]$	$4.86\to 7.46$
256	$4.33\to 5.03$	$+0.70\ [-0.21,1.57]$	$5.25\to 7.63$
1024	$4.38\to 5.08$	$+0.70\ [-0.16,1.63]$	$6.04\to 7.51$

Appendix

Table 12: DiDi-Instruct’s perplexity gain does not yield a judge-robust quality gain. Same layout as Table 11 , with 40 outputs per cell. GPT-OSS and Terra have no resolved quality gain; Luna favors the standard sampler at two primary budgets.

$M$	Sampler	GPT-2-Large ratio	Qwen ratio	GEQ $\uparrow$	$\Delta$ GEQ [95% interval]	GED $\uparrow$
16	Std.	4.58	4.69	3.77 / 3.77	0.00 [-1.03, 1.02]	6.88 / 4.27
16	Sharp.	2.33	2.71	4.34 / 3.16	1.19 [0.41, 1.99]	6.61 / 1.96
64	Std.	3.78	3.68	4.30 / 4.09	0.20 [-0.85, 1.21]	8.02 / 5.27
64	Sharp.	2.19	2.42	4.64 / 4.23	0.41 [-0.26, 1.12]	9.24 / 7.45
256	Std.	3.49	3.58	5.70 / 4.20	1.50 [0.39, 2.65]	8.70 / 6.59
256	Sharp.	1.76	1.95	5.41 / 4.69	0.72 [-0.21, 1.64]	9.27 / 6.23

Appendix

Table 13: Likelihood and judged quality disagree on the same outputs. gPPL columns are MDLM / DiDi ratios, with ratio

>1

favoring DiDi. GEQ / GED entries are MDLM / DiDi, and

\Delta

GEQ is MDLM minus DiDi, with positive values favoring MDLM.

Task	Automatic success L / D	c GEQ L / D	$\Delta$ c GEQ D–L [95% CI]	AUROC [95% CI]	c GEQ (pass)–c GEQ (fail) [95% CI]
GSM8K	80.9 / 75.4%	9.54 / 8.99	-0.55 [-0.77, -0.34]	0.91 [0.87, 0.95]	3.04 [2.54, 3.53] (56)
HumanEval	40.2 / 51.2%	8.13 / 8.40	0.27 [-0.13, 0.65]	0.96 [0.94, 0.98]	3.07 [2.53, 3.66] (42)

Appendix

Table 14: Conditional GEQ tracks task success and the task-dependent model ordering. L / D: LLaDA / Dream. Intervals: 10,000-replicate condition-cluster bootstrap, conditional on one GPT-OSS judgment per response pair. AUROC pools both models; the last column uses conditions with exactly one passing response (counts in parentheses). Task-specific c GEQ is not comparable with unconditional GEQ ; GED is undefined with one response per model and condition.

Explore similar work

Jul 22, 2026cs.CL

Multi-Mask Diffusion Language Models for Few-Step Generation

Masked diffusion models (MDMs) are a promising family of language generators, but achieving high-quality few-step generation remains challenging. In MDMs, all forward trajectories collapse to a single fully masked state, leaving no terminal entropy for consistency-style few-step generation. While recent few-step alternatives based on uniform-state diffusion avoid this degeneracy, it becomes harder to distinguish clean tokens from noise than MDMs, which usually harms modeling quality and training efficiency. In this work, we propose a multi-mask diffusion model (MultiMDM) that preserves the masking structure towards few-step generation. In the forward process, each clean token is first pushed towards a designated mask and then gradually mixes over the mask set. As a result, the backward process has a drafting capability by predicting a designated mask before refining to a clean token. We derive a closed-form ELBO training objective for MultiMDM that supports continual training from pretrained MDMs. In addition, we formulate a purely discrete-state consistency distillation scheme, with a shared-Gumbel coupling to reduce pathwise entropy. Experiments on pretraining and distillation show that MultiMDM provides an effective foundation for principled few-step generation.

Sijin Chen, Yinuo Ren, Heyang Zhao +3

ByteDance Seed · Princeton University · Stanford University +2

Masked Diffusion Language Models Diffusion Language Models Few-Step Generation

Ask PDF

Jun 17, 2026cs.AI

Diffusion Language Models: An Experimental Analysis

Large Language Models (LLMs) have revolutionized language modeling through autoregressive generation, enabling strong performance across a wide range of tasks. Recently, Diffusion Language Models (DLMs) have emerged as an alternative paradigm that generates text through iterative denoising rather than next-token prediction, allowing parallel refinement of entire sequences. While numerous diffusion-based architectures have been proposed, differences in evaluation protocols, datasets, inference budgets, and generation hyperparameters make it difficult to compare their capabilities and understand the trade-offs they offer. In this work, we present a systematic experimental analysis of modern DLMs. Specifically, we evaluate eight state-of-the-art DLMs across eight benchmarks spanning reasoning, coding, translation, knowledge, and structured problem solving, while explicitly considering both generation quality and computational efficiency. Beyond downstream evaluation, we analyze the impact of key inference-time factors, including denoising steps, context length, block size, and parallel unmasking strategies, and complement large-scale experiments with controlled comparisons of smaller models trained under identical conditions. Our analysis highlights the strengths and limitations of diffusion-based language modeling across different tasks, architectures, and inference budgets. We show that the behavior of DLMs is strongly influenced by generation-time design choices, leading to distinct trade-offs between performance and computational efficiency. Overall, our study provides practical insights into the capabilities and deployment characteristics of contemporary DLMs.

Thomas Bertolani, Davide Bucciarelli, Leonardo Zini +2

University of Modena and Reggio Emilia, Italy · University of Pisa, Italy

Diffusion Language Models Large Language Model Benchmarks Diffusion Models Autoregressive Generation

Ask PDF

Sep 17, 2026cs.LG

Parallelism, critical windows, and separations among diffusion language models

A popular selling point of diffusion large language models (dLLMs) is their capacity for parallelism: the ability to generate sequences of text far more efficiently than autoregressive models, which require one forward pass per token. Yet among the many competing paradigms for dLLMs, from masked to uniform to Gaussian diffusion, principled understanding of how these different proposals compare in parallelism remains limited. In this work, we initiate a fine-grained comparison of the capacity for parallelism among these three leading approaches and prove the following: - Uniform and Gaussian diffusion can sample in a number of forward passes which scales with the dual total correlation of the underlying distribution, a measure of intrinsic complexity which can be much smaller than the context length. Previously, it was only known how to achieve this using masked diffusion. - For a certain family of random empirical measures, we show that

\widetildeΘ(\sqrt{d})

forward passes are necessary and sufficient to sample using uniform or Gaussian diffusion, yet there exist approximate score oracles for which

\widetildeΩ(d)

forward passes are needed for masked diffusion. This establishes the first provable separation in parallelism between the three prevailing dLLM paradigms. Contrary to popular intuition that masked diffusions are harder to parallelize because they must commit to token values, the latter separation instead comes from the fact that the critical windows in masked diffusion sampling are asymptotically narrower than those in uniform and Gaussian diffusion sampling.

Sitan Chen, Liye Wang

SEAS, Harvard University · Tsinghua University

Diffusion Language Models Diffusion Sampling Structural Complexity

Ask PDF