DUEL: Exact Likelihood for Masked Diffusion via Deterministic Unmasking

Gilad Turok; Chris De Sa; Volodymyr Kuleshov

cs.LGMar 2, 2026

DUEL: Exact Likelihood for Masked Diffusion via Deterministic Unmasking

Authors: Gilad Turok, Chris De Sa, Volodymyr Kuleshov

Organizations: Department of Computer Science, Cornell University, New York, New York, USA.

Abstract

Masked diffusion models (MDMs) generate text by iteratively selecting positions to unmask and then predicting tokens at those positions. Yet MDMs lack proper likelihood evaluation: the evidence lower bound (ELBO) is not only a loose bound on log-likelihood, but, as we show, is also computed under the training distribution rather than the test-time distribution. We resolve this within our DUEL framework, which unifies leading MDM sampling strategies that employ $\textit{deterministic}$ position selection. We prove that DUEL samplers admit $\textbf{exact likelihood computation under the test-time distribution}$ -- giving MDMs $\textit{proper}$ likelihood, and hence proper perplexity, for the first time. This proper perplexity is the natural analogue of autoregressive perplexity and lets us revisit key questions about MDMs. $\textbf{MDMs are substantially better than previously thought}$ : the MDM-autoregressive perplexity gap shrinks by up to $32\%$ on in-domain data and $82\%$ on zero-shot benchmarks. DUEL enables the first principled comparison of fast,parallel samplers across compute budgets -- an analysis impossible with the ELBO and unreliable with generative perplexity -- identifying a strong default method. Finally, oracle search over position orderings reveals MDMs can far surpass autoregressive models -- achieving $36.47$ vs. $52.11$ perplexity on AG News -- demonstrating the ceiling of MDM performance has not yet been reached.

Figures & tables

Figure 1: One step of MDM generation. Given a partially masked sequence, the unmasking policy

\pi

performs position selection (choosing position 2), and the denoising network

p_{\theta}

performs token prediction at that position (sampling “big”). This repeats until no masked positions remain.

Unmask Rule $F$	Selected Positions $F(\mathbf{z})$
Left-to-Right	$\{k\text{ smallest }\ell:z_{\ell}=\mathtt{m}\}$
Greedy Conf.	$\arg\max_{\|\mathcal{I}\|=k}\sum_{\ell\in\mathcal{I}}P_{\ell}^{(1)}$
Prob. Margin	$\arg\max_{\|\mathcal{I}\|=k}\sum_{\ell\in\mathcal{I}}(P_{\ell}^{(1)}-P_{\ell}^{(2)})$
Conf. Thresh.	$\{\ell\in\mathcal{M}:P_{\ell}^{(1)}\geq\mu\}$
KLASS	$\{\ell\in\mathcal{M}:P_{\ell}^{(1)}\geq\mu,\;\mathrm{KL}(\hat{P}_{\ell}\\|P_{\ell})\leq\nu\}$

Table 1: Unmasking rules defining deterministic policies.

\mathcal{M}=\mathcal{M}(\mathbf{z})

denotes masked positions;

\mathbf{P}=\mathrm{softmax}(x_{\theta}(\mathbf{z}))

is the token probability matrix.

Figure 2: Unmasking trajectory for

\sigma=(\{1,3\},\{2\},\{4\})

. Starting from a fully masked sequence (

t{=}0

), positions are progressively revealed according to the ordered partition. At

t{=}1

, positions 1 and 3 are unmasked in parallel; subsequent steps unmask one position each.

Figure 3: DUEL sampling and exact likelihood computation. Highlights mark the only differences. Both algorithms iterate with a denoising network

x_{\theta}

and unmasking rule

F

: compute token probabilities

\mathbf{P}=\mathrm{softmax}(x_{\theta}(\mathbf{z}))

, select positions via

F(\mathbf{z})

, then either sample tokens or accumulate log-probabilities. Section 4 proves that Algorithm 2 computes the exact likelihood under the distribution induced by

F

Model	ELBO	DUEL	Gap Closed
SEDD	$\leq$ 24.10	22.58	23.2%
MDLM	$\leq$ 22.98	21.86	20.6%
BD3-LM $L^{\prime}$ =4	$\leq$ 20.73	19.73	31.3%
BD3-LM $L^{\prime}$ =8	$\leq$ 21.68	20.37	31.6%
BD3-LM $L^{\prime}$ =16	$\leq$ 22.27	20.76	31.9%

Table 2: In-domain test perplexity (

\downarrow

) on OWT. ARM baseline: 17.54. Bold : best MDM. Gap closed defined in Equation 7 .

Model	ELBO	DUEL	Gap Closed
SEDD	$\leq$ 32.68	31.51	11.9%
MDLM	$\leq$ 31.78	29.03	30.7%
BD3-LM $L^{\prime}$ =4	$\leq$ 28.23	26.97	23.3%
BD3-LM $L^{\prime}$ =8	$\leq$ 29.83	27.85	28.3%
BD3-LM $L^{\prime}$ =16	$\leq$ 30.60	28.39	28.4%

Table 3: In-domain test perplexity (

\downarrow

) on LM1B. ARM baseline: 22.83. Bold : best MDM. Gap closed defined in Equation 7 .

Dataset	SEDD	MDLM	BD3-LM
PTB	31.3%	34.3%	81.8%
Wikitext	40.8%	28.5%	31.4%
LM1B	22.1%	29.0%	29.8%
AG News	25.7%	27.8%	51.7%
Average	30.0%	29.9%	48.7%

Table 4: Gap closed (%) by DUEL on zero-shot evaluation ( Equation 7 ). Models trained on OWT; BD3-LM uses

L^{\prime}=4

. Full perplexity values in Table 8 .

Model	Method	Wikitext	Lambada	AG News
Llama3	Exact	7.94	32.40	41.29
LLaDA	ELBO	$\leq$ 15.29	$\leq$ 39.04	$\leq$ 85.17
LLaDA	DUEL	14.50	36.00	78.91

Table 5: Zero-shot perplexity (

\downarrow

) for 8B models. Bold : better of ELBO/ DUEL .

Unmask Rule $F$	128	256	512	1024
Left-to-Right	240.27	109.71	45.99	21.46
Greedy Conf.	164.94	66.64	34.74	22.03
Prob. Margin	140.38	57.48	32.24	22.05
Conf. Thresh. †	226.97	116.83	43.48	22.05

Table 6: DUEL perplexity (

\downarrow

) by unmasking rule on OWT. Model: BD3-LM (

L^{\prime}

=16). ELBO:

\leq

23.52. Bold : best per NFE.

Model	Method	Unmask Rule $F$	Perplexity
ARM	Exact	—	52.11
BD3-LM	ELBO	—	$\leq$ 61.67
BD3-LM	DUEL	Left-to-Right	54.94
	DUEL	Greedy Conf.	56.73
	DUEL	Prob. Margin	57.80
	DUEL	Conf. Thresh. †	55.68

Table 7: Oracle perplexity (

\downarrow

) on AG News for BD3-LM (

L^{\prime}=4

). All DUEL evaluations use block-restricted unmasking with

L^{\prime}=4

. The oracle searches all

4!=24

block permutations.

Figure 4: Comparing fast samplers. Top: DUEL perplexity. Bottom: Generative perplexity. Dashed line: ELBO (

\leq

23.52). DUEL yields consistent rankings across NFE (probability margin best at low NFE, convergence at high NFE). Generative perplexity rankings cross repeatedly, making it unreliable.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Method	PTB	Wikitext	LM1B	Lambada	AG News	PubMed	ArXiv
ARM	Exact	81.07	25.32	51.14	52.13	52.11	48.59	41.22
SEDD	ELBO	$\leq$ 96.33	$\leq$ 35.98	$\leq$ 68.14	$\leq$ 48.93	$\leq$ 67.82	$\leq$ 43.13	$\leq$ 37.89
SEDD	DUEL	91.56	31.63	64.39	46.01	63.78	39.31	35.48
MDLM	ELBO	$\leq$ 90.96	$\leq$ 33.22	$\leq$ 64.94	$\leq$ 48.29	$\leq$ 62.78	$\leq$ 43.13	$\leq$ 37.89
MDLM	DUEL	87.57	30.97	60.94	48.42	59.81	41.96	37.57
BD3-LM $L^{\prime}$ =4	ELBO	$\leq$ 96.81	$\leq$ 31.31	$\leq$ 60.88	$\leq$ 50.03	$\leq$ 61.67	$\leq$ 42.52	$\leq$ 39.20

Appendix

Table 8: Zero-shot perplexity (

\downarrow

) for models trained on OWT, evaluated on unseen datasets. ARM baseline in first row. Bold : better of ELBO/ DUEL per model-dataset pair.

Figure 5: Comparing fast samplers across four metrics. Dashed line in DUEL perplexity: ELBO (

\leq

23.52). DUEL perplexity yields consistent rankings across NFE (probability margin best at low NFE), while generative perplexity rankings cross repeatedly. Entropy reveals that left-to-right produces degenerate low-entropy text at low NFE despite achieving favorable generative perplexity. MAUVE saturates near zero at low NFE for all rules, providing little discriminative signal.

Explore similar work

Understanding and Accelerating the Training of Masked Diffusion Language Models
May 13, 2026Chunsan Hong, Sanghyun Lee, Chieh-Hsin Lai +5Masked Diffusion Models Diffusion Model Acceleration
1KAIST · 2Sony AI · University of Tokyo +1
Improving Sampling for Masked Diffusion Models via Information Gain
Feb 20, 2026Kaisen Yang, Jayden Teoh, Kaicheng Yang +2Diffusion Sampling Diffusion Language Model Decoding
Department of Computer Science and Technology, Tsinghua University · Massachusetts Institute of Technology · Shanghai Jiao Tong University +2
Scaling Beyond Masked Diffusion Language Models
Feb 16, 2026Subham Sekhar Sahoo, Jean-Marie Lemercier, Zhihan Yang +4Diffusion Language Models Diffusion Models
NVIDIA, Santa Clara, USA · Department of Computer Science, Cornell Tech, NYC, USA · Department of Computer Science, Cornell University, Ithaca, USA +2

May 13, 2026J/K move · Enter open · S save

Understanding and Accelerating the Training of Masked Diffusion Language Models

Chunsan Hong, Sanghyun Lee, Chieh-Hsin Lai, Satoshi Hayakawa+4

1KAIST · 2Sony AI · University of Tokyo+1

Masked Diffusion Models Diffusion Model Acceleration Diffusion Language Models

Open paper AskPDF arXiv

Masked diffusion models (MDMs) have emerged as a promising alternative to autoregressive models (ARMs) for language modeling. However, MDMs are known to learn substantially more slowly than ARMs, which may become problematic when scaling MDMs to larger models. Therefore, we ask the following question: how can we accelerate standard MDM training while maintaining its final performance? To this end, we first provide a detailed analysis of why MDM training is slow. We find that the main factor is the locality bias of language: the predictive information for a token is concentrated in nearby positions. We further investigate how this bias slows learning and suggest a simple yet effective remedy: bell-shaped time sampling as a training strategy. Notably, MDMs trained with our training recipe reach the same validation negative log-likelihood (NLL) up to

\sim4\times

faster than standard training on One Billion Word Benchmark (LM1B). We also show faster improvements in generative perplexity, zero-shot perplexity, and downstream task performance on various benchmarks.

Abstract

Explore similar work

Understanding and Accelerating the Training of Masked Diffusion Language Models

Improving Sampling for Masked Diffusion Models via Information Gain

Scaling Beyond Masked Diffusion Language Models