Is Your Diffusion Sampler Actually Correct? A Sampler-Centric Evaluation of Discrete Diffusion Language Models

Luhan Tang; Longxuan Yu; Shaorong Zhang; Greg Ver Steeg

cs.LGFeb 23, 2026

Is Your Diffusion Sampler Actually Correct? A Sampler-Centric Evaluation of Discrete Diffusion Language Models

Authors: Luhan Tang, Longxuan Yu, Shaorong Zhang, Greg Ver Steeg

Organizations: University of California, Riverside

Abstract

Discrete diffusion language models (dLLMs) provide a fast and flexible alternative to autoregressive models (ARMs) via iterative denoising with parallel updates. However, their evaluation is challenging: existing metrics conflate denoiser approximation error with sampler-induced error from the sampling dynamics, a problem that does not arise for ARMs whose autoregressive sampling exactly reflects the learned probability model. We introduce a sampler-centric oracle framework that replaces learned denoisers with an exact Hidden Markov Model posterior derived from a ground-truth Markov chain, isolating sampler-induced error in a controlled setting. We show that few-step discrete diffusion samplers are not distributionally correct even under an oracle denoiser, with transition-level mismatch that vanishes only as the number of steps approaches the sequence length. Moreover, improvements in negative log-likelihood (NLL), generative perplexity (GenPPL), or MAUVE do not imply correct sampling. Code is available at https://luhantang.github.io/dllm_sampler

Figures & tables

Figure 1: Overview of our sampler-centric oracle evaluation framework for dLLMs. We construct a controlled oracle world using a ground-truth Markov chain and exact HMM posterior inference to isolate sampler-induced error. Different samplers (SEDD, MDLM, LLaDA, and ReMDM) are then evaluated under the same oracle posterior using transition-level correctness, diversity, and external metrics.

Figure 2: Transition-level metrics on OpenWebText (OWT) under an oracle denoiser. We report transition KL, NLL, entropy rate, and $n$ -gram diversity as functions of the number of sampling steps for several discrete diffusion samplers, using an exact oracle posterior to isolate sampler-induced error. For all samplers, substantial transition-level error persists at small step counts, with convergence to the autoregressive baseline occurring only when the number of steps approaches the sequence length $T$ (except LLaDA). Notably, LLaDA attains very low sequence NLL at few steps, despite severe degradation in transition KL, entropy, and diversity, demonstrating that NLL alone is not a reliable indicator of distributional correctness. All samplers show zero exact duplication.

Figure 3: Temperature-induced score sharpening in SEDD under an oracle denoiser (OWT). As $\beta$ increases, transition KL rises and 3-gram diversity falls, while sequence NLL and GenPPL decrease. MAUVE drops sharply at low temperatures, indicating severe distributional degradation. This shows that likelihood-style metrics can improve under score sharpening even as transition-level correctness and diversity degrade. Duplication remains negligible except under extreme sharpening (Appendix F.2 ).

Figure 4: GenPPL under controlled local sharpening in an autoregressive bigram generator (OWT). We introduce a sharpening factor $\beta$ ( $\beta=1$ corresponds to accurate sampling) and evaluate fixed samples using a pretrained GPT-2 Large model. As $\beta$ increases, GenPPL decreases monotonically, while 3-gram diversity collapses and sentence entropy steadily declines, indicating increasing concentration of probability mass. Exact duplication remains negligible until extreme sharpening.

Figure 5: Step-wise evaluation of oracle ReMDM variants on OWT. Across transition-level metrics, the ReMDM-loop sampler exhibits larger deviations from the AR baseline than ReMDM-conf, indicating higher sampler error, while ReMDM-conf remains consistently closer to the oracle transition kernel. Applying nucleus sampling reduces transition KL, NLL, and entropy, indicating improved alignment with the oracle kernel, but decreases the support fraction due to truncation of low-probability tail transitions. External MAUVE scores remain high and vary only mildly across diffusion steps. A full breakdown of all reported metrics is provided in Appendix Figure 9 .

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Distribution of effective sparsity $k_{i}^{*}$ in OpenWebText (OWT) under the 99% cumulative mass criterion. The distribution is strongly right-skewed: most tokens require only a small number of successors, while a small fraction exhibit heavy-tailed behavior. The chosen global sparsity level $K=206$ corresponds to the 90th percentile of this distribution.

Dataset	Type	Model	Steps	Seed	NLL	KL rate	TV rate	Ent rate	Unigram L1	2-gram Diversity	3-gram Diversity	Other mass	Support frac
OpenWebText (BPE-4096)	Baseline	AR	—	—	4.1179	0.2454	0.2012	3.8724	0.0489	0.9459	0.9975	0.0001	0.2380
OpenWebText (BPE-4096)	Diffusion	LLaDA	8	123	4.3019	1.3182	0.4458	2.9837	0.9113	0.6351	0.9092	0.0855	0.1070
OpenWebText (BPE-4096)	Diffusion	LLaDA	16	123	3.5183	1.1849	0.5342	2.3334	1.1441	0.4561	0.7321	0.0442	0.0654
OpenWebText (BPE-4096)	Diffusion	LLaDA	32	123	3.1247	1.1996	0.5947	1.9251	1.2824	0.3548	0.5941	0.0251	0.0414
OpenWebText (BPE-4096)	Diffusion	LLaDA	64	123	2.9198	1.2414	0.6294	1.6784	1.3543	0.2940	0.5054	0.0154	0.0265
OpenWebText (BPE-4096)	Diffusion	LLaDA	128	123	2.8082	1.2840	0.6507	1.5243	1.3959	0.2555	0.4484	0.0104	0.0172

Appendix

Table 1: Metrics on OpenWebText (BPE-4096) for five samplers/models (AR, SEDD, ReMDM, MDLM, LLaDA). We report a collection of transition- and sample-level metrics (e.g., NLL/tok, KL/TV/entropy rates,

n

-gram diversity, duplication rate, other-mass, and support fraction).

Dataset	Type	Model	Steps	Seed	NLL	KL rate	TV rate	Ent rate	Unigram L1	2-gram Diversity	3-gram Diversity	Support frac
Text8 (Char)	Baseline	AR	—	—	2.3780	0.0026	0.0181	2.3754	0.0119	0.2394	0.6949	0.0042
Text8 (Char)	Diffusion	LLaDA	8	123	3.4649	1.9778	0.4448	1.4871	0.7657	0.1018	0.2489	0.0038
Text8 (Char)	Diffusion	LLaDA	16	123	2.8683	1.7279	0.5148	1.1404	0.8939	0.0720	0.1588	0.0027
Text8 (Char)	Diffusion	LLaDA	32	123	2.6452	1.7054	0.5551	0.9398	0.9411	0.0540	0.1105	0.0018
Text8 (Char)	Diffusion	LLaDA	64	123	2.4194	1.6167	0.5883	0.8028	0.9664	0.0395	0.0784	0.0010
Text8 (Char)	Diffusion	LLaDA	128	123	2.1635	1.4670	0.6071	0.6966	0.9836	0.0307	0.0596	0.0007

Appendix

Table 2: Metrics on Text8 (Char) for five samplers/models (AR, SEDD, ReMDM, MDLM, LLaDA). We report a collection of transition- and sample-level metrics (e.g., NLL/tok, KL/TV/entropy rates,

n

-gram diversity, duplication rate, other-mass, and support fraction).

Figure 7: Transition-level overview on Text8 (top row) and OpenWebText (bottom row). Despite differences in vocabulary size and tokenization granularity, both datasets exhibit the same qualitative trends: SEDD, MDLM, and ReMDM converge toward the AR baseline with increasing diffusion steps, whereas LLaDA shows persistent deviation and entropy collapse.

Figure 8: Behavior of SEDD under large temperature scaling factors on Text8. As $\beta$ increases beyond the moderate regime, the sampler exhibits qualitatively different behaviors. For intermediate values (e.g., $\beta\!\approx\!10$ ), transition KL and NLL become large at small diffusion steps and entropy and 3-gram diversity drop sharply, indicating near-deterministic but non-repeating transition dynamics. For larger values ( $\beta\!\geq\!20$ ), the sampler enters a degenerate regime characterized by entropy collapse and transient exact duplication at small diffusion steps, followed by reduced duplication due to repeated masking and resampling. These extreme settings illustrate the limiting behavior of temperature scaling and are not intended to model realistic language generation.

Figure 9: Step-wise evaluation of oracle ReMDM variants on OpenWebText (OWT). We report transition-level metrics (KL, per-token NLL, entropy, support fraction), surface statistics (3-gram diversity, duplication rate), and external language-model metrics (MAUVE and GenPPL) across diffusion steps. ReMDM-conf remains consistently closer to the oracle transition kernel, while ReMDM-loop exhibits larger transition-level deviations. Nucleus (top- $p$ ) sampling reduces KL, NLL, and entropy but decreases support fraction due to truncation of low-probability transitions. MAUVE remains uniformly high and varies only mildly across steps, despite substantial changes in transition KL. The dashed horizontal line indicates the autoregressive (AR) baseline.

Diffusion Step	ReMDM sample (same seed, index = 7)
GT	elroidified ... They talked until a Carl React. She was hub executive chap, Mr McGs, making his personal read the past rather than the hrey. ... According to gy on Gresist “In acknowled” expressive information, community ... Ver at 10 minutes [...]
32	, although I am ... The two countries restrictions about fakeholds a Improducue group led us staying a big city of the work Prime ... It would get behind by the GOP nomination. Even though the year which facts for grinalsced worsteep survey [...]
128	seasons of them out on the Duckabb ... firing individual’s on how much work very nice Jews is greatest to put in front of the NBooking ... No. I do one million in any unatticestercig is to make a public opinion [...]
1024	or even at least the world of good rockets every room last ... The Early as expected surround contained modific. Pennyard. Proble, he’s not that when I can be said ... the Rick of the New York Times [...]

Appendix

Table 3: Qualitative samples of GT and ReMDM under increasing diffusion steps on OpenWebText. We show one ground-truth (GT) sample from the oracle Markov chain and ReMDM samples at different diffusion steps (same seed, index = 7). Note that the GT distribution is defined using a fixed ByteBPE vocabulary with a top-

K

approximation of the transition probabilities, which induces mild subword-level fragmentation in decoded text. As the number of diffusion steps increases, ReMDM samples become more consistent with the fragmented GT distribution, without exhibiting template-level repetition.

Diffusion Step	LLADA sample (same seed, index = 7)
32	to the last season. The government’s ... the same time, the first half of the team in the city ... one of the same time, but the same way, and the rest of the world, in the most of the number of the time, in the same time, in the country [...]
128	one of the same time, in the back in the same time, the country’s of the most of the same time to the U. ... the same amount of the same value of the government’s, the world, and the world. That’s, the University of the same time [...]
1024	the same time, and the same time, and the same time, in the first time of the same time ... the country’s of the same time, in the world’s of the rest of the time in the same way, in the world’s, the first time in the first appeared [...]

Appendix

Table 4: Qualitative degeneration of LLADA under increasing diffusion steps on OpenWebText. All samples use the same seed (index = 7). While LLADA can initially generate locally fluent text, increasing the number of diffusion steps amplifies template-level repetition and collapse, resulting in increasingly repetitive and low-diversity outputs.

Figure 10: Transition-level evaluation of oracle LLaDA on Text8 (character-level). We compare unconditional generation with three single-token conditioning variants: pi conditions on a token sampled from the stationary distribution, head on a high-frequency token, and tail on a low-frequency token. Across all metrics, these variants are nearly indistinguishable, indicating that one-token conditioning does not materially affect the sampling dynamics. This is expected because the underlying Markov prior mixes quickly, so a single observed token is rapidly forgotten and does not impose a persistent constraint on subsequent transitions.

Explore similar work

The Diffusion Duality, Chapter II: $Ψ$ -Samplers
Feb 24, 2026Justin Deschenaux, Caglar Gulcehre, Subham Sekhar SahooDiffusion Models Diffusion Sampling
EPFL, Lausanne, Switzerland · Microsoft AI · Cornell Tech, NY
Fork-dLLM: Avoiding the Flexibility Trap in Diffusion Language Models
Sep 30, 2026Stipe Frković, Metod Jazbec, Christian A. NaessethMasked Diffusion Models Speculative Decoding for Diffusion Language Models
University of Amsterdam · UvA-Bosch Delta Lab, University of Amsterdam
LESS Is More: Mutual-Stability Sampling for Diffusion Language Models
Jun 15, 2026Amr Mohamed, Guokan Shang, Michalis VazirgiannisMasked Diffusion Models Diffusion Language Model Decoding
1MBZUAI · 2Ecole Polytechnique

Feb 24, 2026J/K move · Enter open · S save

The Diffusion Duality, Chapter II: $Ψ$ -Samplers

Justin Deschenaux, Caglar Gulcehre, Subham Sekhar Sahoo

EPFL, Lausanne, Switzerland · Microsoft AI · Cornell Tech, NY

Diffusion Models Diffusion Sampling Diffusion Language Models

Open paper AskPDF arXiv

Uniform-state discrete diffusion models excel at few-step generation and guidance due to their ability to self-correct, making them preferred over autoregressive or Masked diffusion models in these settings. However, their sampling quality plateaus with ancestral samplers as the number of steps increases. We introduce a family of Predictor-Corrector (PC) samplers for discrete diffusion that generalize prior methods and apply to arbitrary noise processes. When paired with uniform-state diffusion, our samplers outperform ancestral sampling on both language and image modeling, achieving lower generative perplexity at matched unigram entropy on OpenWebText and better FID/IS scores on CIFAR10. Crucially, unlike conventional samplers, our PC methods continue to improve with more sampling steps. Taken together, these findings call into question the assumption that Masked diffusion is the inevitable future of diffusion-based language modeling. Beyond sampling, we develop a memory-efficient curriculum for the Gaussian relaxation training phase, reducing training time by 25% and memory by 33% compared to Duo while maintaining comparable perplexity on OpenWebText and LM1B and strong downstream performance. We release code, checkpoints, and a video-tutorial on: https://s-sahoo.com/duo-ch2

Abstract

Explore similar work

The Diffusion Duality, Chapter II: ΨΨΨ-Samplers

Fork-dLLM: Avoiding the Flexibility Trap in Diffusion Language Models

LESS Is More: Mutual-Stability Sampling for Diffusion Language Models

The Diffusion Duality, Chapter II: $Ψ$ -Samplers