Why Gaussian Diffusion Models Fail on Discrete Data and How to Prevent It?

Alexander Shabalin; Simon Elistratov; Viacheslav Meshchaninov; Ildus Sadrtdinov; Dmitry Vetrov

cs.CLApr 2, 2026

Why Gaussian Diffusion Models Fail on Discrete Data and How to Prevent It?

Authors: Alexander Shabalin, Simon Elistratov, Viacheslav Meshchaninov, Ildus Sadrtdinov, Dmitry Vetrov

Organizations: Constructor University, Bremen, Germany · Lomonosov Moscow State University, Moscow, Russia

Abstract

Diffusion models have become a standard approach for generative modeling in continuous domains, yet their application to discrete data remains challenging. We investigate why Gaussian diffusion models with the DDPM solver struggle to sample from discrete distributions that are represented as a mixture of delta-distributions in the continuous space. Using a toy Random Hierarchy Model, we identify a critical sampling interval in which the density of noisified data becomes multimodal. In this regime, DDPM occasionally enters low-density regions between modes producing out-of-distribution inputs for the model and degrading sample quality. We show that existing heuristics, including self-conditioning and a solver we term q-sampling, help alleviate this issue. Furthermore, we demonstrate that combining self-conditioning with switching from DDPM to q-sampling within the critical interval improves generation quality on real data. We validate these findings across conditional and unconditional tasks in multiple domains, including text, programming code, and proteins.

Figures & tables

Panel of Figure 1 — Figure 1: Toy visualization of sampling with continuous diffusion models in continuous and discrete data domains. Orange contours visualize the density of noisified data $p(\textbf{x}_{t})$ . Blue points and arrows show in-distribution sampling trajectories, red — out-of-distribution trajectories. White diamonds show original data points in the discrete setting.

Figure 2: (a) correctness of predictions $\hat{\mathbf{x}}_{\theta}(\mathbf{x}_{t})$ and (b) mean probability density $p(\mathbf{x}_{t})$ along correct, incorrect and optimal sampling trajectories; (c) share of pairs of trajectories lying within the same mode (separately for all learned and optimal trajectories). Gray vertical stripes show the transition interval.

Figure 3: left: ratio of candidate sets $\mathcal{S}$ with at least one correct sample vs. its size $|\mathcal{S}|$ for different activation timesteps $t_{\mathrm{act}}$ ; right: correctness and diversity of MBR with $|\mathcal{S}|=5$ , self-conditioning (SC), and q-sampling vs. their activation timestep $t_{\mathrm{act}}$ . Gray vertical stripes show the transition interval.

Figure 4: Comparison of one-step updates for DDPM and q-sampling. For each plotted timestep $t$ , we run DDPM from $t^{\prime}=T$ to $t^{\prime}=t$ . Then, we divide the trajectories into correct and incorrect based on $\hat{\mathbf{x}}_{\theta}(\mathbf{x}_{t})$ . The next step $t^{\prime}=t-1$ is obtained with either DDPM or q-sampling. The plots show (a) correctness of predictions $\hat{\mathbf{x}}_{\theta}(\mathbf{x}_{t-1})$ , (b) mean distance $\|\mathbf{x}_{t}-\mathbf{x}_{t-1}\|$ , (c) mean density $p(\mathbf{x}_{t-1})$ , (d) mean magnitude of predictions $\|\hat{\mathbf{x}}_{\theta}(\mathbf{x}_{t-1})\|$ .

Figure 5: (a) mean distance between predictions $\|\hat{\mathbf{x}}_{\theta}(\mathbf{x}_{t})-\hat{\mathbf{x}}_{\theta}(\mathbf{x}_{t-1})\|$ for correct and incorrect trajectories without SC and for the optimal model; (b) mean distance between predictions $\|\hat{\mathbf{x}}_{\theta}(\mathbf{x}_{t})-\hat{\mathbf{x}}_{\theta}(\mathbf{x}_{t-1})\|$ with and without SC; (c) correctness along the trajectory with and without SC. Gray vertical stripes show the transition interval.

	ROCStories			Wikipedia
Model	Mauve $\uparrow$	PPL $\downarrow$	Div $\uparrow$	Mauve $\uparrow$	PPL $\downarrow$	Div $\uparrow$
Diffusion-LM	$16.2_{{\color[rgb]{0.5,0.5,0.5}\pm 2.9}}$	$148.9_{{\color[rgb]{0.5,0.5,0.5}\pm 2.7}}$	$\textbf{33.7}_{{\color[rgb]{0.5,0.5,0.5}\pm 0.4}}$	$1.2_{{\color[rgb]{0.5,0.5,0.5}\pm 0.2}}$	$419.8_{{\color[rgb]{0.5,0.5,0.5}\pm 9.3}}$	$49.5_{{\color[rgb]{0.5,0.5,0.5}\pm 0.6}}$
+ SC 0.5	$71.3_{{\color[rgb]{0.5,0.5,0.5}\pm 3.8}}$	$73.8_{{\color[rgb]{0.5,0.5,0.5}\pm 0.8}}$	$30.2_{{\color[rgb]{0.5,0.5,0.5}\pm 0.3}}$	$12.1_{{\color[rgb]{0.5,0.5,0.5}\pm 1.9}}$	$300.2_{{\color[rgb]{0.5,0.5,0.5}\pm 6.7}}$	$\textbf{58.7}_{{\color[rgb]{0.5,0.5,0.5}\pm 0.4}}$
+ QS 0.3	$72.7_{{\color[rgb]{0.5,0.5,0.5}\pm 4.3}}$	$42.7_{{\color[rgb]{0.5,0.5,0.5}\pm 0.4}}$	$17.4_{{\color[rgb]{0.5,0.5,0.5}\pm 0.2}}$	$5.8_{{\color[rgb]{0.5,0.5,0.5}\pm 0.8}}$	$166.6_{{\color[rgb]{0.5,0.5,0.5}\pm 3.3}}$	$40.0_{{\color[rgb]{0.5,0.5,0.5}\pm 0.6}}$
+ QS 0.3 + SC 0.5	$\textbf{81.6}_{{\color[rgb]{0.5,0.5,0.5}\pm 2.7}}$	$\textbf{41.0}_{{\color[rgb]{0.5,0.5,0.5}\pm 0.6}}$	$22.9_{{\color[rgb]{0.5,0.5,0.5}\pm 0.3}}$	$36.4_{{\color[rgb]{0.5,0.5,0.5}\pm 4.4}}$	$\textbf{114.2}_{{\color[rgb]{0.5,0.5,0.5}\pm 2.4}}$	$48.4_{{\color[rgb]{0.5,0.5,0.5}\pm 0.4}}$
+ SC 0.95	$75.2_{{\color[rgb]{0.5,0.5,0.5}\pm 2.4}}$	$59.5_{{\color[rgb]{0.5,0.5,0.5}\pm 1.1}}$	$29.0_{{\color[rgb]{0.5,0.5,0.5}\pm 0.4}}$	$27.8_{{\color[rgb]{0.5,0.5,0.5}\pm 3.2}}$	$197.4_{{\color[rgb]{0.5,0.5,0.5}\pm 4.8}}$	$56.6_{{\color[rgb]{0.5,0.5,0.5}\pm 0.5}}$

Table 1: Impact of q-sampling (QS) and self-conditioning (SC) on ROCStories and Wikipedia. SC trained with probability

p

is denoted SC p ; QS with optimal

t_{\mathrm{act}}

is denoted QS

{}_{t_{\mathrm{act}}/T}

Method	Mauve $\uparrow$	PPL $\downarrow$	Div $\uparrow$
Diffusion-LM	$16.2_{{\color[rgb]{0.5,0.5,0.5}\pm 3.0}}$	$148.9_{{\color[rgb]{0.5,0.5,0.5}\pm 2.7}}$	$33.7_{{\color[rgb]{0.5,0.5,0.5}\pm 0.4}}$
+ MBR 0.3	$20.8_{{\color[rgb]{0.5,0.5,0.5}\pm 3.5}}$	$136.3_{{\color[rgb]{0.5,0.5,0.5}\pm 1.8}}$	$32.2_{{\color[rgb]{0.5,0.5,0.5}\pm 0.3}}$
TEncDM Emb	$15.7_{{\color[rgb]{0.5,0.5,0.5}\pm 2.9}}$	$123.2_{{\color[rgb]{0.5,0.5,0.5}\pm 1.8}}$	$36.6_{{\color[rgb]{0.5,0.5,0.5}\pm 0.3}}$
+ MBR 0.3	$18.9_{{\color[rgb]{0.5,0.5,0.5}\pm 1.8}}$	$101.5_{{\color[rgb]{0.5,0.5,0.5}\pm 1.9}}$	$31.6_{{\color[rgb]{0.5,0.5,0.5}\pm 0.5}}$
TEncDM Enc	$70.1_{{\color[rgb]{0.5,0.5,0.5}\pm 3.2}}$	$54.2_{{\color[rgb]{0.5,0.5,0.5}\pm 1.1}}$	$31.2_{{\color[rgb]{0.5,0.5,0.5}\pm 0.4}}$
+ MBR 0.3	$72.3_{{\color[rgb]{0.5,0.5,0.5}\pm 2.0}}$	$48.0_{{\color[rgb]{0.5,0.5,0.5}\pm 44.5}}$	$28.3_{{\color[rgb]{0.5,0.5,0.5}\pm 0.3}}$

Table 2: MBR on ROCStories (

|\mathcal{S}|=10

Panel of Figure 6 — Figure 6: XSum (top) and CoNaLa (bottom) quality for varying q-sampling activation timestep with and without SC.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

	RHM	ROCStories	Wikipedia	OWT
Max length	$8$	$80$	$128$	$512$
	XSum	QQP	IWSLT14	CoNaLa
Max input length	$512$	$50$	$64$	$64$
Max target length	$64$	$50$	$64$	$64$

Appendix

Table 3: Max sequence lengths used for each dataset.

	$t_{\mathrm{act}}/T=0.1$	$t_{\mathrm{act}}/T=0.3$	$t_{\mathrm{act}}/T=0.5$	$t_{\mathrm{act}}/T=0.7$
MBR candidate	b l b b j b j k	b l f b j b j k	i n f b l d d i	k n j i o a p a
Candidate #1	b l b b j b j k	i c b b j b j k	i n f b l d d i	k g o c k i h n
Candidate #2	b l b b j b j k	i n f b j b j k	n o b h k o k i	h n c e f b l e
Candidate #3	b l b b j b j k	b l f b j b o k	e n f b o o m b	k n j i o a p a
Candidate #4	b l b b j b j k	b l f b j b j k	i c b b l d j b	j k j m a i o a
Candidate #5	b l b b j b j k	b l f b j b j k	b g f b j b h b	d e j d p b c b

Appendix

Table 4: Five MBR candidates and chosen samples for different activation timesteps

t_{\mathrm{act}}

. We use latin letters to represent each token. Correct sequences are highlighted in green and incorrect in red . Activating MBR at

t_{\mathrm{act}}/T=0.3

t_{\mathrm{act}}/T=0.5

ensures that the correct sequence is sampled; this would not be the case without MBR.

$t_{\mathrm{act}}/T=0.3$
MBR	\tikz [baseline=(a.base),inner sep=0pt] \node fill=Green!25John; and John decided to build a lemonade stand. They \tikz [baseline=(a.base),inner sep=0pt] \node fill=Green!25got; together in \tikz [baseline=(a.base),inner sep=0pt] \node fill=Green!25a; big lemonade stand in their yard. picking out the lemonade, they put it in a \tikz [baseline=(a.base),inner sep=0pt] \node fill=Green!25jar;. They ate the lemonade for a week. The neighbor told them to go home without their lemonade.
#1	\tikz [baseline=(a.base),inner sep=0pt] \node fill=red!25Amy; and John decided to build a lemonade stand. They \tikz [baseline=(a.base),inner sep=0pt] \node fill=red!25walked; together in \tikz [baseline=(a.base),inner sep=0pt] \node fill=Green!25a; big lemonade stand in their yard. picking out the lemonade, they put it in a \tikz [baseline=(a.base),inner sep=0pt] \node fill=red!25bowl;. They ate the lemonade for a week. The neighbor told them to go home without their lemonade.
#2	\tikz [baseline=(a.base),inner sep=0pt] \node fill=red!25Sam; and John decided to make a lemonade stand. They \tikz [baseline=(a.base),inner sep=0pt] \node fill=Green!25got; together in \tikz [baseline=(a.base),inner sep=0pt] \node fill=red!25the; big lemonade stand in their yard. picking out the lemonade, they put it in a \tikz [baseline=(a.base),inner sep=0pt] \node fill=red!25bowl;. They ate the lemonade for a week. The neighbor told them to go home without their lemonade.
#3	\tikz [baseline=(a.base),inner sep=0pt] \node fill=Green!25John; and John decided to build a lemonade stand. They \tikz [baseline=(a.base),inner sep=0pt] \node fill=red!25sat; together in \tikz [baseline=(a.base),inner sep=0pt] \node fill=Green!25a; large lemonade stand in their yard. picking out the lemonade, they put it in a \tikz [baseline=(a.base),inner sep=0pt] \node fill=red!25box;. They ate the lemonade for a week. The neighbor told them to go home without their lemonade.
#4	\tikz [baseline=(a.base),inner sep=0pt] \node fill=Green!25John; and John decided to make a lemonade stand. They \tikz [baseline=(a.base),inner sep=0pt] \node fill=Green!25got; together in \tikz [baseline=(a.base),inner sep=0pt] \node fill=Green!25a; large lemonade stand in their yard. picking out the lemonade, they put it in a \tikz [baseline=(a.base),inner sep=0pt] \node fill=Green!25jar;. They ate the lemonade for a week. The neighbor told them to go home without their lemonade.
$t_{\mathrm{act}}/T=0.5$

Appendix

Table 5: Four MBR candidates and the chosen sample for different activation timesteps

t_{\mathrm{act}}

. Parts matching the selected candidate are highlighted in \tikz [baseline=(a.base),inner sep=0pt] \node fill=Green!25green;; differing parts in \tikz [baseline=(a.base),inner sep=0pt] \node fill=red!25red;. Parts shared across all candidates are not highlighted.

Figure 7: Quality metrics with and without self-conditoning (SC) for different q-sampling activation timestep on ROCStories and Wikipedia. We use the baseline SC model with

p=0.5

Figure 8: Quality metrics with and without self-conditoning (SC) for different q-sampling activation timestep on OpenWebText. We use the baseline SC model with

p=0.5

Figure 9: Quality metrics for TEncDM-based models without self-conditining (SC) for different q-sampling activation timesteps and varying amount of generation steps on ROCStories and Wikipedia. We use the baseline SC model with

p=0.5

Figure 10: Quality metrics for TEncDM-based models with self-conditining (SC) for different q-sampling activation timesteps and varying amount of generation steps on ROCStories and Wikipedia. We use the baseline SC model with

p=0.5

	ROCStories			Wikipedia			OpenWebText
Model	Mauve $\uparrow$	PPL $\downarrow$	Div $\uparrow$	Mauve $\uparrow$	PPL $\downarrow$	Div $\uparrow$	Mauve $\uparrow$	PPL $\downarrow$	Div $\uparrow$
Diffusion-LM
+ SC 0.5	$71.3_{{\color[rgb]{0.5,0.5,0.5}\pm 3.8}}$	$74.5_{{\color[rgb]{0.5,0.5,0.5}\pm 0.9}}$	$\textbf{30.2}_{{\color[rgb]{0.5,0.5,0.5}\pm 0.3}}$	$12.1_{{\color[rgb]{0.5,0.5,0.5}\pm 1.9}}$	$300.2_{{\color[rgb]{0.5,0.5,0.5}\pm 6.7}}$	$58.7_{{\color[rgb]{0.5,0.5,0.5}\pm 0.4}}$	—	—	—
+ SC 0.8	$74.0_{{\color[rgb]{0.5,0.5,0.5}\pm 4.3}}$	$66.5_{{\color[rgb]{0.5,0.5,0.5}\pm 1.0}}$	$30.1_{{\color[rgb]{0.5,0.5,0.5}\pm 0.4}}$	$16.8_{{\color[rgb]{0.5,0.5,0.5}\pm 1.6}}$	$271.2_{{\color[rgb]{0.5,0.5,0.5}\pm 7.4}}$	$\textbf{58.8}_{{\color[rgb]{0.5,0.5,0.5}\pm 0.5}}$	—	—	—
+ SC 0.9	$74.0_{{\color[rgb]{0.5,0.5,0.5}\pm 1.9}}$	$61.8_{{\color[rgb]{0.5,0.5,0.5}\pm 0.5}}$	$29.2_{{\color[rgb]{0.5,0.5,0.5}\pm 0.3}}$	$24.3_{{\color[rgb]{0.5,0.5,0.5}\pm 3.5}}$	$229.5_{{\color[rgb]{0.5,0.5,0.5}\pm 5.1}}$	$\textbf{58.8}_{{\color[rgb]{0.5,0.5,0.5}\pm 0.5}}$	—	—	—
+ SC 0.95	$\textbf{75.2}_{{\color[rgb]{0.5,0.5,0.5}\pm 2.4}}$	$59.5_{{\color[rgb]{0.5,0.5,0.5}\pm 1.1}}$	$29.0_{{\color[rgb]{0.5,0.5,0.5}\pm 0.4}}$	$27.8_{{\color[rgb]{0.5,0.5,0.5}\pm 3.2}}$	$197.4_{{\color[rgb]{0.5,0.5,0.5}\pm 4.8}}$	$56.6_{{\color[rgb]{0.5,0.5,0.5}\pm 0.5}}$	—	—	—

Appendix

Table 6: Impact of self-conditioning (SC) probability on ROCStories, Wikipedia and OpenWebText. SC trained with probability

p

is denoted SC p . The SC probability giving the highest Mauve is highlighted in \tikz [baseline=(a.base),inner sep=0pt] \node fill=blue!15blue;.

	OpenWebText
Model	Mauve $\uparrow$	PPL $\downarrow$	Div $\uparrow$
TEncDM Emb	$2.6_{{\color[rgb]{0.5,0.5,0.5}\pm 0.4}}$	$185.3_{{\color[rgb]{0.5,0.5,0.5}\pm 2.4}}$	$\textbf{34.6}_{{\color[rgb]{0.5,0.5,0.5}\pm 0.2}}$
+ SC 0.5	$59.8_{{\color[rgb]{0.5,0.5,0.5}\pm 4.7}}$	$70.6_{{\color[rgb]{0.5,0.5,0.5}\pm 0.8}}$	$26.7_{{\color[rgb]{0.5,0.5,0.5}\pm 0.3}}$
+ QS 0.4	$6.2_{{\color[rgb]{0.5,0.5,0.5}\pm 0.8}}$	$106.8_{{\color[rgb]{0.5,0.5,0.5}\pm 1.7}}$	$27.7_{{\color[rgb]{0.5,0.5,0.5}\pm 0.3}}$
+ QS 0.4 + SC 0.5	$\textbf{67.3}_{{\color[rgb]{0.5,0.5,0.5}\pm 4.5}}$	$\textbf{53.9}_{{\color[rgb]{0.5,0.5,0.5}\pm 0.7}}$	$23.4_{{\color[rgb]{0.5,0.5,0.5}\pm 0.3}}$
TEncDM Enc	$20.8_{{\color[rgb]{0.5,0.5,0.5}\pm 3.0}}$	$176.1_{{\color[rgb]{0.5,0.5,0.5}\pm 2.0}}$	$\textbf{39.7}_{{\color[rgb]{0.5,0.5,0.5}\pm 0.3}}$

Appendix

Table 7: Impact of q-sampling (QS) and self-conditioning (SC) on OpenWebText. SC trained with probability

p

is denoted SC p ; QS with optimal activation timestep

t_{\mathrm{act}}

is denoted QS

{}_{t_{\mathrm{act}}/T}

	FD-seq $\downarrow$	pppl $\downarrow$	CD 0.5 $\uparrow$
DiMA Emb	$1.086_{{\color[rgb]{0.5,0.5,0.5}\pm 0.006}}$	$12.62_{{\color[rgb]{0.5,0.5,0.5}\pm 0.25}}$	$87.9_{{\color[rgb]{0.5,0.5,0.5}\pm 0.9}}$
+ SC 0.5	$1.015_{{\color[rgb]{0.5,0.5,0.5}\pm 0.024}}$	$11.33_{{\color[rgb]{0.5,0.5,0.5}\pm 0.21}}$	$\textbf{92.5}_{{\color[rgb]{0.5,0.5,0.5}\pm 1.6}}$
+ QS 0.4	$1.045_{{\color[rgb]{0.5,0.5,0.5}\pm 0.009}}$	$12.04_{{\color[rgb]{0.5,0.5,0.5}\pm 0.25}}$	$87.1_{{\color[rgb]{0.5,0.5,0.5}\pm 1.3}}$
+ QS 0.4 + SC 0.5	$\textbf{1.004}_{{\color[rgb]{0.5,0.5,0.5}\pm 0.026}}$	$\textbf{10.89}_{{\color[rgb]{0.5,0.5,0.5}\pm 0.23}}$	$91.5_{{\color[rgb]{0.5,0.5,0.5}\pm 1.5}}$

Appendix

Table 8: Impact of q-sampling (QS) and self-conditioning (SC) on SwissProt protein generation. SC trained with probability

p

is denoted SC p ; QS with optimal activation timestep

t_{\mathrm{act}}

is denoted QS

{}_{t_{\mathrm{act}}/T}

Figure 11: Quality metrics with and without self-conditoning (SC) for different q-sampling activation timestep on SwissProt protein generation. We use the baseline SC model with

p=0.5

	IWSLT14 (SeqDiffuSeq)		QQP (SeqDiffuSeq)			XSum (TEncDM Emb)
Solver	BLEU $\uparrow$	R-1/2/L $\uparrow$	BS $\uparrow$	BLEU $\uparrow$	R-1/2/L $\uparrow$	BS $\uparrow$	R-1/2/L $\uparrow$
DDPM	29.19	$\textbf{57.4}/\textbf{32.3}/\textbf{54.3}$	82.8	$33.58$	$\textbf{60.2}/\textbf{35.7}/\textbf{57.8}$	$59.3$	$27.2/6.7/20.7$
+ SC 0.5	$28.93$	$56.8/31.8/53.7$	82.8	34.13	$60.1/35.6/57.5$	$66.3$	$30.5/9.1/23.4$
+ QS 1.0	$28.60$	$56.4/31.7/53.5$	$82.1$	$33.50$	$59.6/35.1/57.0$	$63.4$	$30.1/8.4/23.1$
+ QS 1.0 + SC 0.5	$28.79$	$56.8/31.9/53.7$	$82.7$	34.11	$59.8/35.4/57.3$	67.5	$\textbf{31.9}/\textbf{9.9}/\textbf{24.5}$

Appendix

Table 9: Impact of q-sampling (QS) and self-conditioning (SC) on IWSLT14, QQP, and XSum with their corresponding models. SC trained with probability

p

is denoted SC p ; QS with optimal activation timestep

t_{\mathrm{act}}

is denoted QS

{}_{t_{\mathrm{act}}/T}

. For IWSLT-14 and QQP, q-sampling does not give a significant improvement for any

t_{\mathrm{act}}

, thus we report results for pure q-sampling

t_{\mathrm{act}}/T=1.0

, which is also optimal for XSum.

Method	FID $\downarrow$	Precision $\uparrow$	Recall $\uparrow$	Density $\uparrow$	Coverage $\uparrow$
SongUNet	$8.47$	$0.811$	$0.673$	$1.127$	$0.956$
+ QS 0.025	$37.40$	$0.693$	$0.434$	$0.713$	$0.630$
+ QS 0.05	$56.72$	$0.661$	$0.251$	$0.581$	$0.455$
+ QS 0.075	$64.22$	$0.647$	$0.195$	$0.557$	$0.412$
+ QS 0.1	$70.07$	$0.630$	$0.172$	$0.529$	$0.392$
+ SC	$16.48$	$0.853$	$0.565$	$1.467$	$0.922$

Appendix

Table 11: Impact of q-sampling (QS), self-conditioning, and MBR on image diffusion on CIFAR-10.

Panel of Figure 13 — Figure 13: Qualitative comparison on CIFAR-10. Generation is initialized from the same seeds. The first column corresponds to MBR-selected samples, while the last column corresponds to q-sampling (QS). As can be seen, q-sampling produces significantly smoother images.

Figure 14: Correctness (left axis, blue) and diversity (right axis, green) as functions of the q-sampling activation time

t_{\mathrm{act}}

, shown for the three patch-based CIFAR-10 experiments. Solid lines correspond to the base model, dashed lines to self-conditioning (SC).

Panel of Figure 15 — Figure 15: Examples of patch-based training images.

Diffusion Models

Diffusion models learn to reverse gradual data corruption, enabling flexible generation, editing, forecasting, and inference.

See all 1,169 papers

Sep 8, 2026

Let It Go or Learn to Self-Correct: Continuous Diffusion for Constrained Discrete Tasks

Denoising Diffusion Probabilistic Models (DDPMs) generate samples by starting from noise and repeatedly denoising while keeping each update close to the current noisy state. This behavior is effective in many continuous domains, but its role is less clear for globally constrained discrete tasks, such as Sudoku, graph connectivity, Latin squares, and N-queens. In such settings, early discrete errors can be difficult to undo. As a result, standard diffusion sampling may preserve early mistakes, even when the model's clean predictions are informative. We compare standard samplers to sampling directly from the model's clean prediction. Without retraining, this single change improves Sudoku validity from 31% to 95%, with consistent gains across the other discrete tasks. We hypothesize that staying close to the current noisy state is harmful because the reverse trajectory can drift off the forward noising distribution the model was trained on. To reduce this train-test mismatch, we further introduce self-correction training, which exposes the model to its own predictions, improving robustness to errors that arise during inference. This substantially improves the performance of standard samplers. Our results suggest that continuous diffusion models can learn nontrivial global constraints, but discrete reasoning tasks require better alignment between training and inference: either through samplers that reduce commitment to early decisions, or through training that teaches the model to correct its own inference-time errors.

Mariia Drozdova, Stéphane Liem Nguyen, François Fleuret

University of Geneva

Constrained Generative Modeling Diffusion Sampling Diffusion Models

Ask PDF

Oct 8, 2026

Ambient Discrete Diffusion: Using the Wrong Data at the Right Time for Data Efficient Learning

We introduce RefineMix, a framework for training discrete diffusion models under severe data scarcity, a common constraint in scientific applications. RefineMix uses out-of-distribution data at selected diffusion times to improve generalization without biasing the sampling distribution. Although this strategy has been explored in continuous diffusion, discrete diffusion presents a distinct challenge: unlike Gaussian noise, masking preserves domain information in surviving tokens, limiting the use of related data at high noise levels. At low noise levels, however, the domains effectively disjoint supports become an advantage, allowing the model to learn from both in-domain and out-of-distribution data without biasing the sampler. We formalize these intuitions and provide a theoretical analysis for the proposed method. Experimentally, across five domain-shift settings, RefineMix matches or outperforms in-domain finetuning and data mixing. For protein sequence generation, finetuning with just 197 in-domain examples nearly doubles the fraction of generated proteins that are simultaneously novel, foldable, and in-family compared to standard finetuning.

Julian Kleutgens, Mauricio Tec, Claudio Battiloro +2

ETH Zürich · Harvard University · MIT

Diffusion Models Masked Diffusion Models Diffusion Model Fine-Tuning

Ask PDF

Mar 20, 2026

Beyond Single Tokens: Distilling Discrete Diffusion Models via Discrete MMD

It is currently difficult to distill discrete diffusion models. In contrast, continuous diffusion literature has many distillation approaches methods that can reduce sampling steps to a handful. Our method, Discrete Moment Matching Distillation (D-MMD), leverages ideas that have been highly successful in the continuous domain. Whereas previous discrete distillation methods collapse, D-MMD maintains high quality and diversity (given sufficient sampling steps). This is demonstrated on both text and image datasets. Moreover, the newly distilled generators can outperform their teachers.

Emiel Hoogeboom, David Ruhe, Jonathan Heek +2

Google DeepMind Amsterdam

Diffusion Model Distillation Diffusion Models Diffusion Language Model Distillation

Ask PDF

Abstract

Diffusion Models

Explore similar work

Let It Go or Learn to Self-Correct: Continuous Diffusion for Constrained Discrete Tasks

Ambient Discrete Diffusion: Using the Wrong Data at the Right Time for Data Efficient Learning

Beyond Single Tokens: Distilling Discrete Diffusion Models via Discrete MMD