Improved Sampling Schedules for Discrete Diffusion Models

Alberto Foresti; Mustapha Bounoua; Giulio Franzese; Luca Ambrogioni; Pietro Michiardi

cs.LGFeb 6, 2026

Improved Sampling Schedules for Discrete Diffusion Models

Authors: Alberto Foresti, Mustapha Bounoua, Giulio Franzese, Luca Ambrogioni, Pietro Michiardi

Organizations: Department of Data Science, EURECOM, 450 Route des Chappes, 06410 Biot, France · Donders Institute for Brain, Cognition and Behaviour, Radboud University, Postbus 9102, 6500 HC Nijmegen, Netherlands

Abstract

Uniform Discrete Diffusion Models have emerged as a powerful paradigm for generative modeling on sequence data; however, the information-theoretic principles governing their reverse processes remain underexplored. In this work, we bridge this gap by analyzing the reverse process dynamics through the lens of thermodynamic entropy production and dynamical activity. We use a variational expression of the discrete Wasserstein distance to characterize the dynamics of the generation process. Leveraging these insights, we first introduce the Entropic Discrete Schedule (EDS), which aligns with entropy production, and later construct the Wasserstein Discrete Schedule (WDS), which combines entropy production and dynamical activity to act as a geometric cost. Our experimental results on language modeling show that WDS provide the same results as uniform while using up to half of the computations on the MBPP benchmark. The code is available at https://github.com/AlbertoForesti/impscheds.

Figures & tables

Panel of Figure 1 — Figure 1: Binomial distribution benchmark. We report the mean performance across 5 seeds, the bars indicate the standard error of the mean.

Panel of Figure 2 — Figure 2: Sumi reasoning benchmark.

Panel of Figure 3 — Figure 3: Performance comparisons on piano note generation across different metrics.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Samples ( $N$ )	Timesteps ( $N_{t}$ )
Binomial Distributions	1024	1024
Monophonic Music	64	1024
Sumi-7B (MBPP)	64	512

Appendix

Table 1: Parameters used for estimating neural entropy across different datasets.

Dataset	JYS (s)	Entropy production (s)	JYS / Entropy production
Binomial ( $n=255$ , linear)	13.72	12.70	1.08 $\times$
Binomial ( $n=255$ , loglinear)	12.37	10.91	1.13 $\times$
Binomial ( $n=31$ , linear)	12.24	10.51	1.16 $\times$
Binomial ( $n=31$ , loglinear)	12.27	11.07	1.11 $\times$
Piano	40.65	21.66	1.88 $\times$

Appendix

Table 2: Schedule computation time across different datasets

Task	NFE	Uniform	WDS	Performance Improvement (WDS / Uniform)
GSM8K	4	0.1327	0.1660	1.25 $\times$
	8	0.2456	0.2472	1.01 $\times$
	16	0.2987	0.2949	0.99 $\times$
	32	0.3017	0.3124	1.04 $\times$
MBPP	16	0.0900	0.1420	1.58 $\times$
	32	0.1800	0.2160	1.20 $\times$

Appendix

Table 3: Numerical results for WDS on reasoning tasks

Panel of Figure 4 — Figure 4: Entropy production rate (top) and Wasserstein rate (bottom) for Binomial, Monophonic Music, and Sumi-7B.

Diffusion Sampling

Diffusion sampling studies how to turn a trained diffusion model into high-quality samples efficiently and reliably.

See all 272 papers

Generalized Discrete Diffusion from Snapshots
Mar 22, 2026Oussama Zekri, Théo Uscidda, Nicolas Boullé +1Diffusion Models Diffusion Language Models
CREST, ENSAE, Institut Polytechnique de Paris, Paris, France · Department of Mathematics, Imperial College London, London, SW7 2AZ, UK
Forward-Learned Discrete Diffusion: Learning how to noise to denoise faster
May 18, 2026Grigory Bartosh, Teodora Pandeva, Sushrut Karmalkar +1Few-Step Diffusion Sampling Diffusion Model Acceleration
University of Amsterdam · Microsoft Research, Cambridge
Acceleration of Diffusion Language Model through Discrete Average Generator
Sep 29, 2026Yidong Ouyang, Zhengyan Wan, Themis Haris +8Few-Step Diffusion Sampling Diffusion Language Model Inference
University of California, Los Angeles · Google · Work done as a research intern at Google

Mar 22, 2026J/K move · Enter open · S save

Generalized Discrete Diffusion from Snapshots

Oussama Zekri, Théo Uscidda, Nicolas Boullé, Anna Korba

CREST, ENSAE, Institut Polytechnique de Paris, Paris, France · Department of Mathematics, Imperial College London, London, SW7 2AZ, UK

Diffusion Models Diffusion Language Models

Open paper AskPDF arXiv

We introduce Generalized Discrete Diffusion from Snapshots (GDDS), a unified framework for discrete diffusion modeling that supports arbitrary noising processes over large discrete state spaces. Our formulation encompasses all existing discrete diffusion approaches, while allowing significantly greater flexibility in the choice of corruption dynamics. The forward noising process relies on uniformization and enables fast arbitrary corruption. For the reverse process, we derive a simple evidence lower bound (ELBO) based on snapshot latents, instead of the entire noising path, that allows efficient training of standard generative modeling architectures with clear probabilistic interpretation. Our experiments on large-vocabulary discrete generation tasks suggest that the proposed framework outperforms existing discrete diffusion methods in terms of training efficiency and generation quality, and beats autoregressive models for the first time at this scale. We provide the code along with a blog post on the project page : https://oussamazekri.fr/gdds.

Abstract

Diffusion Sampling

Explore similar work

Generalized Discrete Diffusion from Snapshots

Forward-Learned Discrete Diffusion: Learning how to noise to denoise faster

Acceleration of Diffusion Language Model through Discrete Average Generator