Does Uniform Discrete Diffusion Need Time?

Chunsan Hong; Chieh-Hsin Lai; Satoshi Hayakawa; Yuhta Takida; Jong Chul Ye; Yuki Mitsufuji

cs.LGSep 25, 2026

Does Uniform Discrete Diffusion Need Time?

Authors: Chunsan Hong, Chieh-Hsin Lai, Satoshi Hayakawa, Yuhta Takida, Jong Chul Ye, Yuki Mitsufuji

Organizations: KAIST · Sony Group Corporation · The University of Tokyo

Abstract

Uniform discrete diffusion models (UDMs) commonly use explicit time conditioning, but we find that it can often be unnecessary in practice. In this paper, we first show that the population-optimal UDM predictor generally depends on time: time controls how much the model should trust the observed context. We then show that this dependence can become negligible in finite-data settings relevant to language. When a corrupted training sequence remains much closer to its original clean sequence than to competing training sequences, the empirical-optimal predictor is nearly insensitive to time over most of the diffusion trajectory, where the guarantee weakens toward the high-noise endpoint. Empirically, trained language UDMs exhibit limited time sensitivity over most of the trajectory, while time-agnostic predictors remain competitive with, and often outperform, time-conditioned models across datasets and training objectives. These results challenge the use of explicit time conditioning in UDMs: although the population optimum depends on time, explicitly conditioning on it may often be unnecessary in practice.

Figures & tables

Figure 1: Empirical separation of OWT training data. Left : distribution of the minimum Hamming distance $d({\bm{x}})$ from each training sequence to the remaining data, with median $d({\bm{x}})=988$ . Right : distribution of the largest post-corruption separation margin $\Delta$ across diffusion times. The dashed line marks $\Delta=6$ , which implies an oracle Fisher-sensitivity bound below $10^{-5}$ for $t\leq 0.933$ . Small separation occurs primarily when ${\bm{z}}$ is sampled near the extreme high-noise endpoint.

Figure 2: Left : Fisher sensitivity of Duality and LOO+CE , varying

{\bm{z}}

sampling time

t_{\mathrm{corruption}}\in\{0.01,0.50,0.99\}

and evaluating Fisher sensitivity on every time coordinate. Right : Per-time NLL difference between time-agnostic and time-variant UDMs.

	Agnostic	Variant
Duality	24.33	25.54
LOO+CE	25.29	25.29

Table 2: Validation PPL on OWT.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 3 — Figure 3: Sampling performance of time-variant and time-agnostic UDMs across different numbers of function evaluations (NFEs). We compare generative perplexity as the sampling budget varies under (a) Duality and (b) LOO+CE ; the annotated values report the entropy of generated samples.

Figure 4: Fisher sensitivity under different time-input reparameterizations, where ${\bm{z}}$ sampling time and Fisher sensitivity evaluation time are equal. All sensitivities are measured with respect to the original diffusion time $t$ , not reparametrized time $\hat{t}$ for model input.

Diffusion Language Models

Diffusion language models generate and refine text through iterative denoising, enabling bidirectional context and parallel token updates.

See all 475 papers

Uniform Diffusion Models Revisited: Leave-One-Out Denoiser and Absorbing State Reformulation
May 21, 2026Samson Gourevitch, Yazid Janati, Dario Shariatian +4Diffusion Model Sampling Diffusion Language Models
CMAP, Ecole polytechnique · Institute of Foundation Models · Inria, PSL Research University +2
Less Uniform Discrete Diffusion is More Powerful and Scalable
Sep 21, 2026Kaibo Wang, Ding Ding, Fangyu Ding +5Diffusion Language Models Diffusion Language Model Decoding
The Hong Kong University of Science and Technology · Huawei Foundation Model Department
Language Diffusion Models are Associative Memories Capable of Retrieving Unseen Data
Apr 29, 2026Bao Pham, Mohammed J. Zaki, Luca Ambrogioni +2Memorization in Generative Models Neural Network Memorization
1Rensselaer Polytechnic Institute (RPI) · 2Donders Institute for Brain, Cognition, and Behaviour, Radboud University · 3Dynamical Mind +1

May 21, 2026J/K move · Enter open · S save

Uniform Diffusion Models Revisited: Leave-One-Out Denoiser and Absorbing State Reformulation

Samson Gourevitch, Yazid Janati, Dario Shariatian, Umut Simsekli+3

CMAP, Ecole polytechnique · Institute of Foundation Models · Inria, PSL Research University+2

Diffusion Model Sampling Diffusion Language Models Diffusion Models

Open paper AskPDF arXiv

Discrete diffusion models are often trained through clean-data prediction, but the prediction can be used in different ways to define the reverse dynamics. In Masked Diffusion Models (MDM) these choices largely coincide, whereas in Uniform Diffusion Models (UDM) they do not. We show that the standard plug-in bridge parameterization for UDM is not optimized by the denoising posterior, but by a leave-one-out posterior that predicts each clean token without using its own noisy observation. This identifies a mismatch between the plug-in ELBO and the usual cross-entropy denoising objective. We characterize the leave-one-out target and derive exact conversions between the denoiser, the leave-one-out posterior, and the score. These conversions allow us to disentangle parameterization and training objective. Our results also lead to inference improvements without any additional training through an informed predictor-corrector sampler and improved temperature sampling based on the leave-one-out predictor. We further introduce an absorbing-state reformulation of uniform diffusion that preserves the UDM joint law while decomposing it into masked-diffusion-like sampling operations, with simpler denoising posteriors, carry-over unmasking, and a natural remasking mechanism. On language modeling, leave-one-out parameterizations consistently improve UDM generation, while the absorbing construction matches or surpasses masked diffusion. These results suggest that the empirical gap between masked and uniform diffusion is driven less by the choice of marginals themselves than by parameterization and sampling design. The code and models can be found at https://github.com/samsongourevitch/rev_udm.

Abstract

Diffusion Language Models

Explore similar work

Uniform Diffusion Models Revisited: Leave-One-Out Denoiser and Absorbing State Reformulation

Less Uniform Discrete Diffusion is More Powerful and Scalable

Language Diffusion Models are Associative Memories Capable of Retrieving Unseen Data