A Ticket from Marginals to Joints: Coupled-Noise Distillation for One-Step Block Generation in Diffusion Language Models

Lin Yao

cs.CLSep 6, 2026

A Ticket from Marginals to Joints: Coupled-Noise Distillation for One-Step Block Generation in Diffusion Language Models

Authors: Lin Yao

Organizations: School of Computer Science, Shanghai Jiao Tong University, Shanghai, 200240, China · Zhongguancun Academy, Beijing, 100097, China

Abstract

Can a diffusion language model generate a coherent token block in one forward pass? Masked models already predict every position at once, but each prediction is the marginal distribution given the visible context, so the tokens can be mutually inconsistent and later steps revise those already committed. We introduce CONDOR (Coupled-Noise Distillation for One-Step Readout), trained from scratch to map different noise samples to different coherent blocks. Initially, random noise is not naturally paired with a target. Winner-take-all supervision lets different samples specialize, and self-distillation trains the one-pass output to match the refined coherent block. TinyStories experiments show diverse, coherent continuations over successive blocks, one forward pass each. Qualitative MNIST experiments show that the same approach can extend to multimodal generation, such as text-to-image and unconditional text-and-image generation.

Figures & tables

Figure 1: Committing a

B{=}8

-token block. (a) AR, one token per pass. (b) Continuous-latent dLM,

n

denoising steps. (c) Masked dLLM, commit then edit. (d) Ours, one noise ticket per pass.

Figure 2: Training Condor . Left: the winner-take-all task. Right: self-distillation.

WTA	Condor
Prefix: At the market, Lily saw many fruits
. She were to apples,,, . She saw to apples,,,	inside. She said, “Wow, on the ground. She said, “
Multi-block text continuation. One day, a little girl named Sue went to the park. She saw a big tree and wanted to climb it. Sue started to climb the tree, but she was very small. Sue saw a big bird in the tree. The bird said, “Hi, Sue! Do you need help?” Sue said, “Yes, I want to climb the tree.” The bird said, “I can help you, Sue.” Sue and the bird climbed the tree. They were happy and said, “Thank you, bird!” The bird said, “You’re welcome, Sue.” Sue and the bird became good friends. They played in the park every day.
Multi-block text continuation. Sara and Tom are friends. They like to play with toys in Sara’s room. Sara has many toys. She has dolls, cars, blocks and books. Tom likes to play with toys. One day, Sara says, “Tom, let’s play together.” Tom says, “OK, Sara, I will play with you.” Sara and Tom play with the toys together. They have a lot of fun. They are happy.

Table 5: Qualitative text-generation examples. Top: one-pass continuations of the same eight-token prefix; underlining marks errors. Bottom: selected stories from the reference

0.5

B student at step

16{,}000

. Each attempt uses one forward pass; these stories are not unfiltered rollouts.

Panel of Figure 3 — Figure 3: Qualitative multimodal generation on MNIST, with one forward pass per generated block.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Angle	Tangent	Generated continuation	Conf. (%)	$\log P$
A. One paired rotation trajectory
$0^{\circ}$	—	to play. The fish said, "	78.4	$-2.163$
$30^{\circ}$	$v_{0}$	to play. Fin said, "Yes	89.2	$-1.063$
$90^{\circ}$	$v_{0}$	to swim.\nOne day, the	91.0	$-0.961$
$180^{\circ}$	—	to be friends.\nOne day,	86.5	$-1.259$
B. Multiple directions with the same angle to the same reference

Appendix

Table S2: Unedited continuations of They would swim, splash, and pretend at norm

r_{0}

. Panel A follows one tangent; panel B uses other directions at

30^{\circ}

One-Step Generative Modeling

One-step generative modeling maps noise or conditions to data in one network evaluation, replacing iterative denoising or numerical integration.

See all 142 papers

Multi-Mask Diffusion Language Models for Few-Step Generation
Jul 22, 2026Sijin Chen, Yinuo Ren, Heyang Zhao +3Masked Diffusion Models Diffusion Model Distillation
ByteDance Seed · Princeton University · Stanford University +2
Multi-Block Diffusion Language Models
Jun 28, 2026Yijie Jin, Jiajun Xu, Yuxuan Liu +8Blockwise Diffusion Language Models Diffusion Language Model Decoding
Shanghai Jiao Tong University · Xi’an Jiao Tong University · Huawei
DiLaDiff: Distilled Latent-Augmented Diffusion for Language Modeling
May 22, 2026Jean-Marie Lemercier, Tomas Geffner, Karsten Kreis +3Diffusion Language Models Diffusion Language Model Distillation
NVIDIA

Jul 22, 2026J/K move · Enter open · S save

Multi-Mask Diffusion Language Models for Few-Step Generation

Sijin Chen, Yinuo Ren, Heyang Zhao, Ziheng Cheng+2

ByteDance Seed · Princeton University · Stanford University+2

Masked Diffusion Models Diffusion Model Distillation Few-Step Diffusion Sampling Consistency Distillation Diffusion Language Models Diffusion Models

Open paper AskPDF arXiv

Masked diffusion models (MDMs) are a promising family of language generators, but achieving high-quality few-step generation remains challenging. In MDMs, all forward trajectories collapse to a single fully masked state, leaving no terminal entropy for consistency-style few-step generation. While recent few-step alternatives based on uniform-state diffusion avoid this degeneracy, it becomes harder to distinguish clean tokens from noise than MDMs, which usually harms modeling quality and training efficiency. In this work, we propose a multi-mask diffusion model (MultiMDM) that preserves the masking structure towards few-step generation. In the forward process, each clean token is first pushed towards a designated mask and then gradually mixes over the mask set. As a result, the backward process has a drafting capability by predicting a designated mask before refining to a clean token. We derive a closed-form ELBO training objective for MultiMDM that supports continual training from pretrained MDMs. In addition, we formulate a purely discrete-state consistency distillation scheme, with a shared-Gumbel coupling to reduce pathwise entropy. Experiments on pretraining and distillation show that MultiMDM provides an effective foundation for principled few-step generation.

Abstract

One-Step Generative Modeling

Explore similar work

Multi-Mask Diffusion Language Models for Few-Step Generation

Multi-Block Diffusion Language Models

DiLaDiff: Distilled Latent-Augmented Diffusion for Language Modeling