Can a diffusion language model generate a coherent token block in one forward pass? Masked models already predict every position at once, but each prediction is the marginal distribution given the visible context, so the tokens can be mutually inconsistent and later steps revise those already committed. We introduce CONDOR (Coupled-Noise Distillation for One-Step Readout), trained from scratch to map different noise samples to different coherent blocks. Initially, random noise is not naturally paired with a target. Winner-take-all supervision lets different samples specialize, and self-distillation trains the one-pass output to match the refined coherent block. TinyStories experiments show diverse, coherent continuations over successive blocks, one forward pass each. Qualitative MNIST experiments show that the same approach can extend to multimodal generation, such as text-to-image and unconditional text-and-image generation.
Figures & tables
Figure 1: Committing a B=8 -token block. (a) AR, one token per pass. (b) Continuous-latent dLM, n denoising steps. (c) Masked dLLM, commit then edit. (d) Ours, one noise ticket per pass.
Figure 2: Training Condor . Left: the winner-take-all task. Right: self-distillation.
WTA
Condor
Prefix: At the market, Lily saw many fruits
. She were to apples,,, . She saw to apples,,,
inside. She said, “Wow, on the ground. She said, “
Multi-block text continuation. One day, a little girl named Sue went to the park. She saw a big tree and wanted to climb it. Sue started to climb the tree, but she was very small. Sue saw a big bird in the tree. The bird said, “Hi, Sue! Do you need help?” Sue said, “Yes, I want to climb the tree.” The bird said, “I can help you, Sue.” Sue and the bird climbed the tree. They were happy and said, “Thank you, bird!” The bird said, “You’re welcome, Sue.” Sue and the bird became good friends. They played in the park every day.
Multi-block text continuation. Sara and Tom are friends. They like to play with toys in Sara’s room. Sara has many toys. She has dolls, cars, blocks and books. Tom likes to play with toys. One day, Sara says, “Tom, let’s play together.” Tom says, “OK, Sara, I will play with you.” Sara and Tom play with the toys together. They have a lot of fun. They are happy.
Table 5: Qualitative text-generation examples. Top: one-pass continuations of the same eight-token prefix; underlining marks errors. Bottom: selected stories from the reference 0.5 B student at step 16,000 . Each attempt uses one forward pass; these stories are not unfiltered rollouts.
Figure 3: Qualitative multimodal generation on MNIST, with one forward pass per generated block.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Angle
Tangent
Generated continuation
Conf. (%)
logP
A. One paired rotation trajectory
0∘
—
to play. The fish said, "
78.4
−2.163
30∘
v0
to play. Fin said, "Yes
89.2
−1.063
90∘
v0
to swim.\nOne day, the
91.0
−0.961
180∘
—
to be friends.\nOne day,
86.5
−1.259
B. Multiple directions with the same angle to the same reference
Appendix
Table S2: Unedited continuations of They would swim, splash, and pretend at norm r0 . Panel A follows one tangent; panel B uses other directions at 30∘ .
Masked diffusion models (MDMs) are a promising family of language generators, but achieving high-quality few-step generation remains challenging. In MDMs, all forward trajectories collapse to a single fully masked state, leaving no terminal entropy for consistency-style few-step generation. While recent few-step alternatives based on uniform-state diffusion avoid this degeneracy, it becomes harder to distinguish clean tokens from noise than MDMs, which usually harms modeling quality and training efficiency. In this work, we propose a multi-mask diffusion model (MultiMDM) that preserves the masking structure towards few-step generation. In the forward process, each clean token is first pushed towards a designated mask and then gradually mixes over the mask set. As a result, the backward process has a drafting capability by predicting a designated mask before refining to a clean token. We derive a closed-form ELBO training objective for MultiMDM that supports continual training from pretrained MDMs. In addition, we formulate a purely discrete-state consistency distillation scheme, with a shared-Gumbel coupling to reduce pathwise entropy. Experiments on pretraining and distillation show that MultiMDM provides an effective foundation for principled few-step generation.
Sijin Chen, Yinuo Ren, Heyang Zhao +3
ByteDance Seed · Princeton University · Stanford University +2
Block Diffusion Language Models (BD-LMs) improve diffusion-based text generation with KV caching and flexible-length generation. A natural next step is to extend them from Single-Block Diffusion (SingleBD) to Multi-Block Diffusion (MultiBD), where a running-set of consecutive blocks is decoded concurrently for inter-block parallelism. However, existing BD-LMs are mostly trained under teacher forcing, where the model observes only one noisy block conditioned on a clean prefix. While the recent diffusion forcing strategy introduces visibility among multiple noisy blocks, its training states still differ from MultiBD inference, where decoding operates on a bounded running-set with heterogeneous slot-wise noise patterns. To bridge this gap, we propose Multi-Block Diffusion Language Models (MBD-LMs), obtained by post-training BD-LMs with Multi-block Teacher Forcing (MultiTF). MultiTF integrates teacher forcing and diffusion forcing by training on bounded noise-groups conditioned on clean prefixes, with randomized noise-schedulers that better match MultiBD inference states. To make MultiBD practically executable, we further introduce an optimized decoding algorithm based on the Block Buffer mechanism that preserves prefix-cache reuse, keeps input shapes static, and translates increased decoding parallelism into wall-clock acceleration. Empirically, MBD-LLaDA2-Mini increases average Tokens Per Forward pass (TPF) from 3.47 to 6.19 and improves average accuracy from 79.95% to 81.03%; when combined with DMax, MBD-LLaDA2-Mini-DMax reaches an average TPF of 9.34 with only a 1.02% accuracy drop on math and code benchmarks.
Yijie Jin, Jiajun Xu, Yuxuan Liu +8
Shanghai Jiao Tong University · Xi’an Jiao Tong University · Huawei
Diffusion language models intrinsically fail to capture correlations between decoded tokens, which leads to a harsh trade-off between sampling quality and throughput. To solve this issue, we propose DiLaDiff, a variant of masked diffusion language models with three components: (1) a continuous latent space with semantic capabilities, learned by an auto-encoder fine-tuned from an existing masked diffusion language model; (2) a latent diffusion model learning the prior over the encoder distribution; (3) a consistency model distilling the learned prior into a few-step latent generative model. We show that, even without distillation, our latent-guided diffusion model outperforms the masked diffusion baseline while significantly accelerating inference. Consistency distillation further lowers the computational overhead of continuous diffusion, such that the latent is generated in negligible time compared to discrete decoding.
Jean-Marie Lemercier, Tomas Geffner, Karsten Kreis +3