Authors: David Li, Nikita Gushchin, Dmitry Abulkhanov, Eric Moulines, Ivan Oseledets, Maxim Panov, Alexander Korotin
Organizations: Mohamed Bin Zayed University of AI, Abu Dhabi, UAE · Applied AI Institute, Moscow, Russia · AXXX, Russia · EPITA, Laboratoire Recherche de l’EPITA, Paris, France
Diffusion Language Models (DLMs) have recently achieved strong results in text generation. However, their multi-step sampling leads to slow inference, limiting practical use. To address this, we extend Inverse Distillation, a technique originally developed to accelerate continuous diffusion models, to the discrete setting. Nonetheless, this extension introduces both theoretical and practical challenges. From a theoretical perspective, the inverse distillation objective lacks uniqueness guarantees, which may lead to suboptimal solutions. From a practical standpoint, backpropagation in the discrete space is non-trivial and often unstable. To overcome these challenges, we first provide a theoretical result demonstrating that our inverse formulation admits a unique solution, thereby ensuring valid optimization. We then introduce gradient-stable relaxations to support effective training. As a result, experiments on multiple DLMs show that our method, Inverse-distilled Diffusion Language Models (IDLM), reduces the number of inference steps by 4x-64x, while preserving the teacher model's generation quality. We provide the code, model checkpoints, and video tutorials on the project page: https://david-cripto.github.io/idlm-project-page
Figures & tables
Figure 1: Diffusion training vs inverse distillation. DLMs learn a slow teacher f∗ from data. IDLM fixes f∗ and learns pθ for fast few-step generation.
Figure 2: IDLM training loop. The student generator Gθ maps noised data to a clean sample, which is forward-diffused and passed as input to the frozen teacher f∗ and the learnable fake model f . Training alternates between fitting the fake model to the current student distribution and updating the generator with the IDLM loss. This loss pushes Gθ toward samples supported by the teacher more than by the fake model.
Figure 3: OWT distillation. IDLM preserves the quality/diversity tradeoff while using far fewer reverse steps for both masked and uniform diffusion. The strongest gains appear in the low-step regime.
Figure 4: GenPPL–Entropy tradeoff on OWT. Each point denotes a Method/Sampling steps. Lower GenPPL and higher entropy are preferred.
Figure 5: Qualitative examples. Representative IDLM samples. Additional qualitative samples are provided in Appendix G .
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: IDLM-SEDD comparison with SEDD. Left: IDLM-SEDD matches SEDD generation quality and diversity while reducing sampling from 1024 to 256 steps ( 4× ). Right: IDLM-SEDD improves GenPPL across sampling steps while preserving high entropy.
Figure 7: Masked forward process. The transition qt(⋅∣x0) preserves the clean-token mass with probability αt and moves the remaining mass to the mask token. Because this map is linear, the same operation applies to a one-hot token and to a differentiable simplex output Gθ(ϵ) .
Figure 8: Masked-diffusion sampling procedure. The sampler starts from a fully masked sequence and reveals each position once; after a token is revealed, later reverse steps copy it. Thus the schedule decides when a position is revealed, and the model decides which token to put there.
Figure 9: Simplex relaxation for IDLM-MDLM, following Section 3.2 . A generated token is ultimately interpreted as a one-hot vocabulary choice, but the generator update uses Gθ(ϵ)∈Δ so gradients can weight several candidate tokens before a hard sample is chosen.
Figure 10: Loss evaluation under the simplex relaxation. The one-hot case selects a single log-probability from logf(xt,t) , while the simplex case evaluates the weighted sum ⟨Gθ(ϵ),logf(xt,t)⟩ .
Figure 11: Mask-only cancellation under subs . The masked forward process produces either an unmasked token with probability αt or the mask state with probability 1−αt . The unmasked branch is copied by subs , so teacher and fake predictions agree and its IDLM contribution cancels. The remaining generator signal comes from the masked branch.
Figure 12: Uniform diffusion sampling. Unlike masked diffusion, a position can be revised multiple times, so each step must decide both whether to hold or change the current token and which token should replace it.
Steps
GenPPL ↓
MAUVE ↑
Entropy ↑
FLM ( Lee et al., 2026 )
1024
62.23
–
5.33
Duo a (Teacher) ( Sahoo et al., 2025 )
77.69
0.96 ± 0.01
5.55
ELF ( Hu et al., 2026 )
32
24.08 ± 0.16
–
5.14
FMLM ( Lee et al., 2026 )
45.09
–
5.25
Duo-DCD a ( Sahoo et al., 2025 )
61.31
0.97 ± 0.01
5.52
IDLM-DCD a ( Ours )
42.13 ± 3.65
0.94 ± 0.01
5.41 ± 0.05
Appendix
Table 2: Duo a distillation comparison. We report GenPPL ↓ , MAUVE ↑ , and Entropy ↑ . Methods are grouped by Steps, with the best metrics bolded in each group.
Figure 13: OWT GenPPL and entropy vs. sampling steps. Each panel shows how generation quality and diversity change as the reverse sampling budget is reduced. GenPPL measures sample quality, where lower is better, while entropy tracks output diversity, where higher is better. The panels provide a unified comparison of step-scaling behavior for our IDLM-MDLM, IDLM-DCD g , and IDLM-DCD a models.
Method ( 863 M params)
Steps
GenPPL ↓
Entropy ↑
MDLM (Teacher) ( Sahoo et al., 2024 )
1024
34.99
5.24
SDTT ( Deschenaux and Gulcehre, 2025 )
32
31.16
5.12
IDLM-MDLM (Ours)
13.71 ± 1.70
5.17 ± 0.10
SDTT ( Deschenaux and Gulcehre, 2025 )
16
52.13
5.20
IDLM-MDLM (Ours)
24.07 ± 2.13
5.36 ± 0.04
SDTT ( Deschenaux and Gulcehre, 2025 )
8
121.51
5.25
Appendix
Table 3: Scaling to an 863M MDLM teacher. We report GenPPL ↓ and Entropy ↑ .
Diffusion language models intrinsically fail to capture correlations between decoded tokens, which leads to a harsh trade-off between sampling quality and throughput. To solve this issue, we propose DiLaDiff, a variant of masked diffusion language models with three components: (1) a continuous latent space with semantic capabilities, learned by an auto-encoder fine-tuned from an existing masked diffusion language model; (2) a latent diffusion model learning the prior over the encoder distribution; (3) a consistency model distilling the learned prior into a few-step latent generative model. We show that, even without distillation, our latent-guided diffusion model outperforms the masked diffusion baseline while significantly accelerating inference. Consistency distillation further lowers the computational overhead of continuous diffusion, such that the latent is generated in negligible time compared to discrete decoding.
Jean-Marie Lemercier, Tomas Geffner, Karsten Kreis +3
Diffusion language models (DLMs) are an attractive alternative to autoregressive models because they promise sublinear-time, parallel generation, yet practical gains remain elusive as high-quality samples still demand hundreds of refinement steps. In continuous domains, consistency training along the probability-flow ODE is a popular recipe to accelerate diffusion. For discrete diffusion, no analogous sample-space ODE exists, making direct adaptation ill-defined. We argue that the right discrete substitute is the exact posterior bridge, the closed-form conditional law linking any two noise levels, which is available for broad corruptions including masked and uniform diffusion. Building on this observation, we introduce Multi-Path Discrete Consistency (MPDC), a new principle that trains a denoiser to be path-invariant in expectation across these stochastic bridges, and instantiate it as the Consistent Diffusion Language Model (CDLM), a single-stage training framework that does not require an already trained teacher model. Our CDLM objective recovers masked diffusion, continuous consistency models, and progressive or discrete distillation as analytic limits or empirical approximations of one common view. Empirically, CDLM establishes a new state of the art on both conditional and unconditional text-generation, consistently outperforming strong base discrete diffusion models and often even multi-stage distilled baselines across sampling budgets, with the largest gains in the few-step regime. Together, these results position CDLM as a principled and scalable foundation for the next generation of fast, high-fidelity discrete generative modeling.
Hasan Amin, Yuan Gao, Yaser Souri +4
Department of Computer Science, Purdue University · Microsoft
Continuous diffusion language models generate all tokens in parallel, yet high-quality generation can still require hundreds of network evaluations (NFEs). We study how distributional distillation can reduce this cost by exploiting the student's probabilistic token outputs. Our unified formulation connects the student's output parameterization to the resulting gradient estimators and yields two methods with the same student architecture and reverse-KL matching objective: Simplex-DMD uses continuous token relaxations and pathwise gradients, while Reinforce-DMD uses categorical sampling and REINFORCE with a learned density ratio. We develop both methods for multi-step generation and investigate the training and sampling choices associated with each parameterization. On OpenWebText, for sequences of 1,024 tokens, Simplex-DMD achieves a generative perplexity of 45.6 at a unigram entropy of 5.44 nats in just 4 NFEs, a 49% reduction relative to the strongest evaluated diffusion baseline at matched entropy and sampling budget. Reinforce-DMD improves the frontier at larger budgets, reaching a generative perplexity of 14.9 at an entropy of 5.00 nats with 256 NFEs, a 20% reduction under the same comparison protocol.
Paul Le Van Kiem, Dario Shariatian, Umut Simsekli +1
Inria, PSL Research University · Cohere · CMAP, Ecole Polytechnique