IDLM: Inverse-distilled Diffusion Language Models

David Li; Nikita Gushchin; Dmitry Abulkhanov; Eric Moulines; Ivan Oseledets; Maxim Panov; Alexander Korotin

cs.LGFeb 22, 2026

IDLM: Inverse-distilled Diffusion Language Models

Authors: David Li, Nikita Gushchin, Dmitry Abulkhanov, Eric Moulines, Ivan Oseledets, Maxim Panov, Alexander Korotin

Organizations: Mohamed Bin Zayed University of AI, Abu Dhabi, UAE · Applied AI Institute, Moscow, Russia · AXXX, Russia · EPITA, Laboratoire Recherche de l’EPITA, Paris, France

Abstract

Diffusion Language Models (DLMs) have recently achieved strong results in text generation. However, their multi-step sampling leads to slow inference, limiting practical use. To address this, we extend Inverse Distillation, a technique originally developed to accelerate continuous diffusion models, to the discrete setting. Nonetheless, this extension introduces both theoretical and practical challenges. From a theoretical perspective, the inverse distillation objective lacks uniqueness guarantees, which may lead to suboptimal solutions. From a practical standpoint, backpropagation in the discrete space is non-trivial and often unstable. To overcome these challenges, we first provide a theoretical result demonstrating that our inverse formulation admits a unique solution, thereby ensuring valid optimization. We then introduce gradient-stable relaxations to support effective training. As a result, experiments on multiple DLMs show that our method, Inverse-distilled Diffusion Language Models (IDLM), reduces the number of inference steps by 4x-64x, while preserving the teacher model's generation quality. We provide the code, model checkpoints, and video tutorials on the project page: https://david-cripto.github.io/idlm-project-page

Figures & tables

Figure 1: Diffusion training vs inverse distillation. DLMs learn a slow teacher $f^{*}$ from data. IDLM fixes $f^{*}$ and learns $p_{\theta}$ for fast few-step generation.

Figure 2: IDLM training loop. The student generator $G_{\theta}$ maps noised data to a clean sample, which is forward-diffused and passed as input to the frozen teacher $f^{*}$ and the learnable fake model $f$ . Training alternates between fitting the fake model to the current student distribution and updating the generator with the IDLM loss. This loss pushes $G_{\theta}$ toward samples supported by the teacher more than by the fake model.

Panel of Figure 3 — Figure 3: OWT distillation. IDLM preserves the quality/diversity tradeoff while using far fewer reverse steps for both masked and uniform diffusion. The strongest gains appear in the low-step regime.

Figure 4: GenPPL–Entropy tradeoff on OWT. Each point denotes a Method/Sampling steps. Lower GenPPL and higher entropy are preferred.

Panel of Figure 5 — Figure 5: Qualitative examples. Representative IDLM samples. Additional qualitative samples are provided in Appendix G .

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6: IDLM-SEDD comparison with SEDD. Left: IDLM-SEDD matches SEDD generation quality and diversity while reducing sampling from $1024$ to $256$ steps ( $4\times$ ). Right: IDLM-SEDD improves GenPPL across sampling steps while preserving high entropy.

Panel of Figure 7 — Figure 7: Masked forward process. The transition $q_{t}(\cdot\mid x_{0})$ preserves the clean-token mass with probability $\alpha_{t}$ and moves the remaining mass to the mask token. Because this map is linear, the same operation applies to a one-hot token and to a differentiable simplex output $G_{\theta}(\epsilon)$ .

Figure 8: Masked-diffusion sampling procedure. The sampler starts from a fully masked sequence and reveals each position once; after a token is revealed, later reverse steps copy it. Thus the schedule decides when a position is revealed, and the model decides which token to put there.

Panel of Figure 9 — Figure 9: Simplex relaxation for IDLM-MDLM, following Section 3.2 . A generated token is ultimately interpreted as a one-hot vocabulary choice, but the generator update uses $G_{\theta}(\epsilon)\in\Delta$ so gradients can weight several candidate tokens before a hard sample is chosen.

Panel of Figure 10 — Figure 10: Loss evaluation under the simplex relaxation. The one-hot case selects a single log-probability from $\log f(x_{t},t)$ , while the simplex case evaluates the weighted sum $\langle G_{\theta}(\epsilon),\log f(x_{t},t)\rangle$ .

Figure 11: Mask-only cancellation under subs . The masked forward process produces either an unmasked token with probability $\alpha_{t}$ or the mask state with probability $1-\alpha_{t}$ . The unmasked branch is copied by subs , so teacher and fake predictions agree and its IDLM contribution cancels. The remaining generator signal comes from the masked branch.

Figure 12: Uniform diffusion sampling. Unlike masked diffusion, a position can be revised multiple times, so each step must decide both whether to hold or change the current token and which token should replace it.

	Steps	GenPPL $\downarrow$	MAUVE $\uparrow$	Entropy $\uparrow$
FLM ( Lee et al., 2026 )	1024	62.23	–	5.33
Duo a (Teacher) ( Sahoo et al., 2025 )	1024	77.69	0.96 $\pm$ 0.01	5.55
ELF ( Hu et al., 2026 )	32	24.08 $\pm$ 0.16	–	5.14
FMLM ( Lee et al., 2026 )		45.09	–	5.25
Duo-DCD a ( Sahoo et al., 2025 )		61.31	0.97 $\pm$ 0.01	5.52
IDLM-DCD a ( Ours )		42.13 $\pm$ 3.65	0.94 $\pm$ 0.01	5.41 $\pm$ 0.05

Appendix

Table 2: Duo a distillation comparison. We report GenPPL

\downarrow

, MAUVE

\uparrow

, and Entropy

\uparrow

. Methods are grouped by Steps, with the best metrics bolded in each group.

Panel of Figure 13 — Figure 13: OWT GenPPL and entropy vs. sampling steps. Each panel shows how generation quality and diversity change as the reverse sampling budget is reduced. GenPPL measures sample quality, where lower is better, while entropy tracks output diversity, where higher is better. The panels provide a unified comparison of step-scaling behavior for our IDLM-MDLM, IDLM-DCD g , and IDLM-DCD a models.

Method ( $863$ M params)	Steps	GenPPL $\downarrow$	Entropy $\uparrow$
MDLM (Teacher) ( Sahoo et al., 2024 )	1024	34.99	5.24
SDTT ( Deschenaux and Gulcehre, 2025 )	32	31.16	5.12
IDLM-MDLM (Ours)	32	13.71 $\pm$ 1.70	5.17 $\pm$ 0.10
SDTT ( Deschenaux and Gulcehre, 2025 )	16	52.13	5.20
IDLM-MDLM (Ours)	16	24.07 $\pm$ 2.13	5.36 $\pm$ 0.04
SDTT ( Deschenaux and Gulcehre, 2025 )	8	121.51	5.25

Appendix

Table 3: Scaling to an 863M MDLM teacher. We report GenPPL

\downarrow

and Entropy

\uparrow

Diffusion Language Model Inference

Diffusion language model inference studies how to generate, guide, and serve text efficiently through iterative denoising and parallel token updates.

See all 165 papers

DiLaDiff: Distilled Latent-Augmented Diffusion for Language Modeling
May 22, 2026Jean-Marie Lemercier, Tomas Geffner, Karsten Kreis +3Diffusion Language Models Diffusion Language Model Distillation
NVIDIA
Consistent Diffusion Language Models
Apr 30, 2026Hasan Amin, Yuan Gao, Yaser Souri +4Few-Step Diffusion Sampling Diffusion Model Acceleration
Department of Computer Science, Purdue University · Microsoft
Distribution Matching Distillation for Continuous Diffusion Language Models
Sep 30, 2026Paul Le Van Kiem, Dario Shariatian, Umut Simsekli +1Continuous Diffusion Language Models Diffusion Language Model Distillation
Inria, PSL Research University · Cohere · CMAP, Ecole Polytechnique

May 22, 2026J/K move · Enter open · S save

DiLaDiff: Distilled Latent-Augmented Diffusion for Language Modeling

Jean-Marie Lemercier, Tomas Geffner, Karsten Kreis, Morteza Mardani+2

NVIDIA

Diffusion Language Models Diffusion Language Model Distillation Diffusion Language Model Inference Masked Diffusion Models Continuous Diffusion Language Models Diffusion Model Inference Acceleration+1

Open paper AskPDF arXiv

Diffusion language models intrinsically fail to capture correlations between decoded tokens, which leads to a harsh trade-off between sampling quality and throughput. To solve this issue, we propose DiLaDiff, a variant of masked diffusion language models with three components: (1) a continuous latent space with semantic capabilities, learned by an auto-encoder fine-tuned from an existing masked diffusion language model; (2) a latent diffusion model learning the prior over the encoder distribution; (3) a consistency model distilling the learned prior into a few-step latent generative model. We show that, even without distillation, our latent-guided diffusion model outperforms the masked diffusion baseline while significantly accelerating inference. Consistency distillation further lowers the computational overhead of continuous diffusion, such that the latent is generated in negligible time compared to discrete decoding.

Abstract

Diffusion Language Model Inference

Explore similar work

DiLaDiff: Distilled Latent-Augmented Diffusion for Language Modeling

Consistent Diffusion Language Models

Distribution Matching Distillation for Continuous Diffusion Language Models