Diffusion Language Model Distillation

Follow this topic Ask about this topic

Diffusion language model distillation studies how to transfer a teacher’s knowledge or a model’s own later predictions so that text can be generated in fewer denoising steps. Common approaches include trajectory and on-policy distillation, autoregressive-to-diffusion transfer, and consistency or distribution-matching methods for discrete and continuous token representations. These methods aim to preserve reasoning, coding, and language quality while enabling more parallel decoding and lower inference cost. A central challenge is keeping training supervision aligned with the student’s inference-time states and preserving dependencies among tokens as the number of generation steps shrinks.

Latest papers

37 papers11 papers in the last four weeks, up 120% on the four weeks before. 0.1% of all new papers.

Infinite Mask Diffusion for Few-Step Distillation
May 11, 2026Jaehoon Yoo, Wonjung Kim, Chanhyuk Lee +1Masked Diffusion Models Diffusion Model Distillation
Korea Advanced Institute of Science and Technology (KAIST)
TAD: Temporal-Aware Trajectory Self-Distillation for Fast and Accurate Diffusion LLM
May 10, 2026Haoyang Zhou, Li Kong, Shijie Ren +4Diffusion Model Distillation Diffusion Language Model Inference
Gaoling School of Artificial Intelligence, Renmin University of China · Ant Group
Turning the TIDE: Cross-Architecture Distillation for Diffusion Large Language Models
Apr 29, 2026Gongbo Zhang, Wen Wang, Ye Tian +1Cross-Tokenizer Knowledge Distillation Diffusion Model Distillation
Peking University · Zhejiang University
ECHO: Efficient Chest X-ray Report Generation with One-step Block Diffusion
Apr 10, 2026Lifeng Chen, Tianqi You, Hao Liu +8Radiology Report Generation Radiology VLMs
AIRC, Midea Group · Beijing Jiaotong University · Dalian University of Technology
Beyond Single Tokens: Distilling Discrete Diffusion Models via Discrete MMD
Mar 20, 2026Emiel Hoogeboom, David Ruhe, Jonathan Heek +2Diffusion Model Distillation Diffusion Models
Google DeepMind Amsterdam
IDLM: Inverse-distilled Diffusion Language Models
Feb 22, 2026David Li, Nikita Gushchin, Dmitry Abulkhanov +4Diffusion Language Model Inference Diffusion Language Model Distillation
Mohamed Bin Zayed University of AI, Abu Dhabi, UAE · Applied AI Institute, Moscow, Russia · AXXX, Russia +1
Few-Step Diffusion Language Models via Trajectory Self-Distillation
Feb 12, 2026Tunyu Zhang, Xinxi Zhang, Ligong Han +10Diffusion Model Distillation Few-Step Diffusion Sampling
Rutgers University · Red Hat AI Innovation · MIT-IBM Watson AI Lab

May 11, 2026J/K move · Enter open · S save

Infinite Mask Diffusion for Few-Step Distillation

Jaehoon Yoo, Wonjung Kim, Chanhyuk Lee, Seunghoon Hong

Korea Advanced Institute of Science and Technology (KAIST)

Masked Diffusion Models Diffusion Model Distillation Few-Step Diffusion Sampling Diffusion Language Models Diffusion Models Diffusion Language Model Distillation

Open paper AskPDF arXiv

Masked Diffusion Models (MDMs) have emerged as a promising alternative to autoregressive models in language modeling, offering the advantages of parallel decoding and bidirectional context processing within a simple yet effective framework. Specifically, their explicit distinction between masked tokens and data underlies their simple framework and effective conditional generation. However, MDMs typically require many sampling iterations due to factorization errors stemming from simultaneous token updates. We observe that a theoretical lower bound of the factorization error exists, which standard MDMs cannot reduce due to their use of a deterministic single-state mask. In this paper, we propose the Infinite Mask Diffusion Model (IMDM), which introduces a stochastic infinite-state mask to mitigate the theoretical bound while directly inheriting the benefits of MDMs, including the compatibility with pre-trained weights. We empirically demonstrate that MDM fails to perform few-step generation even in a simple synthetic task due to the factorization error bound, whereas IMDM can find an efficient solution for the same task. Finally, when equipped with appropriate distillation methods, IMDM surpasses existing few-step distillation methods at small step counts on LM1B and OpenWebText. Code is available at https://Ugness.github.io/official_imdm.