Diffusion Language Model Distillation

Ask about this topic

Overview

Diffusion language model distillation studies how to transfer a teacher’s knowledge or a model’s own later predictions so that text can be generated in fewer denoising steps. Common approaches include trajectory and on-policy distillation, autoregressive-to-diffusion transfer, and consistency or distribution-matching methods for discrete and continuous token representations. These methods aim to preserve reasoning, coding, and language quality while enabling more parallel decoding and lower inference cost. A central challenge is keeping training supervision aligned with the student’s inference-time states and preserving dependencies among tokens as the number of generation steps shrinks.

Latest papers

CardsList

37 papers11 papers in the last four weeks, up 120% on the four weeks before. 0.1% of all new papers.

  1. Self-Regularized Flow Language Models

    Oct 27, 2026Theodoros Kouzelis, Spyros Gidaris, Nikos KomodakisDiffusion Language Model TrainingContinuous Diffusion Language Models

    Archimedes, Athena RC · National Technical University of Athens · University of Crete +1

  2. HiLo-RD: Cross-Budget Self-Distillation for Masked Diffusion Language Models

    Oct 27, 2026Xiang Li, Chengting Yu, Jungang Li +5Diffusion Language Model DecodingDiffusion Language Models

    The Hong Kong University of Science and Technology (Guangzhou) · Zhejiang University · Nanyang Technological University +2

  3. Nemotron-Labs-Diffusion-Uni: Elastic Self-Refining Unified Diffusion LLM with Cross-Paradigm On-Policy Distillation

    Oct 27, 2026Shufan Li, Hanrong Ye, Greg Heinrich +5Diffusion VLMsDiffusion Language Models

  4. Learning When to Commit: Schedule Distillation for Fast Diffusion Language Models

    Oct 26, 2026Aidar Myrzakhan, Zhiqiang ShenDiffusion Language Model InferenceDiffusion Language Model Decoding

    VILA Lab, MBZUAI

  5. What Makes a Good Unmasking Policy? A Controlled Study of Learned Unmasking in Diffusion Language Models

    Oct 22, 2026Jinya Sakurai, Xun Xu, Jaehong YoonDiffusion Language Model DecodingDiffusion Language Models

    NTU Singapore · Institute for Advanced Intelligence and Computing (IAIC), A*STAR · Centre for Frontier AI Research (CFAR), A*STAR

  6. DStill: Trajectory Consistency Distillation for Accelerated One-Step Drafting in Block Diffusion Speculative Decoding

    Oct 19, 2026Shreyas Kulkarni, Uttam Gurram, Madhav Kanda +5Speculative Decoding for Diffusion Language ModelsLLM Inference Acceleration

    AI Foundations, Capital One

  7. Outcome-Routed Distillation and Reinforcement Learning for Diffusion Language Models

    Oct 13, 2026Wentao Lu, Tianyu Zhu, Jesse ClarkDiffusion Language Model DistillationDiffusion Language Models

    Celeris San Francisco, CA

  8. JustOPD: On-Policy Distillation from Autoregressive to Diffusion Language Models

    Oct 12, 2026Zikun Zhang, Jiayuan Sheng, David D. Yao +1Diffusion Language Model DistillationOn-Policy Distillation

    Department of Industrial Engineering and Operations Research, Columbia University, New York, NY 10027.

  9. Omni-Diffusion-Distill: Few-Step Distillation of Unified Multimodal Diffusion Large Language Models

    Oct 7, 2026Hong Huang, Chenhongyi Yang, Junzhe Sun +3Diffusion Language Model DistillationDiffusion Model Inference Acceleration

    Simon Fraser University · Meta

  10. Towards Unbiased On-Policy Distillation for Block Diffusion Language Models

    Oct 4, 2026Zaiquan Yang, Fei Wei, Yong Wang +6Blockwise Diffusion Language ModelsOn-Policy Distillation

    City University of Hong Kong · Alibaba Group · Beijing Institute of Technology +2

  11. Stabilizing language models under continual learning via condition-anchored distillation

    Oct 3, 2026Huan Li, Zhe Cao, Qinlei Xie +2Continual Learning for LLMsLanguage Model Distillation

    Institute for AI Industry Research (AIR), Tsinghua University · Nanjing University

  12. Pivot-SD: Efficient Self-Distillation for Masked Diffusion Language Models

    Oct 2, 2026Seo Hyun Kim, Sunwoo Hong, Younwoo Choi +3Diffusion Language Model DistillationMasked Diffusion Models

    KAIST AI · University of Toronto & Vector Institute

  13. Distribution Matching Distillation for Continuous Diffusion Language Models

    Sep 30, 2026Paul Le Van Kiem, Dario Shariatian, Umut Simsekli +1Continuous Diffusion Language ModelsDiffusion Language Model Distillation

    Inria, PSL Research University · Cohere · CMAP, Ecole Polytechnique

  14. d-OPD: Future-Aware On-Policy Distillation for Block Diffusion Language Models

    Sep 28, 2026Ruitao Liu, Qinghao Hu, Song HanDiffusion Model DistillationBlockwise Diffusion Language Models

    Tsinghua University · MIT · NVIDIA

  15. DreamingGoose: Staged Distillation from Autoregressive Transformers to Bidirectional Recurrent Diffusion Language Models

    Sep 28, 2026Julian Boesch, Andrew Wee, Alexander StranzlIn-Context RetrievalRecurrent Transformers

    Purdue University · Obit Research · State University of New York at Stony Brook.

  16. Hesitation-Aware On-Policy Distillation for Diffusion Language Models

    Sep 27, 2026Jianguo Huang, Lipeng Wan, Yanchen Deng +1Diffusion Language Model DecodingUncertainty-Aware Knowledge Distillation

    Nanyang Technological University, Singapore · Xi’an Jiaotong University, China

  17. ForkLeft: Entropy-First Rollouts for Prefix-Aligned Autoregressive-to-Diffusion Distillation

    Sep 26, 2026Junming Liu, Jicheng Wang, Yifeng He +2Diffusion Language Model DistillationDiffusion Language Models

    University of Melbourne · University of California, Davis · University of Hong Kong

  18. Temporal Self-Distillation: Faster Inference in Discrete Diffusion Language Models

    Sep 14, 2026Shijian Xu, Andrea Miele, Metod Jazbec +3Diffusion Model DistillationDiffusion Language Model Inference

    University of Basel · University of Amsterdam · Johns Hopkins University

  19. A Ticket from Marginals to Joints: Coupled-Noise Distillation for One-Step Block Generation in Diffusion Language Models

    Sep 6, 2026Lin YaoOne-Step Generative ModelingDiffusion Language Models

    School of Computer Science, Shanghai Jiao Tong University, Shanghai, 200240, China · Zhongguancun Academy, Beijing, 100097, China

  20. Distilled Continuous Diffusion Language Models Can Write Code in Few Steps---or One

    Sep 3, 2026Fred Zhangzhi Peng, Kaiwen Zheng, Anru R. ZhangDiffusion Model DistillationOne-Step Generative Modeling

    Duke University · Tsinghua University

  21. SelFusion: Self-distillation for Diffusion Language Models

    Aug 24, 2026Hyeongsoo Lim, Jinyoung Kim, Eunseo Seo +2Diffusion Language Model TrainingDiffusion Language Models

    Department of Artificial Intelligence, Chung-Ang University

  22. CForce: Boosting Parallel Decoding for dLLMs via Consistency Forcing

    Aug 14, 2026Yuji Ren, Chenkai Xu, Zhuocheng Gong +2Parallel Decoding for Diffusion Language ModelsDiffusion Language Model Decoding

    Shanghai Jiao Tong University · Ant Group

  23. OPTD: On-Policy Transition Distillation with Consistency-Guided Adaptive Compression for Few-Step Diffusion Language Models

    Aug 3, 2026Xiaocheng Lu, Hualei Zhang, Shuhan Guo +8Diffusion Model DistillationDiffusion Language Model Decoding

    HKUST · NWPU · PolyU

  24. Trace-Based On-Policy Distillation for Masked Diffusion Language Models

    Jul 18, 2026Haolin Ren, Ziyang Huang, Chenhao Yuan +2Masked Diffusion ModelsDiffusion Model Distillation

    Institute of Automation, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China

  25. dOPSD: On-Policy Self-Distillation for Diffusion Language Models

    Jul 5, 2026Phuong Tuan Dat, Qi Li, Xinchao WangOn-Policy Self-DistillationLanguage Model Distillation

    National University of Singapore

  26. Data-Efficient Autoregressive-to-Diffusion Language Models via On-Policy Distillation

    Jun 4, 2026Xingyu Su, Jacob Helwig, Shubham Parashar +6Language Model DistillationDiffusion Language Models

    Department of Computer Science and Engineering, Texas A&M University · Department of Bioinformatics and Systems Medicine,2026 University of Texas Health Science Center at Houston · Department of Electrical and Computer Engineering, Texas A&M UniversityJun 4

  27. GDSD: Reinforcement Learning as Guided Denoiser Self-Distillation for Diffusion Language Models

    May 28, 2026Xiaohang Tang, Keyue Jiang, Che Liu +4Diffusion Model DistillationRL for Diffusion Models

    UNIST · University of Basel

  28. DiLaDiff: Distilled Latent-Augmented Diffusion for Language Modeling

    May 22, 2026Jean-Marie Lemercier, Tomas Geffner, Karsten Kreis +3Diffusion Language ModelsDiffusion Language Model Distillation

    NVIDIA

  29. Roll Out and Roll Back: Diffusion LLMs are Their Own Efficiency Teachers

    May 16, 2026Fanqin Zeng, Feng Hong, Geng Yu +6Diffusion Model DistillationDiffusion Language Model Decoding

    Shanghai Jiao Tong University, Shanghai, China · Apple MLR, Cupertino, CA, USA · Tongji University, Shanghai, China +2

  30. Self-Distilled Trajectory-Aware Boltzmann Modeling: Bridging the Training-Inference Discrepancy in Diffusion Language Models

    May 12, 2026Kecheng Chen, Ziru Liu, Xijia Tao +9Diffusion Model DistillationDiffusion Model Fine-Tuning

    1City University of Hong Kong · 2Huawei Research · 3The University of Hong Kong