Diffusion Language Models

Follow this topic Ask about this topic

Diffusion language models generate text by starting from corrupted or masked sequences and repeatedly refining them, using context from both earlier and later positions. Main approaches include discrete masked diffusion, continuous latent or flow-based models, and block or hybrid designs that combine parallel refinement with autoregressive structure. Work explores adaptive token ordering, remasking, distillation, and caching to improve reasoning, code generation, multimodal tasks, and inference speed. A central tension is balancing faster, more parallel decoding against token dependencies and output quality, while reliable evaluation and the cost of training from scratch remain open challenges.

Latest papers

473 papers99 papers in the last four weeks, up 130% on the four weeks before. 0.6% of all new papers.

Self-Distilled Trajectory-Aware Boltzmann Modeling: Bridging the Training-Inference Discrepancy in Diffusion Language Models
May 12, 2026Kecheng Chen, Ziru Liu, Xijia Tao +9Diffusion Model Distillation Diffusion Model Fine-Tuning
1City University of Hong Kong · 2Huawei Research · 3The University of Hong Kong
Block-R1: Rethinking the Role of Block Size in Multi-domain Reinforcement Learning for Diffusion Large Language Models
May 12, 2026Yan Jiang, Ruihong Qiu, Zi HuangRL Benchmarks Reinforcement Learning
The University of Queensland
DiffScore: Text Evaluation Beyond Autoregressive Likelihood
May 12, 2026Wen Lai, Yingli Shen, Dingnan Jin +4LLM Evaluation Language Model Generation Evaluation
1Ant Group · 2Tsinghua University · 3Technical University of Munich
BitLM: Unlocking Multi-Token Language Generation with Bitwise Continuous Diffusion
May 12, 2026Shaobin Zhuang, Yuang Ai, Jiaming Han +7Multi-Token Prediction Autoregressive Language Modeling
Shanghai Jiao Tong University · MMLab, The Chinese University of Hong Kong · Institute of Automation, Chinese Academy of Sciences +1
ELF: Embedded Language Flows
May 11, 2026Keya Hu, Linlu Qiu, Yiyang Lu +5Flow-Based Generative Modeling Diffusion Language Models
MIT
Infinite Mask Diffusion for Few-Step Distillation
May 11, 2026Jaehoon Yoo, Wonjung Kim, Chanhyuk Lee +1Masked Diffusion Models Diffusion Model Distillation
Korea Advanced Institute of Science and Technology (KAIST)
Relative Score Policy Optimization for Diffusion Language Models
May 11, 2026Zichao Yu, Shengze Xu, Bingqing Jiang +2RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards
University of Science and Technology of China · The Chinese University of Hong Kong · The University of Hong Kong
TrajDLM: Topology-Aware Block Diffusion Language Model for Trajectory Generation
May 11, 2026Wilson Wongso, Lihuan Li, Arian Prabowo +4Trajectory Generation Synthetic Data Generation
University of New South Wales · Hong Kong University of Science and Technology (Guangzhou)
Dystruct: Dynamically Structured Diffusion Language Model Decoding via Bayesian Inference
May 10, 2026Bian Sun, Kevin Zhai, Mubarak Shah +1Diffusion Language Model Decoding Diffusion Language Models
University of Central Florida
Edit-Based Refinement for Parallel Masked Diffusion Language Models
May 10, 2026Houxing Ren, Mingjie Zhan, Zimu Lu +5Masked Diffusion Models Parallel Decoding
CUHK MMLab · SenseTime Research · Shenzhen Loop Area Institute +1
TAD: Temporal-Aware Trajectory Self-Distillation for Fast and Accurate Diffusion LLM
May 10, 2026Haoyang Zhou, Li Kong, Shijie Ren +4Diffusion Model Distillation Diffusion Language Model Inference
Gaoling School of Artificial Intelligence, Renmin University of China · Ant Group
JUMP: Efficient Membership Inference on Fine-Tuned Diffusion Language Models
May 9, 2026Yeachan Jun, Albert NoMembership Inference Attacks Privacy Leakage in Language Models
Department of Artificial Intelligence Yonsei University, Seoul, Korea
Steering Without Breaking: Mechanistically Informed Interventions for Discrete Diffusion Language Models
May 8, 2026Hanhan Zhou, Shamik Roy, Rashmi GangadharaiahLanguage Model Steering Diffusion Language Model Decoding
AWS AI Labs Santa Clara, CA 95054
Fast Byte Latent Transformer
May 8, 2026Julie Kallini, Artidoro Pagnoni, Tomasz Limisiewicz +5Speculative Decoding LLM Inference Acceleration
FAIR at Meta · Stanford University · University of Washington
How to Train Your Latent Diffusion Language Model Jointly With the Latent Space
May 8, 2026Viacheslav Meshchaninov, Alexander Shabalin, Egor Chimbulatov +4Non-Autoregressive Text Generation Latent Diffusion Models
Constructor University, Bremen, Germany · HSE University, Moscow, Russia · Applied AI Institute, Moscow, Russia +2
TextLDM: Language Modeling with Continuous Latent Diffusion
May 8, 2026Jiaxiu Jiang, Jingjing Ren, Wenbo Li +10Diffusion Transformer Latent Diffusion Models
Joy Future Academy · HIT · HKUST(GZ)
Guidance Is Not a Hyperparameter: Learning Dynamic Control in Diffusion Language Models
May 8, 2026Fan Zhou, Tim Van de CruysClassifier-Free Guidance Controllable Text Generation
KU Leuven
GPO-V: Jailbreak Diffusion Vision Language Model by Global Probability Optimization
May 8, 2026Yu Pan, Andi Zhang, Yi Wang +2Diffusion Model Alignment Adversarial Attacks on VLMs
ShanghaiTech University · University of Warwick · SUN YAT-SEN UNIVERSITY
DiffRetriever: Parallel Representative Tokens for Retrieval with Diffusion Language Models
May 8, 2026Shuai Wang, Yu Yin, Shengyao Zhuang +2Multi-Vector Retrieval Diffusion Language Models
The University of Queensland, Brisbane, QLD, Australia · CSIRO
DACA-GRPO: Denoising-Aware Credit Assignment for Reinforcement Learning in Diffusion Language Models
May 8, 2026Amin Karimi Monsefi, Dominic Culver, Nikhil Bhendawade +4Group Relative Policy Optimization Credit Assignment in RL
The Ohio State University · Apple
CoBit: Language Modeling with Bitstream Diffusion
May 7, 2026Georgios Batzolis, Mark Girolami, Luca AmbrogioniDiffusion Language Model Decoding Diffusion Language Models
Department of Engineering, University of Cambridge. · Devotion AI Labs Ltd. · Donders Institute for Brain, Cognition, and Behaviour, Radboud University.
Don't Retrain, Align: Adapting Autoregressive LMs to Diffusion LMs via Representation Alignment
May 7, 2026Fred Zhangzhi Peng, Alexis Fox, Anru R. Zhang +1Masked Diffusion Models Diffusion Language Models
Duke University · AITHYRA
Continuous Latent Diffusion Language Model
May 7, 2026Hongcan Guo, Qinyu Zhao, Yian Zhao +8Non-Autoregressive Text Generation Diffusion Transformer
ByteDance Seed · The Australian National University · Peking University +2
Layer Collapse in Diffusion Language Models
May 7, 2026Alexander Conzelmann, Albert Catalan-Tatjer, Shiwei LiuLLM Compression Diffusion Language Models
Tübingen AI Center · Max Planck Institute for Intelligent Systems · ELLIS Institute Tübingen
Masked Diffusion Language Models are Strong and Steerable Text-Based World Models for Agentic RL
May 7, 2026Darshan DeshpandeLLM World Models World Models
Patronus AI
Chainwash: Multi-Step Rewriting Attacks on Diffusion Language Model Watermarks
May 6, 2026Mohd Ruhul Ameen, Akif Islam, Nadim Mahmud +1Diffusion Model Watermarking Watermark Removal
College of Engineering and Computer Science, Marshall University, Huntington, WV, USA · Department of Computer Science and Engineering, University of Rajshahi, Rajshahi, Bangladesh · Miami University, Oxford, OH 45056, USA
Leveraging Pretrained Language Models as Energy Functions for Glauber Dynamics Text Diffusion
May 5, 2026Tarun Kathuria, Sachin KumarLanguage Modeling Diffusion Language Models
UC Berkeley · Ohio State University
Towards A Generative Protein Evolution Machine with DPLM-Evo
Apr 30, 2026Xinyou Wang, Liang Hong, Jiasheng Ye +4Protein Language Models Molecular Generation
ByteDance Seed · Nanjing University · CUHK +1
Consistent Diffusion Language Models
Apr 30, 2026Hasan Amin, Yuan Gao, Yaser Souri +4Few-Step Diffusion Sampling Diffusion Model Acceleration
Department of Computer Science, Purdue University · Microsoft
Turning the TIDE: Cross-Architecture Distillation for Diffusion Large Language Models
Apr 29, 2026Gongbo Zhang, Wen Wang, Ye Tian +1Cross-Tokenizer Knowledge Distillation Diffusion Model Distillation
Peking University · Zhejiang University

May 12, 2026J/K move · Enter open · S save

Self-Distilled Trajectory-Aware Boltzmann Modeling: Bridging the Training-Inference Discrepancy in Diffusion Language Models

Kecheng Chen, Ziru Liu, Xijia Tao, Hui Liu+8

1City University of Hong Kong · 2Huawei Research · 3The University of Hong Kong

Diffusion Model Distillation Diffusion Model Fine-Tuning Diffusion Language Model Decoding Self-Distillation Diffusion Language Models Diffusion Language Model Distillation

Open paper AskPDF arXiv

Diffusion Language Models (DLMs) have recently emerged as a promising alternative to autoregressive language models, offering stronger global awareness and highly parallel generation. However, post-training DLMs with standard Negative Evidence Lower Bound (NELBO)-based supervised fine-tuning remains inefficient: training reconstructs randomly masked tokens in a single step, whereas inference follows a confidence-guided, multi-step easy-to-hard denoising trajectory. Recent trajectory-based self-distillation methods exploit such inference trajectories mainly for sampling-step compression and acceleration, often improving decoding efficiency without substantially enhancing the model's underlying capability, and may even degrade performance under full diffusion decoding. In this work, we ask whether self-distilled trajectories can be used not merely for faster inference, but for genuine knowledge acquisition. Although these trajectories lie on the pretrained DLM's own distributional manifold and thus offer a potentially lower optimization barrier, we find that naively fine-tuning on them with standard NELBO objectives yields only marginal gains. To address this limitation, we propose Trajectory-Aligned optimization via Boltzmann Modeling (TABOM), a self-distilled trajectory-based post-training framework that aligns training with the easy-to-hard structure of inference. TABOM models the inference unmasking preference as a Boltzmann distribution over predictive entropies and derives a tractable pairwise ranking objective to align the model's certainty ordering with the observed decoding trajectory. Empirically, TABOM achieves substantial gains in new domains, expands the effective knowledge boundary of DLMs, and significantly mitigates catastrophic forgetting compared with standard SFT.