Diffusion Language Models

Follow this topic Ask about this topic

Momentum

55 papers in the last four weeks, up 450% on the four weeks before. 0.5% of all new papers.

Jul 13Week of Sep 28

Latest papers 306

All topics

Enabling Approximate Joint Sampling in Diffusion LMs
Sep 25, 2025Parikshit Bansal, Sujay SanghaviDiffusion Language Models Diffusion Models
UT Austin
Lavida-O: Elastic Large Masked Diffusion Models for Unified Multimodal Understanding and Generation
Sep 23, 2025Shufan Li, Jiuxiang Gu, Kangning Liu +4Multimodal Understanding Masked Diffusion Language Models
Adobe · UCLA
Any-Order GPT as Masked Diffusion Model: Decoupling Formulation and Architecture
Jun 24, 2025Shuchen Xue, Tianyu Xie, Tianyang Hu +5Diffusion Language Models Masked Diffusion Language Models
University of Chinese Academy of Sciences · Academy of Mathematics and Systems Science, Chinese Academy of Sciences · School of Mathematical Sciences, Peking University +2
Dimple: Discrete Diffusion Multimodal Large Language Model with Parallel Decoding
May 22, 2025Runpeng Yu, Xinyin Ma, Xinchao WangDiffusion Language Models Autoregressive Model
National University of Singapore
Distilled Continuous Diffusion Language Models Can Write Code in Few Steps---or One
Date pendingFred Zhangzhi Peng, Kaiwen Zheng, Anru R. ZhangCode Generation Diffusion Language Models
Duke University · Tsinghua University
SAC-Copula: Quality-Preserving Watermarking for Diffusion Language Models via Smooth Correlated Gumbel Fields
Date pendingBaixin Li, Haiyun HeWatermarking Diffusion Language Models
The Hong Kong University of Science and Technology (Guangzhou)

Sep 25, 2025, cs.CLJ/K move · Enter open · S save

Enabling Approximate Joint Sampling in Diffusion LMs

Parikshit Bansal, Sujay Sanghavi

UT Austin

Diffusion Language Models Diffusion Models Accelerated Sampling Forward Propagation Dream

Open paper AskPDF arXiv

In autoregressive language models, each token is sampled by conditioning on all the past tokens; the overall string has thus been sampled from the correct underlying joint distribution represented by the model. In contrast, masked diffusion language models generate text by unmasking tokens out of order and potentially in parallel. Generating an overall string sampled from the correct underlying joint distribution would (again) require exactly one token unmasking in every full-model forward pass. The more tokens unmasked in parallel, the further away the string is from the true joint; this can be seen in the resulting drop in accuracy (but, increase in speed). In this paper we devise a way to {\em approximately} sample multiple tokens from the joint distribution in a single full-model forward pass; we do so by developing a new lightweight single-layer ``sampler" on top of an existing large diffusion LM. One forward pass of the full model can now be followed by multiple forward passes of only this sampler layer, to yield multiple unmasked tokens. Our sampler is trained to mimic exact joint sampling from the (frozen) full model. We show the effectiveness of our approximate joint sampling for both pretrained-only (Dream-7B-Base, Llada-7B-Base) and instruction-tuned (Dream-7B-Instruct, Dream-7B-Coder) models on language modeling and math & coding tasks. When four tokens are unmasked for each full-model denoising step, our sampling algorithm achieves a MAUVE score of 0.87 (vs marginal baseline of 0.31) with respect to the true joint distribution.

Momentum

Related topics

Latest papers 306

Enabling Approximate Joint Sampling in Diffusion LMs

Lavida-O: Elastic Large Masked Diffusion Models for Unified Multimodal Understanding and Generation

Any-Order GPT as Masked Diffusion Model: Decoupling Formulation and Architecture

Dimple: Discrete Diffusion Multimodal Large Language Model with Parallel Decoding

Distilled Continuous Diffusion Language Models Can Write Code in Few Steps---or One

SAC-Copula: Quality-Preserving Watermarking for Diffusion Language Models via Smooth Correlated Gumbel Fields