Blockwise Diffusion Language Models

Follow this topic Ask about this topic

These models generate sequences by denoising groups of masked or noisy tokens, combining parallel refinement within a block with sequential progress across blocks. Research explores fixed and adaptive block boundaries, training objectives that align denoising with inference, and samplers that use confidence, speculation, or reinforcement learning to balance parallelism and coherence. Systems work adds attention caching, sparse or linear attention, and block-level batching to speed up long-context generation and reduce memory use. The central challenge is gaining speed over autoregressive models without losing accuracy or coherent long-range reasoning, especially when training and inference use different block sizes or decoding schedules.

Latest papers

55 papers15 papers in the last four weeks, up 200% on the four weeks before. 0.1% of all new papers.

Nemotron-Labs-TwoTower: Diffusion Language Modeling with Pretrained Autoregressive Context
Jun 25, 2026Fitsum Reda, John Kamalu, Roger Waleffe +3Autoregressive Diffusion Blockwise Diffusion Language Models
NVIDIA
DreamReasoner-8B: Block-Size Curriculum Learning for Diffusion Reasoning Models
Jun 17, 2026Zirui Wu, Lin Zheng, Jiacheng Ye +5Blockwise Diffusion Language Models Efficient Language Model Reasoning
The University of Hong Kong · Peking University
Teaching Diffusion to Speculate Left-to-Right
Jun 10, 2026Lexington Whalen, Yuki Ito, Ryo SakamotoSpeculative Decoding for Diffusion Language Models Blockwise Diffusion Language Models
SB Intuitions, Tokyo, Japan
AsyncLane: Decoupling Refinement from Advancement in Diffusion Language Model Decoding
Jun 7, 2026Yingxuan Ren, Yuxuan Lou, Yong Liu +4LLM Inference Scheduling Blockwise Diffusion Language Models
1National University of Singapore · University of Southampton
SemBlock: Semantic Boundary Dynamic Blocks for Diffusion LLMs
Jun 3, 2026Xinrui Song, Zhuoran Wang, Mingju Gao +1Blockwise Diffusion Language Models Diffusion Language Model Decoding
School of Computer Science, Peking University
BlockGen: Flexible Blockwise Sequence Modeling with Hybrid Samplers
Jun 1, 2026Justin Deschenaux, Caglar GulcehreMasked Diffusion Models Diffusion Sampling
EPFL Lausanne, Switzerland · EPFL, Lausanne, Switzerland · Microsoft AI
BlockBatch: Multi-Scale Consensus Decoding for Efficient Diffusion Language Model Inference
May 28, 2026Xiaoyou Wu, Cheng-Jhih Shih, Binfei Ji +2Blockwise Diffusion Language Models Diffusion Language Model Inference
Georgia Institute of Technology
Fast-dDrive: Efficient Block-Diffusion VLM for Autonomous Driving
May 22, 2026Kewei Zhang, Jin Wang, Sensen Gao +9VLMs for Autonomous Driving End-to-End Autonomous Driving
1Peking University · 3The University of Hong Kong · 2NVIDIA +1
Dynamic Chunking for Diffusion Language Models
May 15, 2026Yichen Zhu, Xiaoming Shi, Peng Zhao +3Document Chunking Autoregressive Diffusion
CSE, HKUST · Xiaohongshu Inc. · Alibaba group +1
Break the Block: Dynamic-size Reasoning Blocks for Diffusion Large Language Models via Monotonic Entropy Descent with Reinforcement Learning
May 4, 2026Yan Jiang, Ruihong Qiu, Zi HuangBlockwise Diffusion Language Models Diffusion Language Model Decoding
School of Electrical Engineering and Computer Science, The University of Queensland, Brisbane, Queensland, Australia
On the Trainability of Masked Diffusion Language Models via Blockwise Locality
Apr 27, 2026Yuxiang Wang, Yu Xiang, Baojian Zhou +4Masked Diffusion Models Masked Language Modeling
1Fudan University, Shanghai, China · 2Alibaba Group, China.
When to Commit? Towards Variable-Size Self-Contained Blocks for Discrete Diffusion Language Models
Apr 27, 2026Danny Wang, Ruihong Qiu, Zi HuangBlockwise Diffusion Language Models Diffusion Language Model Inference
The University of Queensland
LoSA: Locality Aware Sparse Attention for Block-Wise Diffusion Language Models
Apr 13, 2026Haocheng Xi, Harman Singh, Yuezhou Hu +9Diffusion Language Model Inference Diffusion Model Inference Acceleration
University of California, Berkeley · FuriosaAI · ICSI +2
ECHO: Efficient Chest X-ray Report Generation with One-step Block Diffusion
Apr 10, 2026Lifeng Chen, Tianqi You, Hao Liu +8Radiology Report Generation Radiology VLMs
AIRC, Midea Group · Beijing Jiaotong University · Dalian University of Technology
Breaking Block Boundaries: Anchor-based History-stable Decoding for Diffusion Large Language Models
Apr 10, 2026Shun Zou, Yong Wang, Zehui Chen +4Diffusion Language Model Decoding Diffusion Model Inference Acceleration
MoE Key Lab of BIPC, University of Science and Technology of China · AMAP, Alibaba Group
Fast-dVLM: Efficient Block-Diffusion VLM via Direct Conversion from Autoregressive VLM
Apr 8, 2026Chengyue Wu, Shiyi Lan, Yonggan Fu +9Vision-Language Models Diffusion VLMs
The University of Hong Kong · NVIDIA · MBZUAI +1
S2D2: Fast Decoding for Diffusion LLMs via Training-Free Self-Speculation
Mar 26, 2026Ligong Han, Hao Wang, Han Gao +2Speculative Decoding for Diffusion Language Models Diffusion Language Model Decoding
Red Hat AI Innovation · MIT-IBM Watson AI Lab · Iowa State University +1
MinerU-Diffusion: Rethinking Document OCR as Inverse Rendering via Diffusion Decoding
Mar 23, 2026Hejun Dong, Junbo Niu, Bin Wang +3Optical Character Recognition Diffusion Language Model Decoding
Shanghai Artificial Intelligence Laboratory, OpenDataLab · Peking University
Mask Is What DLLM Needs: A Masked Data Training Paradigm for Diffusion LLMs
Mar 16, 2026Linrui Ma, Yufei Cui, Kai Han +1Diffusion Language Models Blockwise Diffusion Language Models
Noah’s Ark Lab, Huawei Montreal, Canada & Beijing, China
GeoBlock: Inferring Block Granularity from Dependency Geometry in Diffusion Language Models
Mar 4, 2026Lipeng Wan, Junjie Ma, Jianhui Gu +3Diffusion Language Model Decoding Blockwise Diffusion Language Models
Xi’an Jiaotong University Xi’an, Shaanxi, China
LightningRL: Breaking the Accuracy-Parallelism Trade-off of Block-wise dLLMs via Reinforcement Learning
Mar 4, 2026Yanzhe Hu, Yijie Jin, Pengfei Liu +2Parallel Decoding for Diffusion Language Models Blockwise Diffusion Language Models
Shanghai Jiao Tong University · Huazhong University of Science and Technology
Locally Coherent Parallel Decoding in Diffusion Language Models
Mar 3, 2026Michael Hersche, Nicolas Menet, Ronan Tanios +1Parallel Decoding for Diffusion Language Models Diffusion Language Model Decoding
IBM Research - Zurich · Department of Computer Science, ETH Zürich
DODO: Discrete OCR Diffusion Models
Feb 18, 2026Sean Man, Gilad Deutch, Roy Ganz +4Optical Character Recognition Diffusion Language Model Decoding
Technion - Israel Institute of Technology, Haifa, Israel. · Amazon Web Services.
FlashBlock: Attention Caching for Efficient Long-Context Block Diffusion
Feb 5, 2026Zhuokun Chen, Jianfei Cai, Bohan ZhuangVideo Diffusion Models Self-Attention
1Monash University · 2Zhejiang University
Streaming-dLLM: Accelerating Diffusion LLMs via Suffix Pruning and Dynamic Decoding
Jan 25, 2026Zhongyu Xiao, Zhiwei Hao, Jianyuan Guo +4Adaptive Inference Blockwise Diffusion Language Models
School of information and Electronics, Beijing Institute of Technology, Beijing · Department of Computer Science, City University of Hong Kong, Hong Kong · School of Computer Science, Wuhan University, Wuhan +2

Jun 25, 2026J/K move · Enter open · S save

Nemotron-Labs-TwoTower: Diffusion Language Modeling with Pretrained Autoregressive Context

Fitsum Reda, John Kamalu, Roger Waleffe, Mostofa Patwary+2

NVIDIA

Autoregressive Diffusion Blockwise Diffusion Language Models Diffusion Language Model Inference Diffusion Language Models

Open paper AskPDF arXiv

Diffusion language models offer a promising alternative to autoregressive models due to their potential for parallel and iterative generation. However, existing approaches use a single network for both context representation and iterative denoising, forcing one model to serve both roles and limiting its capacity for either role. We propose TwoTower, a block-wise autoregressive diffusion model that decouples these roles into two towers: a frozen AR context tower that causally processes clean tokens, and a trainable diffusion denoiser tower with bidirectional block attention that refines noisy blocks via cross-attention to the context. Built on Nemotron-3-Nano-30B-A3B, an open-weight 30B hybrid Mamba-Transformer MoE model, and trained on approximately 2.1T tokens, Nemotron-Labs-TwoTower retains 98.7% of the autoregressive baseline's quality while offering 2.42X higher wall-clock generation throughput. We release the code and model weights at https://huggingface.co/collections/nvidia/nemotron-labs-twotower.