SemBlock: Semantic Boundary Dynamic Blocks for Diffusion LLMs

Xinrui Song; Zhuoran Wang; Mingju Gao; Hao Tang

cs.CLJun 3, 2026

SemBlock: Semantic Boundary Dynamic Blocks for Diffusion LLMs

Authors: Xinrui Song, Zhuoran Wang, Mingju Gao, Hao Tang

Organizations: School of Computer Science, Peking University

Abstract

Diffusion language models (DLMs) generate text through iterative denoising, and blockwise decoding improves their practicality by committing tokens in local blocks. However, existing blockwise methods typically rely on fixed block sizes or delimiter-based runtime signals, which do not necessarily align with semantic boundaries. In this paper, we propose SemBlock, a semantic-boundary-driven dynamic block decoding framework for diffusion LLMs. SemBlock formulates dynamic block construction as semantic boundary prediction and trains lightweight predictors on frozen LLaDA hidden states. To provide supervision, we construct SemBound, a semantic-boundary dataset that derives boundary labels from discourse units, reasoning steps, and implementation spans across natural language, math, and code tasks. During inference, SemBlock uses predicted boundary probabilities to select the ending position of each dynamic block. Experiments on GSM8K, IFEval, MATH, and HumanEval show that SemBlock consistently improves over fixed-block decoding and AdaBlock. Our code is publicly available: https://github.com/TH-AI-Lab-PKU/SemBlock.

Explore similar work

Multi-Block Diffusion Language Models
Jun 28, 2026Yijie Jin, Jiajun Xu, Yuxuan Liu +8Diffusion Language Models Few-Step Generation
Dynamic Chunking for Diffusion Language Models
May 15, 2026Yichen Zhu, Xiaoming Shi, Peng Zhao +3Diffusion Language Models Chunk
AsyncLane: Decoupling Refinement from Advancement in Diffusion Language Model Decoding
Jun 7, 2026Yingxuan Ren, Yuxuan Lou, Yong Liu +4Diffusion Language Models Autoregressive Decoding

Jun 28, 2026 · cs.LGJ/K move · Enter open · S save

Multi-Block Diffusion Language Models

Yijie Jin, Jiajun Xu, Yuxuan Liu, Chenkai Xu+7

Shanghai Jiao Tong University · Xi’an Jiao Tong University · Huawei

Block Diffusion Language Models (BD-LMs) improve diffusion-based text generation with KV caching and flexible-length generation. A natural next step is to extend them from Single-Block Diffusion (SingleBD) to Multi-Block Diffusion (MultiBD), where a running-set of consecutive blocks is decoded concurrently for inter-block parallelism. However, existing BD-LMs are mostly trained under teacher forcing, where the model observes only one noisy block conditioned on a clean prefix. While the recent diffusion forcing strategy introduces visibility among multiple noisy blocks, its training states still differ from MultiBD inference, where decoding operates on a bounded running-set with heterogeneous slot-wise noise patterns. To bridge this gap, we propose Multi-Block Diffusion Language Models (MBD-LMs), obtained by post-training BD-LMs with Multi-block Teacher Forcing (MultiTF). MultiTF integrates teacher forcing and diffusion forcing by training on bounded noise-groups conditioned on clean prefixes, with randomized noise-schedulers that better match MultiBD inference states. To make MultiBD practically executable, we further introduce an optimized decoding algorithm based on the Block Buffer mechanism that preserves prefix-cache reuse, keeps input shapes static, and translates increased decoding parallelism into wall-clock acceleration. Empirically, MBD-LLaDA2-Mini increases average Tokens Per Forward pass (TPF) from 3.47 to 6.19 and improves average accuracy from 79.95% to 81.03%; when combined with DMax, MBD-LLaDA2-Mini-DMax reaches an average TPF of 9.34 with only a 1.02% accuracy drop on math and code benchmarks.

Diffusion Language Models Few-Step Generation

Open paper Ask about it PDF

Abstract

Explore similar work

Multi-Block Diffusion Language Models

Dynamic Chunking for Diffusion Language Models

AsyncLane: Decoupling Refinement from Advancement in Diffusion Language Model Decoding