GeoBlock: Inferring Block Granularity from Dependency Geometry in Diffusion Language Models

Lipeng Wan; Junjie Ma; Jianhui Gu; Zeyang Liu; Xuyang Lu; Xuguang Lan

cs.CLMar 4, 2026

GeoBlock: Inferring Block Granularity from Dependency Geometry in Diffusion Language Models

Authors: Lipeng Wan, Junjie Ma, Jianhui Gu, Zeyang Liu, Xuyang Lu, Xuguang Lan

Organizations: Xi’an Jiaotong University Xi’an, Shaanxi, China

Abstract

Block diffusion enables efficient parallel refinement in diffusion language models, but its decoding behavior depends critically on block size. Existing block-sizing strategies rely on fixed rules or heuristic signals and do not account for the dependency geometry that determines which tokens can be safely refined together. This motivates a geometry view of diffusion decoding: regions with strong causal ordering require sequential updates, whereas semantically cohesive regions admit parallel refinement. We introduce GeoBlock, a geometry-aware block inference framework that determines block granularity directly from attention-derived dependency geometry. Instead of relying on predefined schedules or local confidence heuristics, GeoBlock analyzes cross-token dependency patterns to identify geometrically stable refinement regions and dynamically determines appropriate block boundaries during decoding. By adapting block granularity to the dependency geometry, GeoBlock preserves the parallel efficiency of block diffusion while enforcing dependency-consistent refinement that exhibits autoregressive reliability. GeoBlock requires no additional training and integrates seamlessly into existing block diffusion architectures. Extensive experiments across multiple benchmarks show that GeoBlock reliably identifies geometry-consistent block boundaries and improves the accuracy of block diffusion with only a small additional computational budget.

Figures & tables

Figure 1. Local dependency patterns in natural language. Tokens with strong mutual interactions are highlighted using the same color. (a) A locally causal-dominated dependency pattern, where tokens follow a strong directional chain and intermediate tokens exhibit weak local interactions that do not form large, densely connected regions. Only small-size parallel updates are stable in such regions. (b) A locally parallel-dominated pattern, where tokens form dense mutual connections, enabling multi-token parallel refinement. These contrasting structures motivate adaptive block-size selection in dependency-aware diffusion decoding.

Figure 2. Dependency geometry of decoding under different regimes. (a) Autoregressive decoding induces a strictly sequential dependency structure, yielding a triangular attention pattern. (b) Block diffusion groups tokens into contiguous blocks, allowing bidirectional interactions within blocks while preserving causal ordering across blocks. (c) GeoBlock decomposes frontier attention into structured regions over historical tokens, candidate blocks, and future tokens, highlighting internal coupling, past conditioning, and future leakage. (d) Geo boundary selection evaluates candidate boundaries using a closure score and selects the right-shifted boundary within a tolerance of the maximum.

Figure 3. Accuracy versus number of function evaluations (NFE) on HumanEval, MBPP, GSM8K, and IFEval under Dream-7B and LLaDA-8B backbones.

		Vanilla			Dynamic			AdaBlock			GeoBlock
Benchmark		B16	B32	B64	B16	B32	B64	$B_{0}16$	$B_{0}32$	$B_{0}64$	$B_{max}16$	$B_{max}32$	$B_{max}64$
Dream-7B
GSM8K	ACC	74.60	73.62	74.83	75.13	74.37	74.68	74.98	75.44	73.31	77.79 $\uparrow$	76.88 $\uparrow$	77.86 $\uparrow$
GSM8K	NFE	256	256	256	151.72	154.69	160.83	167.44	161.62	161.26	169.80	167.96	168.92
HumanEval	ACC	57.93	55.49	57.93	53.66	59.15	55.49	51.83	53.66	57.93	55.49 $\uparrow$	59.15	55.49
HumanEval	NFE	512	512	512	258.98	266.19	267.44	303.15	294.35	289.89	292.10	288.53	292.10

Table 1. Accuracy (Acc) and decoding cost (NFE) across benchmarks. Bold indicates best accuracy per row. Red/green arrows denote improvement/decline of GeoBlock over Dynamic at matched block size.

	Dream		LLaDA
Benchmark	Block Length	Extra NFE	Block Length	Extra NFE
GSM8K	$13.72\pm 5.87$	0.08	$13.42\pm 5.69$	0.08
HumanEval	$14.33\pm 4.95$	0.12	$14.99\pm 5.26$	0.14
MBPP	$16.54\pm 4.79$	0.15	$19.76\pm 5.96$	0.29
MATH	$11.81\pm 5.14$	0.14	$11.67\pm 5.31$	0.15
IFEval	$17.86\pm 6.05$	0.07	$18.65\pm 6.35$	0.08

Table 2. Block length and additional NFE ratio of GeoBlock.

Benchmark	Metric	0	0.25	0.5	1.0
Dream-7B	Acc	76.65	76.88	76.88	76.73
Dream-7B	NFE	169.40	164.15	167.96	172.57
LLaDA-8B	Acc	81.19	81.35	81.88	80.82
LLaDA-8B	NFE	98.72	102.51	100.05	103.05

Table 3. Effect of the anchoring coefficient under matched decoding budgets.

$\delta$	Acc	NFE	Block Len.
0	80.74	115.03	$6.59\pm 3.12$
0.05	80.74	102.99	$10.28\pm 4.21$
0.1	81.88	100.05	$13.42\pm 5.69$
0.2	80.21	98.64	$19.57\pm 9.33$

Table 4. Effect of the right-shift tolerance on boundary selection.

Layer	Weight	Acc	NFE
2-6-10	0.1, 0.3, 0.6	79.15	123.33
	0.6, 0.3, 0.1	81.12	127.56
	0.333, 0.333, 0.334	80.67	126.26
16-21-26	0.1, 0.3, 0.6	80.59	100.61
	0.6, 0.3, 0.1	81.12	105.35
	0.333, 0.333, 0.334	81.88	100.05

Table 5. Layer and weight configurations on GSM8K (LLaDA, threshold=0.9,

\delta=0

Figure 4. Block boundary inference under dependency geometry during diffusion decoding. Top: Fused attention map within the frontier window, revealing internal cohesion and cross-boundary dependencies. Bottom: Block closure score as a function of candidate boundary position. The selected boundary (dashed line) corresponds to the largest candidate block whose closure score remains near-optimal, allowing stable commitment of a structurally coherent token subset.

Diffusion Language Model Decoding

Diffusion language model decoding generates text through iterative parallel refinement, balancing fewer denoising steps with accuracy, coherence, and control.

See all 250 papers

SemBlock: Semantic Boundary Dynamic Blocks for Diffusion LLMs
Jun 3, 2026Xinrui Song, Zhuoran Wang, Mingju Gao +1Blockwise Diffusion Language Models Diffusion Language Model Decoding
School of Computer Science, Peking University
Adaptive Block Diffusion: Resolving Training-Inference Mismatch in Diffusion Language Models
Jun 28, 2026Gagan JainDiffusion Language Model Training Blockwise Diffusion Language Models
Work done while at Microsoft AI.
Multi-Block Diffusion Language Models
Jun 28, 2026Yijie Jin, Jiajun Xu, Yuxuan Liu +8Blockwise Diffusion Language Models Diffusion Language Model Decoding
Shanghai Jiao Tong University · Xi’an Jiao Tong University · Huawei

Jun 3, 2026J/K move · Enter open · S save

SemBlock: Semantic Boundary Dynamic Blocks for Diffusion LLMs

Xinrui Song, Zhuoran Wang, Mingju Gao, Hao Tang

School of Computer Science, Peking University

Blockwise Diffusion Language Models Diffusion Language Model Decoding Diffusion Language Models

Open paper AskPDF arXiv

Diffusion language models (DLMs) generate text through iterative denoising, and blockwise decoding improves their practicality by committing tokens in local blocks. However, existing blockwise methods typically rely on fixed block sizes or delimiter-based runtime signals, which do not necessarily align with semantic boundaries. In this paper, we propose SemBlock, a semantic-boundary-driven dynamic block decoding framework for diffusion LLMs. SemBlock formulates dynamic block construction as semantic boundary prediction and trains lightweight predictors on frozen LLaDA hidden states. To provide supervision, we construct SemBound, a semantic-boundary dataset that derives boundary labels from discourse units, reasoning steps, and implementation spans across natural language, math, and code tasks. During inference, SemBlock uses predicted boundary probabilities to select the ending position of each dynamic block. Experiments on GSM8K, IFEval, MATH, and HumanEval show that SemBlock consistently improves over fixed-block decoding and AdaBlock. Our code is publicly available: https://github.com/TH-AI-Lab-PKU/SemBlock.

Abstract

Diffusion Language Model Decoding

Explore similar work

SemBlock: Semantic Boundary Dynamic Blocks for Diffusion LLMs

Adaptive Block Diffusion: Resolving Training-Inference Mismatch in Diffusion Language Models

Multi-Block Diffusion Language Models