Trajectory-Level Speculative Decoding for Diffusion Language Models

Tianxiang Pan; Baitao Gong; Mo Guang; Hongwei Yong; Tianpeng Jiang; Yaqian Li; Zheng Cao; Kaiwen Long

cs.CLAug 27, 2026

Trajectory-Level Speculative Decoding for Diffusion Language Models

Authors: Tianxiang Pan, Baitao Gong, Mo Guang, Hongwei Yong, Tianpeng Jiang, Yaqian Li, Zheng Cao, Kaiwen Long

Organizations: Li Auto Inc., Shanghai, China

Abstract

Diffusion-based language models (dLLMs) enable parallel token generation through iterative denoising, but existing decoding strategies collapse to single-token generation under low confidence, severely limiting throughput. Unlike autoregressive models where speculative decoding operates on token sequences in a fixed left-to-right order, dLLMs require speculating over denoising trajectories-sequences of multi-token updates with explicit positions and unmasking orders. We develop a trajectory-level speculative framework that constructs draft denoising trajectories via confidence-stratified tree exploration and verifies them through blockwise parallel evaluation with bidirectional attention masking. Our method further introduces inter-block speculation, exploiting diffusion models' bidirectional structure to perform cross-block lookahead. We formally characterize when this approach is exact and identify trajectory drift as the fundamental cost of increased parallelism. Building on Fast-dLLM's dual-cache infrastructure, our framework reduces denoising iterations by 30-40% and increases tokens-per-step from 2.6 to 4.3, achieving 7-14x speedup over vanilla dLLMs and 1.3x over Fast-dLLM with less than 1% accuracy change across reasoning and code benchmarks.

Figures & tables

Figure 1: Visualization of low-confidence degeneration in parallel decoding for dLLMs. When confidence drops below threshold, existing methods collapse to single-token generation, severely limiting throughput despite dLLMs’ inherent parallel capacity.

Property	Self-Spec	Spiffy	Ours
Speculation granularity	Token	Block	Trajectory
Cross-block lookahead	✗	✗	✓
Tree-based exploration	✗	✗	✓
Lossless verification	✓	✓	$\sim$ ✓
Calibration overhead	Low	High	Low
Speedup (7B models)	$\sim$ 3.46 $\times$	$\sim$ 7 $\times$	$\sim$ 14 $\times$

Table 1: Comparison of speculative decoding methods for dLLMs.

Figure 2: Overview of trajectory-level speculative decoding framework. The method constructs a tree of candidate denoising trajectories, verifies them in parallel using blockwise attention masking, and selects the longest valid path.

Figure 3: Blockwise attention mask construction. Each draft block (shown in different colors) attends bidirectionally to cached prefix and suffix tokens (gray), but draft blocks remain isolated from each other to enable parallel verification of independent trajectories.

Figure 4: Inter-block speculation mechanism. When block

B_{t+1}

exhibits high early confidence, we construct compact draft trees for both

B_{t}

and

B_{t+1}

and verify them jointly, exploiting the bidirectional attention structure to enable cross-block lookahead.

Benchmark	Method	TPS↑	Steps↓	Acc.↑	Speedup
GSM8K	Vanilla	4.1	512.0	76.6	1.0 $\times$
	Fast-dLLM (DC)	48.8	110.3	75.6	11.9 $\times$
	Ours	55.7	71.3	75.5	13.6 $\times$
HumanEval	Vanilla	13.8	512.0	43.9	1.0 $\times$
	Fast-dLLM (DC)	70.6	179.2	45.7	5.1 $\times$
	Ours	94.4	106.7	46.3	6.8 $\times$

Table 2: Performance on LLaDA-7B-Instruct (A800).

Benchmark	Method	TPS↑	Steps↓	Acc.↑	Speedup
GSM8K	Vanilla	9.0	512.0	76.8	1.0 $\times$
	Fast-dLLM (DC)	50.6	280.6	74.6	5.6 $\times$
	Ours	62.2	164.9	73.7	6.9 $\times$
HumanEval	Vanilla	16.9	512.0	54.2	1.0 $\times$
	Fast-dLLM (DC)	52.3	295.0	51.2	3.1 $\times$
	Ours	78.1	180.7	54.3	4.6 $\times$

Table 3: Performance on Dream-7B-Instruct (A800).

Configuration	Tokens/Step	Steps	Latency (s)
Fast-dLLM (DC)	2.61	179.2	6.6
+ Intra W1D1(1)	3.51	134.5	5.6
+ Intra W2D2(3)	3.83	124.0	5.3
+ Intra W3D3(6)	3.95	120.1	5.5
+ Inter W2D2(3)	4.28	106.7	5.0
+ Inter W3D3(6)	4.52	105.1	5.6

Table 4: Ablation on LLaDA-7B (HumanEval, A800).

Figure 5: Computational overhead analysis. Per-step latency increases with the number of draft blocks, but the reduction in total steps yields net speedup. Dashed line indicates cumulative step reduction. Optimal tree size depends on hardware capability and the overhead-reduction trade-off.

	Sequence Length
Configuration	32	128	512	2048
w1d1	2.52	3.50	1.27	0.86
w2d1	7.57	6.94	1.69	1.15
w3d1	6.50	5.80	1.54	1.08
w2d2	8.25	6.72	1.86	1.29
w2d3	7.64	4.97	1.80	1.35

Table 5: Tree construction overhead ratio (%) on A100 across sequence lengths.

Configuration	A100 ( $\mu$ s)	H100 ( $\mu$ s)
w1d1	60.38	37.22
w2d1	60.22	36.03
w3d1	63.14	37.76
w2d2	63.23	37.82
w2d3	74.18	43.23
w3d3	97.95	41.95

Table 6: Fused multi-head attention latency (in

\mu

s) on A100 and H100.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Example of tree-based draft trajectory construction. The root block contains high-confidence tokens (positions 0-31). Child blocks explore alternative predictions at low-confidence positions 12, 18, 24. Each path represents a complete denoising trajectory verified in parallel.

Strategy	Steps	Latency/Step (s)	Total Latency (s)
Hybrid W2D2(3)	124.0	0.043	5.3
+ top-2 W2D2(4)	123.9	0.045	5.6
+ full tree W2D2(4)	124.9	0.045	5.6
+ duplicated W2D2(4)	125.0	0.043	5.3
Hybrid W3D3(6)	120.1	0.046	5.5
+ top-2 W4D3(8)	119.3	0.054	6.4

Appendix

Table 7: Ablation of tree expansion strategies (HumanEval, A800).

Inter-Block Setting	Tokens/Step	Steps	Latency (s)
No inter-block	3.83	124.0	5.3
+ 1 draft block	4.29	109.1	5.0
+ 2 draft blocks	4.38	106.7	5.0
+ 3 draft blocks	4.53	103.0	5.1

Appendix

Table 8: Inter-block ablation (LLaDA, HumanEval, A800).

Benchmark	Spiffy	Ours
GSM8K	3.04 $\times$	7.18 $\times$
HumanEval	2.88 $\times$	4.87 $\times$
MBPP	2.93 $\times$	4.42 $\times$
MATH	2.89 $\times$	5.29 $\times$

Appendix

Table 9: Reference comparison with Spiffy on LLaDA-Instruct-7B.

Dataset	Token Match	Logit Error
HumanEval	100%	0.00
GSM8K	100%	0.00

Appendix

Table 10: Empirical exactness verification.

Model	L1	L2	L3	L4	L5	Avg
Fast-dLLM (DC)	61.2	51.4	38.6	30.4	10.4	38.4
Ours	61.3	52.2	38.7	25.1	9.2	37.4

Appendix

Table 11: Accuracy by MATH difficulty level.

Length	Fast-dLLM (DC)	Ours
128	1.5	1.0
256	3.2	2.3
512	6.6	5.0
1024	10.2	8.7
2048	22.4	21.2

Appendix

Table 12: Latency (s) across generation lengths (HumanEval).

Method	TPS $\uparrow$	Steps $\downarrow$	Accuracy	Speedup
Fast-dLLM (DC)	36.8	210.8	53.97 $\pm$ 0.02	1.0 $\times$
Ours (W2D2)	69.1	118.8	54.71 $\pm$ 0.02	1.9 $\times$

Appendix

Table 13: Performance on IFEval (LLaDA-7B-Instruct, H800,

\tau=0.9

). Fast-dLLM exhibits severe low-confidence degeneration (avg. 1.3 tokens/step), enabling our method to achieve nearly

2\times

speedup.

Speculative Decoding for Diffusion Language Models

Speculative decoding for diffusion language models speeds generation by drafting tokens in parallel and verifying them efficiently.

See all 30 papers

SimSD: Simple Speculative Decoding in Diffusion Language Models
Jun 1, 2026Junxia Cui, Haotian Ye, Runchu Tian +9Masked Diffusion Models Speculative Decoding for Diffusion Language Models
University of California San Diego · University of Illinois Urbana-Champaign · 3Google +1
On the SPOT: Enhancing Speculative Decoding for Diffusion Language Models via Commit-Order Prediction
Oct 15, 2026Eliezer Shlomi, Eliron Rahimi, Avi Mendelson +1Speculative Decoding for Diffusion Language Models Diffusion Language Models
Technion – Israel Institute of Technology · INSIGHT Lab, School of Electrical and Computer Engineering, Ben-Gurion University of the Negev
Efficient Diffusion LLMs via Temporal-Spatial Parallel Decoding and Confidence Extrapolation
May 29, 2026Zekai Li, Ji Liu, Yiqing Huang +3Adaptive Computation for Diffusion Models Diffusion Language Model Decoding
Advanced Micro Devices, Inc. (AMD)

Jun 1, 2026J/K move · Enter open · S save

SimSD: Simple Speculative Decoding in Diffusion Language Models

Junxia Cui, Haotian Ye, Runchu Tian, Hongcan Guo+8

University of California San Diego · University of Illinois Urbana-Champaign · 3Google+1

Masked Diffusion Models Speculative Decoding for Diffusion Language Models Speculative Decoding Diffusion Language Model Inference Diffusion Language Model Decoding Diffusion Language Models

Open paper AskPDF arXiv

Diffusion large language models (dLLMs) have recently emerged as a promising alternative to autoregressive (AR) LLMs, offering faster inference through parallel or blockwise decoding. However, their masked language modeling formulation remains incompatible with standard token-level speculative decoding, one of the most effective acceleration techniques for AR models. In AR decoding, the causal mask preserves temporally valid token-level contexts, enabling a target model to verify multiple drafted tokens in a single forward pass. In contrast, dLLMs rely on mask tokens and bidirectional attention, causing the effective context to change across denoising steps and preventing direct token-level speculative verification. To bridge this gap, we propose a simple but effective speculative decoding algorithm for diffusion language models, named SimSD, which mainly adopts a plug-and-play masking strategy that equips dLLMs with temporally valid token-level contexts for speculative decoding. Our method explicitly introduces reference tokens from draft-model predictions and designs an attention mask that regulates their interaction with current-step tokens, allowing dLLMs to compute valid logits for drafted tokens in a single forward pass. This restores the key verification ability provided by causal masking in AR models while preserving the parallel decoding advantages of dLLMs. The proposed method is training-free and can be flexibly integrated with other acceleration techniques such as KV cache and blockwise decoding. Experiments on SDAR-family dLLMs across four benchmarks show that our method achieves up to 7.46x higher decoding throughput while maintaining and even improving average generation quality.

Abstract

Speculative Decoding for Diffusion Language Models

Explore similar work

SimSD: Simple Speculative Decoding in Diffusion Language Models

On the SPOT: Enhancing Speculative Decoding for Diffusion Language Models via Commit-Order Prediction

Efficient Diffusion LLMs via Temporal-Spatial Parallel Decoding and Confidence Extrapolation