Organizations: Shanghai Innovation Institute · Shanghai Academy of AI for Science · Fudan University · Nanjing University · Huawei Technologies Co., Ltd.
Target-conditioned diffusion drafters predict candidate tokens in parallel, yet each new block must wait for updated target-model hidden states. This forces drafting and verification to run sequentially, limiting speedup even though each block is generated in one pass. We introduce D3D, a standalone diffusion drafter that removes this inter-round dependency. D3D combines blockwise denoising with auxiliary autoregressive supervision to achieve long accepted prefixes without target hidden states. This independence lets D3D draft successor blocks during verification in one packed pass, enabling asynchronous speculative decoding. We progressively prune layers according to their effect on acceptance length to reduce the cost of packed drafting. Removing five of 36 layers retains over 99% of mean acceptance length across three benchmarks with Qwen3-8B as the target. Across seven benchmarks, D3D averages 5.00× and 5.71× decoding speedup with Qwen3-8B and Qwen3-32B targets, compared with 4.03× and 3.67× for DFlash. On GSM8K with Qwen3-32B, speedup reaches 7.48×. Code is publicly available at https://github.com/fudan-generative-vision/D3D.
Figures & tables
Figure 1: Overview of D3D. (a) Drafter training. Target-generated responses supervise blockwise denoising and auxiliary autoregressive prediction. Packed attention shares causal prefix computation while isolating diffusion blocks and allowing bidirectional attention within each block. (b) Progressive layer pruning. Each iteration removes the layer causing the smallest decrease in validation acceptance length, followed by recovery and reevaluation of layer importance. Recovery uses distillation from a fixed unpruned drafter whose output distribution is adjusted toward the supervised token. (c) Asynchronous inference. During target verification, one packed drafter pass prepares successor blocks for alternative rejection outcomes and full acceptance. A branch matching the verified prefix is reused; otherwise, the drafter generates a new block synchronously.
Model
Method
Math
Code
Chat
GSM8K
MATH-500
AIME25
HumanEval
MBPP
LCB-v6
MT-Bench
Avg.
Speedup
τ
Speedup
τ
Speedup
τ
Speedup
τ
Speedup
τ
Speedup
τ
Speedup
τ
Speedup
τ
Qwen3 8B
EAGLE-3 (8)
3.43 ×
5.58
3.34 ×
5.38
3.01 ×
4.76
2.95 ×
4.71
2.71 ×
4.31
2.88 ×
4.30
1.87 ×
2.90
2.88 ×
4.56
DFlash (16)
4.59 ×
7.93
5.00 ×
7.44
4.34 ×
6.29
4.34 ×
6.02
4.03 ×
5.29
3.48 ×
5.32
2.46 ×
3.48
4.03 ×
5.96
DSpark (8)
4.93 ×
6.47
4.86 ×
6.40
4.79 ×
5.75
4.70 ×
5.84
4.19 ×
5.48
4.44 ×
5.40
3.17 ×
3.82
4.44 ×
5.59
PARD-2 (16)
6.16 ×
7.26
5.81 ×
7.71
4.44 ×
5.52
3.95 ×
5.46
3.39 ×
4.78
2.89 ×
4.43
1.89 ×
4.15
4.08 ×
5.62
Table 1: Speedup and average acceptance length τ across mathematical reasoning, code generation, and chat benchmarks. LCB denotes LiveCodeBench. Numbers in parentheses indicate block sizes.
Requests
MATH-500
HumanEval
MT-Bench
1
154(6.1×)
135(5.3×)
75(2.9×)
4
470(4.8×)
407(4.3×)
202(2.1×)
8
745(3.9×)
655(3.9×)
327(1.9×)
16
864(2.7×)
748(2.6×)
386(1.5×)
Table 2: Decoding throughput in tokens/s (speedup in parentheses) versus concurrency; D3D-1.7B drafter, Qwen3-32B target.
λ
GSM8K
MATH-500
HumanEval
MBPP
0
7.89
7.86
5.86
5.53
0.5
8.05
7.98
6.02
5.64
1.0
8.29
8.16
6.25
5.77
1.5
8.25
8.13
6.19
5.75
Table 3: Effects of auxiliary AR weight (a) and drafter scale (b) on average acceptance length τ . We evaluate D3D variants after one epoch of training, with Qwen3-8B as the target and B=16 .
Figure 2: Ablations of D3D-8B depth and initialization with Qwen3-8B as the target and B=16 . Left: decoding speedup and acceptance length τ , averaged over MATH-500, HumanEval, and MT-Bench. Right: GSM8K τ during training from pretrained or random initialization.
Figure 3: Decoding speedup and cache hit rate versus asynchronous draft budget on AIME25, using the 31-layer D3D-8B drafter with Qwen3-8B and B=16 . Left: varying position coverage C at F=2 . Right: varying token fan-out F at C=12 .
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: GSM8K acceptance length τ during D3D-1.7B training with different auxiliary autoregressive loss weights λ . The target is Qwen3-8B and the block size is B=16 .
B
MATH-500
HumanEval
MT-Bench
Speedup
τ
Speedup
τ
Speedup
τ
8
4.58×
6.26
4.00×
5.52
2.55×
4.38
16
5.55×
9.89
4.47×
7.85
2.47×
5.72
32
3.16×
12.39
2.56×
8.70
1.42×
6.33
Appendix
Table 4: Decoding speedup and acceptance length τ versus block size; D3D-8B, Qwen3-8B target.
K
MATH-500
HumanEval
MT-Bench
16
7.89
6.00
4.41
64
8.14
6.21
4.45
128
8.15
6.24
4.50
256
8.17
6.24
4.51
Appendix
Table 5: Average acceptance length τ versus the number of sampled training anchors; D3D-1.7B, Qwen3-8B target, B=16 .
Figure 5: Selective prefilling and sequential speculative decoding with D3D. The same standalone drafter scores prompt chunks for target prefilling and generates candidate blocks for verification. Retained chunks preserve their original order, and the drafter and target maintain separate KV caches.
Task
Setting
Score
TTFT
Decode
E2E
Single-doc QA
Full + D3D
47.91
0.84 ×
3.09 ×
1.46 ×
30% + D3D
44.23
2.09 ×
3.81 ×
2.83 ×
20% + D3D
42.46
2.57 ×
3.95 ×
3.22 ×
10% + D3D
41.27
2.91 ×
4.25 ×
3.56 ×
Multi-doc QA
Full + D3D
45.98
0.86 ×
2.24 ×
1.51 ×
30% + D3D
44.32
2.01 ×
2.07 ×
2.05 ×
Appendix
Table 6: LongBench results with a 1.7B D3D drafter and Qwen3-32B target model.
Task
Setting
Score
TTFT
Decode
E2E
Single-doc QA
Full + D3D
47.91
0.72 ×
4.16 ×
1.40 ×
30% + D3D
47.42
1.50 ×
5.49 ×
2.62 ×
20% + D3D
46.41
1.70 ×
5.96 ×
2.93 ×
10% + D3D
47.05
1.93 ×
6.73 ×
3.32 ×
Multi-doc QA
Full + D3D
45.97
0.74 ×
2.82 ×
1.54 ×
30% + D3D
46.25
1.54 ×
3.15 ×
2.40 ×
Appendix
Table 7: LongBench results with an 8B D3D drafter and Qwen3-32B target model.
Department of Computer Science, Technion – Israel Institute of Technology. · Department of Electrical and Computer Engineering, Technion – Israel Institute of Technology.