Speculative decoding speeds up language models by having a lightweight drafter propose future tokens that the target model verifies in parallel. Block diffusion drafters such as DFlash predict an entire block in one forward pass, and DDTree builds draft trees directly from their per-position distributions. Each distribution implicitly assumes that the drafter's own greedy continuation is accepted, whereas a draft tree improves over a chain only on branches that leave this greedy path. Once a path forks, the drafter's top-1 accuracy drops from 89% to 18% and its top candidates often miss the true token. Re-running the drafter from the fork token largely restores its accuracy, showing that the errors stem from conditioning on an outdated prefix rather than from limited model capacity. In this paper, we introduce DForkTree, a training-free method that re-anchors each branch on its own tokens. It adjusts the drafter's scores with pointwise mutual information from target-generated n-grams, weighted more heavily after a fork, and a layer-wise GPU search recovers the exact optimal tree under these scores. On Qwen3.5-4B and 9B, DForkTree outperforms all baselines on every benchmark, by up to 41.5% over DFlash and 15.1% over DDTree, with average speedups of up to 4.75x over autoregressive decoding.
Figures & tables
Figure 1: Speedup over autoregressive decoding at temperature 0 on seven benchmarks (MATH is MATH-500). DFlash drafts a 24-token chain, and all trees use its drafter.
Figure 2: Where the marginals fail on Qwen3.5-4B. (a) Top-1 accuracy of the drafter’s marginal and of DForkTree’s score by node group. (b) Top-1 accuracy at the first three tokens after the first fork, with the drafter re-run from the fork token. (c) Share of positions whose true token is in the top 16 while all earlier ones are.
Figure 3: Overview of DForkTree on an MT-Bench example with illustrative numbers. (a) After the fork to job , the marginals rate hiring and creation almost equally, and DForkTree re-scores each child by the PMI of its grandparent, parent and child, with a larger weight off the spine. (b) The tree grows one depth at a time on the GPU and keeps the best B prefixes. (c) One target pass verifies the tree and commits the accepted path’s state in closed form.
Figure 4: One-step estimate λ^ against the fitted weight λ⋆ for 37 groups of development nodes, fitted as λ⋆≈1.8λ^ ( R2=0.82 ).
Figure 5: System cost on Qwen3.5-4B. (a) One decoding round for the 24-token DFlash chain and four tree methods with the budgets of Table 1 , split into four stages, with DDTree’s host search counted as building. Columns give build and round time in ms, τ and throughput. (b) Throughput with 23 draft positions, labeled with DForkTree’s gain over DDTree. (c) Short convolution and 24 GDN layers for one tree verification, with native snapshot sizes in GiB and our speedup.
T=0
GSM8K
MATH-500
HumanEval
MBPP
LCB
MT-Bench
Alpaca
Avg.
Method
S
τ
S
τ
S
τ
S
τ
S
τ
S
τ
S
τ
S
τ
Qwen3.5-4B
Autoregressive
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
N -gram
1.00
1.68
1.09
1.85
0.80
1.54
0.84
1.44
0.96
1.64
0.79
1.32
0.75
1.25
0.89
1.53
MTP
2.07
4.67
2.05
4.70
1.87
4.62
1.98
4.42
1.85
4.34
1.63
3.88
1.53
3.57
1.85
4.32
DFlash, 16 tokens
3.61
7.09
3.70
7.43
3.67
7.72
2.94
5.91
2.45
4.99
1.74
3.60
1.44
2.86
2.79
5.66
Table 1: Speedup S over autoregressive decoding and acceptance length τ at temperature 0, best per model in bold and best baseline underlined . MTP is Qwen3.5’s built-in head with 4 steps and top-4, and the EAGLE-3 tree is EAGLE’s dynamic tree on the DFlash drafter. All tree methods share the DFlash drafter, the GPU tree packing and our verifier and use 32 nodes, with blocks of 16 for the baselines and 24 for DForkTree.
Path score
Tree
Evidence
Weight
Spine
K
τ
tok/s
Δ
(a)
none
–
free
23
7.72
515
−7.0%
(b)
raw log-prob
single λ
free
23
8.00
531
−4.2%
(c)
bigram PMI
single λ
free
23
8.17
543
−2.1%
(d)
trigram PMI
single λ
free
23
8.18
550
−0.7%
(e)
trigram PMI
fork-aware
free
23
8.36
554
–
Table 2: Ablations on Qwen3.5-4B with 64 nodes. Gray settings are shared with DForkTree, row (e). K counts draft positions, τ is averaged over benchmarks, and Δ is the throughput change.
Figure 6: Replay of the 24-token DFlash chain and of DDTree and DForkTree with 64 nodes and 23 draft positions. (a) Tokens produced per round. A round produces at most 24 tokens, so the hatched bar counts all rounds that reach the end of the block. (b) Accepted draft tokens per round on the spine, at the fork node where the accepted path leaves it, and after the fork.
Correction
Drafter
Trained
Alone
DForkTree
Both
Branch tokens
DFlash
11–16
41–46
–
Distilled head
DFlash
✓
34–38
36–45
41–49
Selector
DFlash 2
✓
46–47
44–45
49–50
Table 3: Share of the gap to the re-anchored drafter closed at the first three tokens after a fork (range, %). Rows use different events, so only numbers within a row compare.
Qwen3.5-4B
Qwen3.5-9B
Method
S
τ
tok/s
S
τ
tok/s
MTP
1.69
3.90
282
2.03
4.04
194
DFlash, 16 tokens
2.15
4.36
325
2.88
4.76
243
DFlash, 24 tokens
2.31
4.53
341
2.91
4.97
241
EAGLE-3 tree
2.80
5.69
429
3.64
6.23
310
DDTree
2.61
5.64
404
3.44
6.16
297
Table 4: Averages at temperature 1, with S the throughput ratio to autoregressive sampling. Appendix A gives all methods per benchmark.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
T=1
GSM8K
MATH-500
HumanEval
MBPP
LCB
MT-Bench
Alpaca
Avg.
Method
S
τ
S
τ
S
τ
S
τ
S
τ
S
τ
S
τ
S
τ
Qwen3.5-4B
Autoregressive
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
N -gram
0.94
1.62
1.04
1.78
0.88
1.52
0.82
1.41
0.90
1.53
0.77
1.33
0.73
1.25
0.87
1.49
MTP
1.90
4.40
1.95
4.48
1.80
4.15
1.76
4.06
1.68
3.83
1.48
3.40
1.28
2.96
1.69
3.90
DFlash, 16 tokens
2.85
5.78
3.05
6.17
2.59
5.26
2.17
4.39
1.81
3.64
1.45
2.94
1.16
2.36
2.15
4.36
Appendix
Table 5: Speedup S and acceptance length τ at temperature 1, with top-p 1 and no top-k (848 prompts per model). Sampled outputs differ between methods, so S is the ratio of decoding throughputs to autoregressive sampling on each benchmark. Autoregressive sampling runs at 169 and 97 tok/s on Qwen3.5-4B and 9B, DForkTree is best in every column, and the best baseline is underlined .
Figure 7: Offline τ of five construction strategies on Qwen3.5-4B with 23 draft positions, replayed along autoregressive outputs.
Table 13
Figure 8: (a) Offline τ as the corpus shrinks, and with 10K matched math and code prompts answered by the target or by published reference solutions. (b) Offline τ pooled by domain when the corpus excludes all math or all code responses.
Figure 15
Figure 11: Development replays grouped by the number of steps since the parent path left the spine. (a) Reduction in the negative log-likelihood of the true next token as a function of λ . (b) Top-1 accuracy of the re-scored candidates as a function of λ .
Figure 12: Overlap of the evidence with the base score against its fitted weight, for the 37 groups of Figure 4 . The overlap is the mean Spearman correlation between the base log-probability and the evidence over the candidates of a position.
DDTree
DForkTree
Nodes / block
32 / 16
64 / 24
64 / 24
Spine nodes
7.3
8.3
7.9
Off-spine nodes
23.7
54.7
55.1
Branching nodes
5.3
10.2
10.6
Maximum depth
7.3
8.3
10.3
Rounds leaving the spine (%)
58.6
71.7
74.2
Appendix
Table 11: Tree shape per round (replay, mean over rounds). Branching nodes have two or more children.
Department of Computer Science, Technion – Israel Institute of Technology. · Department of Electrical and Computer Engineering, Technion – Israel Institute of Technology.