Speculative decoding uses a target model to verify a tree of draft tokens in parallel. We study whether the resulting next-token distributions can help select the next tree. For unrestricted target models, we show that guaranteed exact reuse requires a distribution computed at the next round's complete context. We characterize when retaining this distribution requires sacrificing accepted tokens and establish conditions under which this loss persists across decoding rounds. Across six targets, we find that reranking first-token branches with a separately computed distribution at the current context improves expected acceptance over draft ranking, especially with small tree budgets. Drawing the first token from this distribution before allocating the tree further improves expected acceptance over pruning a preselected tree, even at equal mean tree size. In our experiments, both selection rules also decode faster than draft ranking when the separate target pass overlaps standalone drafting. Our findings show that making target probabilities available before tree allocation can improve the efficiency of speculative decoding. Our code is available at https://github.com/js-lee-AI/TargetRows.
Figures & tables
Figure 1: Address matching across decoding rounds. The dashed link identifies the scored prefix caa with the next root; its row is p(⋅∣caa) .
Exhaustive
Cap 1
Sampled-leaf stop
Scored top M , Pr(C)
Target
E[A]
Pr(C)
E[LT−A]
Pr(C)
E[LT−A]
M=100
M=200
M=710
Greedy decoding
Llama-3.1-8B
4.509
0.817
3.599
0.466
0.466
0.158
0.275
0.328
Qwen3-4B
2.340
0.701
1.452
0.241
0.241
0.128
0.261
0.345
Qwen3-8B
2.221
0.587
1.401
0.239
0.239
0.115
0.226
0.304
Qwen2.5-7B
3.293
0.686
2.459
0.293
0.293
0.123
0.225
0.285
Table 1: Coverage, expected acceptance and shortfall on EAGLE-3’s 59 -node trees, computed as expectations from the same recorded target rows for every rule. Scored sets keep exhaustive acceptance on the tree and add the next M−59 candidates in draft order.
B=8
B=59
Expected acceptance
Root-draw gain, %
Expected acceptance
Root-draw gain, %
Target
Draft
Root
Oracle
Same cap
Equal mean
Draft
Root
Oracle
Same cap
Equal mean
Greedy decoding
Llama-3.1-8B
3.186
3.568
4.899
10.0
2.9
4.509
4.676
4.899
3.5
0.9
Qwen3-4B
1.664
2.002
2.720
12.8
4.5
2.340
2.516
2.720
6.7
2.4
Qwen3-8B
1.556
1.919
2.564
13.8
4.4
2.221
2.399
2.564
6.7
2.2
Table 2: Expected accepted draft tokens and root-draw gains over the PEARL-style decoder at shared contexts. Same cap uses the stated B ; Equal mean enlarges PEARL-style preselection to match the mean verified node count.
Figure 2: Gain over draft selection when exact target rows replace draft factors through increasing depths, averaged over six targets. The upper edge of each band is one selection rule. The dotted line marks EAGLE-3’s tree.
Figure 3: Share of the oracle advantage at B=59 recovered by root selection, split by whether the drafter’s most probable first token is the target’s most probable token. Labels give the share of rounds in which the two differ.
Greedy decoding
Temperature 1
Target
Gain, %
Nodes
Gain, %
Nodes
Llama-3.1-8B
3.3 [2.6, 4.1]
53 / 39
4.1 [1.7, 6.8]
48 / 34
Qwen3-4B
6.7 [6.2, 7.3]
49 / 33
7.1 [6.4, 7.9]
48 / 32
Qwen3-8B
7.3 [6.6, 8.0]
44 / 28
7.3 [6.3, 8.3]
43 / 27
Qwen2.5-7B
4.5 [3.6, 5.5]
49 / 35
5.7 [4.1, 7.4]
46 / 32
Qwen2.5-14B
3.8 [3.1, 4.4]
49 / 36
4.5 [3.0, 5.9]
47 / 32
Table 3: Online decoding at B=59 with a separate root pass. Gain is root-draw selection’s gain in accepted draft tokens over the PEARL-style decoder, with 95% intervals. Nodes gives mean non-root counts as root draw / PEARL-style at each decoder’s own contexts.
B=8
B=59
Target
Draft
Base
Rows
Root
Draft
Base
Rows
Root
Greedy decoding
Llama-3.1-8B
3.186
3.187
3.234
3.568
4.509
4.509
4.536
4.676
Qwen3-4B
1.664
1.674
1.722
2.002
2.340
2.343
2.372
2.516
Qwen3-8B
1.556
1.581
1.650
1.919
2.221
2.241
2.272
2.399
Qwen2.5-7B
2.359
2.360
2.399
2.710
3.293
3.294
3.311
3.450
Table 4: Expected accepted draft tokens at shared contexts for draft selection, cross-fitted rerankers and root selection. Base uses draft and contextual features; Rows adds features from retained rows at other addresses.
Target
Decoding
B=8
B=59
Llama-3.1-8B
Greedy
8.5 [7.3, 9.8]
2.7 [2.4, 3.1]
Qwen3-8B
Greedy
22.6 [20.9, 24.1]
7.6 [7.2, 8.1]
Llama-3.1-8B
Temp. 1
7.6 [6.9, 8.4]
2.8 [2.6, 3.1]
Qwen3-8B
Temp. 1
21.4 [20.4, 22.5]
7.3 [7.0, 7.6]
Table 5: Root selection’s expected-acceptance gain over draft ranking under a 1,024 -token limit, in percent, for rounds starting once the default output limit is reached. Brackets give 95% prompt-bootstrap intervals.
Target
MT-bench
Translation
Summarization
QA
Math
RAG
HumanEval
All
Llama-3.1-8B
4.9 / 3.7
3.9 / 3.6
4.4 / 3.8
6.1 / 4.8
3.7 / 3.6
4.0 / 3.2
2.3 / 1.9
3.7 / 3.0
Qwen3-4B
9.1 / 7.9
12.0 / 11.5
6.9 / 6.5
9.8 / 9.4
7.0 / 6.9
7.6 / 7.0
6.4 / 6.0
7.5 / 7.0
Qwen3-8B
8.8 / 8.9
8.7 / 8.6
7.1 / 7.1
10.2 / 9.4
8.3 / 8.0
8.2 / 7.5
7.2 / 6.8
8.0 / 7.7
Qwen2.5-7B
5.4 / 4.7
6.2 / 5.8
5.4 / 4.9
7.5 / 5.9
4.6 / 4.4
5.3 / 4.6
3.4 / 3.5
4.8 / 4.4
Qwen2.5-14B
5.7 / 4.5
5.6 / 5.5
5.2 / 4.4
7.2 / 5.7
3.6 / 3.5
5.4 / 4.5
3.4 / 3.2
4.6 / 4.0
Qwen2.5-32B
5.3 / 4.1
6.3 / 5.5
5.3 / 4.5
7.2 / 5.6
3.8 / 3.0
5.9 / 4.4
3.1 / 3.1
4.5 / 3.8
Table 6: Root selection’s expected-acceptance gain over draft ranking at B=59 , in percent. Each cell gives greedy decoding / temperature 1 . All aggregates the complete prompt set.
Figure 4: Root-draw selection’s decoding-speed gain over the PEARL-style decoder on two RTX A6000 GPUs. Left: both caps are 8 . Right: the PEARL-style cap is 59 and root-draw caps are 40 – 43 .
Appendix figures & tables25 assets
Supplementary material from the paper’s appendix.
Appendix
Checkpoint
Role
meta-llama/Llama-3.1-8B-Instruct
Target
yuhuili/EAGLE3-LLaMA3.1-Instruct-8B
EAGLE-3 drafter
meta-llama/Llama-3.2-1B-Instruct
Standalone drafter
Qwen/Qwen3-4B
Target
AngelSlim/Qwen3-4B_eagle3
EAGLE-3 drafter
Qwen/Qwen3-8B
Target
Appendix
Table 7: Target checkpoints and their paired EAGLE-3 and standalone drafters. Checkpoint names link to the evaluated Hugging Face revisions.
Target
MT-bench
Translation
Summarization
QA
Math
RAG
HumanEval
All
Greedy decoding
Llama-3.1-8B
5.53
2.65
5.10
4.52
6.05
5.48
6.68
5.51
Qwen3-4B
3.26
2.59
3.05
3.03
3.74
3.38
3.51
3.34
Qwen3-8B
3.05
2.20
2.91
2.69
3.63
3.22
3.59
3.22
Qwen2.5-7B
4.29
3.20
3.62
3.69
5.14
4.03
4.73
4.29
Qwen2.5-14B
4.48
3.24
3.81
3.71
5.26
4.13
5.00
4.49
Appendix
Table 8: Mean tokens emitted in each round, counting accepted draft tokens and the continuation, by task.
Figure 5: Share of the oracle advantage over draft selection recovered by exact rows at increasing depths, for each target and temperature. The dotted line marks EAGLE-3’s 59 -node tree.
Expected
Gain over draft selection
Recovered oracle advantage, %
Target
B
Depth 2
Root
Root draw
Oracle
Root
Depth 2
Greedy decoding
Llama-3.1-8B
8
3.925
0.382 [0.364, 0.401]
0.468 [0.450, 0.485]
1.713 [1.657, 1.766]
22 [21, 23]
43 [42, 44]
59
4.786
0.167 [0.157, 0.176]
0.252 [0.241, 0.265]
0.390 [0.373, 0.406]
43 [42, 44]
71 [70, 72]
Qwen3-4B
8
2.287
0.338 [0.328, 0.347]
0.438 [0.428, 0.449]
1.056 [1.037, 1.076]
32 [31, 33]
59 [58, 60]
59
2.625
0.176 [0.171, 0.182]
0.276 [0.270, 0.284]
0.380 [0.370, 0.389]
46 [45, 48]
75 [74, 76]
Appendix
Table 9: Selection gains in expected accepted draft tokens and recovered shares of the oracle advantage. Depth 2 under Expected gives acceptance with two exact factors. Brackets give 95% prompt-bootstrap intervals.
Root gain
Depth-two gain
Target
Rounds
Equal tasks
Rounds
Equal tasks
Greedy decoding
Llama-3.1-8B
0.167
0.169 [0.160, 0.179]
0.277
0.279 [0.266, 0.292]
Qwen3-4B
0.176
0.181 [0.176, 0.187]
0.285
0.286 [0.279, 0.293]
Qwen3-8B
0.178
0.169 [0.163, 0.174]
0.264
0.244 [0.238, 0.250]
Qwen2.5-7B
0.157
0.162 [0.155, 0.170]
0.254
0.260 [0.251, 0.269]
Appendix
Table 10: Gains over draft selection at B=59 under round weighting and equal task weighting. Brackets give 95% intervals from 1,000 resamples of prompts within tasks.
M=100
M=200
M=400
M=710
Target
Advantage
Root, %
Advantage
Root, %
Advantage
Root, %
Advantage
Root, %
Greedy decoding
Llama-3.1-8B
0.186
58
0.321
48
0.380
43
0.390
43
Qwen3-4B
0.135
63
0.279
52
0.359
47
0.380
46
Qwen3-8B
0.125
67
0.247
58
0.319
53
0.343
52
Qwen2.5-7B
0.147
59
0.267
51
0.327
47
0.343
46
Appendix
Table 11: Oracle advantage over draft selection at B=59 when the pool holds the M highest-ranked candidates, and the share that root selection recovers.
B=8
B=59
Target
Gain, %
Equal size, %
Share, %
Gain, %
Equal size, %
Share, %
Greedy decoding
Llama-3.1-8B
10.0 [9.4, 10.5]
2.9 [2.6, 3.3]
71
3.5 [3.3, 3.7]
0.9 [0.8, 1.0]
64
Qwen3-4B
12.8 [12.4, 13.3]
4.5 [4.2, 4.8]
55
6.7 [6.5, 6.9]
2.4 [2.2, 2.5]
60
Qwen3-8B
13.8 [13.4, 14.2]
4.4 [4.0, 4.7]
48
6.7 [6.5, 6.9]
2.2 [2.1, 2.4]
47
Qwen2.5-7B
10.7 [10.2, 11.2]
4.1 [3.8, 4.4]
55
4.2 [4.0, 4.4]
1.3 [1.2, 1.4]
46
Appendix
Table 12: Gain of root-draw selection over the PEARL-style decoder. In Equal size, it preselects more candidates until mean tree sizes are equal. Brackets give 95% intervals. Share is the part of root-draw selection’s gain over draft selection that Gain makes up.
Reference budget B=59
Reference budget B=8
Target
Draft reaches root
Root reaches draft
Draft reaches root
Root reaches draft
Greedy decoding
Llama-3.1-8B
94 [92, 96]
37 [36, 37]
13 [12, 13]
7 [7, 7]
Qwen3-4B
118 [116, 121]
26 [25, 26]
20 [20, 21]
4 [4, 4]
Qwen3-8B
130 [127, 134]
24 [23, 25]
22 [21, 23]
4 [4, 4]
Qwen2.5-7B
105 [103, 107]
32 [32, 33]
15 [15, 16]
6 [6, 6]
Appendix
Table 13: Smallest budget at which draft selection reaches root selection’s expected accepted draft tokens at the reference budget, and at which root selection reaches draft selection’s. Brackets give 95% prompt-bootstrap intervals.
Root selection
Root-draw selection
Target
EAGLE-3
Accepted
Gain, %
Offline, %
Accepted
Gain, %
Offline, %
Greedy decoding
Llama-3.1-8B
4.509
4.721
4.7 [3.6, 5.9]
3.7
4.842
7.4 [6.3, 8.4]
5.6
Qwen3-4B
2.340
2.507
7.1 [6.5, 7.8]
7.5
2.630
12.4 [11.6, 13.2]
11.8
Qwen3-8B
2.221
2.402
8.2 [7.4, 8.9]
8.0
2.606
17.4 [16.4, 18.3]
15.4
Qwen2.5-7B
3.293
3.463
5.2 [4.2, 6.1]
4.8
3.683
11.9 [10.7, 13.0]
9.5
Appendix
Table 14: Online decoding at B=59 . Accepted is the realized mean acceptance, and EAGLE-3 that of the recorded runs. Gain over EAGLE-3 has 95% prompt-bootstrap intervals in brackets, and Offline is the gain over draft selection at the recorded contexts.
First tokens differ, %
Oracle advantage
Root recovery, %
Depth-two recovery, %
Target
Rounds
Root gain
Agree
Differ
Agree
Differ
Agree
Differ
Greedy decoding
Llama-3.1-8B
24
60
0.371
0.448
24
92
61
96
Qwen3-4B
36
75
0.371
0.395
19
93
62
97
Qwen3-8B
45
78
0.352
0.333
20
94
61
97
Qwen2.5-7B
34
69
0.340
0.349
22
92
62
96
Appendix
Table 15: Selection at B=59 split by whether the drafter’s most probable first token is the target’s most probable token. Oracle advantage is in expected accepted draft tokens, and recovery is the share of it that each rule attains.
Accepted
Target
PEARL-style
Root draw
Gain, %
Offline, %
Root TV
Greedy decoding
Llama-3.1-8B
4.686
4.842
3.3 [2.6, 4.1]
3.5
0.0041
Qwen3-4B
2.463
2.630
6.7 [6.2, 7.3]
6.7
0.0057
Qwen3-8B
2.429
2.606
7.3 [6.6, 8.0]
6.7
0.0063
Qwen2.5-7B
3.523
3.683
4.5 [3.6, 5.5]
4.2
0.0111
Appendix
Table 16: Online decoding at B=59 . Accepted is realized mean acceptance. Gain compares root-draw selection with the PEARL-style decoder; brackets give 95% intervals. Offline evaluates shared recorded contexts. Root TV compares the two passes’ root rows by mean total variation.
Accepted
Target
PEARL-style
Root draw
Gain, %
Offline, %
Root TV
Greedy decoding
Llama-3.1-8B
3.365
3.724
10.7 [9.9, 11.4]
10.0
0.0043
Qwen3-4B
1.881
2.122
12.8 [12.2, 13.4]
12.8
0.0057
Qwen3-8B
1.872
2.130
13.7 [13.1, 14.4]
13.8
0.0063
Qwen2.5-7B
2.647
2.940
11.1 [10.1, 12.0]
10.7
0.0113
Appendix
Table 17: Online decoding at B=8 . Gain compares root-draw selection with the PEARL-style decoder in realized acceptance; brackets give 95% intervals. Offline evaluates shared recorded contexts. Root TV compares the two passes’ root rows by mean total variation.
Greedy decoding
Sampling at temperature 1
Target
Online, %
Offline, %
Online, %
Offline, %
Llama-3.1-8B
1.0 [0.9, 1.1]
0.9 [0.8, 1.0]
1.4 [1.3, 1.5]
1.4 [1.3, 1.5]
Qwen3-4B
2.3 [2.1, 2.4]
2.4 [2.2, 2.5]
2.4 [2.3, 2.5]
2.4 [2.2, 2.5]
Qwen3-8B
2.1 [2.0, 2.3]
2.2 [2.1, 2.4]
2.3 [2.1, 2.4]
2.3 [2.2, 2.5]
Qwen2.5-7B
1.3 [1.2, 1.4]
1.3 [1.2, 1.4]
1.9 [1.8, 2.0]
1.8 [1.7, 1.9]
Qwen2.5-14B
1.2 [1.1, 1.3]
1.2 [1.1, 1.3]
1.7 [1.6, 1.8]
1.8 [1.7, 1.9]
Appendix
Table 18: Gain in expected acceptance of root-draw selection over the PEARL-style tree enlarged to equal mean size at B=59 , with 95% intervals. Online evaluates both trees at online contexts; Offline uses the shared recorded contexts.
B=8
B=59
Calibrated
Reranked
Calibrated
Reranked
Target
Draft
First
All
Base
Rows
Root
Draft
First
All
Base
Rows
Root
Greedy decoding
Llama-3.1-8B
3.186
3.188
3.168
3.187
3.234
3.568
4.509
4.510
4.493
4.509
4.536
4.676
Qwen3-4B
1.664
1.667
1.708
1.674
1.722
2.002
2.340
2.342
2.356
2.343
2.372
2.516
Qwen3-8B
1.556
1.563
1.564
1.581
1.650
1.919
2.221
2.229
2.223
2.241
2.272
2.399
Appendix
Table 19: Expected accepted draft tokens after cross-fitting calibrated or reranked draft factors on complementary prompt halves. Base uses draft and contextual features; Rows adds retained rows from other addresses. Blue cells mark selection with the exact root row.
Late rounds
Whole outputs
Target
Late, %
Rounds
B=59
B=8
B=59
B=8
Greedy decoding
Llama-3.1-8B
41
7,374
2.7 [2.4, 3.1]
8.5 [7.3, 9.8]
3.5 [3.3, 3.7]
11.0 [10.3, 11.7]
Qwen3-8B
48
19,346
7.6 [7.2, 8.1]
22.6 [20.9, 24.1]
7.8 [7.6, 8.0]
23.0 [22.2, 23.9]
Sampling at temperature 1
Llama-3.1-8B
41
12,334
2.8 [2.6, 3.1]
7.6 [6.9, 8.4]
3.0 [2.9, 3.1]
8.4 [8.0, 8.9]
Appendix
Table 20: Root selection’s gains over draft ranking under a 1,024 -token limit, in percent, with 95% prompt-bootstrap intervals in brackets. Late is the share of outputs with rounds starting once the default output limit is reached; Rounds counts these rounds.
Coverage, %, with E scored frontier candidates
Target
Frontier size
E=10
E=20
E=40
E=80
E=160
All
Greedy decoding
Llama-3.1-8B
407
5.9
10.4
16.9
23.7
29.4
32.8
Qwen3-4B
388
4.4
7.9
13.7
21.4
28.5
34.5
Qwen3-8B
381
4.0
7.1
11.9
18.2
24.6
30.4
Qwen2.5-7B
416
4.9
8.5
13.4
19.1
24.5
28.5
Appendix
Table 21: Coverage of exhaustive traversal when the pass also scores the first E frontier candidates in draft order. Frontier size is the mean number of pool candidates outside the tree whose parents lie in it.
Tree rows
All scored rows
Target
Successor rounds
Token, %
Token, %
Address, %
Token only, %
Greedy decoding
Llama-3.1-8B
19,340
41.7
61.1
25.1
36.0
Qwen3-4B
35,971
27.8
47.4
22.9
24.5
Qwen3-8B
36,856
29.1
46.9
20.4
26.5
Qwen2.5-7B
27,251
31.3
46.7
20.5
26.3
Appendix
Table 22: Successor rounds whose root’s final token or root address appears among the prefixes scored in the previous round. Tree rows are EAGLE-3’s 60 verified prefixes, and all scored rows are the roughly 170 prefixes of our verification pass.
Matched address
Matched token only
Target
Pairs
Mean TV
99th pct. TV
Argmax agree, %
Pairs
Mean TV
Argmax agree, %
Greedy decoding
Llama-3.1-8B
4,862
0.0036
0.0316
99.7
54,630
0.416
59.1
Qwen3-4B
8,254
0.0043
0.0603
99.5
106,260
0.390
61.3
Qwen3-8B
7,505
0.0045
0.0602
99.4
96,948
0.378
62.7
Qwen2.5-7B
5,575
0.0087
0.0734
99.2
62,228
0.457
54.1
Appendix
Table 23: Agreement between retained rows and the next round’s root row. Matched rows share the complete emitted prefix, while token-only rows share only its final token. Total variation uses softmax probabilities.
B=8
B=59
Target
Drafter
Draft
Root, %
Root draw, %
Draft
Root, %
Root draw, %
Greedy decoding
Llama-3.1-8B
Llama-3.2-1B
4.843
11.3 [10.6, 12.0]
9.8 [9.2, 10.4]
6.706
2.7 [2.5, 2.9]
2.6 [2.4, 2.8]
Qwen3-8B
Qwen3-0.6B
3.556
16.8 [15.8, 17.8]
13.2 [12.5, 14.0]
5.234
5.0 [4.6, 5.3]
4.6 [4.3, 4.9]
Sampling at temperature 1
Llama-3.1-8B
Llama-3.2-1B
3.391
6.2 [5.8, 6.7]
13.4 [12.8, 14.0]
5.045
2.4 [2.3, 2.6]
4.8 [4.5, 5.0]
Appendix
Table 24: Standalone drafters in pools of EAGLE-3’s shape. Draft is draft selection’s expected acceptance. Root is root selection’s gain over draft selection, and Root draw root-draw selection’s gain over the PEARL-style decoder, with 95% prompt-bootstrap intervals in brackets.
Gain over draft selection
Target
B
Exact
Cache
Same token
Preceding
Greedy decoding
Llama-3.1-8B
8
0.275 [0.259, 0.291]
− 0.595 [ − 0.632, − 0.559]
− 0.220 [ − 0.243, − 0.198]
− 1.760 [ − 1.819, − 1.695]
59
0.117 [0.110, 0.125]
− 0.325 [ − 0.348, − 0.302]
− 0.111 [ − 0.124, − 0.099]
− 0.803 [ − 0.838, − 0.770]
Qwen3-4B
8
0.315 [0.306, 0.323]
− 0.289 [ − 0.304, − 0.275]
− 0.040 [ − 0.047, − 0.033]
− 0.936 [ − 0.954, − 0.919]
59
0.166 [0.161, 0.172]
− 0.185 [ − 0.195, − 0.176]
− 0.022 [ − 0.026, − 0.017]
− 0.519 [ − 0.532, − 0.506]
Appendix
Table 25: Gains over draft selection in expected accepted draft tokens, with 95% intervals, when the softmax probabilities of the exact root row or of a row at another address replace the first draft factor. Rounds without one keep the draft ranking.
Available, %
Mean TV
Argmax agree, %
Target
Cache
Same token
Cache
Same token
Preceding
Cache
Same token
Preceding
Greedy decoding
Llama-3.1-8B
76
40
0.46
0.36
0.94
54
64
6
Qwen3-4B
71
27
0.51
0.36
0.95
49
64
5
Qwen3-8B
71
29
0.52
0.35
0.94
49
65
6
Qwen2.5-7B
74
31
0.50
0.43
0.94
51
57
5
Appendix
Table 26: Agreement of rows at other addresses with the next round’s root row. Availability is the share of roots with such a row, and agreement is computed over those roots. The preceding row is always available. Total variation uses softmax probabilities.
Figure 6: Gain over draft selection, averaged over six targets, when the softmax probabilities of the exact root row or of a row at another address replace the first draft factor. The dotted line marks EAGLE-3’s tree.
Tokens for each pass
Target
B
EAGLE-3
Covering
Change, %
Covered, %
Gain
Greedy decoding
Llama-3.1-8B
8
4.171
3.859
− 7.5 [ − 8.2, − 6.8]
49
0.62
59
5.505
5.059
− 8.1 [ − 8.6, − 7.6]
52
0.25
Qwen3-4B
8
2.654
2.476
− 6.7 [ − 7.1, − 6.3]
31
0.49
59
3.323
3.106
− 6.5 [ − 6.9, − 6.2]
29
0.28
Appendix
Table 27: Covering decoder against EAGLE-3’s decoder, in tokens emitted for each target pass. Change is the relative change with 95% intervals, Covered the share of covered rounds, and Gain root-draw selection’s token gain over draft selection after covered rounds.
Over draft selection
Over PEARL-style
Target
Drafter
Root draw, %
PEARL-style, %
Root draw, %
Greedy decoding
Llama-3.1-8B
Llama-3.2-1B
12.0 [11.3, 12.7]
2.1 [1.6, 2.7]
9.6 [9.0, 10.3]
Qwen3-8B
Qwen3-0.6B
17.4 [16.6, 18.3]
4.9 [4.3, 5.5]
12.0 [11.3, 12.6]
Sampling at temperature 1
Llama-3.1-8B
Llama-3.2-1B
19.3 [17.6, 21.1]
7.3 [5.9, 8.8]
11.2 [10.0, 12.3]
Appendix
Table 28: Gain in decoding speed at B=8 with each standalone drafter in its own process on a second GPU, over draft selection and over the PEARL-style decoder, with 95% prompt-bootstrap intervals in brackets.
Nodes
Tree pass, ms
Target
Cap
Root draw
PEARL-style
Root draw
PEARL-style
Expected, %
Speed, %
Greedy decoding
Llama-3.1-8B
43
42.0
41.6
32.9
33.1
1.1 [1.0, 1.3]
1.3 [0.9, 1.7]
Qwen3-8B
41
38.4
37.7
34.9
34.5
1.9 [1.7, 2.2]
1.7 [1.2, 2.2]
Sampling at temperature 1
Llama-3.1-8B
40
36.1
35.7
31.8
32.1
2.1 [1.9, 2.2]
2.2 [1.5, 3.0]
Appendix
Table 29: Root-draw selection at its lowered cap against the PEARL-style decoder at B=59 . Nodes and Tree pass are means over rounds. Expected is its gain in expected acceptance on the recorded rounds, Speed its gain in decoding speed, with 95% intervals.
Speculative decoding accelerates language model inference by using a fast drafter to propose candidate tokens that are then verified by a larger target model. Existing theory largely studies the stochastic, distribution-preserving setting, where the goal is to exactly sample from the target distribution. In contrast, many practical systems use greedy decoding, relaxed acceptance rules, or tree-based candidate sets, where success is governed by local ranking and threshold events rather than exact distributional equality. We develop a theory for these regimes. We identify that many common acceptance criteria have rejection regions that can be characterized as lower level sets of the target distribution. For these, we characterize the exact KL divergence required for rejection yielding exact certificates and sharp margin-based bounds for strict greedy decoding, additive and multiplicative relaxed acceptance, top-(m) relaxed criteria, and entropy-thresholded acceptance. We then extend the framework to greedy tree decoding, deriving exact and margin-only certificates for when the target greedy token remains covered by the drafter's top-(m) candidates. Finally, we evaluate the resulting certificates on Qwen3 models, showing that relaxed and tree-based criteria substantially enlarge the region of certified acceptance, especially on decoding steps with low target model distribution margin. These results complement existing distribution-preserving analyses of speculative decoding by characterizing the deterministic local acceptance events common in practical inference systems.
Speculative decoding (SD) accelerates autoregressive generation by letting a lightweight draft model propose tokens that a target model verifies in parallel. However, its standard acceptance rule, designed for exact distribution correction, rejects tokens to which the draft assigns excess probability, even when the target finds them highly plausible: over half of the rejected tokens lie inside the target's own nucleus. Since verification dominates the cost of SD, these rejections waste its most expensive operation. We introduce Nucleus Speculative Decoding (NSD), a plausibility-aware verification rule that accepts a draft token if it passes the standard test or belongs to the target nucleus, while keeping SD's residual correction and bonus sampling unchanged. NSD requires no training or extra forward passes and is compatible with any drafting mechanism. Because it preserves SD's recovery rule, its deviation from the target distribution can be characterized exactly: we prove that the single-step total variation error equals the draft's excess probability inside the target nucleus, which is precisely NSD's gain in acceptance probability. NSD thus trades fidelity for acceptance at an exact one-to-one rate, and we further derive sequence-level fidelity bounds. Across six target models and five mathematical reasoning and code generation benchmarks, NSD achieves the highest throughput among all compared methods, with speedups of up to 5.16× over autoregressive decoding and 3.15× over standard SD, while improving accuracy over SD on five of the six models and nearly halving verification time per output token. Our code is available at https://github.com/EIT-NLP/Nucleus-Speculative-Decoding.
Shuhao Li, Fanghua Ye, Wanyu Lin +2
EIT-NLP Lab, Eastern Institute of Technology, Ningbo · The Hong Kong Polytechnic University · Independent Researcher
Speculative decoding's effectiveness is governed by two quantities: the acceptance rate of drafted tokens and the cost of producing them. We show the two objectives can be co-optimized with algorithm-system co-design, and present JetSpec++, which retains a strong learned draft model while hiding its cost almost entirely. JetSpec++ combines three components: intra-block self refinement, which trains the draft model to iteratively correct its own parallel predictions within a block of tokens; inter-block foresight drafting, which begins the next block before the current one is verified; and pipelined drafting and verification, which overlaps the two stages so drafting's marginal wall-clock cost approaches zero. In comparison with state-of-the-art speculative decoding methods, JetSpec++ attains a wall-clock speedup of up to 7.45× over autoregressive (AR) decoding and 1.54× over DSpark, with up to 1,361 tokens per second for a single request on GB200, and an average acceptance length of 8.94 across seven benchmarks, while preserving the target model's output distribution. Our code and models are available at https://github.com/hao-ai-lab/JetSpec.