Understanding Target Probability Reuse in Speculative Decoding

Jungseob Lee; Giyoon Jang; Chailin Nam; Jiwoo Kum; Sihyeon Lee; Kyojun Choo; Jihun Song; Eunseo Song

cs.CLOct 21, 2026

Understanding Target Probability Reuse in Speculative Decoding

Authors: Jungseob Lee, Giyoon Jang, Chailin Nam, Jiwoo Kum, Sihyeon Lee, Kyojun Choo, Jihun Song, Eunseo Song,

Organizations: Korea University · Soongsil University

Abstract

Speculative decoding uses a target model to verify a tree of draft tokens in parallel. We study whether the resulting next-token distributions can help select the next tree. For unrestricted target models, we show that guaranteed exact reuse requires a distribution computed at the next round's complete context. We characterize when retaining this distribution requires sacrificing accepted tokens and establish conditions under which this loss persists across decoding rounds. Across six targets, we find that reranking first-token branches with a separately computed distribution at the current context improves expected acceptance over draft ranking, especially with small tree budgets. Drawing the first token from this distribution before allocating the tree further improves expected acceptance over pruning a preselected tree, even at equal mean tree size. In our experiments, both selection rules also decode faster than draft ranking when the separate target pass overlaps standalone drafting. Our findings show that making target probabilities available before tree allocation can improve the efficiency of speculative decoding. Our code is available at https://github.com/js-lee-AI/TargetRows.

Figures & tables

Figure 1: Address matching across decoding rounds. The dashed link identifies the scored prefix $caa$ with the next root; its row is $p(\cdot\mid caa)$ .

Greedy decoding
	Exhaustive	Cap 1		Sampled-leaf stop		Scored top $M$ , $\Pr(C)$
Target	$\mathbb{E}[A]$	$\Pr(C)$	$\mathbb{E}[L_{T}-A]$	$\Pr(C)$	$\mathbb{E}[L_{T}-A]$	$M=100$	$M=200$	$M=710$
Llama-3.1-8B	4.509	0.817	3.599	0.466	0.466	0.158	0.275	0.328
Qwen3-4B	2.340	0.701	1.452	0.241	0.241	0.128	0.261	0.345
Qwen3-8B	2.221	0.587	1.401	0.239	0.239	0.115	0.226	0.304
Qwen2.5-7B	3.293	0.686	2.459	0.293	0.293	0.123	0.225	0.285

Table 1: Coverage, expected acceptance and shortfall on EAGLE-3’s

59

-node trees, computed as expectations from the same recorded target rows for every rule. Scored sets keep exhaustive acceptance on the tree and add the next

M-59

candidates in draft order.

Greedy decoding
	$B=8$					$B=59$
	Expected acceptance			Root-draw gain, %		Expected acceptance			Root-draw gain, %
Target	Draft	Root	Oracle	Same cap	Equal mean	Draft	Root	Oracle	Same cap	Equal mean
Llama-3.1-8B	3.186	3.568	4.899	10.0	2.9	4.509	4.676	4.899	3.5	0.9
Qwen3-4B	1.664	2.002	2.720	12.8	4.5	2.340	2.516	2.720	6.7	2.4
Qwen3-8B	1.556	1.919	2.564	13.8	4.4	2.221	2.399	2.564	6.7	2.2

Table 2: Expected accepted draft tokens and root-draw gains over the PEARL-style decoder at shared contexts. Same cap uses the stated

B

; Equal mean enlarges PEARL-style preselection to match the mean verified node count.

Figure 2: Gain over draft selection when exact target rows replace draft factors through increasing depths, averaged over six targets. The upper edge of each band is one selection rule. The dotted line marks EAGLE-3’s tree.

Figure 3: Share of the oracle advantage at

B=59

recovered by root selection, split by whether the drafter’s most probable first token is the target’s most probable token. Labels give the share of rounds in which the two differ.

	Greedy decoding		Temperature $1$
Target	Gain, %	Nodes	Gain, %	Nodes
Llama-3.1-8B	3.3 [2.6, 4.1]	53 / 39	4.1 [1.7, 6.8]	48 / 34
Qwen3-4B	6.7 [6.2, 7.3]	49 / 33	7.1 [6.4, 7.9]	48 / 32
Qwen3-8B	7.3 [6.6, 8.0]	44 / 28	7.3 [6.3, 8.3]	43 / 27
Qwen2.5-7B	4.5 [3.6, 5.5]	49 / 35	5.7 [4.1, 7.4]	46 / 32
Qwen2.5-14B	3.8 [3.1, 4.4]	49 / 36	4.5 [3.0, 5.9]	47 / 32

Table 3: Online decoding at

B=59

with a separate root pass. Gain is root-draw selection’s gain in accepted draft tokens over the PEARL-style decoder, with

95\%

intervals. Nodes gives mean non-root counts as root draw / PEARL-style at each decoder’s own contexts.

Greedy decoding
	$B=8$				$B=59$
Target	Draft	Base	Rows	Root	Draft	Base	Rows	Root
Llama-3.1-8B	3.186	3.187	3.234	3.568	4.509	4.509	4.536	4.676
Qwen3-4B	1.664	1.674	1.722	2.002	2.340	2.343	2.372	2.516
Qwen3-8B	1.556	1.581	1.650	1.919	2.221	2.241	2.272	2.399
Qwen2.5-7B	2.359	2.360	2.399	2.710	3.293	3.294	3.311	3.450

Table 4: Expected accepted draft tokens at shared contexts for draft selection, cross-fitted rerankers and root selection. Base uses draft and contextual features; Rows adds features from retained rows at other addresses.

Target	Decoding	$B=8$	$B=59$
Llama-3.1-8B	Greedy	8.5 [7.3, 9.8]	2.7 [2.4, 3.1]
Qwen3-8B	Greedy	22.6 [20.9, 24.1]	7.6 [7.2, 8.1]
Llama-3.1-8B	Temp. $1$	7.6 [6.9, 8.4]	2.8 [2.6, 3.1]
Qwen3-8B	Temp. $1$	21.4 [20.4, 22.5]	7.3 [7.0, 7.6]

Table 5: Root selection’s expected-acceptance gain over draft ranking under a

1{,}024

-token limit, in percent, for rounds starting once the default output limit is reached. Brackets give

95\%

prompt-bootstrap intervals.

Target	MT-bench	Translation	Summarization	QA	Math	RAG	HumanEval	All
Llama-3.1-8B	4.9 / 3.7	3.9 / 3.6	4.4 / 3.8	6.1 / 4.8	3.7 / 3.6	4.0 / 3.2	2.3 / 1.9	3.7 / 3.0
Qwen3-4B	9.1 / 7.9	12.0 / 11.5	6.9 / 6.5	9.8 / 9.4	7.0 / 6.9	7.6 / 7.0	6.4 / 6.0	7.5 / 7.0
Qwen3-8B	8.8 / 8.9	8.7 / 8.6	7.1 / 7.1	10.2 / 9.4	8.3 / 8.0	8.2 / 7.5	7.2 / 6.8	8.0 / 7.7
Qwen2.5-7B	5.4 / 4.7	6.2 / 5.8	5.4 / 4.9	7.5 / 5.9	4.6 / 4.4	5.3 / 4.6	3.4 / 3.5	4.8 / 4.4
Qwen2.5-14B	5.7 / 4.5	5.6 / 5.5	5.2 / 4.4	7.2 / 5.7	3.6 / 3.5	5.4 / 4.5	3.4 / 3.2	4.6 / 4.0
Qwen2.5-32B	5.3 / 4.1	6.3 / 5.5	5.3 / 4.5	7.2 / 5.6	3.8 / 3.0	5.9 / 4.4	3.1 / 3.1	4.5 / 3.8

Table 6: Root selection’s expected-acceptance gain over draft ranking at

B=59

, in percent. Each cell gives greedy decoding / temperature

1

. All aggregates the complete prompt set.

Figure 4: Root-draw selection’s decoding-speed gain over the PEARL-style decoder on two RTX A6000 GPUs. Left: both caps are

8

. Right: the PEARL-style cap is

59

and root-draw caps are

40

–

43

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Checkpoint	Role
meta-llama/Llama-3.1-8B-Instruct	Target
yuhuili/EAGLE3-LLaMA3.1-Instruct-8B	EAGLE-3 drafter
meta-llama/Llama-3.2-1B-Instruct	Standalone drafter
Qwen/Qwen3-4B	Target
AngelSlim/Qwen3-4B_eagle3	EAGLE-3 drafter
Qwen/Qwen3-8B	Target

Appendix

Table 7: Target checkpoints and their paired EAGLE-3 and standalone drafters. Checkpoint names link to the evaluated Hugging Face revisions.

Greedy decoding
Target	MT-bench	Translation	Summarization	QA	Math	RAG	HumanEval	All
Llama-3.1-8B	5.53	2.65	5.10	4.52	6.05	5.48	6.68	5.51
Qwen3-4B	3.26	2.59	3.05	3.03	3.74	3.38	3.51	3.34
Qwen3-8B	3.05	2.20	2.91	2.69	3.63	3.22	3.59	3.22
Qwen2.5-7B	4.29	3.20	3.62	3.69	5.14	4.03	4.73	4.29
Qwen2.5-14B	4.48	3.24	3.81	3.71	5.26	4.13	5.00	4.49

Appendix

Table 8: Mean tokens emitted in each round, counting accepted draft tokens and the continuation, by task.

Figure 5: Share of the oracle advantage over draft selection recovered by exact rows at increasing depths, for each target and temperature. The dotted line marks EAGLE-3’s

59

-node tree.

Greedy decoding
		Expected	Gain over draft selection			Recovered oracle advantage, %
Target	$B$	Depth 2	Root	Root draw	Oracle	Root	Depth 2
Llama-3.1-8B	8	3.925	0.382 [0.364, 0.401]	0.468 [0.450, 0.485]	1.713 [1.657, 1.766]	22 [21, 23]	43 [42, 44]
	59	4.786	0.167 [0.157, 0.176]	0.252 [0.241, 0.265]	0.390 [0.373, 0.406]	43 [42, 44]	71 [70, 72]
Qwen3-4B	8	2.287	0.338 [0.328, 0.347]	0.438 [0.428, 0.449]	1.056 [1.037, 1.076]	32 [31, 33]	59 [58, 60]
	59	2.625	0.176 [0.171, 0.182]	0.276 [0.270, 0.284]	0.380 [0.370, 0.389]	46 [45, 48]	75 [74, 76]

Appendix

Table 9: Selection gains in expected accepted draft tokens and recovered shares of the oracle advantage. Depth 2 under Expected gives acceptance with two exact factors. Brackets give

95\%

prompt-bootstrap intervals.

Greedy decoding
	Root gain		Depth-two gain
Target	Rounds	Equal tasks	Rounds	Equal tasks
Llama-3.1-8B	0.167	0.169 [0.160, 0.179]	0.277	0.279 [0.266, 0.292]
Qwen3-4B	0.176	0.181 [0.176, 0.187]	0.285	0.286 [0.279, 0.293]
Qwen3-8B	0.178	0.169 [0.163, 0.174]	0.264	0.244 [0.238, 0.250]
Qwen2.5-7B	0.157	0.162 [0.155, 0.170]	0.254	0.260 [0.251, 0.269]

Appendix

Table 10: Gains over draft selection at

B=59

under round weighting and equal task weighting. Brackets give

95\%

intervals from

1{,}000

resamples of prompts within tasks.

	$M=100$		$M=200$		$M=400$		$M=710$
Target	Advantage	Root, %	Advantage	Root, %	Advantage	Root, %	Advantage	Root, %
Greedy decoding
Llama-3.1-8B	0.186	58	0.321	48	0.380	43	0.390	43
Qwen3-4B	0.135	63	0.279	52	0.359	47	0.380	46
Qwen3-8B	0.125	67	0.247	58	0.319	53	0.343	52
Qwen2.5-7B	0.147	59	0.267	51	0.327	47	0.343	46

Appendix

Table 11: Oracle advantage over draft selection at

B=59

when the pool holds the

M

highest-ranked candidates, and the share that root selection recovers.

	$B=8$			$B=59$
Target	Gain, %	Equal size, %	Share, %	Gain, %	Equal size, %	Share, %
Greedy decoding
Llama-3.1-8B	10.0 [9.4, 10.5]	2.9 [2.6, 3.3]	71	3.5 [3.3, 3.7]	0.9 [0.8, 1.0]	64
Qwen3-4B	12.8 [12.4, 13.3]	4.5 [4.2, 4.8]	55	6.7 [6.5, 6.9]	2.4 [2.2, 2.5]	60
Qwen3-8B	13.8 [13.4, 14.2]	4.4 [4.0, 4.7]	48	6.7 [6.5, 6.9]	2.2 [2.1, 2.4]	47
Qwen2.5-7B	10.7 [10.2, 11.2]	4.1 [3.8, 4.4]	55	4.2 [4.0, 4.4]	1.3 [1.2, 1.4]	46

Appendix

Table 12: Gain of root-draw selection over the PEARL-style decoder. In Equal size, it preselects more candidates until mean tree sizes are equal. Brackets give

95\%

intervals. Share is the part of root-draw selection’s gain over draft selection that Gain makes up.

Greedy decoding
	Reference budget $B=59$		Reference budget $B=8$
Target	Draft reaches root	Root reaches draft	Draft reaches root	Root reaches draft
Llama-3.1-8B	94 [92, 96]	37 [36, 37]	13 [12, 13]	7 [7, 7]
Qwen3-4B	118 [116, 121]	26 [25, 26]	20 [20, 21]	4 [4, 4]
Qwen3-8B	130 [127, 134]	24 [23, 25]	22 [21, 23]	4 [4, 4]
Qwen2.5-7B	105 [103, 107]	32 [32, 33]	15 [15, 16]	6 [6, 6]

Appendix

Table 13: Smallest budget at which draft selection reaches root selection’s expected accepted draft tokens at the reference budget, and at which root selection reaches draft selection’s. Brackets give

95\%

prompt-bootstrap intervals.

Greedy decoding
		Root selection			Root-draw selection
Target	EAGLE-3	Accepted	Gain, %	Offline, %	Accepted	Gain, %	Offline, %
Llama-3.1-8B	4.509	4.721	4.7 [3.6, 5.9]	3.7	4.842	7.4 [6.3, 8.4]	5.6
Qwen3-4B	2.340	2.507	7.1 [6.5, 7.8]	7.5	2.630	12.4 [11.6, 13.2]	11.8
Qwen3-8B	2.221	2.402	8.2 [7.4, 8.9]	8.0	2.606	17.4 [16.4, 18.3]	15.4
Qwen2.5-7B	3.293	3.463	5.2 [4.2, 6.1]	4.8	3.683	11.9 [10.7, 13.0]	9.5

Appendix

Table 14: Online decoding at

B=59

. Accepted is the realized mean acceptance, and EAGLE-3 that of the recorded runs. Gain over EAGLE-3 has

95\%

prompt-bootstrap intervals in brackets, and Offline is the gain over draft selection at the recorded contexts.

	First tokens differ, %		Oracle advantage		Root recovery, %		Depth-two recovery, %
Target	Rounds	Root gain	Agree	Differ	Agree	Differ	Agree	Differ
Greedy decoding
Llama-3.1-8B	24	60	0.371	0.448	24	92	61	96
Qwen3-4B	36	75	0.371	0.395	19	93	62	97
Qwen3-8B	45	78	0.352	0.333	20	94	61	97
Qwen2.5-7B	34	69	0.340	0.349	22	92	62	96

Appendix

Table 15: Selection at

B=59

split by whether the drafter’s most probable first token is the target’s most probable token. Oracle advantage is in expected accepted draft tokens, and recovery is the share of it that each rule attains.

Greedy decoding
	Accepted
Target	PEARL-style	Root draw	Gain, %	Offline, %	Root TV
Llama-3.1-8B	4.686	4.842	3.3 [2.6, 4.1]	3.5	0.0041
Qwen3-4B	2.463	2.630	6.7 [6.2, 7.3]	6.7	0.0057
Qwen3-8B	2.429	2.606	7.3 [6.6, 8.0]	6.7	0.0063
Qwen2.5-7B	3.523	3.683	4.5 [3.6, 5.5]	4.2	0.0111

Appendix

Table 16: Online decoding at

B=59

. Accepted is realized mean acceptance. Gain compares root-draw selection with the PEARL-style decoder; brackets give

95\%

intervals. Offline evaluates shared recorded contexts. Root TV compares the two passes’ root rows by mean total variation.

Greedy decoding
	Accepted
Target	PEARL-style	Root draw	Gain, %	Offline, %	Root TV
Llama-3.1-8B	3.365	3.724	10.7 [9.9, 11.4]	10.0	0.0043
Qwen3-4B	1.881	2.122	12.8 [12.2, 13.4]	12.8	0.0057
Qwen3-8B	1.872	2.130	13.7 [13.1, 14.4]	13.8	0.0063
Qwen2.5-7B	2.647	2.940	11.1 [10.1, 12.0]	10.7	0.0113

Appendix

Table 17: Online decoding at

B=8

. Gain compares root-draw selection with the PEARL-style decoder in realized acceptance; brackets give

95\%

intervals. Offline evaluates shared recorded contexts. Root TV compares the two passes’ root rows by mean total variation.

	Greedy decoding		Sampling at temperature $1$
Target	Online, %	Offline, %	Online, %	Offline, %
Llama-3.1-8B	1.0 [0.9, 1.1]	0.9 [0.8, 1.0]	1.4 [1.3, 1.5]	1.4 [1.3, 1.5]
Qwen3-4B	2.3 [2.1, 2.4]	2.4 [2.2, 2.5]	2.4 [2.3, 2.5]	2.4 [2.2, 2.5]
Qwen3-8B	2.1 [2.0, 2.3]	2.2 [2.1, 2.4]	2.3 [2.1, 2.4]	2.3 [2.2, 2.5]
Qwen2.5-7B	1.3 [1.2, 1.4]	1.3 [1.2, 1.4]	1.9 [1.8, 2.0]	1.8 [1.7, 1.9]
Qwen2.5-14B	1.2 [1.1, 1.3]	1.2 [1.1, 1.3]	1.7 [1.6, 1.8]	1.8 [1.7, 1.9]

Appendix

Table 18: Gain in expected acceptance of root-draw selection over the PEARL-style tree enlarged to equal mean size at

B=59

, with

95\%

intervals. Online evaluates both trees at online contexts; Offline uses the shared recorded contexts.

Greedy decoding
	$B=8$						$B=59$
		Calibrated		Reranked				Calibrated		Reranked
Target	Draft	First	All	Base	Rows	Root	Draft	First	All	Base	Rows	Root
Llama-3.1-8B	3.186	3.188	3.168	3.187	3.234	3.568	4.509	4.510	4.493	4.509	4.536	4.676
Qwen3-4B	1.664	1.667	1.708	1.674	1.722	2.002	2.340	2.342	2.356	2.343	2.372	2.516
Qwen3-8B	1.556	1.563	1.564	1.581	1.650	1.919	2.221	2.229	2.223	2.241	2.272	2.399

Appendix

Table 19: Expected accepted draft tokens after cross-fitting calibrated or reranked draft factors on complementary prompt halves. Base uses draft and contextual features; Rows adds retained rows from other addresses. Blue cells mark selection with the exact root row.

Greedy decoding
			Late rounds		Whole outputs
Target	Late, %	Rounds	$B=59$	$B=8$	$B=59$	$B=8$
Llama-3.1-8B	41	7,374	2.7 [2.4, 3.1]	8.5 [7.3, 9.8]	3.5 [3.3, 3.7]	11.0 [10.3, 11.7]
Qwen3-8B	48	19,346	7.6 [7.2, 8.1]	22.6 [20.9, 24.1]	7.8 [7.6, 8.0]	23.0 [22.2, 23.9]
Sampling at temperature $1$
Llama-3.1-8B	41	12,334	2.8 [2.6, 3.1]	7.6 [6.9, 8.4]	3.0 [2.9, 3.1]	8.4 [8.0, 8.9]

Appendix

Table 20: Root selection’s gains over draft ranking under a

1{,}024

-token limit, in percent, with

95\%

prompt-bootstrap intervals in brackets. Late is the share of outputs with rounds starting once the default output limit is reached; Rounds counts these rounds.

Greedy decoding
		Coverage, %, with $E$ scored frontier candidates
Target	Frontier size	$E=10$	$E=20$	$E=40$	$E=80$	$E=160$	All
Llama-3.1-8B	407	5.9	10.4	16.9	23.7	29.4	32.8
Qwen3-4B	388	4.4	7.9	13.7	21.4	28.5	34.5
Qwen3-8B	381	4.0	7.1	11.9	18.2	24.6	30.4
Qwen2.5-7B	416	4.9	8.5	13.4	19.1	24.5	28.5

Appendix

Table 21: Coverage of exhaustive traversal when the pass also scores the first

E

frontier candidates in draft order. Frontier size is the mean number of pool candidates outside the tree whose parents lie in it.

Greedy decoding
		Tree rows	All scored rows
Target	Successor rounds	Token, %	Token, %	Address, %	Token only, %
Llama-3.1-8B	19,340	41.7	61.1	25.1	36.0
Qwen3-4B	35,971	27.8	47.4	22.9	24.5
Qwen3-8B	36,856	29.1	46.9	20.4	26.5
Qwen2.5-7B	27,251	31.3	46.7	20.5	26.3

Appendix

Table 22: Successor rounds whose root’s final token or root address appears among the prefixes scored in the previous round. Tree rows are EAGLE-3’s

60

verified prefixes, and all scored rows are the roughly

170

prefixes of our verification pass.

Greedy decoding
	Matched address				Matched token only
Target	Pairs	Mean TV	99th pct. TV	Argmax agree, %	Pairs	Mean TV	Argmax agree, %
Llama-3.1-8B	4,862	0.0036	0.0316	99.7	54,630	0.416	59.1
Qwen3-4B	8,254	0.0043	0.0603	99.5	106,260	0.390	61.3
Qwen3-8B	7,505	0.0045	0.0602	99.4	96,948	0.378	62.7
Qwen2.5-7B	5,575	0.0087	0.0734	99.2	62,228	0.457	54.1

Appendix

Table 23: Agreement between retained rows and the next round’s root row. Matched rows share the complete emitted prefix, while token-only rows share only its final token. Total variation uses softmax probabilities.

Greedy decoding
		$B=8$			$B=59$
Target	Drafter	Draft	Root, %	Root draw, %	Draft	Root, %	Root draw, %
Llama-3.1-8B	Llama-3.2-1B	4.843	11.3 [10.6, 12.0]	9.8 [9.2, 10.4]	6.706	2.7 [2.5, 2.9]	2.6 [2.4, 2.8]
Qwen3-8B	Qwen3-0.6B	3.556	16.8 [15.8, 17.8]	13.2 [12.5, 14.0]	5.234	5.0 [4.6, 5.3]	4.6 [4.3, 4.9]
Sampling at temperature 1
Llama-3.1-8B	Llama-3.2-1B	3.391	6.2 [5.8, 6.7]	13.4 [12.8, 14.0]	5.045	2.4 [2.3, 2.6]	4.8 [4.5, 5.0]

Appendix

Table 24: Standalone drafters in pools of EAGLE-3’s shape. Draft is draft selection’s expected acceptance. Root is root selection’s gain over draft selection, and Root draw root-draw selection’s gain over the PEARL-style decoder, with

95\%

prompt-bootstrap intervals in brackets.

		Gain over draft selection
Target	$B$	Exact	Cache	Same token	Preceding
Greedy decoding
Llama-3.1-8B	8	0.275 [0.259, 0.291]	$-$ 0.595 [ $-$ 0.632, $-$ 0.559]	$-$ 0.220 [ $-$ 0.243, $-$ 0.198]	$-$ 1.760 [ $-$ 1.819, $-$ 1.695]
	59	0.117 [0.110, 0.125]	$-$ 0.325 [ $-$ 0.348, $-$ 0.302]	$-$ 0.111 [ $-$ 0.124, $-$ 0.099]	$-$ 0.803 [ $-$ 0.838, $-$ 0.770]
Qwen3-4B	8	0.315 [0.306, 0.323]	$-$ 0.289 [ $-$ 0.304, $-$ 0.275]	$-$ 0.040 [ $-$ 0.047, $-$ 0.033]	$-$ 0.936 [ $-$ 0.954, $-$ 0.919]
	59	0.166 [0.161, 0.172]	$-$ 0.185 [ $-$ 0.195, $-$ 0.176]	$-$ 0.022 [ $-$ 0.026, $-$ 0.017]	$-$ 0.519 [ $-$ 0.532, $-$ 0.506]

Appendix

Table 25: Gains over draft selection in expected accepted draft tokens, with

95\%

intervals, when the softmax probabilities of the exact root row or of a row at another address replace the first draft factor. Rounds without one keep the draft ranking.

	Available, %		Mean TV			Argmax agree, %
Target	Cache	Same token	Cache	Same token	Preceding	Cache	Same token	Preceding
Greedy decoding
Llama-3.1-8B	76	40	0.46	0.36	0.94	54	64	6
Qwen3-4B	71	27	0.51	0.36	0.95	49	64	5
Qwen3-8B	71	29	0.52	0.35	0.94	49	65	6
Qwen2.5-7B	74	31	0.50	0.43	0.94	51	57	5

Appendix

Table 26: Agreement of rows at other addresses with the next round’s root row. Availability is the share of roots with such a row, and agreement is computed over those roots. The preceding row is always available. Total variation uses softmax probabilities.

Figure 6: Gain over draft selection, averaged over six targets, when the softmax probabilities of the exact root row or of a row at another address replace the first draft factor. The dotted line marks EAGLE-3’s tree.

		Tokens for each pass
Target	$B$	EAGLE-3	Covering	Change, %	Covered, %	Gain
Greedy decoding
Llama-3.1-8B	8	4.171	3.859	$-$ 7.5 [ $-$ 8.2, $-$ 6.8]	49	0.62
	59	5.505	5.059	$-$ 8.1 [ $-$ 8.6, $-$ 7.6]	52	0.25
Qwen3-4B	8	2.654	2.476	$-$ 6.7 [ $-$ 7.1, $-$ 6.3]	31	0.49
	59	3.323	3.106	$-$ 6.5 [ $-$ 6.9, $-$ 6.2]	29	0.28

Appendix

Table 27: Covering decoder against EAGLE-3’s decoder, in tokens emitted for each target pass. Change is the relative change with

95\%

intervals, Covered the share of covered rounds, and Gain root-draw selection’s token gain over draft selection after covered rounds.

Greedy decoding
		Over draft selection		Over PEARL-style
Target	Drafter	Root draw, %	PEARL-style, %	Root draw, %
Llama-3.1-8B	Llama-3.2-1B	12.0 [11.3, 12.7]	2.1 [1.6, 2.7]	9.6 [9.0, 10.3]
Qwen3-8B	Qwen3-0.6B	17.4 [16.6, 18.3]	4.9 [4.3, 5.5]	12.0 [11.3, 12.6]
Sampling at temperature 1
Llama-3.1-8B	Llama-3.2-1B	19.3 [17.6, 21.1]	7.3 [5.9, 8.8]	11.2 [10.0, 12.3]

Appendix

Table 28: Gain in decoding speed at

B=8

with each standalone drafter in its own process on a second GPU, over draft selection and over the PEARL-style decoder, with

95\%

prompt-bootstrap intervals in brackets.

Greedy decoding
		Nodes		Tree pass, ms
Target	Cap	Root draw	PEARL-style	Root draw	PEARL-style	Expected, %	Speed, %
Llama-3.1-8B	43	42.0	41.6	32.9	33.1	1.1 [1.0, 1.3]	1.3 [0.9, 1.7]
Qwen3-8B	41	38.4	37.7	34.9	34.5	1.9 [1.7, 2.2]	1.7 [1.2, 2.2]
Sampling at temperature 1
Llama-3.1-8B	40	36.1	35.7	31.8	32.1	2.1 [1.9, 2.2]	2.2 [1.5, 3.0]

Appendix

Table 29: Root-draw selection at its lowered cap against the PEARL-style decoder at

B=59

. Nodes and Tree pass are means over rounds. Expected is its gain in expected acceptance on the recorded rounds, Speed its gain in decoding speed, with

95\%

intervals.

Speculative Decoding

Speculative decoding accelerates generation by drafting multiple tokens and verifying them in parallel with a larger target model.

See all 308 papers

When Is a Draft Accepted? A Theory of Acceptance in Speculative Decoding
Jun 29, 2026Aaryam SharmaSpeculative Decoding LLM Inference Acceleration
Independent Researcher Waterloo, Ontario, Canada
Nucleus Speculative Decoding: Plausibility-Aware Verification Beyond Exact Distribution
Oct 6, 2026Shuhao Li, Fanghua Ye, Wanyu Lin +2Speculative Decoding LLM Inference Acceleration
EIT-NLP Lab, Eastern Institute of Technology, Ningbo · The Hong Kong Polytechnic University · Independent Researcher
JetSpec++: Foresight Parallel Drafting with Self Refinements
Oct 14, 2026Zhaoxiang Feng, Lanxiang Hu, Haoran Yuan +6Parallel Speculative Decoding Speculative Decoding
UC San Diego · UIUC · Nanjing University +2

Jun 29, 2026J/K move · Enter open · S save

When Is a Draft Accepted? A Theory of Acceptance in Speculative Decoding

Aaryam Sharma

Independent Researcher Waterloo, Ontario, Canada

Speculative Decoding LLM Inference Acceleration

Open paper AskPDF arXiv

Speculative decoding accelerates language model inference by using a fast drafter to propose candidate tokens that are then verified by a larger target model. Existing theory largely studies the stochastic, distribution-preserving setting, where the goal is to exactly sample from the target distribution. In contrast, many practical systems use greedy decoding, relaxed acceptance rules, or tree-based candidate sets, where success is governed by local ranking and threshold events rather than exact distributional equality. We develop a theory for these regimes. We identify that many common acceptance criteria have rejection regions that can be characterized as lower level sets of the target distribution. For these, we characterize the exact KL divergence required for rejection yielding exact certificates and sharp margin-based bounds for strict greedy decoding, additive and multiplicative relaxed acceptance, top-(m) relaxed criteria, and entropy-thresholded acceptance. We then extend the framework to greedy tree decoding, deriving exact and margin-only certificates for when the target greedy token remains covered by the drafter's top-(m) candidates. Finally, we evaluate the resulting certificates on Qwen3 models, showing that relaxed and tree-based criteria substantially enlarge the region of certified acceptance, especially on decoding steps with low target model distribution margin. These results complement existing distribution-preserving analyses of speculative decoding by characterizing the deterministic local acceptance events common in practical inference systems.

Abstract

Speculative Decoding

Explore similar work

When Is a Draft Accepted? A Theory of Acceptance in Speculative Decoding

Nucleus Speculative Decoding: Plausibility-Aware Verification Beyond Exact Distribution

JetSpec++: Foresight Parallel Drafting with Self Refinements