When Does Better Acceptance Pay? A Controlled Study of Speculative Decoding

Yongchao Long; Chunrui Zheng; Yuxuan Zhou; Guangkun Nie; Tian-Chi Ma; Chao Pang; Shuohuan Wang; Yuchen Ding

cs.CLOct 18, 2026

When Does Better Acceptance Pay? A Controlled Study of Speculative Decoding

Authors: Yongchao Long, Chunrui Zheng, Yuxuan Zhou, Guangkun Nie, Tian-Chi Ma, Chao Pang, Shuohuan Wang, Yuchen Ding,

Organizations: Department of Computer Science, Tianjin University of Technology · Baidu Inc. · School of Information Science and Engineering, Shandong University · School of Intelligence Science and Technology, Peking University ·

Abstract

The throughput gains of speculative decoding depend on the trade-off between acceptance length and execution latency. We ask when additional acceptance is worth the cost of obtaining it. We establish a comparison protocol spanning training, inference, and evaluation, remeasuring released checkpoints and retraining six drafter recipes on shared target-generated data across ten benchmarks and two target models. Alignment changes the picture: gaps that appear large in published comparisons shrink substantially under shared training and inference conditions. This narrowing motivates two deeper questions: how is acceptance improved at different costs, and when do these gains improve throughput? We find that acceptance can be improved through nearly free objective and context changes or more expensive token-to-token conditioning, which pays off most clearly under sampled drafting. DCore illustrates the former route as a strong baseline with little additional measured drafting overhead. The payoff is conditional: task uncertainty and serving conditions affect acceptance returns, execution costs, and the preferred drafter and budget. These findings inform future drafter design and deployment choices.

Figures & tables

Panel of Figure 1 — Figure 1: Observed gaps narrow after inference and training alignment. Four drafters on GSM8K, representing the low-entropy half of our benchmark suite, and MT-Bench, representing the high-entropy half, on Qwen3-8B. Pale bars: acceptance length $\tau$ (left axis); solid bars: speedup over the target-only baseline (right axis). As reported : each paper’s own published numbers. Re-measured : its released checkpoint re-run in our engine. Re-trained : a drafter we train on the same training data.

Panel of Figure 2 — Figure 2: Key design ablation roadmap on Qwen3-8B ( $B{=}16,T_{t}|T_{d}{=}1|0$ , weight-averaged over ten benchmarks). Most points are single design-search runs; branches illustrate design changes. $\blacksquare$ / $\blacksquare$ : architecture/loss-ablation speedup; $\blacksquare$ : acceptance length; $\bigstar$ / $\bigstar$ : proposal latency, gold $=$ DCore. L: backbone layer count; HS: number of target layers supplying hidden states; Curr.: curriculum learning; Anchor Pred.: anchor-position prediction. Cand. Pool restricts conditioned predictions to top- $K$ marginal candidates following Oda et al. (2026) . KL $+$ TV is an LK-style objective ( Samarin et al., 2026 ) ; D-PAKL pairs KL supervision with D-PACE position weights.

			$T_{t}\|T_{d}=0\|0$		$T_{t}\|T_{d}=1\|0$		$T_{t}\|T_{d}=1\|1$
Tgt.	Bgt.	Method	Low-Ent.	High-Ent.	Low-Ent.	High-Ent.	Low-Ent.	High-Ent.
Qwen3-4B	8	Eagle-3 (7/1/8)	$616.1_{(4.69)}$	$435.0_{(3.20)}$	$514.9_{(4.18)}$	$347.9_{(2.77)}$	$526.6_{(4.16)}$	$358.5_{(2.76)}$
		Eagle-3 (7/4/8)	$591.9_{(4.93)}$	$439.6_{(3.57)}$	$509.5_{(4.48)}$	$366.7_{(3.16)}$	—	—
		DFlash	$809.3_{(5.37)}$	$595.4_{(3.86)}$	$697.4_{(4.73)}$	$487.7_{(3.23)}$	$703.3_{(4.69)}$	$487.4_{(3.19)}$
		DFlare	$815.5_{(5.43)}$	$604.3_{(3.94)}$	$707.4_{(4.77)}$	$495.7_{(3.28)}$	$712.5_{(4.75)}$	$497.2_{(3.26)}$
		D-PACE	$\smash{\underline{825.8}}_{(5.49)}$	$611.0_{(3.96)}$	$\smash{\underline{715.0}}_{(4.83)}$	$494.3_{(3.27)}$	$729.6_{(4.88)}$	$510.7_{(3.35)}$

Table 1: Main results. Cells are

\textsc{TPS}_{(\tau)}

: tokens/s with acceptance length

\tau

subscripted, averaged over each entropy half’s five benchmarks. Entropy split, temperature pair

T_{t}|T_{d}

, budget Bgt. , Eagle-3 configuration triple and the missing

1|1

trees (the dashes) all follow § 3 . Bold / underline : best/second-best in a column of one target– Bgt. block, the only comparable unit, TPS and

\tau

ranked apart. Per-benchmark values: App. J .

Panel of Figure 3 — Figure 3: Accepted paths tend to have lower target entropy than their rejected endpoints across three drafter families. Qwen3-8B at $1|0$ : Eagle-3 (chain), DFlash, and DSpark. Cells show mean raw- $T{=}1$ target entropy at draft position $j$ ; rows show committed tokens $n_{c}+1$ , where $n_{c}$ is the number of accepted drafts. Black borders mark rejected positions; grey bars show the share of steps at each length (full strip $36\%$ ). $B{=}16$ is the main grid and $B{=}8$ the inset. Other targets and temperatures are in Fig. 13 .

Panel of Figure 4 — Figure 4: Budget elasticity: what a larger budget buys, and what it costs (Qwen3-8B, $1|0$ ). We use log changes to express throughput gains as the sum of acceptance gains and efficiency changes, revealing whether additional acceptance repays its execution cost. Axes: log change in acceptance length $\tau$ and throughput from $B{=}8$ to $16$ . Eagle-3(tree) compares $7/4/8\rightarrow 7/10/16$ ; Eagle-3(chain) compares $7/1/8\rightarrow 15/1/16$ . Parallel guide labels give $100\,\Delta\ln\eta$ via Eq. ( 1 ), not ordinary percentage changes. Negative TPS changes indicate throughput loss.

Panel of Figure 5 — Figure 5: Speedup under concurrency (Qwen3-8B, budget $8$ ). Request output throughput relative to the target-only baseline at the same temperature and concurrency; dashed is $1\times$ . $C$ is per replica and doubles at each tick. Eagle-3 is chain-decoded. Insets use local $y$ ranges to resolve method ordering at $C=4$ and $8$ . Fig. 10 adds the other target and budget; App. K defines the metric and documents repeats.

Panel of Figure 6 — Figure 6: Where sampled drafting gains or loses acceptance (Qwen3-8B). (a) Two-factor Shapley split of the change in mean committed tokens $\Delta\tau$ from $1|0$ to $1|1$ into the first draft position ( $j{=}0$ , light) and all later positions ( $j{\geq}1$ , solid); the bar pair sums to $\Delta\tau$ (black line). (b) Acceptance at $j{\geq}1$ under $1|1$ after an argmax predecessor (filled: the previous accepted token is the target’s argmax) or a non-argmax predecessor (open: it is not), standardized to a common target-entropy mix. Block-parallel recipes (first four of each group) lose at $j{\geq}1$ and lose far more after non-argmax predecessors than semi-AR recipes. App. L gives the estimators, intervals, and per-position curves.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

	BP / Semi-AR	Eagle-3
Epochs	$3$	$3$
Learning rate	$6\mathrm{e}{-4}$	$1\mathrm{e}{-4}$
Warmup ratio	$0.04$	$0.015$
Gradient clip	$1.0$	$0.5$
Optimiser	AdamW $(0.9,\,0.999)$
Weight decay	$0$

Appendix

Table 2: Training settings. Common settings such as epochs, optimizer, and precision are unified across the two columns. The left column is shared by DFlash, DFlare, D-PACE, Domino, DSpark, and DCore; Eagle-3 keeps its family-specific defaults.

Drafter	Family	Architectural distinction	Training target
Eagle-3	AR	Token-conditioned decoder with target-layer fusion and a reduced draft vocabulary	KL to target next-token distributions
DFlash	Block Parallel	Shared projection over selected target layers; masked block input	Decayed cross-entropy
DFlare	Block Parallel	Per-draft-layer target-state mixing	Decayed cross-entropy
D-PACE	Block Parallel	Parallel backbone with acceptance-oriented position weighting	D-PACE weighted cross-entropy
Domino	Semi-AR	Parallel backbone plus a correction head and curriculum	Base/final cross-entropy curriculum
DSpark	Semi-AR	Parallel backbone with a token-dependent correction path	Decayed CE, distribution matching, and confidence BCE

Appendix

Table 3: The evaluated drafter recipes viewed through the common framework. We use AR for autoregressive, Block Parallel for block-parallel, and Semi-AR for semi-autoregressive.

		Qwen3-4B target ( $d{=}2560$ )					Qwen3-8B target ( $d{=}4096$ )
Drafter	Layers	Fusion	Backbone	Corr.	Head	Total	Fusion	Backbone	Corr.	Head	Total
Eagle-3	1	19.7	116.7	—	81.9	218.2	50.3	218.1	—	131.1	399.5
DFlash	5	32.8	504.7	—	389.0 †	926.4	83.9	964.7	—	622.3 †	1671.0
DFlare	5	26.2	504.7	—	389.0 †	919.8	41.9	964.7	—	622.3 †	1629.0
D-PACE	5	32.8	504.7	—	389.0 †	926.4	83.9	964.7	—	622.3 †	1671.0
Domino	5	32.8	504.7	50.8	389.0 †	977.2	83.9	964.7	55.9	622.3 †	1726.9

Appendix

Table 4: Drafter parameters (millions) under both targets. Layers is the number of drafter decoder layers. Fusion is the projection from the selected target layers into the drafter width: a shared projection over five layers for DFlash, D-PACE, Domino, and DSpark; per-draft-layer projections over nine layers for DFlare and DCore; a three-layer fusion FC for Eagle-3. Backbone is the transformer decoder stack. Corr. is the within-block correction module (Domino’s correction head, DSpark’s token-dependent correction path). Head is the output unembedding; entries marked with † are the frozen LM head shared with the target model—not trained, but executed in every draft forward pass and therefore included in Total . All other columns are trainable. The target embedding table reused by all drafters is excluded. The counts are independent of the draft block size, which only sets the number of mask placeholders per drafted block.

		$0\|0$		$1\|0$		$1\|1$
Seed	Anchors	Low	High	Low	High	Low	High
$0$	$256$	5.53	3.96	4.84	3.27	4.88	3.34
$1$	$256$	5.53	3.96	4.83	3.27	4.87	3.33
$42$	$256$	5.53	3.96	4.82	3.29	4.89	3.34
Tab. 1	$512$	5.57	4.05	4.91	3.33	4.97	3.42

Appendix

Table 5: Training-seed robustness (DCore, Qwen3-4B, budget

8

). Mean

\tau

over each entropy half. Each seed sets both data shuffling and initialization. The last row is the Tab. 1 checkpoint.

Stage	Autoregressive drafter	Block-parallel and semi-AR drafters	Execution
Setup	Allocates the drafter’s key–value slots for the whole rollout and builds its positions and forward metadata in one pass, sized from an upper bound so that no per-step count is read back.	Writes the draft block — the previous step’s bonus token, then mask tokens — with its positions and cache slots, and gathers its input embeddings from the target’s embedding table.	Metadata kernels ahead of the replay. Timed for the autoregressive drafter, part of Resid. for the others.
Rollout	Every drafter forward and, between consecutive forwards, the top- $k$ expansion that turns one step’s logits into the next step’s input, with the running score and parent bookkeeping.	One forward over the whole block, with the output head and argmax that turn its hidden states into the proposal; Domino’s and DSpark’s correction paths run here as fused kernels.	One replayed graph, holding all of the work in this row.
Proposal assembly	One kernel writes the verification mask in the attention backend’s bit-packed layout, the proposal’s positions, and the three retrieval arrays acceptance indexes with.	— The block is contiguous and causal, so verification needs no custom mask and no retrieval arrays.	One kernel launch, writing straight into the verification graph’s buffers.
Target forward	The target verifies the entire proposal in one forward using the shared target model and serving framework.		CUDA-graph replay for the target forward.
Acceptance	Constructs the target’s distribution where the protocol needs one, runs the tree or chain acceptance kernel, then extracts the bonus token; above top- $k$ one the accepted path is compacted to the front of the cache.	Compares the block against the target’s argmax, or against its distribution at a sampled target; one kernel writes the accepted length, the commit length, the bonus token, the output tokens and the new sequence lengths.	Device-resident kernels writing preallocated buffers. The host’s view of the accepted length is deliberately lagging.
Resync	A drafter forward over the tokens verification has just committed, which both refreshes its key–value cache and emits the first candidate of the next rollout.	The target’s hidden states for the committed prefix, projected to the drafter’s width and written into the drafter’s key–value cache.	A third replayed graph for the autoregressive drafter, one fused kernel for the others.

Appendix

Table 6: Execution stages of one speculative decode step, using the labels in Tab. 7 . AR and parallel/semi-AR drafters differ in proposal generation and cache resynchronization; a dash denotes an absent stage. Model forwards use CUDA-graph replay and accepted-length bookkeeping stays on device. The table describes greedy drafts; sampled paths also use graph replay, with distribution-dependent sampling tails outside the rollout graph.

	Proposal		Verification		Drafter resync		Step
Drafter	Rollout	Total	Target fwd.	Acceptance	KV write	Resync fwd.	Resid.	Device	$\Delta$
Qwen3-4B , budget $8$
Eagle-3 (7/1/8)	$2.02$	$2.14$	$4.57$	$0.025$	—	$0.320$	$0.010$	$7.07$	$0.020$
Eagle-3 (7/4/8)	$2.44$	$2.47$	$4.82$	$0.026$	—	$0.430$	$0.000$	$7.75$	$0.020$
DFlash	$1.28$	$1.31$	$4.54$	$0.018$	$0.076$	—	$0.266$	$6.21$	$0.090$
DFlare	$1.28$	$1.31$	$4.55$	$0.018$	$0.070$	—	$0.262$	$6.21$	$0.090$

Appendix

Table 7: Device-stage latency (ms), greedy decoding (

0|0

), one request per replica. Entries are medians over timed steps with CUDA graphs enabled. Rollout measures proposal compute; Total includes its bracketing stages. Drafter resync updates the cache through KV writes or an AR forward. A dash denotes an absent stage. Eagle-3’s Target fwd. includes bonus handling and tree cache compaction, giving an upper bound on forward time. Resid. captures device work outside named timers; Device records the complete device step.

\Delta

is host minus device step time. These profiles diagnose stage costs; Fig. 5 reports end-to-end speedup from unprofiled runs. App. C defines the timing boundaries.

$T_{t}$	$T_{d}$	Acceptance rule	Resampling rule	Lossless	Coupling	Dataset	Samples
Eagle-3 — SafeAILab/EAGLE @ cb7e0841 ; third-party weights Qwen3-4B , Qwen3-8B
$0$	$0$	$\mathbf{1}[\hat{x}=\arg\max p]$	— (bonus $=\arg\max p$ )	✓	$\delta$	n/r	n/r
$T$	$0$	$\min\{1,\,p(\hat{x})/1\}=p(\hat{x})$ ; $q(\hat{x})$ is the constant $1$	$\dfrac{p(v)\,\mathbf{1}[v\notin R]}{1-\sum_{u\in R}p(u)}$	✓	$\delta$		n/r
DFlash , authors’ repository — z-lab/dflash @ 07ebd93d ; weights Qwen3-4B , Qwen3-8B
$0$	$0$	$\mathbf{1}[\hat{x}=\arg\max p]$	— (bonus $=\arg\max p$ )	✓	$\delta$	NPT2 $+$ CA	$800$ K
$1$	$1$	$\min\{1,\,p(\hat{x})/q(\hat{x})\}$	$\dfrac{(p-q)_{+}}{\sum_{v}(p-q)_{+}}$	✓	maximal		$800$ K

Appendix

Table 8: Verification protocols, training sets and released weights of the published baselines.

T_{t}

and

T_{d}

are the target and draft temperatures the pinned commit’s code actually reaches, not the flag value; one row per pair.

\hat{x}

is the drafted token,

p,q

the target and drafter distributions,

R

the rejected set. NPT2/CA/S3.5/OPB

=

Nemotron-Post-Training-Dataset-v2 / CodeAlpaca / Step-3.5-Flash-SFT / Open-PerfectBlend; n/r

=

unreported. ∗ D-PACE’s 4B setting; its 8B corpus is ShareGPT.

Qwen3-4B
		Low-Entropy															High-Entropy
		GSM8K			MATH			HE			MBPP			LCB			AIME25			MT-B
$T_{t}\|T_{d}$	Method
$0\|0$	Eagle-3(16)	$3.30$	$3.33$	$6.33$	$3.08$	$3.14$	$6.09$	$3.05$	$3.11$	$5.51$	$2.95$	$2.98$	$5.34$	$2.91$	$3.01$	$5.28$	$3.05$	$3.10$	$5.52$	$3.02$	$2.93$	$3.75$
	Eagle-3(60)	$3.77$	$3.85$	$7.05$	$3.52$	$3.63$	$6.91$	$3.47$	$3.53$	$6.36$	$3.38$	$3.43$	$6.21$	$3.22$	$3.39$	$6.11$	$3.51$	$3.59$	$6.49$	$3.49$	$3.33$	$4.51$
	DFlash(16)	$6.53$	$6.23$	$8.05$	$7.84$	$7.87$	$7.56$	$6.64$	$6.49$	$6.00$	$6.09$	$5.70$	$5.50$	$7.09$	$6.70$	$5.88$	$7.27$	$7.08$	$6.41$	$4.35$	$3.27$	$3.50$

Appendix

Table 9: Acceptance length, as published, as re-measured and as re-trained. Three columns per benchmark: is

\tau

exactly as reported by the source paper, the only column taken from the literature; re-runs that method’s released checkpoint, and a drafter we retrain on one common training set, both under the single protocol of App. B .

T_{t}|T_{d}

is the target and draft temperature pair set by each method’s own evaluation code (Tab. 8 ); parenthesised numbers are verification budgets. ‘—’ marks a configuration the paper does not report ( ) or a drafter with no released weights ( ). Shading shows within-row changes : each measured cell is filled by its percentage change in

\tau

against the cell to its left (legend below), grey where that cell is ‘—’. Cross-method comparisons use matched budgets and temperature pairs.

Dataset	Qwen3-4B	Qwen3-8B
LCB	$0.064\pm 0.057$	$0.069\pm 0.065$
GSM8K	$0.124\pm 0.054$	$0.142\pm 0.059$
MATH-500	$0.171\pm 0.088$	$0.192\pm 0.094$
MBPP	$0.128\pm 0.066$	$0.195\pm 0.076$
HumanEval	$0.133\pm 0.050$	$0.209\pm 0.071$
Alpaca	$0.261\pm 0.149$	$0.313\pm 0.172$

Appendix

Table 10: Target-model token-distribution entropy

H

per benchmark, in nats (

\pm

std. across prompts), sorted by the Qwen3-8B column. The rule separates the five Low-Entropy from the five High-Entropy benchmarks of Tab. 1 : that grouping is exactly the top and bottom half of this ordering, under either target.

Panel of Figure 7 — Figure 7: Continuous task variation, without an entropy split. Each method has one point per benchmark (Qwen3-8B, $1|0$ ). Left: $B{=}16$ acceptance length. Right: throughput change $100(\mathrm{TPS}_{16}/\mathrm{TPS}_{8}-1)$ , using three-repeat means from the evaluation logs. Entropy follows Tab. 10 . Eagle-3 widens a tree using one checkpoint; DFlash and DSpark use budget-specific checkpoints. Points summarize benchmark-level associations.

Panel of Figure 8 — Figure 8: DCore recipe. Orange: DFlash backbone; blue: DFlare context mixture; purple: D-PACE objective; grey: frozen target and replaced defaults. (a) Draft layer $i$ mixes nine target layers using its own $\alpha^{(i)}=\mathrm{softmax}(W_{\mathrm{fuse}}[i,:])$ . (b) Position weights $\bar{w}_{j}=\sum_{m\geq j}\prod_{i\leq m}\tilde{q}_{i}$ use smoothed draft confidences $\tilde{q}_{i}$ . Bars illustrate the normalized weights for uniform early/late confidences; the inset shows the corresponding prefix products. Fusion shading and confidence curves are schematic, not measured. App. G specifies the recipe.

Panel of Figure 9 — Figure 9: Eagle-3 configurations under concurrency (Qwen3-8B, $1|0$ ). Settings are depth/top- $k$ /budget; filled markers denote chains and open markers trees. Lines show unprofiled speedup relative to the target-only baseline at the same concurrency. Bars show drafting and verification GPU-stage medians ( $\lambda_{D}+\lambda_{V}$ ) through $C=64$ , measured separately from the throughput runs; they provide supplementary stage-cost diagnostics. Acceptance is approximately stable across load within each setting ( $3.17$ , $3.56$ , $3.34$ , $4.09$ , $4.80$ in legend order).

			Low-Entropy					High-Entropy
Tgt.	$T_{t}\|T_{d}$	Method	GSM8K	MATH	HE	MBPP	LCB	AIME25	SWE-B	MT-B	Alpaca	Arena-H
Qwen3-4B	$0\|0$	Target only	$229.6$
		Eagle-3 (7/1/8) ${}^{\textsc{a}}$	$2.97_{(5.29)}$	$2.94_{(5.00)}$	$2.62_{(4.53)}$	$2.42_{(4.22)}$	$2.46_{(4.41)}$	$2.60_{(4.37)}$	$1.72_{(2.91)}$	$1.68_{(2.84)}$	$1.66_{(2.85)}$	$1.81_{(3.04)}$
		Eagle-3 (7/4/8) ${}^{\textsc{a}}$	$2.82_{(5.47)}$	$2.82_{(5.26)}$	$2.49_{(4.72)}$	$2.37_{(4.51)}$	$2.39_{(4.70)}$	$2.47_{(4.59)}$	$1.78_{(3.31)}$	$1.76_{(3.28)}$	$1.73_{(3.25)}$	$1.84_{(3.40)}$
		DFlash ${}^{\textsc{b}}$	$3.91_{(5.99)}$	$3.90_{(5.83)}$	$3.38_{(5.09)}$	$3.21_{(4.84)}$	$3.23_{(5.08)}$	$3.57_{(5.29)}$	$2.43_{(3.63)}$	$2.29_{(3.41)}$	$2.19_{(3.28)}$	$2.49_{(3.69)}$
		DFlare ${}^{\textsc{b}}$	$3.94_{(6.08)}$	$3.94_{(5.93)}$	$3.40_{(5.14)}$	$3.23_{(4.90)}$	$3.24_{(5.10)}$	$3.61_{(5.37)}$	$2.45_{(3.68)}$	$2.33_{(3.50)}$	$2.23_{(3.34)}$	$2.53_{(3.77)}$

Appendix

Table 11: Per-benchmark speedup and acceptance length on both targets (

B{=}8

C{=}1

). Cells and ranking marks follow Tab. 12 , using the same target-only references. Eagle-3 includes both the

7/1/8

chain and the

7/4/8

tree; sampled

1|1

results use the chain.

			Low-Entropy					High-Entropy
Tgt.	$T_{t}\|T_{d}$	Method	GSM8K	MATH	HE	MBPP	LCB	AIME25	SWE-B	MT-B	Alpaca	Arena-H
Qwen3-4B	$0\|0$	Target only	$229.6$
		Eagle-3 ${}^{\textsc{a}}$	$3.14_{(6.33)}$	$3.15_{(6.09)}$	$2.81_{(5.51)}$	$2.70_{(5.34)}$	$2.60_{(5.28)}$	$2.90_{(5.52)}$	$1.97_{(3.77)}$	$1.94_{(3.75)}$	$1.93_{(3.74)}$	$2.06_{(3.90)}$
		Eagle-3(60) ${}^{\textsc{a}}$	$3.21_{(7.05)}$	$3.31_{(6.91)}$	$2.99_{(\smash{\underline{6.36}})}$	$2.89_{(6.21)}$	$2.75_{(6.11)}$	$3.15_{(6.49)}$	$2.14_{(\mathbf{4.45})}$	$2.17_{(\mathbf{4.51})}$	$2.13_{(\mathbf{4.48})}$	$2.24_{(\mathbf{4.60})}$
		DFlash ${}^{\textsc{b}}$	$5.07_{(8.05)}$	$4.94_{(7.56)}$	$3.89_{(6.00)}$	$3.56_{(5.50)}$	$3.63_{(5.88)}$	$4.21_{(6.41)}$	$2.43_{(3.71)}$	$2.30_{(3.50)}$	$2.14_{(3.27)}$	$2.52_{(3.81)}$
		DFlare ${}^{\textsc{b}}$	$5.15_{(8.17)}$	$5.00_{(7.68)}$	$3.92_{(6.07)}$	$3.59_{(5.58)}$	$3.70_{(5.96)}$	$4.18_{(6.37)}$	$2.45_{(3.75)}$	$\smash{\underline{2.36}}_{(3.59)}$	$\smash{\underline{2.18}}_{(3.33)}$	$2.57_{(3.89)}$

Appendix

Table 12: Per-benchmark speedup and acceptance length on both targets (

B{=}16

C{=}1

), with an additional budget-

60

Eagle-3 tree. Cells are

\mathcal{S}_{(\tau)}

, normalized to the Target only reference in the same target–temperature block. Bold / underline mark best/second-best values within a benchmark column; speedup and acceptance are ranked separately. Superscripts denote AR ( a ), block-parallel ( b ), and semi-AR ( s ). All recipes use shared-data retraining. The Eagle-3 rows use tree decoding; its budget-

16

chain is summarized in Tab. 1 .

Budget $B{=}8$
	Committed tokens			Shapley terms			$1\|1$ , $j\geq 1$			$1\|0$	$f$ (%)
Drafter	$\tau^{1\|0}$	$\tau^{1\|1}$	$\Delta\tau$	$j{=}0$	$j{\geq}1$	$\Delta\alpha_{0}$	A	N	gap	gap	$1\|0$	$1\|1$	$\Delta$ KL
Eagle-3	$3.18$	$3.27$	$0.095$	$0.138$	${-}0.043$	$0.035$	$0.738$	$0.643$	${-}0.095$	${-}0.043$	$2.9$	$9.6$	$1.16$
DFlash	$3.57$	$3.60$	$0.026$	$0.246$	${-}0.220$	$0.064$	$0.758$	$0.333$	${-}0.425$	${-}0.196$	$2.5$	$8.7$	$9.94$
DFlare	$3.63$	$3.71$	$0.076$	$0.271$	${-}0.195$	$0.070$	$0.768$	$0.337$	${-}0.430$	${-}0.198$	$2.5$	$8.5$	$10.08$
D-PACE	$3.69$	$3.86$	$0.170$	$0.248$	${-}0.078$	$0.058$	$0.796$	$0.396$	${-}0.400$	${-}0.159$	$2.5$	$7.3$	$9.38$

Appendix

Table 13: Sampled-draft decomposition on Qwen3-8B (

T_{t}{=}1

\tau

columns and the Shapley terms (Eq. ( 9 )) use realized chains;

\Delta\alpha_{0}

and the prefix-state columns are entropy-standardized (Eq. ( 8 )) and pool

j\geq 1

. A/N:

1|1

acceptance after an argmax or non-argmax predecessor; gap is N minus A.

f

: raw N share at

j\geq 1

(%).

\Delta

KL: N minus A

\mathrm{KL}(q\|p)

under

1|1

, in nats. Eagle-3 has no

B{=}16

1|1

dump.

Panel of Figure 11 — Figure 11: Per-position change from $1|0$ to $1|1$ (Qwen3-8B). Top: acceptance change $\Delta\alpha_{j}$ at each reached draft position, standardized to a common target-entropy mix (Eq. ( 8 )). Bottom: change in the survival probability $\Delta S_{k}=\Delta\Pr[\text{at least }k\text{ drafts accepted}]$ ; each curve sums over $k$ to that method’s $\Delta\tau$ (Eq. ( 9 )). Every recipe gains at $j{=}0$ ; block-parallel curves turn negative from $j{=}1$ or $2$ , semi-AR curves stay positive through $j{=}5$ ( $B{=}8$ ) and $j{=}7$ ( $B{=}16$ ). Eagle-3 has no $B{=}16$ sampled-draft dump.

Panel of Figure 14 — Figure 14: Acceptance during training. Checkpoints from $0.2$ to $3.0$ epochs on a fixed Open-PerfectBlend corpus, Qwen3-8B, $B=16$ , $1|0$ , $C=1$ , along one training trajectory per recipe.

Speculative Decoding

Speculative decoding accelerates generation by drafting multiple tokens and verifying them in parallel with a larger target model.

See all 308 papers

Jun 29, 2026

When Is a Draft Accepted? A Theory of Acceptance in Speculative Decoding

Speculative decoding accelerates language model inference by using a fast drafter to propose candidate tokens that are then verified by a larger target model. Existing theory largely studies the stochastic, distribution-preserving setting, where the goal is to exactly sample from the target distribution. In contrast, many practical systems use greedy decoding, relaxed acceptance rules, or tree-based candidate sets, where success is governed by local ranking and threshold events rather than exact distributional equality. We develop a theory for these regimes. We identify that many common acceptance criteria have rejection regions that can be characterized as lower level sets of the target distribution. For these, we characterize the exact KL divergence required for rejection yielding exact certificates and sharp margin-based bounds for strict greedy decoding, additive and multiplicative relaxed acceptance, top-(m) relaxed criteria, and entropy-thresholded acceptance. We then extend the framework to greedy tree decoding, deriving exact and margin-only certificates for when the target greedy token remains covered by the drafter's top-(m) candidates. Finally, we evaluate the resulting certificates on Qwen3 models, showing that relaxed and tree-based criteria substantially enlarge the region of certified acceptance, especially on decoding steps with low target model distribution margin. These results complement existing distribution-preserving analyses of speculative decoding by characterizing the deterministic local acceptance events common in practical inference systems.

Aaryam Sharma

Independent Researcher Waterloo, Ontario, Canada

Speculative Decoding LLM Inference Acceleration

Ask PDF

Sep 21, 2026

Acceptance-Aware Draft Model Training for Speculative Decoding

Speculative decoding accelerates large language model (LLM) inference by using a lightweight draft model to generate multiple candidate tokens that are verified by the target model in a single forward pass. Its speedup is largely determined by the acceptance length, yet existing draft-model training methods mainly optimize cross-entropy or Kullback-Leibler (KL) divergence as proxies. These objectives encourage distribution matching but do not directly optimize acceptance length, and the acceptance mechanism also differs between greedy and sampling-based decoding. In this work, we propose acceptance-length-aware training losses that directly optimize the expected number of accepted tokens within a speculative window. For greedy verification, we derive an expected accepted length (EAL) loss that explicitly maximizes expected acceptance length. For sampling-based decoding, we introduce a window total variation (WTV) loss that optimizes the overlap between temperature-scaled draft and target distributions while accounting for sequential acceptance dependencies. Both objectives can be further combined with a group-relative reinforcement learning stage (GRPO) using simulated acceptance length as the reward. Experiments across different target and draft models, tasks, and decoding settings show that our losses consistently improve acceptance length over KL-based training. WTV provides particularly strong gains under sampling-based decoding, while EAL better matches greedy verification. These results show that directly optimizing the acceptance objective, with losses tailored to the decoding mode, is more effective than conventional distribution-matching objectives.

Tianhua Xia, Mugilan Ganesan, Yifei Feng +3

New York University · University of Waterloo · Independent Researcher

Speculative Decoding LLM Inference Acceleration

Ask PDF

Oct 20, 2026

Beyond the Point Mass: Acceptance-Derived Weights and Distributional Losses for Speculative Decoding

In speculative decoding, a lightweight draft model proposes tokens that the target model verifies in parallel, speeding up generation without changing the target distribution. D-PACE (Wu et al., 2026) trains such drafters with a loss that weights each position by its contribution to the expected accepted length. Both its weights and its cross-entropy (CE) loss use only the draft's probability of the recorded target token, treating the target as a point mass. Under rejection sampling, however, a sampled draft token is accepted with probability equal to the overlap of the draft and target distributions, one minus their total variation (TV). We derive D-PAL (Dynamic Position-wise Acceptance Loss) from the exact expected accepted length, using this overlap as the acceptance signal. This yields overlap-based position weights with log acceptance (LA) as the per-position loss, and the objective reduces to D-PACE for a point-mass target. Since the weights are detached, any per-position loss can replace LA in practice. We show that forward KL divergence and Rényi divergence of order 1/2 match or exceed LA, and that Rényi-1/2 provably bounds the true acceptance rate from above and below. Replacing CE with Rényi-1/2 improves acceptance length under both D-PACE's weights and ours. On six benchmarks with a Qwen3-4B target, D-PAL raises mean acceptance length over D-PACE from 4.50 to 4.57 at T=0 and from 4.20 to 4.28 at T=1, and consistently achieves higher throughput at both temperatures.

Tianyu Wu, Zhuohan Wang, Jonathan Geuter +5

Harvard University · Massachusetts Institute of Technology

Speculative Decoding LLM Inference Acceleration

Ask PDF

Abstract

Speculative Decoding

Explore similar work

When Is a Draft Accepted? A Theory of Acceptance in Speculative Decoding

Acceptance-Aware Draft Model Training for Speculative Decoding

Beyond the Point Mass: Acceptance-Derived Weights and Distributional Losses for Speculative Decoding