Goose: Anisotropic Speculation Trees for Training-Free Speculative Decoding

Tao Jin; Phuong Minh Nguyen; Naoya Inoue

cs.CLApr 2, 2026

Goose: Anisotropic Speculation Trees for Training-Free Speculative Decoding

Authors: Tao Jin, Phuong Minh Nguyen, Naoya Inoue

Organizations: Japan Advanced Institute of Science and Technology (JAIST)

Abstract

Speculative decoding accelerates large language model inference by drafting multiple candidate tokens and verifying them in a single forward pass. Candidates are organized as a tree: deeper trees accept more tokens per step, but adding depth requires sacrificing breadth (fallback options) under a fixed verification budget. Existing training-free methods draft from a single token source and shape their trees without distinguishing candidate quality across origins. We observe that two common training-free token sources -- n-gram matches copied from the input context, and statistical predictions from prior forward passes -- differ sharply in acceptance rate (~6x median gap, range 2-18x across five models and five benchmarks). We prove that when such a quality gap exists, the optimal tree is anisotropic (asymmetric): reliable tokens should form a deep chain while unreliable tokens spread as wide branches, raising the depth ceiling of balanced trees. We realize this structure in GOOSE, a training-free framework that builds an adaptive spine tree: a deep chain of high-acceptance context-matched tokens with wide branches of low-acceptance alternatives at each node. The resulting tree provably accepts at least as many tokens per step as either source alone. On five LLMs (7B-33B) and five benchmarks, GOOSE achieves 1.9-4.3x lossless speedup, outperforming balanced-tree baselines by 12-33% under the same budget.

Figures & tables

Figure 3: Goose pipeline overview. From anchor . , Stage 1 draws from a unified candidate pool : context matching produces the spine ( blue ), the adjacency table supplies branches ( orange ). Stage 2 verifies all candidates via one LLM forward pass. Stage 3 selects the longest accepted path via a greedy walk, discovering spine continuation : “ 2 ” $\,\to\,$ “ ) ” extends beyond the spine mismatch, yielding 8 tokens per call.

Figure 4: Wall-clock speedup over autoregressive (AR) decoding across five models and five benchmarks. All methods are lossless (greedy decoding, identical output); Goose values are annotated above each group and the sixth panel is the per-model mean. SuffixDecoding is the strongest training-free system reported in prior work, EAGLE-2 the strongest trained one shown (absent on Qwen3-8B, which has no released head). Per-benchmark

\tau

and Sequoia: Table 3 ; EAGLE-3: Section D.4 . † Vicuna-33B runs on 2

\times

A100-40GB; all others on a single A40.

Table 4

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

		ClassEval		GSM8K		HumanEval		MBPP		MT-Bench		Avg.
Model	Method	Spd	$\tau$	Spd	$\tau$	Spd	$\tau$	Spd	$\tau$	Spd	$\tau$	Spd
Vicuna-7B	AR	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00
	PLD	2.34	2.50	1.43	1.46	1.74	1.78	1.59	1.62	1.47	1.48	1.71
	TR	2.58	3.29	2.46	2.96	2.53	3.06	2.41	2.86	2.19	2.60	2.43
	Lookahead	2.27	2.41	1.63	1.67	1.72	1.77	1.75	1.78	1.58	1.60	1.79
	REST	1.68	1.73	1.17	1.19	1.16	1.17	1.05	1.05	1.08	1.08	1.23

Appendix

Table 3: Full main results. Speedup (Spd): wall-clock tok/s ratio vs. AR measured in the same run.

\tau

: compression ratio (tokens per target-model forward; drafter forwards are not counted, which favors the trained rows). Bold : best per column. All methods produce identical output (lossless). Shaded rows denote methods requiring per-model training. Sequoia appears on the two targets with released modern drafters, at

N{=}60

to match our budget

B

; its

\tau

follows Sequoia’s official convention (prefill call excluded), at most

1.7\%

above the table’s convention. EAGLE-2 is absent on Qwen3-8B, which has no released head. † Vicuna-33B evaluated on NVIDIA A100 (2

\times

40GB); all other models on A40. The Vicuna-33B GSM8K EAGLE-2 cell comes from a separate run of that environment, so its Spd is taken against that run’s own AR;

\tau

is unaffected.

Benchmark	Domain	Samples	Max tokens
HumanEval	Code	164	512
MBPP	Code	500	512
ClassEval	Code	100	512
GSM8K	Math	1,319	1,024
MT-Bench	Dialogue	80	1,024

Appendix

Table 4: Benchmark details.

Table 7

Figure 5: (a) Goose ’s mean

\tau

decomposed into the best standalone baseline

\max(\tau_{\textsc{PLD}{}},\tau_{\textsc{TR}{}})

(gray) and the synergy gain from the spine-tree topology (green); the gain ranges from

+16\%

(Vicuna-13B) to

+45\%

(Llama-3-8B). (b) Hyperparameter sensitivity (Qwen3-8B): each panel sweeps one parameter;

\tau

plateaus at

B{\geq}60

and

D{\geq}6

, and varies by fewer than 0.6

\tau

units across

r

and

\rho

. Panels (c)/(d) sweep the spine ratio

r

and the spine-branch ratio

\rho

of Section 4.1 .

Panel of Figure 6 — Figure 6: (a) $\tau$ across four A40 models and five datasets grouped by Goose ’s affinity. CV below each column shows cross-model coefficient of variation. (b) Per-model mean $\tau_{\text{eq5}}/\tau_{\text{iso}}$ across 24 settings; all bars exceed 1.0, confirming spine dominance (Pearson $r{=}0.91$ ).

	Goose		EAGLE-3
Dataset	$\tau$	tok/s	$\tau$	tok/s
HumanEval	2.59	66.6	5.79	109.0
ClassEval	5.35	131.3	5.34	94.5
MT-Bench	2.22	57.0	5.36	102.0
MBPP	2.36	61.4	5.69	108.7
GSM8K	2.76	70.3	5.69	107.8

Appendix

Table 7: Goose vs. EAGLE-3 under EAGLE’s native chat-template protocol (Llama-3.1-8B, A40), the primary protocol for this baseline ( Section 5.2 ).

\tau

/ tok/s.

	Goose		EAGLE-2		EAGLE-3
Dataset	$\tau$	tok/s	$\tau$	tok/s	$\tau$	tok/s
HumanEval	4.50	114.8	4.14	62.2	3.29	62.6
ClassEval	5.54	133.4	4.11	58.9	4.18	74.8
MT-Bench	3.63	90.7	3.51	52.2	3.87	72.6
MBPP	3.51	91.0	4.19	63.6	3.55	68.5
GSM8K	8.90	221.7	4.28	63.6	4.62	86.2

Appendix

Table 8: Raw-prompt protocol (our paper’s protocol, applied to every baseline): Llama-3.1-8B, A40, full

n

\tau

/ tok/s.

Model	ClassEval	MT-Bench	HumanEval	GSM8K	MBPP
Qwen3-8B	70.8	53.3	33.0	48.3	38.3
Llama-3-8B	62.1	53.5	53.4	83.8	45.2
Llama-3.2-1B	61.4	35.1	60.7	37.9	36.7
TinyLlama-1.1B	66.1	84.5	33.2	59.8	33.4

Appendix

Table 9: PLD share of accepted tokens (%); TR supplies the remainder. Llama-3-8B and Qwen3-8B are paper models (full

n

); the two

{\sim}1

B models are robustness checks (

n{=}30

$\tau$ : PLD $\to$ Suffix spine	HumanEval	MBPP	GSM8K	MT-Bench	ClassEval
Llama-3.1-8B (full $n$ )	4.21 $\to$ 4.80	3.30 $\to$ 3.63	6.28 $\to$ 9.26	3.22 $\to$ 3.91	5.00 $\to$ 5.89
Llama-3.2-1B ( $n{=}30$ )	3.75 $\to$ 4.33	3.04 $\to$ 3.27	3.38 $\to$ 3.60	2.55 $\to$ 2.63	4.29 $\to$ 5.00
TinyLlama-1.1B ( $n{=}30$ )	2.84 $\to$ 3.08	2.88 $\to$ 3.09	4.56 $\to$ 5.41	6.19 $\to$ 8.92	5.03 $\to$ 5.74

Appendix

Table 10:

\tau

with the PLD spine swapped for SuffixDecoding’s suffix tree (TR branches and topology unchanged, bypass off on both sides).

GPU / task	Method	bs=1	bs=4	bs=8	bs=16	bs=32
A40, HumanEval	Goose	59.0	129.8	141.5	149.4	143.0
	AR	31.0	96.5	168.2	265.0	366.1
A40, GSM8K	Goose	62.4	138.5	155.7	168.7	156.4
	AR	31.1	79.0	142.1	244.9	382.2
RTX 3090, HumanEval	Goose	73.6	132.0	131.1	124.1	—
	AR	42.4	136.1	228.3	339.0	—

Appendix

Table 11: Aggregate tok/s (total tokens / wall-clock) vs. batch size, Llama-3-8B HumanEval (

n{=}164

) and GSM8K (

n{=}256

), greedy, fp16.

	HumanEval	MBPP	MT-Bench	ClassEval	GSM8K ∗
Goose	4.50 / 114.8	3.51 / 91.0	3.63 / 90.7	5.54 / 133.4	8.90 / 221.7
SuffixDecoding (tree)	2.98/86.6	2.14/63.5	2.33/68.0	3.97/108.9	5.59/162.0

Appendix

Table 12: Goose vs. SuffixDecoding (tree mode, the stronger of its two training-free modes) on Llama-3.1-8B (A40, greedy, full

n

\tau

/ tok/s. ∗ GSM8K fixed-budget.

Model	Dataset	Goose	Suf. (cold)	Suf. (warm)	Merged (warm)
Llama-3-8B	ClassEval	4.92	3.53	3.66	3.55
	GSM8K	5.90	3.38	3.79	3.68
	HumanEval	4.13	2.76	2.94	2.88
	MBPP	3.58	2.25	2.54	2.54
	MT-Bench	2.71	1.63	1.73	1.71
Llama-3.1-8B	ClassEval	5.50	3.97	4.06	4.05

Appendix

Table 13:

\tau

on the full grid: Goose (cold) vs. SuffixDecoding official (cold/warm) and the merged-tree variant (warm). Bold = column best per row.

Dataset	$\tau$	tok/s	$\tau$	tok/s	Sequoia paper $\tau$
	Goose		Sequoia (official)
C4	4.16	116.9	4.79	107.8	5.08
CNN/DM	3.04	86.6	3.91	91.1	4.05
OpenWebText	3.28	93.4	3.95	91.9	3.86

Appendix

Table 14: Goose vs. official Sequoia on Llama-2-7B (A40, greedy,

n{=}50

; Section 5.2 ). Sequoia uses its released trained 68M drafter; Goose is training-free. Wall-clock margins carry a

{\sim}2

-point protocol uncertainty from the generation-length asymmetry described above.

		$N{=}60$		$N{=}128$
Target	Dataset	$\tau$	Spd	$\tau$	Spd
Llama-3-8B	ClassEval	8.63	1.52	8.95	1.52
	GSM8K	8.89	1.56	9.12	1.54
	HumanEval	8.56	1.53	8.88	1.51
	MBPP	8.42	1.48	8.75	1.48
	MT-Bench	7.57	1.36	8.00	1.38

Appendix

Table 15: Modern-stack Sequoia:

\tau

and wall-clock speedup vs. AR at tree sizes

N{=}60/128

(

N

= node count in Sequoia’s notation;

N{=}60

matches our budget

B

; A40, greedy, full

n

). Drafters: Llama-3.2-1B for Llama targets, Qwen3-0.6B for Qwen3-8B.

Model	Dataset	Heuristic	Allocator	Allocator (inst.)
Llama-3-8B	ClassEval	4.80	5.39	5.24
	GSM8K	8.84	7.48	7.25
	HumanEval	4.03	4.47	4.50
	MBPP	3.58	3.92	3.98
	MT-Bench	3.41	3.56	3.47
Llama-3.1-8B	ClassEval	5.46	5.95	5.81

Appendix

Table 16: Equal-budget allocation comparison,

\tau

(full

n

except GSM8K at

n{=}500

). Bold = row best. The two GSM8K/8B-Llama regressions are diagnosed above.

Speculative Decoding

Speculative decoding accelerates generation by drafting multiple tokens and verifying them in parallel with a larger target model.

See all 308 papers

TreeGraft: Adaptive Multi-Drafter Grafting for Tree-Based Speculative Decoding
May 28, 2026Jiaming Fan, Daming Cao, Canchen Huang +6Speculative Decoding LLM Inference Acceleration
Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China · Southeast University · Ant Group +1
SpecForge: A Flexible and Efficient Open-Source Training Framework for Speculative Decoding
Mar 19, 2026Shenggui Li, Chao Wang, Yikai Zhu +14Speculative Decoding LLM Inference Acceleration
Nanyang Technological University · SpecForge Team · Meituan +2
DIVERSED: Relaxed Speculative Decoding via Dynamic Ensemble Verification
Apr 8, 2026Ziyi Wang, Siva Rajesh Kasa, Ankith M S +6Speculative Decoding LLM Inference Acceleration
Purdue University · Amazon Inc. · University of Illinois Urbana-Champaign +1

May 28, 2026J/K move · Enter open · S save

TreeGraft: Adaptive Multi-Drafter Grafting for Tree-Based Speculative Decoding

Jiaming Fan, Daming Cao, Canchen Huang, Jiale Fu+5

Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China · Southeast University · Ant Group+1

Speculative Decoding LLM Inference Acceleration LLM Inference Scheduling

Open paper AskPDF arXiv

Speculative decoding accelerates large language model inference through a draft-then-verify paradigm. Building on this, tree-structured methods improve inference by organizing proposals into multiple candidate paths, increasing the accepted length. However, existing tree-structured methods use a single drafter for all drafting steps, creating a dilemma: a smaller drafter is fast but yields lower-quality trees, whereas a larger drafter improves tree quality but suffers from high latency. To address this, we propose TreeGraft, a multi-drafter framework in which drafters of different costs jointly construct a shared draft tree. TreeGraft uses the stronger drafter to rescore candidates by updating scores assigned by the weaker drafter, reselect grafting positions, and recover promising paths left unexplored. It also integrates stronger drafter expansions non-destructively, preserving existing branches that may still be accepted by the target model. Together, these designs improve the quality of the shared draft tree. To control the drafting cost, TreeGraft introduces a lightweight scheduler distilled from an offline value system to decide when to call the stronger drafter. Across 10 model pairs and 6 benchmarks, TreeGraft outperforms the better of the two fixed single-drafter endpoint strategies by 15.1% on average, reaching a maximum gain of 26.6%. Our code is available at https://github.com/fjm9933/TreeGraft.

Abstract

Speculative Decoding

Explore similar work

TreeGraft: Adaptive Multi-Drafter Grafting for Tree-Based Speculative Decoding

SpecForge: A Flexible and Efficient Open-Source Training Framework for Speculative Decoding

DIVERSED: Relaxed Speculative Decoding via Dynamic Ensemble Verification