Goose: Anisotropic Speculation Trees for Training-Free Speculative Decoding

Tao Jin; Phuong Minh Nguyen; Naoya Inoue

cs.CLApr 2, 2026

Goose: Anisotropic Speculation Trees for Training-Free Speculative Decoding

Authors: Tao Jin, Phuong Minh Nguyen, Naoya Inoue

Organizations: Japan Advanced Institute of Science and Technology (JAIST)

Abstract

Speculative decoding accelerates large language model inference by drafting multiple candidate tokens and verifying them in a single forward pass. Candidates are organized as a tree: deeper trees accept more tokens per step, but adding depth requires sacrificing breadth (fallback options) under a fixed verification budget. Existing training-free methods draft from a single token source and shape their trees without distinguishing candidate quality across origins. We observe that two common training-free token sources -- n-gram matches copied from the input context, and statistical predictions from prior forward passes -- differ sharply in acceptance rate (~6x median gap, range 2-18x across five models and five benchmarks). We prove that when such a quality gap exists, the optimal tree is anisotropic (asymmetric): reliable tokens should form a deep chain while unreliable tokens spread as wide branches, raising the depth ceiling of balanced trees. We realize this structure in GOOSE, a training-free framework that builds an adaptive spine tree: a deep chain of high-acceptance context-matched tokens with wide branches of low-acceptance alternatives at each node. The resulting tree provably accepts at least as many tokens per step as either source alone. On five LLMs (7B-33B) and five benchmarks, GOOSE achieves 1.9-4.3x lossless speedup, outperforming balanced-tree baselines by 12-33% under the same budget.

Figures & tables

Figure 3: Goose pipeline overview. From anchor . , Stage 1 draws from a unified candidate pool : context matching produces the spine ( blue ), the adjacency table supplies branches ( orange ). Stage 2 verifies all candidates via one LLM forward pass. Stage 3 selects the longest accepted path via a greedy walk, discovering spine continuation : “ 2 ” $\,\to\,$ “ ) ” extends beyond the spine mismatch, yielding 8 tokens per call.

Figure 4: Wall-clock speedup over autoregressive (AR) decoding across five models and five benchmarks. All methods are lossless (greedy decoding, identical output); Goose values are annotated above each group and the sixth panel is the per-model mean. SuffixDecoding is the strongest training-free system reported in prior work, EAGLE-2 the strongest trained one shown (absent on Qwen3-8B, which has no released head). Per-benchmark

\tau

and Sequoia: Table 3 ; EAGLE-3: Section D.4 . † Vicuna-33B runs on 2

\times

A100-40GB; all others on a single A40.

Table 4

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

		ClassEval		GSM8K		HumanEval		MBPP		MT-Bench		Avg.
Model	Method	Spd	$\tau$	Spd	$\tau$	Spd	$\tau$	Spd	$\tau$	Spd	$\tau$	Spd
Vicuna-7B	AR	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00
	PLD	2.34	2.50	1.43	1.46	1.74	1.78	1.59	1.62	1.47	1.48	1.71
	TR	2.58	3.29	2.46	2.96	2.53	3.06	2.41	2.86	2.19	2.60	2.43
	Lookahead	2.27	2.41	1.63	1.67	1.72	1.77	1.75	1.78	1.58	1.60	1.79
	REST	1.68	1.73	1.17	1.19	1.16	1.17	1.05	1.05	1.08	1.08	1.23

Appendix

Table 3: Full main results. Speedup (Spd): wall-clock tok/s ratio vs. AR measured in the same run.

\tau

: compression ratio (tokens per target-model forward; drafter forwards are not counted, which favors the trained rows). Bold : best per column. All methods produce identical output (lossless). Shaded rows denote methods requiring per-model training. Sequoia appears on the two targets with released modern drafters, at

N{=}60

to match our budget

B

; its

\tau

follows Sequoia’s official convention (prefill call excluded), at most

1.7\%

above the table’s convention. EAGLE-2 is absent on Qwen3-8B, which has no released head. † Vicuna-33B evaluated on NVIDIA A100 (2

\times

40GB); all other models on A40. The Vicuna-33B GSM8K EAGLE-2 cell comes from a separate run of that environment, so its Spd is taken against that run’s own AR;

\tau

is unaffected.

Benchmark	Domain	Samples	Max tokens
HumanEval	Code	164	512
MBPP	Code	500	512
ClassEval	Code	100	512
GSM8K	Math	1,319	1,024
MT-Bench	Dialogue	80	1,024

Appendix

Table 4: Benchmark details.

Table 7

Figure 5: (a) Goose ’s mean

\tau

decomposed into the best standalone baseline

\max(\tau_{\textsc{PLD}{}},\tau_{\textsc{TR}{}})

(gray) and the synergy gain from the spine-tree topology (green); the gain ranges from

+16\%

(Vicuna-13B) to

+45\%

(Llama-3-8B). (b) Hyperparameter sensitivity (Qwen3-8B): each panel sweeps one parameter;

\tau

plateaus at

B{\geq}60

and

D{\geq}6

, and varies by fewer than 0.6

\tau

units across

r

and

\rho

. Panels (c)/(d) sweep the spine ratio

r

and the spine-branch ratio

\rho

of Section 4.1 .

Panel of Figure 6 — Figure 6: (a) $\tau$ across four A40 models and five datasets grouped by Goose ’s affinity. CV below each column shows cross-model coefficient of variation. (b) Per-model mean $\tau_{\text{eq5}}/\tau_{\text{iso}}$ across 24 settings; all bars exceed 1.0, confirming spine dominance (Pearson $r{=}0.91$ ).

	Goose		EAGLE-3
Dataset	$\tau$	tok/s	$\tau$	tok/s
HumanEval	2.59	66.6	5.79	109.0
ClassEval	5.35	131.3	5.34	94.5
MT-Bench	2.22	57.0	5.36	102.0
MBPP	2.36	61.4	5.69	108.7
GSM8K	2.76	70.3	5.69	107.8

Appendix

Table 7: Goose vs. EAGLE-3 under EAGLE’s native chat-template protocol (Llama-3.1-8B, A40), the primary protocol for this baseline ( Section 5.2 ).

\tau

/ tok/s.

	Goose		EAGLE-2		EAGLE-3
Dataset	$\tau$	tok/s	$\tau$	tok/s	$\tau$	tok/s
HumanEval	4.50	114.8	4.14	62.2	3.29	62.6
ClassEval	5.54	133.4	4.11	58.9	4.18	74.8
MT-Bench	3.63	90.7	3.51	52.2	3.87	72.6
MBPP	3.51	91.0	4.19	63.6	3.55	68.5
GSM8K	8.90	221.7	4.28	63.6	4.62	86.2

Appendix

Table 8: Raw-prompt protocol (our paper’s protocol, applied to every baseline): Llama-3.1-8B, A40, full

n

\tau

/ tok/s.

Model	ClassEval	MT-Bench	HumanEval	GSM8K	MBPP
Qwen3-8B	70.8	53.3	33.0	48.3	38.3
Llama-3-8B	62.1	53.5	53.4	83.8	45.2
Llama-3.2-1B	61.4	35.1	60.7	37.9	36.7
TinyLlama-1.1B	66.1	84.5	33.2	59.8	33.4

Appendix

Table 9: PLD share of accepted tokens (%); TR supplies the remainder. Llama-3-8B and Qwen3-8B are paper models (full

n

); the two

{\sim}1

B models are robustness checks (

n{=}30

$\tau$ : PLD $\to$ Suffix spine	HumanEval	MBPP	GSM8K	MT-Bench	ClassEval
Llama-3.1-8B (full $n$ )	4.21 $\to$ 4.80	3.30 $\to$ 3.63	6.28 $\to$ 9.26	3.22 $\to$ 3.91	5.00 $\to$ 5.89
Llama-3.2-1B ( $n{=}30$ )	3.75 $\to$ 4.33	3.04 $\to$ 3.27	3.38 $\to$ 3.60	2.55 $\to$ 2.63	4.29 $\to$ 5.00
TinyLlama-1.1B ( $n{=}30$ )	2.84 $\to$ 3.08	2.88 $\to$ 3.09	4.56 $\to$ 5.41	6.19 $\to$ 8.92	5.03 $\to$ 5.74

Appendix

Table 10:

\tau

with the PLD spine swapped for SuffixDecoding’s suffix tree (TR branches and topology unchanged, bypass off on both sides).

GPU / task	Method	bs=1	bs=4	bs=8	bs=16	bs=32
A40, HumanEval	Goose	59.0	129.8	141.5	149.4	143.0
	AR	31.0	96.5	168.2	265.0	366.1
A40, GSM8K	Goose	62.4	138.5	155.7	168.7	156.4
	AR	31.1	79.0	142.1	244.9	382.2
RTX 3090, HumanEval	Goose	73.6	132.0	131.1	124.1	—
	AR	42.4	136.1	228.3	339.0	—

Appendix

Table 11: Aggregate tok/s (total tokens / wall-clock) vs. batch size, Llama-3-8B HumanEval (

n{=}164

) and GSM8K (

n{=}256

), greedy, fp16.

	HumanEval	MBPP	MT-Bench	ClassEval	GSM8K ∗
Goose	4.50 / 114.8	3.51 / 91.0	3.63 / 90.7	5.54 / 133.4	8.90 / 221.7
SuffixDecoding (tree)	2.98/86.6	2.14/63.5	2.33/68.0	3.97/108.9	5.59/162.0

Appendix

Table 12: Goose vs. SuffixDecoding (tree mode, the stronger of its two training-free modes) on Llama-3.1-8B (A40, greedy, full

n

\tau

/ tok/s. ∗ GSM8K fixed-budget.

Model	Dataset	Goose	Suf. (cold)	Suf. (warm)	Merged (warm)
Llama-3-8B	ClassEval	4.92	3.53	3.66	3.55
	GSM8K	5.90	3.38	3.79	3.68
	HumanEval	4.13	2.76	2.94	2.88
	MBPP	3.58	2.25	2.54	2.54
	MT-Bench	2.71	1.63	1.73	1.71
Llama-3.1-8B	ClassEval	5.50	3.97	4.06	4.05

Appendix

Table 13:

\tau

on the full grid: Goose (cold) vs. SuffixDecoding official (cold/warm) and the merged-tree variant (warm). Bold = column best per row.

Dataset	$\tau$	tok/s	$\tau$	tok/s	Sequoia paper $\tau$
	Goose		Sequoia (official)
C4	4.16	116.9	4.79	107.8	5.08
CNN/DM	3.04	86.6	3.91	91.1	4.05
OpenWebText	3.28	93.4	3.95	91.9	3.86

Appendix

Table 14: Goose vs. official Sequoia on Llama-2-7B (A40, greedy,

n{=}50

; Section 5.2 ). Sequoia uses its released trained 68M drafter; Goose is training-free. Wall-clock margins carry a

{\sim}2

-point protocol uncertainty from the generation-length asymmetry described above.

		$N{=}60$		$N{=}128$
Target	Dataset	$\tau$	Spd	$\tau$	Spd
Llama-3-8B	ClassEval	8.63	1.52	8.95	1.52
	GSM8K	8.89	1.56	9.12	1.54
	HumanEval	8.56	1.53	8.88	1.51
	MBPP	8.42	1.48	8.75	1.48
	MT-Bench	7.57	1.36	8.00	1.38

Appendix

Table 15: Modern-stack Sequoia:

\tau

and wall-clock speedup vs. AR at tree sizes

N{=}60/128

(

N

= node count in Sequoia’s notation;

N{=}60

matches our budget

B

; A40, greedy, full

n

). Drafters: Llama-3.2-1B for Llama targets, Qwen3-0.6B for Qwen3-8B.

Model	Dataset	Heuristic	Allocator	Allocator (inst.)
Llama-3-8B	ClassEval	4.80	5.39	5.24
	GSM8K	8.84	7.48	7.25
	HumanEval	4.03	4.47	4.50
	MBPP	3.58	3.92	3.98
	MT-Bench	3.41	3.56	3.47
Llama-3.1-8B	ClassEval	5.46	5.95	5.81

Appendix

Table 16: Equal-budget allocation comparison,

\tau

(full

n

except GSM8K at

n{=}500

). Bold = row best. The two GSM8K/8B-Llama regressions are diagnosed above.

Speculative Decoding

Speculative decoding accelerates generation by drafting multiple tokens and verifying them in parallel with a larger target model.

See all 309 papers

May 28, 2026

TreeGraft: Adaptive Multi-Drafter Grafting for Tree-Based Speculative Decoding

Speculative decoding accelerates large language model inference through a draft-then-verify paradigm. Building on this, tree-structured methods improve inference by organizing proposals into multiple candidate paths, increasing the accepted length. However, existing tree-structured methods use a single drafter for all drafting steps, creating a dilemma: a smaller drafter is fast but yields lower-quality trees, whereas a larger drafter improves tree quality but suffers from high latency. To address this, we propose TreeGraft, a multi-drafter framework in which drafters of different costs jointly construct a shared draft tree. TreeGraft uses the stronger drafter to rescore candidates by updating scores assigned by the weaker drafter, reselect grafting positions, and recover promising paths left unexplored. It also integrates stronger drafter expansions non-destructively, preserving existing branches that may still be accepted by the target model. Together, these designs improve the quality of the shared draft tree. To control the drafting cost, TreeGraft introduces a lightweight scheduler distilled from an offline value system to decide when to call the stronger drafter. Across 10 model pairs and 6 benchmarks, TreeGraft outperforms the better of the two fixed single-drafter endpoint strategies by 15.1% on average, reaching a maximum gain of 26.6%. Our code is available at https://github.com/fjm9933/TreeGraft.

Jiaming Fan, Daming Cao, Canchen Huang +6

Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China · Southeast University · Ant Group +1

Speculative Decoding LLM Inference Acceleration LLM Inference Scheduling

Ask PDF

Mar 19, 2026

SpecForge: A Flexible and Efficient Open-Source Training Framework for Speculative Decoding

Large language models incur high inference latency due to sequential autoregressive decoding. Speculative decoding alleviates this bottleneck by using a lightweight draft model to propose multiple tokens for batched verification. However, its adoption has been limited by the lack of high-quality draft models and scalable training infrastructure. We introduce SpecForge, an open-source, production-oriented framework for training speculative decoding models with full support for EAGLE-3. SpecForge incorporates target-draft decoupling, hybrid parallelism, optimized training kernels, and integration with production-grade inference engines, enabling up to 9.9x faster EAGLE-3 training for Qwen3-235B-A22B. In addition, we release SpecBundle, a suite of production-grade EAGLE-3 draft models trained with SpecForge for mainstream open-source LLMs. Through a systematic study of speculative decoding training recipes, SpecBundle addresses the scarcity of high-quality drafts in the community, and our draft models achieve up to 4.48x end-to-end inference speedup on SGLang, establishing SpecForge as a practical foundation for real-world speculative decoding deployment.

Shenggui Li, Chao Wang, Yikai Zhu +14

Nanyang Technological University · SpecForge Team · Meituan +2

Speculative Decoding LLM Inference Acceleration Efficient Language Model Training

Ask PDF

Apr 8, 2026

DIVERSED: Relaxed Speculative Decoding via Dynamic Ensemble Verification

Speculative decoding is an effective technique for accelerating large language model inference by drafting multiple tokens in parallel. In practice, its speedup is often bottlenecked by a rigid verification step that strictly enforces the accepted token distribution to exactly match the target model. This constraint leads to the rejection of many plausible tokens, lowering the acceptance rate and limiting overall time speedup. To overcome this limitation, we propose Dynamic Verification Relaxed Speculative Decoding (DIVERSED), a relaxed verification framework that improves time efficiency while preserving generation quality. DIVERSED learns an ensemble-based verifier that blends the draft and target model distributions with a task-dependent and context-dependent weight. We provide theoretical justification for our approach and demonstrate empirically that DIVERSED achieves substantially higher inference efficiency compared to standard speculative decoding methods. Code is available at: https://github.com/comeusr/diversed.

Ziyi Wang, Siva Rajesh Kasa, Ankith M S +6

Purdue University · Amazon Inc. · University of Illinois Urbana-Champaign +1

Speculative Decoding LLM Inference Acceleration Parallel Speculative Decoding

Ask PDF

Abstract

Speculative Decoding

Explore similar work

TreeGraft: Adaptive Multi-Drafter Grafting for Tree-Based Speculative Decoding

SpecForge: A Flexible and Efficient Open-Source Training Framework for Speculative Decoding

DIVERSED: Relaxed Speculative Decoding via Dynamic Ensemble Verification