Speculative Speculative Decoding

Tanishq Kumar; Tri Dao; Avner May

cs.LGMar 3, 2026

Speculative Speculative Decoding

Authors: Tanishq Kumar, Tri Dao, Avner May

Organizations: Department of Computer Science, Stanford University · Department of Computer Science, Princeton University · Together AI

Abstract

Autoregressive decoding is bottlenecked by its sequential nature. Speculative decoding has become a standard way to accelerate inference by using a fast draft model to predict upcoming tokens from a slower target model, and then verifying them in parallel with a single target model forward pass. However, speculative decoding itself relies on a sequential dependence between speculation and verification. We introduce speculative speculative decoding (SSD) to parallelize these operations. While a verification is ongoing, the draft model predicts likely verification outcomes and prepares speculations pre-emptively for them. If the actual verification outcome is then in the predicted set, a speculation can be returned immediately, eliminating drafting overhead entirely. We identify three key challenges presented by speculative speculative decoding, and suggest principled methods to solve each. The result is Saguaro, an optimized SSD algorithm. Our implementation is on average 30% faster than optimized speculative decoding baselines and up to 5x faster than autoregressive decoding with open source inference engines.

Figures & tables

Figure 1: (Left) Ordinary speculative decoding (SD) requires the verifier to wait idly for the draft to speculate. (Center) In our algorithm, speculation runs on a separate device (1

\times

H100) in parallel with verification; the draft precomputes speculations for many possible verification outcomes and returns the speculated tokens immediately if one occurs. (Right) End-to-end performance of SSD, SD, and autoregressive (AR) decoding averaged over four datasets spanning math, code and chat, for Llama-3.1-70B on TP=4 H100s, batch size 1, greedy decoding, Llama-3.2-1B draft model.

Figure 2: Schematic of speculation cache strategy. We allocate fan-out

F_{k}

(bonus token guesses) over sequence length

K+1

according to Theorem 12 .

Figure 3: Scaling of cache hit rates with fan out. (Left, middle) The rejection rates (1 -

p_{hit,*}(F)

) conditioned on the prior speculation coming from the primary vs backup speculator, respectively. Rejection rates (cache misses) fall as a power law in the draft-fan out, demonstrating that cache hit rates increase with cache size. (Right) The overall cache hit rate

p_{\text{hit}}(F)

Figure 4: Advantage of geometric fan out strategy increases at higher temperatures, improving both speculation cache hit rate (right) and thus end-to-end speed (left). Results averaged over four datasets. At all temperatures, SSD with either fan out strategy outperforms ordinary speculative decoding.

Figure 5: We introduce Saguaro sampling, a novel sampling scheme designed specifically for SSD. (Left) It interpolates between high cache hit rate and high speculative acceptance rate. (Right) Illustrative schematic for how Saguaro sampling increases residual probability mass on the top draft tokens, encouraging the sampled bonus token to lie in the speculation cache by construction.

Figure 6: (Left) Optimal backup speculator (fast vs neural) depends on batch size, as Theorem 17 predicts (temperature 0). (Right) We forecast how decoding speed improves with draft compute (and thus, fan-out) for batch size 16 with the fast backup speculator. At large batch sizes we become compute bound, speedups more GPUs to drafting.

Figure 7: (Left) End-to-end decoding speed comparison of SSD compared to SD and standard autoregressive decoding on Llama-3.1-Instruct 70B across four datasets. (Right) SSD improves the throughput-latency Pareto frontier.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Custom attention mask for multi-query decoding of all $BF({K+1})$ verification branches in parallel. This mask is for $B=1$ , $K=3$ , uniform fan-out $F=4$ , at depth $i=2$ . Black indicates tokens that can be attended to. The left block shows attention to the verified prefix; diagonal bands show each branch attending only within its forking path.

Model	Dataset	AR tok/s	SD tok/s	SSD tok/s	SSD/SD	SSD/AR
Llama-3.1-Instruct
70B/1B	HumanEval	54.7	176	283	1.60 $\times$	5.17 $\times$
	Ultrafeedback	54.7	138	215	1.55 $\times$	3.93 $\times$
	Alpaca	54.7	145	224	1.55 $\times$	4.10 $\times$
	GSM8k	54.7	188	301	1.60 $\times$	5.50 $\times$
	Average	54.7	161.8	255.8	1.58 $\times$	4.68 $\times$

Appendix

Table 9

SGLang EAGLE-3 ( steps $\times$ topk )
steps $\backslash$ topk	1	2	3	4	5
2	122.8	124.7	127.3	$\dagger$	$\dagger$
4	157.1	157.8	164.9	$\dagger$	$\dagger$
6	171.3	178.9	183.0	$\dagger$	$\dagger$
8	172.4	183.9	192.8	190.2	$\dagger$
10	168.7	183.0	192.5	$\dagger$	186.9

Appendix

Table 1: SGLang 0.5.9 Llama-3.1-70B/1B baseline sweeps (decode tok/s).

Method	$K=2$	$K=4$	$K=6$	$K=8$	$K=10$	$K=12$
vLLM EAGLE-3	143.7	181.4	197.6	197.0	191.8	187.9
vLLM Standalone	136.0	117.9	100.9	89.1	78.5	70.5

Appendix

Table 2: vLLM 0.16.0 Llama-3.1-70B/1B baseline sweeps over speculative lookahead

K

(decode tok/s).

Figure 9: Comparison of activation conditioning in EAGLE-3 vs. SSD-EAGLE-3. In standard EAGLE-3 (bottom), the round

T

draft conditions almost entirely on target activations extracted from the already-verified round

T{-}1

. In SSD-EAGLE-3 (top), round

T

speculation begins before round

T{-}1

verification completes, so the draft must substitute its own activations (light nodes) for the not-yet-available target activations. Conditioning on more draft activations can degrade quality of EAGLE speculations if the draft model is not trained to use draft self-conditioning effectively.

Figure 10: Rejection rate and cache hit rate scaling for Qwen-3 model family. Like the Llama-3 model family, we see it is an approximate power law in the fan-out, so that cache hit rates increase steadily as we grow

F

, the size of our cache.

Figure 11: End-to-end decoding speed comparison of SSD compared to SD and standard autoregressive decoding on Qwen-3 32B across four datasets.

Figure 12: End-to-end speed for Qwen-3 model family compared to synchronous baselines.

Figure 13: Batch size scaling of Qwen-3 models compared to synchronous baselines.

Speculative Decoding

Speculative decoding accelerates generation by drafting multiple tokens and verifying them in parallel with a larger target model.

See all 291 papers

DIVERSED: Relaxed Speculative Decoding via Dynamic Ensemble Verification
Apr 8, 2026Ziyi Wang, Siva Rajesh Kasa, Ankith M S +6Speculative Decoding LLM Inference Acceleration
Purdue University · Amazon Inc. · University of Illinois Urbana-Champaign +1
LibraSpec: Dynamic Diffusion-Based Speculative Decoding via Marginal-Gain-Driven Optimization
Aug 9, 2026Zexun Lin, Yuan Feng, Junlin Lv +2Inference-Time Optimization Speculative Decoding for Diffusion Language Models
Suzhou Institute for Advanced Research, University of Science and Technology of China Suzhou, Jiangsu, China
Approximate Speculative Decoding
Aug 4, 2026Yuannuo Feng, Zegang Peng, Yuxin Xie +5Speculative Decoding LLM Inference Acceleration
School of Integrated Circuit Science and Engineering, Beihang University, Beijing, China · Department of Precision Instrument, Tsinghua University, Beijing, China · Faculty of Engineering, The University of Hong Kong, Hong Kong SAR, China +2

Apr 8, 2026J/K move · Enter open · S save

DIVERSED: Relaxed Speculative Decoding via Dynamic Ensemble Verification

Ziyi Wang, Siva Rajesh Kasa, Ankith M S, Santhosh Kumar Kasa+5

Purdue University · Amazon Inc. · University of Illinois Urbana-Champaign+1

Speculative Decoding LLM Inference Acceleration Parallel Speculative Decoding

Open paper AskPDF arXiv

Speculative decoding is an effective technique for accelerating large language model inference by drafting multiple tokens in parallel. In practice, its speedup is often bottlenecked by a rigid verification step that strictly enforces the accepted token distribution to exactly match the target model. This constraint leads to the rejection of many plausible tokens, lowering the acceptance rate and limiting overall time speedup. To overcome this limitation, we propose Dynamic Verification Relaxed Speculative Decoding (DIVERSED), a relaxed verification framework that improves time efficiency while preserving generation quality. DIVERSED learns an ensemble-based verifier that blends the draft and target model distributions with a task-dependent and context-dependent weight. We provide theoretical justification for our approach and demonstrate empirically that DIVERSED achieves substantially higher inference efficiency compared to standard speculative decoding methods. Code is available at: https://github.com/comeusr/diversed.

Abstract

Speculative Decoding

Explore similar work

DIVERSED: Relaxed Speculative Decoding via Dynamic Ensemble Verification

LibraSpec: Dynamic Diffusion-Based Speculative Decoding via Marginal-Gain-Driven Optimization

Approximate Speculative Decoding