Speculative decoding accelerates memory-bound LLM decoding with a lightweight drafter, and parallel drafters extend this speedup by drafting a whole block in one forward pass. However, it remains unclear whether their released systems (stock) attain the full speedup that parallel drafting permits. We identify three challenges. First, with one request, each step accepts few tokens but reads every target weight. Second, with batching, per-request verification work roughly halves the speedup over autoregressive (AR) decoding. Third, speculative decoding is deployment-specific, and no universal optimization fits every serving regime, drafter, target model and GPU. To address these challenges, we present SpecPilot, an autonomous agentic loop that discovers algorithms and systems optimizations for each deployment. In each round, a cost model locates the largest gaps between the current system and its ceiling. Parallel agents explore directions and integrate the successful ones. Token forcing checks every output token against stock to detect reward hacking. A structured memory carries validated mechanisms to new rounds and deployments. SpecPilot discovered a general lossless 12-bit weight format, tree verification for hybrid recurrent targets, two-level verification and three batching mechanisms. We evaluate it across four target models, three drafters, two GPU platforms and five benchmarks. Without training or quality loss, it raises the speedup over AR from stock's 4.46× to 10.87× with one request, and from 1.94× to 3.93× with batching.
Figures & tables
Figure 1: Speedup over AR with one request. SpecPilot more than doubles stock’s speedup in each setting.
Figure 2: (a) Single-request step time split into target weights and the rest, at peak bandwidth and as measured after the first optimization round. Reading the weights dominates the step. (b) Stock speedup over AR falls as the batch size grows, for Qwen3.8-27B in FP8 and Qwen3.5-4B in BF16. (c) Task accuracy change against stock, with 95% CIs, for a lossless system and a faulty one with an injected logit bias. All four intervals include zero, so accuracy cannot flag the faulty systems.
Figure 3: One round of the SpecPilot loop. 0 The cost model measures the gap between the current system and its ceiling. 1 Directions run in parallel, one per GPU. 2 Validated pieces stack into one system. 3 A kernel sub-agent tunes the kernels furthest from their byte floors. 4 Token forcing checks the sealed final against stock at every teacher-forced token. Each round distills its mechanisms into memory entries, which carry to later rounds and new deployments.
Figure 4: Token forcing on lossless and faulty systems in sealed finals. Bars give the mismatch rate with 95% CIs, and dashed lines the rate-rule limit. Above the bars are the verdict and the count of large-margin events. Faulty systems add a logit bias of 2 to every 7th vocabulary id after 2048 tokens.
Figure 5: W12. (a) Each BF16 weight becomes three 4-bit planes: A holds the sign and the top three mantissa bits, E a 4-bit exponent offset within a 16-binade window per output channel, and B the low mantissa bits. A and E form the 8-bit view that the Q pass of two-level verification reads. The remaining weights, a fraction of 7.3×10−5 on Qwen3.8-27B, go to an exact escape list. (b) The GEMM decodes W12 records in registers and feeds them to wgmma as the register operand.
Figure 6: Best-first tree on a hybrid target. On the same lattice, stock verifies only the greedy chain, which stops at its first miss. The best-first tree keeps the N nodes with the highest path score ∑logp(child∣parent) , so another candidate can continue. On hybrid targets, each root path recomputes its GDN state. The right panel compares the path and WY verify kernels over 48 GDN layers on H100.
Figure 7: Two-level verification. A low-precision Q pass checks the draft tree and returns chain C, the path that matches the Q pass’s argmax plus a bonus token, without updating the target’s state. The drafter drafts again from the end of C, and one exact pass over the W12 weights verifies C and the new subtree. Times are from one final H100 cycle, and tokens per exact pass from TLV’s round. The right panel shows the 8-bit view nested in each W12 record, which the Q pass reads.
Table 1: Speedup over AR and τ , the tokens accepted per exact target pass, with one request on BF16 targets. SpecPilot uses two-level verification, so each exact pass verifies two chained drafts and accepts far more tokens than stock.
Autoregressive decoding is bottlenecked by its sequential nature. Speculative decoding has become a standard way to accelerate inference by using a fast draft model to predict upcoming tokens from a slower target model, and then verifying them in parallel with a single target model forward pass. However, speculative decoding itself relies on a sequential dependence between speculation and verification. We introduce speculative speculative decoding (SSD) to parallelize these operations. While a verification is ongoing, the draft model predicts likely verification outcomes and prepares speculations pre-emptively for them. If the actual verification outcome is then in the predicted set, a speculation can be returned immediately, eliminating drafting overhead entirely. We identify three key challenges presented by speculative speculative decoding, and suggest principled methods to solve each. The result is Saguaro, an optimized SSD algorithm. Our implementation is on average 30% faster than optimized speculative decoding baselines and up to 5x faster than autoregressive decoding with open source inference engines.
Tanishq Kumar, Tri Dao, Avner May
Department of Computer Science, Stanford University · Department of Computer Science, Princeton University · Together AI
Large language model (LLM)-based agent applications often incur high response time. Speculative decoding is a promising solution to improve the inference efficiency of LLM agents without impacting generation quality. However, state-of-the-art speculative decoding algorithms exhibit substantial speed degradation under large batch sizes, limiting their effectiveness to deploy in real-world agent applications. In this work, we first present a systematic analysis of speculative decoding for LLM agents and identify two dominant factors of speedup degradation: high rejection rate of speculative tokens, and under-utilization of dynamic token budgets.B ased on these observations, we propose AgentSpec, a speculative decoding algorithm that addresses the limitations of existing methods for LLM agents. AgentSpec incorporates structure-isolated drafting that constrains speculation to semantically coherent segments of the agent workflow, reducing the drafts of irrelevant semantic paths and achieving an extremely low rejection rate. Moreover, AgentSpec adopts redundancy-aware budget allocation that exploits agent-level information to better utilize the dynamically-free token budget during the agent inference. We implement and evaluate AgentSpec on five different workloads and four different models from four different LLM families in vLLM. Our results demonstrate the superiority of AgentSpec over state-of-the-arts.
Xin Wang, Ziming Miao, Yi Zhu +4
The Ohio State University · Microsoft Research · University of Michigan
Single-stream autoregressive decoding of large language models is bound by memory bandwidth: each generated token requires one full forward pass through the target model, and successive passes cannot be parallelized. Speculative decoding restructures this computation: a small draft model proposes K tokens autoregressively, the target model scores all of them in one batched pass, and a rejection-sampling rule provably preserves the target model's output distribution. We present a from-scratch, device-agnostic (CUDA/MPS/CPU) implementation and an empirical study across five draft/target backend configurations on a consumer Apple-silicon laptop. Distribution equivalence is verified at three levels, culminating in a two-sample test over roughly 9,200 real-model tokens per method (χ2=162.5, dof =200, p=0.976) and exact greedy-sequence agreement. The best configuration reaches a measured 1.61× wall-clock speedup at K=6, on an acceptance profile declining from 69.7% at K=1 to 37.8% at the optimum, while three of five configurations decelerate, either because the draft fails to out-speed a small target or because the quantized Metal backend executes "parallel" verification serially, an effect we isolate and quantify. The failures are as instructive as the successes: speculative decoding pays off only when verification is genuinely batch-parallel and the draft/target latency gap is real.