Speculative decoding accelerates memory-bound LLM decoding with a lightweight drafter, and parallel drafters extend this speedup by drafting a whole block in one forward pass. However, it remains unclear whether their released systems (stock) attain the full speedup that parallel drafting permits. We identify three challenges. First, with one request, each step accepts few tokens but reads every target weight. Second, with batching, per-request verification work roughly halves the speedup over autoregressive (AR) decoding. Third, speculative decoding is deployment-specific, and no universal optimization fits every serving regime, drafter, target model and GPU. To address these challenges, we present SpecPilot, an autonomous agentic loop that discovers algorithms and systems optimizations for each deployment. In each round, a cost model locates the largest gaps between the current system and its ceiling. Parallel agents explore directions and integrate the successful ones. Token forcing checks every output token against stock to detect reward hacking. A structured memory carries validated mechanisms to new rounds and deployments. SpecPilot discovered a general lossless 12-bit weight format, tree verification for hybrid recurrent targets, two-level verification and three batching mechanisms. We evaluate it across four target models, three drafters, two GPU platforms and five benchmarks. Without training or quality loss, it raises the speedup over AR from stock's 4.46× to 10.87× with one request, and from 1.94× to 3.93× with batching.
Figures & tables
Figure 1: Speedup over AR with one request. SpecPilot more than doubles stock’s speedup in each setting.
Figure 2: (a) Single-request step time split into target weights and the rest, at peak bandwidth and as measured after the first optimization round. Reading the weights dominates the step. (b) Stock speedup over AR falls as the batch size grows, for Qwen3.8-27B in FP8 and Qwen3.5-4B in BF16. (c) Task accuracy change against stock, with 95% CIs, for a lossless system and a faulty one with an injected logit bias. All four intervals include zero, so accuracy cannot flag the faulty systems.
Figure 3: One round of the SpecPilot loop. 0 The cost model measures the gap between the current system and its ceiling. 1 Directions run in parallel, one per GPU. 2 Validated pieces stack into one system. 3 A kernel sub-agent tunes the kernels furthest from their byte floors. 4 Token forcing checks the sealed final against stock at every teacher-forced token. Each round distills its mechanisms into memory entries, which carry to later rounds and new deployments.
Figure 4: Token forcing on lossless and faulty systems in sealed finals. Bars give the mismatch rate with 95% CIs, and dashed lines the rate-rule limit. Above the bars are the verdict and the count of large-margin events. Faulty systems add a logit bias of 2 to every 7th vocabulary id after 2048 tokens.
Figure 5: W12. (a) Each BF16 weight becomes three 4-bit planes: A holds the sign and the top three mantissa bits, E a 4-bit exponent offset within a 16-binade window per output channel, and B the low mantissa bits. A and E form the 8-bit view that the Q pass of two-level verification reads. The remaining weights, a fraction of 7.3×10−5 on Qwen3.8-27B, go to an exact escape list. (b) The GEMM decodes W12 records in registers and feeds them to wgmma as the register operand.
Figure 6: Best-first tree on a hybrid target. On the same lattice, stock verifies only the greedy chain, which stops at its first miss. The best-first tree keeps the N nodes with the highest path score ∑logp(child∣parent) , so another candidate can continue. On hybrid targets, each root path recomputes its GDN state. The right panel compares the path and WY verify kernels over 48 GDN layers on H100.
Figure 7: Two-level verification. A low-precision Q pass checks the draft tree and returns chain C, the path that matches the Q pass’s argmax plus a bonus token, without updating the target’s state. The drafter drafts again from the end of C, and one exact pass over the W12 weights verifies C and the new subtree. Times are from one final H100 cycle, and tokens per exact pass from TLV’s round. The right panel shows the 8-bit view nested in each W12 record, which the Q pass reads.
Table 1: Speedup over AR and τ , the tokens accepted per exact target pass, with one request on BF16 targets. SpecPilot uses two-level verification, so each exact pass verifies two chained drafts and accepts far more tokens than stock.