cs.AINov 3, 2026

Hybrid Spark: Adaptive Speculative Decoding That Lets the Content Choose the Drafter

Authors: Ekagra Ranjan, Bharat Venkitesh

Organizations: Cohere

Abstract

Speculative decoding (SD) speeds up large language model (LLM) serving, but its gain depends on the workload's shape, such as batch size and context length, and on its content. A model-based drafter proposes a short block on any content, while a model-free drafter such as n-gram lookup proposes long copies at almost no cost when the output repeats its input, as in code editing. Adaptive verification spends a batch-wide verification budget on the draft tokens most likely to be accepted, but it supports only one model-based drafter. Model-free drafters provide no per-token confidence, so they cannot take part in budget allocation, and deployments must either fix one drafter or split their GPUs between drafters. We present \sys, a framework that runs a model-based and a model-free drafter on every step and gives each request the better of their chains, within adaptive verification and entirely on the GPU, which keeps asynchronous scheduling intact. The key idea is a price per verified draft token: the goodput that slot would earn elsewhere in the batch. Once the model-free drafter's tokens carry calibrated acceptance estimates, fit once on general text, this price puts both drafters' chains on one scale. Instantiated with \dspark and an n-gram drafter in vLLM, \sys achieves 1.10--1.62\x the output throughput of \dspark on code editing at batch size4 and 1.04--1.27\x at batch size128, while staying at 0.97--1.01\x of it on chat at temperature~0. A single deployment can therefore serve both workloads.

Figures & tables

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation

    Jul 6, 2026Xin Cheng, Xingkai Yu, Chenze Shao +30LLM ServingSpeculative Decoding

  2. An Interpretable Latency Model for Speculative Decoding in LLM Serving

    May 14, 2026Linghao Kong, Megan Flynn, Michael Peng +3LLM ServingSpeculative Decoding