Hybrid Spark: Adaptive Speculative Decoding That Lets the Content Choose the Drafter
Organizations: Cohere
Abstract
Speculative decoding (SD) speeds up large language model (LLM) serving, but its gain depends on the workload's shape, such as batch size and context length, and on its content. A model-based drafter proposes a short block on any content, while a model-free drafter such as n-gram lookup proposes long copies at almost no cost when the output repeats its input, as in code editing. Adaptive verification spends a batch-wide verification budget on the draft tokens most likely to be accepted, but it supports only one model-based drafter. Model-free drafters provide no per-token confidence, so they cannot take part in budget allocation, and deployments must either fix one drafter or split their GPUs between drafters. We present \sys, a framework that runs a model-based and a model-free drafter on every step and gives each request the better of their chains, within adaptive verification and entirely on the GPU, which keeps asynchronous scheduling intact. The key idea is a price per verified draft token: the goodput that slot would earn elsewhere in the batch. Once the model-free drafter's tokens carry calibrated acceptance estimates, fit once on general text, this price puts both drafters' chains on one scale. Instantiated with \dspark and an n-gram drafter in vLLM, \sys achieves 1.10--1.62\x the output throughput of \dspark on code editing at batch size4 and 1.04--1.27\x at batch size128, while staying at 0.97--1.01\x of it on chat at temperature~0. A single deployment can therefore serve both workloads.
Figures & tables
| NextCoder | MT-Bench | ||||||
|---|---|---|---|---|---|---|---|
| 15–20% | 40–45% | ||||||
| BS | Policy | T=0 | T=1 | T=0 | T=1 | T=0 | T=1 |
| 4 | Proposed | 1.30 | 1.28 | 1.13 | 1.13 | 0.99 | 0.98 |
| N-gram-first | 1.27 | 1.23 | 1.07 | 1.06 | 0.98 | 1.01 | |
| Extension | 1.27 | 1.22 | 1.13 | 1.10 | 1.00 ∘ | 1.02 | |
| DSpark (prob.) | 0.98 | 0.97 | 1.03 | ||||
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
| Symbol | Meaning |
|---|---|
| batch size (number of running requests) | |
| , | DSpark chain length (7); maximum n-gram chain length (15) |
| , , | n-gram match length (5–15); runner-up agreement (0/1); copy length (capped at 256) |
| , | conditional acceptance of position : DSpark’s confidence; Hybrid Spark’s n-gram estimate |
| survival, of the conditional acceptances | |
| , | verification budget (slots per step); budget chosen by AV |
| Target model | DSpark drafter |
|---|---|
| Qwen/Qwen3-4B | deepseek-ai/dspark_qwen3_4b_block7 |
| Qwen/Qwen3-8B | deepseek-ai/dspark_qwen3_8b_block7 |
| google/gemma-4-12B-it | deepseek-ai/dspark_gemma4_12b_block7 † |
| CohereLabs/North-Mini-Code-1.0-w4a16 | sdougbrown/North-Mini-Code-1.0-dspark |
| meta-llama/Llama-3.1-8B-Instruct | weifanjiang/llama3.1-8b-it.speculators. dspark-ce01tv09-bs8-conf-swa |
| Workload | Size | Prompts per run | Max out |
|---|---|---|---|
| Blazedit 5k / 10k | 21 / 19 | 64 / 256 / 512 | 4,096 |
| CanItEdit | 30 | 64 / 256 / 512 | per prompt |
| NextCoder | 4,000 | 14 buckets, 0–70% | 4,096 |
| EditBench | 520 | 5 buckets, 0–25% | 4,096 |
| InstructCoder | 3,200 | 200 / 800 / 3,200 | 200 |
| MT-Bench | 80 | 160 / 640 / 1,280 | 256 |
| Qwen3-4B | 0.220 | 0.155 | 0.506 | 0.683 | 0.788 | 0.697 | 0.766 | 0.721 | 0.631 | 0.658 | 0.761 | 0.803 | 0.762 | 0.862 | 0.811 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3-8B | 0.414 | 0.006 | 0.246 | 0.457 | 0.551 | 0.372 | 0.613 | 0.496 | 0.501 | 0.448 | 0.516 | 0.573 | 0.554 | 0.479 | 0.744 |
| Gemma 4 12B | 0.630 | 0.227 | 0.087 | 0.086 | 0.456 | 0.263 | 0.340 | 0.541 | 0.401 | 0.443 | 0.566 | 0.678 | 0.759 | 0.779 | 0.932 |
| North-Mini-Code | 0.426 | 0.204 | 0.045 | 0.044 | 0.175 | 0.084 | 0.051 | 0.137 | 0.417 | 0.586 | 0.582 | 0.581 | 0.478 | 0.308 | 0.154 |
| Llama-3.1-8B | 0.070 | 0.287 | 0.544 | 0.726 | 0.950 | 0.958 | 0.897 | 0.420 | 0.607 | 0.485 | 0.602 | 0.537 | 0.611 | 0.362 | 0.762 |
| Terms | 15–20% | 40–45% | Pooled [95% CI] | |
|---|---|---|---|---|
| 1 | 0.973 | 0.973 | 0.973 [0.966, 0.980] | |
| 2 | 0.988 | 0.971 | 0.978 [0.971, 0.985] | |
| 3 | , | 0.976 | 0.990 | 0.984 [0.978, 0.990] |
| 4 | , | 0.999 | 0.993 | 0.996 [0.987, 1.003] |
| 5 | , , | 0.995 | 0.998 | 0.996 [0.991, 1.001] |
| 6 | all ( Equation 1 ) | 1 | 1 | 1 |
| 1 | 2.117 at every position | – | – | – | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2 | 1.158 | 1.702 | 2.104 | 2.325 | 2.466 | 2.420 | 2.536 | 2.544 | 2.515 | 2.605 | 2.757 | 2.854 | 2.877 | 3.036 | 3.046 | – | – | – |
| 3 | 0.793 | 1.293 | 1.658 | 1.845 | 1.962 | 1.882 | 1.960 | 1.923 | 1.841 | 1.876 | 1.981 | 2.025 | 1.992 | 2.098 | 2.053 | 0.196 | – | – |
| 4 | 0.860 | 1.345 | 1.700 | 1.880 | 1.987 | 1.897 | 1.967 | 1.922 | 1.834 | 1.859 | 1.956 | 1.994 | 1.950 | 2.045 | 1.988 | – | 2.070 | – |
| 5 | 0.247 | 0.144 | 0.495 | 0.671 | 0.776 | 0.685 | 0.754 | 0.709 | 0.618 | 0.646 | 0.749 | 0.791 | 0.750 | 0.850 | 0.799 | – | 2.014 | 1.261 |
| Chain | slots | their | ||
|---|---|---|---|---|
| n-gram | 5.87 | 9 | 4.35 | 1.57 |
| DSpark | 5.58 | 7 | 5.58 | 3.42 |
| Target | Drafting | Proposed | N-gram-first | Extension |
|---|---|---|---|---|
| any | yes | yes | yes | |
| greedy | yes | yes | yes | |
| probabilistic | no | yes | yes |
| Speedup | AL, DSpark/ Hybrid Spark | ||||||
| Model | Workload | BS 4 | BS 32 | BS 128 | BS 4 | BS 32 | BS 128 |
| Llama-3.1-8B | Blazedit 5k | 1.97 | 1.71 | 1.41 | 2.4 / 5.1 | 2.4 / 5.0 | 2.4 / 4.6 |
| Blazedit 10k | 2.96 | 2.32 | 1.96 | 2.1 / 6.3 | 2.0 / 6.3 | 2.0 / 6.0 | |
| CanItEdit | 1.84 | 1.49 | 1.30 | 3.2 / 6.2 | 3.2 / 5.9 | 3.0 / 5.2 | |
| NextCoder | 1.69 | 1.60 | 1.46 | 2.9 / 5.5 | 2.9 / 5.4 | 2.8 / 4.9 | |
| InstructCoder | 1.22 | 1.12 | 1.07 | 3.3 / 4.1 | 3.1 / 3.9 | 2.8 / 3.6 | |
| BS 4 | BS 128 | ||||||
|---|---|---|---|---|---|---|---|
| Edit distance | DSpark | n-gram + AV | Hybrid Spark | DSpark | n-gram | n-gram + AV | Hybrid Spark |
| 0–5% | 5.00 | 7.42 | 7.98 | 2.88 | 3.14 | 3.68 | 3.57 |
| 5–10% | 4.87 | 5.07 | 7.11 | 2.69 | 2.20 | 3.00 | 3.30 |
| 10–15% | 4.78 | 4.06 | 6.52 | 2.48 | 1.60 | 2.54 | 2.84 |
| 15–20% | 4.61 | 3.31 | 5.96 | 2.49 | 1.47 | 2.51 | 2.91 |
| 20–25% | 4.54 | 2.88 | 5.65 | 2.35 | 1.12 | 1.99 | 2.54 |
| Model | BS | Blazedit 5k | Blazedit 10k | CanItEdit | NextCoder | EditBench | InstructCoder | MT-Bench |
|---|---|---|---|---|---|---|---|---|
| Qwen3-4B | 4 | 1.34 | 1.50 | 1.35 | 1.24 | 1.62 | 1.14 | 0.99 ∘ |
| 32 | 1.16 | 1.31 | 1.16 | 1.11 | 1.37 | 1.06 | 0.99 | |
| 128 | 1.13 | 1.26 | 1.12 | 1.10 | 1.27 | 0.99 | 0.98 | |
| Qwen3-8B | 4 | 1.31 | 1.45 | 1.33 | 1.19 | 1.59 | 1.11 | 1.00 ∘ |
| 32 | 1.13 | 1.28 | 1.11 | 1.08 | 1.26 | 1.02 | 1.00 ∘ | |
| 128 | 1.08 | 1.24 | 1.08 | 1.05 | 1.19 | 0.98 | 0.99 |
| NextCoder 15–20% | NextCoder 40–45% | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| BS | Policy | T0g | T0p | T1g | T1p | T0g | T0p | T1g | T1p |
| 4 | Proposed | 1.301 | 1.304 | 1.282 | – | 1.127 | 1.123 | 1.126 | – |
| N-gram-first | 1.269 | 1.274 | 1.244 | 1.231 | 1.067 | 1.067 | 1.070 | 1.056 | |
| Extension | 1.269 | 1.271 | 1.241 | 1.222 | 1.128 | 1.131 | 1.124 | 1.098 | |
| DSpark (prob.) | 0.994 ∘ | 0.976 | 0.994 ∘ | 0.974 | |||||
| 128 | Proposed | 1.163 | 1.159 | 1.178 | – | 1.053 | 1.051 | 1.067 | – |
| BS | Policy | T0g | T0p | T1g | T1p |
|---|---|---|---|---|---|
| 4 | Proposed | 0.989 | 0.981 | 0.985 | – |
| N-gram-first | 0.980 | 0.976 | 0.982 | 1.015 | |
| Extension | 0.997 ∘ | 0.990 ∘ | 1.000 ∘ | 1.025 | |
| DSpark (prob.) | 1.000 ∘ | 1.029 | |||
| 128 | Proposed | 0.968 | 0.964 | 0.971 | – |
| N-gram-first | 0.970 | 0.964 | 0.980 ∘ | 0.965 |
| 1k context | 2k context | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Low | Mixed | High | Low | Mixed | High | ||||||||
| BS | Policy | T1g | T1p | T1g | T1p | T1g | T1p | T1g | T1p | T1g | T1p | T1g | T1p |
| 4 | Proposed | 1.063 | – | 0.989 | – | 0.972 | – | 1.055 | – | 0.993 | – | 1.009 ∘ | – |
| N-gram-first | 1.069 | 1.042 | 0.980 | 0.989 | 0.981 | 1.008 | 1.047 | 1.013 | 0.960 | 0.977 | 1.013 | 1.016 | |
| Extension | 1.044 | 1.017 | 0.993 ∘ | 1.010 | 0.987 ∘ | 1.009 | 1.038 | 0.999 ∘ | 1.005 ∘ | 1.017 | 1.007 ∘ | 1.012 | |
| DSpark (prob.) | 0.978 | 1.013 | 1.000 ∘ | 0.974 | 1.013 | 1.012 ∘ | |||||||
| Arm | BS | Pred. | Obs. | Rel. | Lost | |
|---|---|---|---|---|---|---|
| DSpark | 4 | 1.37 | 2.16 | 1.94 | 8.1% | 0.29 |
| 32 | 1.36 | 6.11 | 5.63 | 3.0% | 0.16 | |
| 128 | 1.37 | 12.24 | 11.03 | 1.4% | 0.06 | |
| n-gram + AV | 4 | 2.98 | 4.74 | 4.23 | 39.4% | 7.17 |
| 32 | 3.00 | 13.50 | 13.11 | 16.0% | 1.82 | |
| 128 | 2.96 | 26.60 | 28.04 | 8.5% | 0.35 |
Speculative Decoding
Speculative decoding accelerates generation by drafting multiple tokens and verifying them in parallel with a larger target model.
See all 343 papers