Memory-Efficient Speculative Decoding with Quantized Draft KV Cache
Organizations: Carnegie Mellon University
Abstract
Speculative decoding reduces generation cost, but it stores two KV caches: one for the target model and one for the draft model. We reduce this extra memory by quantizing the draft cache. The target cache and verifier remain in BF16, so that compression cannot change the target model's final correction rule. We first study how to divide precision between keys and values. A naive per-token quantizer makes keys appear much more sensitive: splitting a 12-bit budget, K8V4 beats K4V8 in acceptance by 10.87pp. However, with grouped per-channel keys, the gap becomes -0.25pp. A controlled Qwen test confirms that changing only the key quantizer can erase or reverse the preferred bit split. Hence, following KIVI, we propose K4V4 quantization with per-channel keys and per-token values. Across six model pairs, our method reduces draft cache memory by 66.20%, and combined memory across both caches by 23.95%. Mean acceptance drops from 57.93% to 57.14%, only 1.36% relative. Overall, quantized draft caches can make speculative decoding more memory efficient; as a predictive principle for efficient AI design, key and value precision must be co-designed with the quantizer layout.
Figures & tables
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
| Family | Target | Draft |
|---|---|---|
| Qwen2.5 | Qwen/Qwen2.5-3B | Qwen/Qwen2.5-1.5B |
| Qwen2.5 | Qwen/Qwen2.5-7B | Qwen/Qwen2.5-3B |
| Qwen3 | Qwen/Qwen3-8B | Qwen/Qwen3-4B |
| Llama 3 | meta-llama/Llama-3.1-8B | meta-llama/Llama-3.2-3B |
| OLMo 2 | allenai/OLMo-2-1124-7B | allenai/OLMo-2-0425-1B |
| SmolLM2 | HuggingFaceTB/SmolLM2-1.7B | HuggingFaceTB/SmolLM2-360M |
| Target / draft | BF16 | K4V4 | acceptance (95% CI) | Draft saved | Draft share | Total saved |
|---|---|---|---|---|---|---|
| Qwen2.5 3B/1.5B | 59.78 | 59.02 | 66.33% | 43.8% | 29.02% | |
| Qwen2.5 7B/3B | 61.80 | 62.08 | 66.33% | 39.1% | 25.96% | |
| Qwen3 8B/4B | 54.06 | 52.39 | 66.33% | 50.0% | 33.17% | |
| Llama 3.1/3.2 | 64.95 | 65.09 | 66.33% | 46.7% | 30.96% | |
| OLMo 2 7B/1B | 54.03 | 53.17 | 66.33% | 20.0% | 13.27% | |
| SmolLM2 1.7B/360M | 52.95 | 51.10 | 65.55% | 17.2% | 11.30% |
| Target / draft | Per-token | Grouped-key | Shift |
|---|---|---|---|
| Qwen2.5 3B/1.5B | |||
| Qwen2.5 7B/3B | |||
| Qwen3 8B/4B | |||
| Llama 3.1/3.2 | |||
| OLMo 2 7B/1B | |||
| SmolLM2 1.7B/360M |
| Contrast | Per-token keys | Grouped per-channel keys |
|---|---|---|
| K8V4 K4V8 | ||
| K4V2 K2V4 |
| Draft model | BF16 | K4V2 | K2V4 | (pp) |
|---|---|---|---|---|
| Qwen2.5-1.5B | 186/192 | 189/192 | 158/192 | |
| Llama-3.2-3B | n/a | 185/192 | 183/192 | |
| Qwen3-4B | 191/192 | 192/192 | 192/192 |
| Context | Prompts | Draft saved | Combined saved | acceptance |
|---|---|---|---|---|
| 16K | 36 | 70.83% | 30.99% | |
| 32K | 24 | 70.96% | 31.04% |
| Context | Residual | Seed | BF16 | K8V4 | K4V8 | K4V4 | K4V4 BF16 | K4V4: residual 128 0 |
|---|---|---|---|---|---|---|---|---|
| 1K | 0 | 0 | 76.69 | 77.25 | 77.01 | 76.61 | ||
| 1K | 0 | 1 | 74.01 | 74.18 | 73.58 | 73.45 | ||
| 1K | 128 | 0 | 76.69 | 77.44 | 76.61 | 77.02 | ||
| 1K | 128 | 1 | 74.01 | 73.79 | 73.55 | 73.42 | ||
| 16K | 0 | 0 | 79.62 | 79.54 | 79.64 | 79.20 | ||
| 16K | 128 | 0 | 79.62 | 79.92 | 79.42 | 79.55 |
| Context | Configuration | Latency (s) | vs. BF16 SD (s) | Draft bytes saved | Peak allocated (GiB) | Exact |
|---|---|---|---|---|---|---|
| 1K | Target only | 7.10 | – | – | 5.98 | reference |
| 1K | BF16 SD | 9.68 | – | 0.00% | 8.87 | 3/24 |
| 1K | Packed K4V4, residual 0 | 12.77 | 70.45% | 8.87 | 4/24 | |
| 1K | Packed K4V4, residual 128 | 12.87 | 67.05% | 8.87 | 5/24 | |
| 16K | Target only | 8.65 | – | – | 7.69 | reference |
| 16K | BF16 SD | 14.54 | – | 0.00% | 10.76 | 2/12 |
| Experiment family | GPU allocations | GPU-hours |
|---|---|---|
| Cross-family speculative and quality | 107 | 33.24 |
| Key geometry and residual controls | 47 | 4.60 |
| Passkey retrieval, layers, and policy | 95 | 13.14 |
| LongBench retrieval | 18 | 2.51 |
| Long-context speculative and quality | 19 | 1.90 |
| Objective and precision matrix | 710 | 88.35 |
| Asset | License or access terms |
|---|---|
| Qwen2.5 1.5B, 7B | Apache License 2.0 |
| Qwen2.5 7B-Instruct; Qwen3 4B, 8B | Apache License 2.0 |
| Qwen2.5 3B | Qwen Research License |
| Llama 3.1 8B(-Instruct); Llama 3.2 3B | Llama Community License, acceptable-use terms, and gated access |
| OLMo 2 7B, 1B; SmolLM2 1.7B, 360M | Apache License 2.0 |
| FineWeb-Edu | Open Data Commons Attribution License (ODC-By) v1.0 |
Speculative Decoding
Speculative decoding accelerates generation by drafting multiple tokens and verifying them in parallel with a larger target model.
See all 351 papers