Can KV Cache Be Reused Across Model Updates? A DeepSeek-V4-Flash Case Study
Organizations: Rank Computing
Abstract
Reusing key--value (KV) caches across language model updates could avoid repeated computation of shared context, but changes in model weights raise a fundamental question about what these caches preserve. We investigate direct cache reuse across four model updates, with a detailed study of DeepSeek-V4-Flash to Flash-0731. Our central finding is that caches can retain useful task information without faithfully reproducing the updated model's behavior. Retrieval accuracy varies substantially across updates, ranging from near-native performance to large losses. With source caches already available, reuse achieves median time-to-first-token speedups of -- on three longer-prefix QA benchmarks. To characterize the behavioral differences behind these outcomes, we introduce Functional Cache Divergence, which compares next-token distributions under reused and native caches. Direct KV cache reuse better preserves native model behavior in natural-text continuation, whereas compatibility degrades with increasing prefix length in single-token retrieval tasks. Answer-margin analyses help explain how correct retrieval decisions survive substantial internal state changes. Yet even agreement on the first generated token does not ensure agreement in answer content or stopping behavior. These findings establish task utility and fidelity to native inference as distinct criteria for evaluating cross-version KV cache reuse.
Figures & tables
| Prefix length (tokens) | ||||||
| Configuration | 512 | 2K | 4K | 8K | 16K | 32K |
| GLM-5.2 GLM-5.3 (750B) | ||||||
| Target | 98.44 | 99.48 | 98.44 | 98.44 | 94.79 | 86.46 |
| Source | 96.88 | 96.88 | 95.83 | 97.92 | 92.71 | 89.58 |
| KV Reuse | 99.48 | 99.48 | 98.96 | 98.96 | 95.31 | 90.62 |
| DeepSeek-V4.1-Flash DeepSeek-V4.1-Flash-1017 (552B) | ||||||
| Task | Metric | Full match | Later change | First-token change | Target score | KV Reuse score | score |
|---|---|---|---|---|---|---|---|
| Retrieval | Acc. | 85.4 | 3.1 | 11.5 | 83.33 | 89.58 | |
| BABILong | Acc. | 66.3 | 4.7 | 29.0 | 59.00 | 58.67 | |
| HotpotQA | F1 | 70.0 | 24.0 | 6.0 | 69.28 | 66.28 | |
| 2WikiMQA | F1 | 60.2 | 28.6 | 11.2 | 61.68 | 54.65 | |
| MultiFieldQA-en | F1 | 41.2 | 46.2 | 12.5 | 45.92 | 46.24 |
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
| Component | Example |
|---|---|
| Prefix (excerpt) | Read these records carefully. Every key has exactly one value. Use the records to answer the question. Return only the value. The morning light fell across the quiet garden. … yet:vector … ynt:izio … IELD:Popular … |
| Query | What is the value for key:yet? Answer with only the value. |
| Gold answer | vector |
| Benchmark | Source snapshot | Eligible pool | Main | Length strata |
|---|---|---|---|---|
| BABILong | QA1–QA5, revision ee0d5887 | 300 independent | 300 | 1K / 2K / 4K; 20 per QA–length pair |
| HotpotQA | LongBench v1, revision 5e628be4 | 199 rows; 199 documents | 100 | Four token-length quartiles |
| 2WikiMQA | LongBench v1, revision 5e628be4 | 192 rows; 192 documents | 100 | Four token-length quartiles |
| MultiFieldQA-en | LongBench v1, revision 5e628be4 | 139 rows; 105 documents | 85 | Four token-length quartiles; 15-sample shortfall |
| Benchmark | Min | Q1 | Median | Q3 | Max | |
|---|---|---|---|---|---|---|
| BABILong | 300 | 571 | 735 | 1,712 | 3,539 | 3,852 |
| HotpotQA | 100 | 2,142 | 11,054 | 14,320 | 15,892 | 16,507 |
| 2WikiMQA | 100 | 2,580 | 5,041 | 6,369 | 8,530 | 15,783 |
| MultiFieldQA-en | 85 | 2,703 | 4,827 | 7,666 | 9,030 | 14,636 |
| Cohort | Independent unit | |
|---|---|---|
| Main retrieval diagnostics | 96 | 32 contexts |
| Retrieval confirmation | 48 | 16 contexts |
| Natural-QA diagnostics | 89 | 89 examples |
| Experiment family | Execution | Cache and prefill settings |
|---|---|---|
| 585-sample main study | SGLang commit 1fdd6c89 ; eight H100 GPUs; TP=8; file HiCache | 4,096-token chunks; 256-token pages; write_through |
| NIAH readout | Transformers and PyTorch on eight H100 GPUs | Complete prefix-state transfer |
| Flash ridge-mapper comparison | Transformers 5.20.1; PyTorch 2.11.0+cu129; eight H100 GPUs | Shared FP8 Triton path for all four configurations; 512-token chunks through 4K, 1,024 at 8K–32K |
| GLM additional-update NIAH | SGLang; TP=2, PP=8 across two GPU nodes | File-backed cache handoff |
| Pro additional-update NIAH | PyTorch reference runtime; separate source and target workers | Full prefill; complete prefix-cache handoff |
| V4.1 additional-update NIAH | Official reference runtime; TP=8 | Complete prefix-state restoration |
| Benchmark | Metric | Target | Direct | Source | [95% CI] | Score | Flips | Fail. | |
|---|---|---|---|---|---|---|---|---|---|
| BABILong | 300 | Acc. | 59.0 | 58.7 | 61.0 | 20/19 | 20/19 | 0/0 | |
| HotpotQA | 100 | Token F1 | 69.3 | 66.3 | 69.4 | 8/6 | 5/0 | 0/0 | |
| 2WikiMQA | 100 | Token F1 | 61.7 | 54.7 | 58.9 | 19/3 | 13/3 | 0/2 | |
| MultiFieldQA-en | 85 | Token F1 | 45.9 | 46.2 | 49.8 | 20/19 | 1/2 | 5/5 |
| Benchmark | Pairs | Native p50/p95 | Direct p50/p95 | Speedup p50 | Direct slower |
|---|---|---|---|---|---|
| BABILong | 300/300 | 153.2/286.5 | 192.2/254.4 | 0.855 | 201 |
| HotpotQA | 100/100 | 1,792.3/8,101.1 | 355.9/392.2 | 5.083 | 1 |
| 2WikiMQA | 98/100 | 543.5/1,712.7 | 267.3/524.2 | 2.097 | 3 |
| MultiFieldQA-en | 80/85 | 759.1/1,480.8 | 248.2/762.1 | 2.512 | 6 |
| Group | Fail. | Target/Direct | [95% CI] | Score | Pairs | Speedup | |
|---|---|---|---|---|---|---|---|
| 2Wiki q1 | 25 | 0/0 | 74.6/66.9 | 7/2 | 25/25 | 1.80 | |
| 2Wiki q2 | 25 | 0/1 | 59.5/56.8 | 3/1 | 24/25 | 2.09 | |
| 2Wiki q3 | 25 | 0/0 | 56.2/47.5 | 4/0 | 25/25 | 2.21 | |
| 2Wiki q4 | 25 | 0/1 | 56.4/47.5 | 5/0 | 24/25 | 4.52 | |
| BABILong 1K | 100 | 0/0 | 67.0/67.0 | 7/7 | 100/100 | 0.86 | |
| BABILong 2K | 100 | 0/0 | 57.0/60.0 | 4/7 | 100/100 | 0.79 |
| Benchmark | First match | Full match | Self Native | Direct Self | |
|---|---|---|---|---|---|
| BABILong | 300 | 265 | 255 | ||
| HotpotQA | 100 | 95 | 87 | ||
| 2WikiMQA | 100 | 94 | 84 | ||
| MultiFieldQA-en | 85 | 82 | 52 |
| Benchmark | Valid pairs (original retest) | Target/direct | [95% CI] | Speedup |
|---|---|---|---|---|
| 2WikiMQA | 98/100 100/100 | 0.617/0.548 | 2.097 | |
| MultiFieldQA-en | 80/85 85/85 | 0.501/0.506 | 2.390 |
| Task ( each) | Target F1 | KV Reuse F1 | Loss / unchanged / gain | Change [95% CI] |
|---|---|---|---|---|
| HotpotQA | 0.6553 | 0.6683 | 1 / 18 / 1 | |
| 2WikiMQA | 0.6382 | 0.5571 | 5 / 13 / 2 |
| Benchmark | Pairs | Identical | First token | Later | Later score | Later EM |
|---|---|---|---|---|---|---|
| BABILong | 300 | 199 | 87 | 14 | 0/12/2 | 0 |
| HotpotQA | 100 | 70 | 6 | 24 | 7/11/6 | 5 |
| 2WikiMQA | 98 | 59 | 11 | 28 | 12/15/1 | 9 |
| MultiFieldQA-en | 80 | 33 | 10 | 37 | 16/6/15 | 0 |
| Prefix | Target Reuse outcome | Queries | Mean FCD | Median FCD |
|---|---|---|---|---|
| 16K | Correct correct | 89 | 4.56 | 1.04 |
| Incorrect correct | 36 | 66.42 | 73.12 | |
| Correct incorrect | 8 | 44.34 | 33.09 | |
| Incorrect incorrect | 59 | 25.03 | 21.79 | |
| 32K | Correct correct | 36 | 9.99 | 6.99 |
| Incorrect correct | 24 | 63.23 | 65.31 |
| Prefix | Arm | Prefix | First token | Exact | Median rank | Mean prob. | Top-10 (%) |
|---|---|---|---|---|---|---|---|
| 512 | Native | 0 | 0 | 0 | 2151.5 | 0.0019 | 7.8 |
| 512 | Direct | 169 | 0 | 169 | 2.0 | 0.1056 | 97.9 |
| 512 | Source | 190 | 190 | 186 | 1.0 | 0.9841 | 100.0 |
| 2,048 | Native | 0 | 0 | 0 | 18508.0 | 0.0 | |
| 2,048 | Direct | 175 | 15 | 174 | 2.0 | 0.1792 | 99.0 |
| 2,048 | Source | 187 | 187 | 186 | 1.0 | 0.9621 | 99.5 |
| Layer (zero-based) | Replace local KV cache | Replace incoming hidden state | Replace both |
|---|---|---|---|
| 27 | 0.012 | 0.142 | 0.139 |
| 37 | 0.019 | 0.473 | 0.530 |
| 42 | 0.005 | 0.064 | 0.077 |
| Replacement in the direct path | Layers | Answers restored |
|---|---|---|
| Native attention outputs | 36–42 | 4/4 |
| Complete native prefix caches | 36–42 | 1/4 |
| Complete native prefix caches | 0–27 | 2/4 |
| Native compressor state bundles | 0–27 | 3/4 |
KV-Cache Reuse
Key-value (KV) cache reuse lets language models reuse previously computed attention states to cut repeated prefill work and serving latency.
See all 81 papers