Reusing key--value (KV) caches across language model updates could avoid repeated computation of shared context, but changes in model weights raise a fundamental question about what these caches preserve. We investigate direct cache reuse across four model updates, with a detailed study of DeepSeek-V4-Flash to Flash-0731. Our central finding is that caches can retain useful task information without faithfully reproducing the updated model's behavior. Retrieval accuracy varies substantially across updates, ranging from near-native performance to large losses. With source caches already available, reuse achieves median time-to-first-token speedups of 2.10--5.08× on three longer-prefix QA benchmarks. To characterize the behavioral differences behind these outcomes, we introduce Functional Cache Divergence, which compares next-token distributions under reused and native caches. Direct KV cache reuse better preserves native model behavior in natural-text continuation, whereas compatibility degrades with increasing prefix length in single-token retrieval tasks. Answer-margin analyses help explain how correct retrieval decisions survive substantial internal state changes. Yet even agreement on the first generated token does not ensure agreement in answer content or stopping behavior. These findings establish task utility and fidelity to native inference as distinct criteria for evaluating cross-version KV cache reuse.
Figures & tables
Figure 2: Three inference configurations on the same prefix and question. Target and Source use one model throughout. KV Reuse combines the source-produced prefix cache with target-model continuation.
Table 1: NIAH answer-prefix accuracy (%; higher is better) across model updates. Each prefix length contains 192 queries from eight histories. Sizes in parentheses denote nominal backbone parameters. Target and Source use their own caches; KV Reuse uses the target model with the source cache; Ridge Mapper maps that cache before target-model continuation.
Figure 3: NIAH retrieval under KV reuse. (a) Harmful flip rate in the original sweep across four model updates over 512–32K prefixes, conditional on Target answering correctly. (b) Matched Target, Source, KV Reuse, and Ridge Mapper answer-prefix accuracy for V4.1-Flash → Flash-1017 over 512–512K prefixes. Each length has 192 queries from eight histories; error bars in (b) show 95% history-cluster bootstrap intervals. Data construction and runtime details are given in Appendices A.1 and B.2 .
Figure 5: Flash KV reuse in Base and Flash-0731. (a) Model development and cache-transfer directions. (b) NIAH accuracy across 512–32K prefixes. Solid and dashed curves denote KV Reuse and Target. Shading marks their accuracy difference. The green curve shows Flash’s source accuracy.
Task
Metric
Full match
Later change
First-token change
Target score ↑
KV Reuse score ↑
Δ score
Retrieval
Acc.
85.4
3.1
11.5
83.33
89.58
+6.25
BABILong
Acc.
66.3
4.7
29.0
59.00
58.67
−0.33
HotpotQA
F1
70.0
24.0
6.0
69.28
66.28
−2.99
2WikiMQA
F1
60.2
28.6
11.2
61.68
54.65
−7.03
MultiFieldQA-en
F1
41.2
46.2
12.5
45.92
46.24
+0.32
Table 2: Generation agreement and answer scores for Flash → Flash-0731. All agreement rates and scores are percentages. Δ score is KV Reuse minus Target, in percentage points. Retrieval uses the separate diagnostic cohort, and QA uses the main study in Section 3.2 . Agreement uses successful request pairs, and failed requests score zero. Acc. denotes accuracy.
Figure 6: Two diagnostic QA cases illustrate (a) stopping changes and (b) answer-content changes under KV Reuse. Scores are full-response F1.
Figure 7: Mean FCD for Flash → Flash-0731 on NIAH and FineWeb-Edu. Curves share a vertical scale, with a FineWeb-Edu zoom inset. Bands show cluster-bootstrap 95% intervals.
Figure 8: Flash KV reuse in Flash-0731 and Flash-Base. (a, b) Layerwise NMSE over all query positions: medians and 25th–75th percentiles of selected cases. (c, d) Native gold margin versus its change under reuse. Dashed lines mark zero reuse gold margin, and colors show paired answer-prefix correctness. One point per margin panel lies outside the view; all queries enter summaries.
Figure 9: Indexer selection under Flash → Flash-Base reuse. (a) Mean top- k overlap with Target across layers. (b) Layer-band means after correcting for uniform-random overlap. Measurements use the first response prediction on four selected retrieval-recovery cases per length. Layer indices are zero-based.
Figure 10: Replacing the local KV cache, the incoming hidden state, or both on the same retrieval cohort. The cache is replaced with the source cache, and the hidden state with the target’s representation after earlier layers have reused source caches. Bars show median attention-output NMSE relative to Target. These comparisons do not add up.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Component
Example
Prefix P (excerpt)
Read these records carefully. Every key has exactly one value. Use the records to answer the question. Return only the value. The morning light fell across the quiet garden. … yet:vector … ynt:izio … IELD:Popular …
Query Q
What is the value for key:yet? Answer with only the value.
Gold answer
vector
Appendix
Table 3: A Flash NIAH example. Prefix text is abbreviated; chat-template tokens and record separators are omitted for readability.
Benchmark
Source snapshot
Eligible pool
Main n
Length strata
BABILong
QA1–QA5, revision ee0d5887
300 independent
300
1K / 2K / 4K; 20 per QA–length pair
HotpotQA
LongBench v1, revision 5e628be4
199 rows; 199 documents
100
Four token-length quartiles
2WikiMQA
LongBench v1, revision 5e628be4
192 rows; 192 documents
100
Four token-length quartiles
MultiFieldQA-en
LongBench v1, revision 5e628be4
139 rows; 105 documents
85
Four token-length quartiles; 15-sample shortfall
Appendix
Table 4: Dataset provenance and main-study selection. “Eligible” counts rows surviving input and tokenizer checks; “independent” counts distinct stories or source documents. BABILong contains 20 examples for every QA-type and length pair.
Benchmark
n
Min
Q1
Median
Q3
Max
BABILong
300
571
735
1,712
3,539
3,852
HotpotQA
100
2,142
11,054
14,320
15,892
16,507
2WikiMQA
100
2,580
5,041
6,369
8,530
15,783
MultiFieldQA-en
85
2,703
4,827
7,666
9,030
14,636
Appendix
Table 5: Tokenized prefix lengths in the 585-sample main study. Q1 and Q3 are the first and third quartiles. Question suffixes are excluded.
Cohort
n
Independent unit
Main retrieval diagnostics
96
32 contexts
Retrieval confirmation
48
16 contexts
Natural-QA diagnostics
89
89 examples
Appendix
Table 6: Primary diagnostic cohorts. Retrieval counts are queries and QA counts are examples.
Shared FP8 Triton path for all four configurations; 512-token chunks through 4K, 1,024 at 8K–32K
GLM additional-update NIAH
SGLang; TP=2, PP=8 across two GPU nodes
File-backed cache handoff
Pro additional-update NIAH
PyTorch reference runtime; separate source and target workers
Full prefill; complete prefix-cache handoff
V4.1 additional-update NIAH
Official reference runtime; TP=8
Complete prefix-state restoration
Appendix
Table 7: Runtime and cache-handoff settings by experiment family.
Benchmark
n
Metric
Target
Direct
Source
Δ [95% CI]
Score ↓/↑
Flips
Fail.
BABILong
300
Acc.
59.0
58.7
61.0
−0.3[−4.3,+3.7]
20/19
20/19
0/0
HotpotQA
100
Token F1
69.3
66.3
69.4
−3.0[−6.6,−0.1]
8/6
5/0
0/0
2WikiMQA
100
Token F1
61.7
54.7
58.9
−7.0[−12.7,−1.5]
19/3
13/3
0/2
MultiFieldQA-en
85
Token F1
45.9
46.2
49.8
+0.3[−3.1,+3.7]
20/19
1/2
5/5
Appendix
Table 8: Task quality on the 585-sample benchmark study. Scores are percentages. Δ is direct minus target-native, in percentage points; intervals use 2,000 document/story-cluster bootstrap resamples. Score ↓/↑ counts paired score decreases and increases. Flips report native-success → direct-failure and native-failure → direct-success. Failures are native/direct and score zero.
Benchmark
Pairs
Native p50/p95
Direct p50/p95
Speedup p50
Direct slower
BABILong
300/300
153.2/286.5
192.2/254.4
0.855 ×
201
HotpotQA
100/100
1,792.3/8,101.1
355.9/392.2
5.083 ×
1
2WikiMQA
98/100
543.5/1,712.7
267.3/524.2
2.097 ×
3
MultiFieldQA-en
80/85
759.1/1,480.8
248.2/762.1
2.512 ×
6
Appendix
Table 9: Prepared-cache TTFT. Only valid target/direct pairs enter the sample count and percentiles. The speedup is computed per sample as target-native/direct TTFT and summarized by its median; it is not the ratio of the two marginal medians.
Group
n
Fail.
Target/Direct
Δ [95% CI]
Score ↓/↑
Pairs
Speedup
2Wiki q1
25
0/0
74.6/66.9
−7.8[−24.6,+9.1]
7/2
25/25
1.80 ×
2Wiki q2
25
0/1
59.5/56.8
−2.7[−10.5,+4.3]
3/1
24/25
2.09 ×
2Wiki q3
25
0/0
56.2/47.5
−8.8[−18.0,−1.3]
4/0
25/25
2.21 ×
2Wiki q4
25
0/1
56.4/47.5
−8.9[−19.2,−0.6]
5/0
24/25
4.52 ×
BABILong 1K
100
0/0
67.0/67.0
0.0[−7.0,+8.0]
7/7
100/100
0.86 ×
BABILong 2K
100
0/0
57.0/60.0
+3.0[−3.0,+9.0]
4/7
100/100
0.79 ×
Appendix
Table 10: Quality and prepared-cache TTFT by actual input-length group. Scores and confidence intervals are percentages. BABILong groups are source-defined lengths; other groups are quartiles of tokenized prefixes.
Benchmark
n
First match
Full match
Self − Native
Direct − Self
BABILong
300
265
255
−0.013
+0.010
HotpotQA
100
95
87
−0.010
−0.020
2WikiMQA
100
94
84
−0.013
−0.058
MultiFieldQA-en
85
82
52
−0.005
+0.008
Appendix
Table 11: Target-self controls in the 585-sample study. “First” and “full” report first-token and complete-recorded-sequence agreement, respectively. Scores show mean self minus native and direct minus self gaps.
Benchmark
Valid pairs (original → retest)
Target/direct
Δ [95% CI]
Speedup
2WikiMQA
98/100 → 100/100
0.617/0.548
−0.069[−0.126,−0.013]
2.097 ×
MultiFieldQA-en
80/85 → 85/85
0.501/0.506
+0.005[−0.036,+0.047]
2.390 ×
Appendix
Table 12: Post hoc cache-tier sensitivity retest. Scores and score changes use each task’s primary metric. The retest isolates seven rows with device/host cache interference and is not pooled with the original main result.
Task ( n=20 each)
Target F1
KV Reuse F1
Loss / unchanged / gain
Change [95% CI]
HotpotQA
0.6553
0.6683
1 / 18 / 1
+0.0129[−0.0088,+0.0476]
2WikiMQA
0.6382
0.5571
5 / 13 / 2
−0.0811[−0.2131,+0.0435]
Appendix
Table 13: Full-response F1 on the preselected random QA subsets. Loss, unchanged, and gain count each example’s paired score change. Intervals use 2,000 bootstrap resamples of source examples.
Benchmark
Pairs
Identical
First token
Later
Later score ↓/=/↑
Later EM 1→0
BABILong
300
199
87
14
0/12/2
0
HotpotQA
100
70
6
24
7/11/6
5
2WikiMQA
98
59
11
28
12/15/1
9
MultiFieldQA-en
80
33
10
37
16/6/15
0
Appendix
Table 14: First-divergence analysis for the four cross-benchmark QA tasks. Only samples with successful native/direct pairs enter this table. “Later” means the first token matched and the recorded sequences still differed.
Prefix
Target → Reuse outcome
Queries
Mean FCD
Median FCD
16K
Correct → correct
89
4.56
1.04
Incorrect → correct
36
66.42
73.12
Correct → incorrect
8
44.34
33.09
Incorrect → incorrect
59
25.03
21.79
32K
Correct → correct
36
9.99
6.99
Incorrect → correct
24
63.23
65.31
Appendix
Table 15: FCD by paired retrieval outcome at 16K and 32K. Categories use the frozen answer-prefix evaluation; larger FCD denotes greater distributional change. Counts and conditional summaries describe the observed groups.
Prefix
Arm
Prefix
First token
Exact
Median rank
Mean prob.
Top-10 (%)
512
Native
0
0
0
2151.5
0.0019
7.8
512
Direct
169
0
169
2.0
0.1056
97.9
512
Source
190
190
186
1.0
0.9841
100.0
2,048
Native
0
0
0
18508.0
2.5×10−5
0.0
2,048
Direct
175
15
174
2.0
0.1792
99.0
2,048
Source
187
187
186
1.0
0.9621
99.5
Appendix
Table 16: Flash-to-Base NIAH readout diagnostics. Gold rank is over the full first-token vocabulary; lower rank is better. The prefix, first-token, and exact columns are counts out of 192.
Layer (zero-based)
Replace local KV cache
Replace incoming hidden state
Replace both
27
0.012
0.142
0.139
37
0.019
0.473
0.530
42
0.005
0.064
0.077
Appendix
Table 17: Crossed input–cache intervention at the prediction position. All values are normalized attention-output differences from the target-native result, median over 96 queries. Columns are not additive because changing both factors also changes their interaction.
Replacement in the direct path
Layers
Answers restored
Native attention outputs
36–42
4/4
Complete native prefix caches
36–42
1/4
Complete native prefix caches
0–27
2/4
Native compressor state bundles
0–27
3/4
Appendix
Table 18: Recovery of four selected retrieval losses after replacing native attention outputs or cache components. Layer numbers are zero-based.