To Cache or Not to Cache: Understanding and Mitigating Safety Incompatibility in Cross-Model KV Cache Reuse

Zhiwei Lin; Jiahao Liu; Yan Liu; Chengpeng Wang

cs.LGOct 11, 2026

To Cache or Not to Cache: Understanding and Mitigating Safety Incompatibility in Cross-Model KV Cache Reuse

Authors: Zhiwei Lin, Jiahao Liu, Yan Liu, Chengpeng Wang

Organizations: National University of Singapore · Ant Group

Abstract

Cross-model KV cache sharing reduces prefill computation by allowing a receiver model to reuse prefix KV states generated by a related sender model. Existing methods optimize such reuse for latency and generation quality, but its safety implications remain unclear. In this work, we systematically evaluate this issue across diverse sender-receiver pairs and reuse configurations. We refer to deviations from the receiver's native safety behavior as safety incompatibility. Our study shows that both the direction and magnitude of safety incompatibility depend on the model pair and reuse configuration, with value reuse causing substantially larger deviations than key reuse across all evaluated pairs. Furthermore, we propose VReparo, a technique that learns per-layer and per-KV-head affine maps offline to align reused sender value tensors with their receiver-native counterparts before decoding. Across two model pairs under full KV reuse and the state-of-the-art selective reuse setting, VReparo reduces the average HarmBench attack success rate (ASR) from 57.28% to 39.21% and the average StrongREJECT score from 0.4215 to 0.2163 upon our benchmarks, while retaining an average Time to First Token (TTFT) reduction of 85.62%.

Figures & tables

Figure 1: One form of safety incompatibility in cross-model KV cache reuse, where reusing sender-generated KV states changes a receiver-native safe response into a harmful one.

Model	(Sender, Receiver)	HarmBench				StrongREJECT
Model	(Sender, Receiver)	$\Delta S$	SD	SI	SDV	$\Delta S$	SD	SI	SDV
Qwen3-4B	(Base, Instruct)	+10.13	13.29	3.16	16.46	+.0495	.0716	.0221	.0938
Qwen3-8B	(Base, Instruct)	-2.27	0.76	3.03	3.79	-.0260	.0077	.0337	.0414
Llama-3.2-3B	(Base, Instruct)	+8.67	13.33	4.67	18.00	+.0781	.1009	.0227	.1236
Llama-3-8B	(Base, Instruct)	+24.84	28.03	3.18	31.21	+.2383	.2387	.0004	.2391
Gemma-2-2B	(Base, Instruct)	+20.25	20.89	0.63	21.52	+.1564	.1576	.0012	.1589

Table 1: Safety incompatibility under full KV cache reuse. HarmBench

\Delta S

is in percentage points (pp) and SD/SI/SDV in %. StrongREJECT metrics use the original score scale.

(Base, Instruct)
Model / Pair	$n$	ASR (%)				Change from Native (pp)
Model / Pair	$n$	Native	K-only	V-only	Full KV	$\Delta_{\mathrm{K}-\mathrm{N}}$	$\Delta_{\mathrm{V}-\mathrm{N}}$	$\Delta_{\mathrm{KV}-\mathrm{N}}$
Qwen3-4B	156	10.90	13.46	25.64	21.15	+2.56	+14.74	+10.26
Qwen3-8B	129	6.98	6.98	9.30	4.65	0.00	+2.33	-2.33
Llama-3.2-3B	147	8.84	7.48	28.57	17.01	-1.36	+19.73	+8.16
Llama-3-8B	138	5.80	18.12	57.25	28.99	+12.32	+51.45	+23.19

Table 2: HarmBench ASRs (%).

\Delta_{\mathrm{K}-\mathrm{N}}

\Delta_{\mathrm{V}-\mathrm{N}}

, and

\Delta_{\mathrm{KV}-\mathrm{N}}

are K-only, V-only, and Full KV ASRs minus Native ASR, respectively, in pp. Positive differences indicate increased harmfulness.

Figure 2: The workflow of VReparo . (a) Matching token positions at each layer and KV head provide paired reused and receiver-native values for affine fitting. Separate behavioral calibration selects

\alpha^{\star}

. (b) The fixed maps transform reused prefix values once. Keys, recomputed values, and newly generated KV states are not transformed. The layer layout illustrates selective reuse. Under full KV reuse, prefix values at all layers are mapped.

Model	Config.	Condition	HarmBench				StrongREJECT
Model	Config.	Condition	$\Delta S$	SD	SI	SDV	$\Delta S$	SD	SI	SDV
LoRI-D	Full KV	Reuse	50.39	54.33	3.94	58.27	0.5212	0.5337	0.0125	0.5462
		VReparo	34.96	38.90	3.94	42.83	0.2746	0.2940	0.0194	0.3134
	DroidSpeak	Reuse	53.54	56.69	3.15	59.84	0.5140	0.5205	0.0065	0.5270
		VReparo	31.34	34.96	3.62	38.58	0.2703	0.2846	0.0143	0.2989
Llama-3-8B	Full KV	Reuse	27.56	30.71	3.15	33.86	0.2398	0.2416	0.0018	0.2434

Table 3: Native-referenced safety metrics. VReparo results are averaged over five runs. HarmBench

\Delta S

is in pp and SD/SI/SDV in %. StrongREJECT uses the original score scale. Native metrics are zero. Bold marks the lower SDV per configuration.

Model	Configuration	TTFT (ms)			Speedup	Generation (ms/token)
Model	Configuration	Native	Reuse	VReparo	Speedup	Reuse	VReparo
LoRI-D	Full KV	1,355.84	54.71	88.98	$15.24\times$	51.15	51.15
	DroidSpeak	1,344.16	201.69	299.11	$4.49\times$	51.15	51.14
Llama-3-8B	Full KV	1,350.37	54.62	88.80	$15.21\times$	51.08	51.07
	DroidSpeak	1,347.80	201.59	298.40	$4.52\times$	51.04	51.03

Table 4: Mean latency with resident sender caches. Speedup is Native TTFT divided by VReparo TTFT. Per-token generation time excludes one-time cache preparation.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Model	(Sender, Receiver)	$n_{\mathrm{HB}}$	ASR (%)		Safety incompatibility
Model	(Sender, Receiver)	$n_{\mathrm{HB}}$	$\mathrm{ASR}_{\mathrm{N}}$	$\mathrm{ASR}_{\mathrm{reuse}}$	$\Delta S$	SD	SI	SDV
Qwen3-4B	(Base, Instruct)	158/159	10.76	20.89	+10.13	13.29	3.16	16.46
Qwen3-8B	(Base, Instruct)	132/159	6.82	4.55	-2.27	0.76	3.03	3.79
Llama-3.2-3B	(Base, Instruct)	150/159	8.67	17.33	+8.67	13.33	4.67	18.00
Llama-3-8B	(Base, Instruct)	157/159	6.37	31.21	+24.84	28.03	3.18	31.21
Gemma-2-2B	(Base, Instruct)	158/159	1.27	21.52	+20.25	20.89	0.63	21.52

Appendix

Table 5: Full KV reuse on HarmBench. ASRs and SD/SI/SDV are in %, and

\Delta S

is in pp.

Model	(Sender, Receiver)	$n_{\mathrm{SR}}$	Average score		Safety incompatibility
Model	(Sender, Receiver)	$n_{\mathrm{SR}}$	Native	Reuse	$\Delta S$	SD	SI	SDV
Qwen3-4B	(Base, Instruct)	306/313	.0614	.1109	+.0495	.0716	.0221	.0938
Qwen3-8B	(Base, Instruct)	296/313	.0447	.0186	-.0260	.0077	.0337	.0414
Llama-3.2-3B	(Base, Instruct)	259/313	.0391	.1173	+.0781	.1009	.0227	.1236
Llama-3-8B	(Base, Instruct)	311/313	.0094	.2477	+.2383	.2387	.0004	.2391
Gemma-2-2B	(Base, Instruct)	313/313	.0094	.1658	+.1564	.1576	.0012	.1589

Appendix

Table 6: StrongREJECT scores and incompatibility metrics under full KV reuse (original scale).

$\alpha$	Unsafe change	Safe change	Max subtype change	Eligible
LoRI-D Full KV, $N=5{,}943$
0.1	2.0	-21.6	0.0	Yes
0.25	4.5	-21.6	0.0	Yes
0.5	18.0	-21.2	0.0	Yes
1.0	51.0	-13.6	12.0	Yes
LoRI-D DroidSpeak, $N=5{,}815$

Appendix

Table 7: XSTest development candidates for the maps used in the timing and additional QA experiments. Unsafe refusal changes are relative to raw reuse. Safe refusal and maximum safe-subtype changes are relative to Native (all in pp). Bold

\alpha

is selected. Each map uses 64 fitting texts.

N

counts sampled positions per head. These are string-refusal development results, not independent test sensitivity.

Model pair	Reuse	Prompt	Prefix	Suffix
LoRI-D	Full KV	1,833/7,262.20/7,530	—	—
	DroidSpeak	1,833/7,261.30/7,529	7,210.04	42/51.26/80
Llama-3-8B	Full KV	1,814/7,243.20/7,511	—	—
	DroidSpeak	1,814/7,242.30/7,510	7,191.04	42/51.26/80

Appendix

Table 8: Token lengths for 50 timing prompts. Prompt and Suffix report min/mean/max. Prefix reports the mean. Full KV prefix and suffix lengths were not recorded separately.

LoRI-D
Config.	HotpotQA			2Wiki			MultiField
Config.	Native	Reuse	VReparo	Native	Reuse	VReparo	Native	Reuse	VReparo
Full KV	28.90	22.48	21.13	27.85	25.19	22.03	50.71	48.71	46.42
DroidSpeak	27.36	27.90	28.92	27.85	27.40	26.99	50.75	51.38	49.55
Llama-3-8B
Full KV	49.64	9.64	11.54	34.27	10.60	11.59	44.45	13.28	13.50

Appendix

Table 9: QA token F1 (

\times 100

). Fitting data are specified in Appendix B.1 . Native and Reuse are shown separately for each configuration. HotpotQA uses 149 examples for full reuse and 248 for DroidSpeak. 2Wiki and MultiField denote 2WikiMultihopQA and MultiFieldQA-en. Bold marks the best F1 per row and dataset.

Model	Condition	Full KV reuse			DroidSpeak
Model	Condition	ASR	Mean	Refusal	ASR	Mean	Refusal
LoRI-D	Native	29.13	0.1219	11.0	29.13	0.1213	11.0
	Reuse	79.53	0.6431	4.0	82.68	0.6353	4.5
	VReparo	64.09	0.3966	12.6	60.47	0.3916	10.8
Llama-3-8B	Native	7.09	0.0107	10.0	7.09	0.0108	9.5
	Reuse	34.65	0.2505	12.5	32.28	0.1569	8.5

Appendix

Table 10: Safety scores and benign refusal. HarmBench ASR and OR-Bench full Refusal are in %. Mean is the average StrongREJECT score (0–1). VReparo results are averaged over five runs.

Model	Config.	Seed	$\alpha$	HB	SR	F1	Refusal
LoRI-D	Full KV	0	1	44.09	0.3149	20.05	12.5
		1	1	40.94	0.3105	20.37	13.0
		2	1	44.09	0.3156	20.25	13.0
		3	1	42.52	0.3094	20.59	12.5
		4	1	42.52	0.3163	19.94	12.0
	DroidSpeak	0	1	40.16	0.3121	28.84	10.5

Appendix

Table 11: Results for each sampling seed. HB is SDV on 127 HarmBench prompts (%). SR is SDV on 313 StrongREJECT prompts. F1 is HotpotQA token F1 (

\times 100

) on 248 prompts, and Refusal is the full-refusal rate on 200 OR-Bench prompts (%).

KV-Cache Reuse

Key-value (KV) cache reuse lets language models reuse previously computed attention states to cut repeated prefill work and serving latency.

See all 73 papers

RaReCache: Bridging the Gap in Cross-Model KV Cache Reuse via Rank disagreement-based Selective Recomputation
Oct 8, 2026Sreetama Sarkar, Saptarshi Mitra, Sitao Huang +2KV-Cache Reuse LLM Inference Acceleration
University of Southern California · University of California, Irvine · Intel Labs
Alignment Collapse Under KV Cache Quantization: Diagnosis and Mitigation
Jun 1, 2026Bruce Changlong Xu, Adarsh Kumarappan, Mu ZhouLLM Quantization KV Caching
†Stanford University · ‡California Institute of Technology
HijackKV: New Threat in Position-Independent KV Cache Reuse
Jul 22, 2026Yichi Zhang, Zhiqi Wang, Huan Zhang +1KV Caching LLM Security
The Pennsylvania State University · University of Illinois Urbana-Champaign

Oct 8, 2026J/K move · Enter open · S save

RaReCache: Bridging the Gap in Cross-Model KV Cache Reuse via Rank disagreement-based Selective Recomputation

Sreetama Sarkar, Saptarshi Mitra, Sitao Huang, Souvik Kundu+1

University of Southern California · University of California, Irvine · Intel Labs

KV-Cache Reuse LLM Inference Acceleration

Open paper AskPDF arXiv

Cross-model KV-cache reuse remains a key challenge in modern LLM serving. Coding agents and multi-model systems increasingly route a shared context across models: a user may switch models mid-session, or a cascade may escalate a difficult query. Because KV caches contain model-specific representations, each switch typically forces the receiving model to prefill the entire context from scratch. Recent work shows that closed-form linear maps can translate KV caches between models in the same family, but transfer accuracy degrades as the model-size gap widens. In this paper, we establish that these transfer failures are concentrated in a small subset of information-dense tokens. To bridge this gap, we introduce RaReCache, a framework that enables a large target model to decode accurately from a cache prefilled by a much smaller source via selective recomputation. RaReCache identifies these critical positions using a novel rank disagreement metric, scoring each token by the energy of its mapped KV in output directions weakly supported by the calibration data. Across two model families and five benchmarks, on a 23x parameter gap (Qwen3-0.6B to 14B) recomputing just 30% of positions retains 95-99% of the target accuracy, whereas on a 8.8x gap (Llama3-8B to 70B), recomputing 40% retains 96.5% of the target accuracy. RaReCache largely removes sensitivity to source-model size, and achieves up to a 3.04x prefill speedup. For online serving, it handles 1.8x the request throughput of target prefill on a single GPU, and at the target's saturation load, reduces median and 99th-percentile time-to-first-token (TTFT) by 5.0x and 6.4x respectively, with a 30% recompute budget. RaReCache establishes an efficient serving paradigm where small models prefill on behalf of massive targets, enabling large models to recompute only critical tokens, drastically reducing prefill latency.

Abstract

KV-Cache Reuse

Explore similar work

RaReCache: Bridging the Gap in Cross-Model KV Cache Reuse via Rank disagreement-based Selective Recomputation

Alignment Collapse Under KV Cache Quantization: Diagnosis and Mitigation

HijackKV: New Threat in Position-Independent KV Cache Reuse