Close Enough is Nowhere Near: Fidelity-Consumability Decoupling in KV Caches

Adam Deng; Shicheng Liu; Yada Zhu

cs.LGOct 15, 2026

Close Enough is Nowhere Near: Fidelity-Consumability Decoupling in KV Caches

Authors: Adam Deng, Shicheng Liu, Yada Zhu

Organizations: MIT ORC · MIT CSAIL · IBM Research

Abstract

Multi-agent LLM systems communicate in text, which is lossy and wasteful: the sender throws away the reading it already did. A growing literature instead transmits the KV cache, the natural carrier of that reading. We study the most appealing form of the idea: both models stay frozen and the sender's cache is substituted for the reader's with nothing trained. It does not work, and the obstruction belongs to the reader. A frozen transformer reads only caches close to the one it would have written itself, a region we call its tolerable set. We show that geometric similarity to that cache does not predict whether the reader can use it, and the failure is not gradual, so a more faithful substitute is not the remedy it appears to be. We explain this phenomenon with competitive attention. Experimentally, we test eight diverse substitution methods against two measured anchors: an uninformative cache baseline and the reader's own accuracy from having read the passage. All sit at the baseline. That turns a long tail of separate negative results into one outcome with a stated condition for overturning it. A different design does work: if the reader keeps its own cache and attends over the sender's as extra context, adapting only its queries, it beats its own baseline, and also clears the baseline with eight senders pooled where undirected attention over the same union falls below it. We offer this as a characterization on one probe, not a leaderboard claim. Our central contribution is a redirection: pursuing faithful substitute caches is the wrong target; adding a cache to what the reader already has is the promising one.

Figures & tables

Figure 1: Cosine similarity does not predict consumability. (a) Isotropic noise on the reader’s own cache falls below the floor by cosine $0.995$ , while the genuine cache the same frozen model produces on the unmasked passage transfers at $0.445$ from a lower cosine of $0.9949$ , an $89\times$ gap at matched fidelity, ordered inversely. (b) Retrieval from a bank of 6000 genuine keys: tolerance is flat in neighbor rank $k$ and collapses in the mixing weight. The nearest real key among six thousand fails.

Figure 2: Two explanations that do not survive measurement. (a) Corrupting the first $\ell$ of 28 layers: off-manifold damage exhausts its budget at $\ell{=}1$ and is flat thereafter, so nothing accumulates over depth, while on-manifold substitution at matched cosine survives eight layers and decays gradually. (b) The off-/on-manifold readout-gain ratio is $1.47\times$ and the logit-stage gain falls below one, so the effect is not constructed from amplification either.

Regime	Production method	Transfer	Note
Substitution	freeform regression	$0.000$	trained, low-rank
Substitution	manifold-loss training	$0.010$	trained toward the key manifold
Substitution	by-construction (column space)	$0.010$	in the column space of $W_{K}$
Substitution	row-space target	$0.085$	trained; see note on spread
Substitution	worst-case hinge	$0.080$	trained, per-layer hinge
Substitution	high-capacity per-layer MLP	$0.200$	best substitution; at the floor

Table 1: The sweep. Substitution methods built from the sender’s side remain at or below the floor regardless of capacity, objective, or whether the produced cache is constrained to lie in the reader’s key column space. Both augmentation methods clear it, and the adapted variant reaches the native-cache reference. Floor

0.180

, native-cache reference

0.665

n{=}200

throughout.

Figure 3: The sweep over how the shared cache is produced and where it is placed. (a) Every substitution method sits at or below the floor (0.180), including an oracle supplied a real key from a bank of 6000. Augmentation escapes, reaching the native-cache reference (0.665). (b) Scaling the sender pool. Details in Table 1 and Appendix 5 .

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Isotropic noise on own cache			Genuine fact-bearing cache
target	achieved cos	transfer	weight	achieved cos	transfer
— (no noise)	$1.0000$	$0.180$	$0.000$	$1.0000$	$0.180$
$0.9999$	$0.9999$	$0.180$	$0.10$	$0.99995$	$0.190$
$0.999$	$0.9990$	$0.120$	$0.25$	$0.99968$	$0.200$
$0.995$	$0.9950$	$\mathbf{0.005}$	$0.50$	$0.99873$	$0.215$
$0.99$	$0.9900$	$0.005$	$0.75$	$0.99713$	$0.325$

Appendix

Table 2: Matched fidelity, opposite outcomes. Noise calibrated to a target cosine (left) against the genuine cache the same frozen model produces on the unmasked passage (right). Floor

0.180

, native-cache reference

0.665

n{=}200

. The genuine cache at cosine

0.9949

transfers

89\times

better than noise at the higher cosine

0.995

; cosine similarity orders the two conditions inversely.

layers corrupted $\ell$	0	1	2	3	4	6	8	12	16	20	24	28
off-manifold (noise)	$.665$	$\mathbf{.030}$	$.035$	$.035$	$.050$	$.045$	$.030$	$.035$	$.025$	$.010$	$.025$	$.005$
on-manifold (genuine)	$.665$	$\mathbf{.695}$	$.740$	$.745$	$.705$	$.685$	$.615$	$.530$	$.300$	$.025$	$.005$	$.000$

Appendix

Table 3: Corrupting the first

\ell

of 28 layers, with the remainder bit-exact. Both corruptions are set to the same target key cosine,

0.822

. Off-manifold noise exhausts its budget at

\ell{=}1

and is flat thereafter; genuine on-manifold keys are harmless at small

\ell

and decay gradually. Floor

0.180

, native-cache reference

0.665

n{=}200

	$\alpha{=}0.0$	$0.1$	$0.25$	$0.5$	$0.75$	$1.0$
$k{=}1$	$0.180$	$0.185$	$0.110$	$0.005$	$0.000$	$0.000$
$k{=}2$	$0.180$	$0.180$	$0.110$	$0.020$	$0.010$	$0.005$
$k{=}4$	$0.180$	$0.205$	$0.120$	$0.010$	$0.005$	$0.000$
$k{=}8$	$0.180$	$0.175$	$0.120$	$0.005$	$0.000$	$0.000$
$k{=}16$	$0.180$	$0.145$	$0.135$	$0.020$	$0.015$	$0.010$

Appendix

Table 4: Retrieval from a bank of 6000 genuine keys per layer. Rows index neighbor rank

k

, columns the interpolation weight toward that neighbor. Every candidate is a real key the reader itself produced and therefore lies on the reader’s manifold. Tolerance is flat in

k

and collapses in

\alpha

. Floor

0.180

, native-cache reference

0.665

n{=}200

	$N{=}1$	$N{=}2$	$N{=}4$	$N{=}8$
raw cross-attention	$0.555$	$0.375$	$0.155$	$0.070$
vs. floor $0.180$	clears	clears	below	below
with query-side adaptation	$\mathbf{0.785}$	$\mathbf{0.735}$	$\mathbf{0.550}$	$\mathbf{0.395}$
strongest control	$0.145$	$0.190$	$0.240$	$0.230$
control-subtracted lift	$+0.640$	$+0.545$	$+0.310$	$+0.165$

Appendix

Table 5: Scaling the sender pool. One sender holds the answer;

N-1

are distractors. Undirected attention degrades below the floor by

N{=}4

; the adapted bridge continues to clear it at

N{=}8

. Controls rise with

N

, so the control-subtracted lift is the appropriate figure. Floor

0.180

, native-cache reference

0.665

n{=}200

KV-Cache Reuse

Key-value (KV) cache reuse lets language models reuse previously computed attention states to cut repeated prefill work and serving latency.

See all 73 papers

KVShareArena: KV-Cache Reuse Across Contexts and Model Checkpoints
Sep 9, 2026Xi Shi, Mengxin Zheng, Qian LouKV Caching LLM Inference Acceleration
University of Central Florida
ATTUNER: Recomputation-Free Agentic Context KV Reuse
Sep 29, 2026Xinghao Chen, Junnan Dong, Cai Ke +8KV-Cache Reuse LLM Inference Acceleration
EIT-NLP Lab, Eastern Institute of Technology, Ningbo · The Hong Kong Polytechnic University · Tencent Youtu Lab
KV Packet: Recomputation-Free Context-Independent KV Caching for LLMs
Apr 14, 2026Chuangtao Chen, Grace Li Zhang, Xunzhao Yin +3KV Caching KV-Cache Reuse
Technical University of Munich Munich, Germany · Technical University of Darmstadt Darmstadt, Germany · Zhejiang University Hangzhou, China +1

Sep 9, 2026J/K move · Enter open · S save

KVShareArena: KV-Cache Reuse Across Contexts and Model Checkpoints

Xi Shi, Mengxin Zheng, Qian Lou

University of Central Florida

KV Caching LLM Inference Acceleration Prefix Caching KV-Cache Reuse

Open paper AskPDF arXiv

Reusing key-value (KV) caches speeds up LLM inference by avoiding repeated computation on shared text. Standard prefix caching reuses a KV cache only when the LLM is the same and all preceding text is identical, but real workloads often break both conditions: RAG systems place different documents before the same one, agents with different system prompts read the same file or tool output, multi-agent workflows use specialized LLMs on shared material, and an updated model reads documents cached by its previous version. Because KV caches depend on both the preceding text and the model weights, direct reuse can reduce answer quality. Many methods repair or compress the reused cache, but each paper uses its own tasks, models, and cost measures, and existing benchmarks mainly test long-context processing or reuse of an unchanged prefix. We introduce KVShareArena, a benchmark and open evaluation framework for comparing them under the same conditions. KVShareArena has (1) reuse tests on 2,150 questions from three QA datasets, where the preceding text, the cache-writing LLM, or both change while the answering LLM and input stay fixed; (2) five dense and mixture-of-experts LLMs (4B-30B) and six LLM pairs where one version of an LLM reads caches written by another, for 33 model-dataset settings; (3) 11 repair and compression methods from six method classes; (4) four evaluation perspectives: answer quality, prefill computation, KV-cache memory, and latency; and (5) a common interface for adding new methods and an interactive leaderboard. Experiments yield two findings. First, both the quality loss from reuse and which repairs help depend on the LLM, even between two 8B models. Second, most repairs keep their quality when another LLM version wrote the cache, but a trained repair adapter loses quality in 12 of 18 pair-dataset tests. Code and data: https://github.com/xishi404/KVShare-Arena

Abstract

KV-Cache Reuse

Explore similar work

KVShareArena: KV-Cache Reuse Across Contexts and Model Checkpoints

ATTUNER: Recomputation-Free Agentic Context KV Reuse

KV Packet: Recomputation-Free Context-Independent KV Caching for LLMs