Evaluating the accuracy of KV cache reuse techniques

Samuel Cestola; Tianxiang Xia; Pengfei Zheng; Weiyan Zheng; Bo Wang; Yi Zhao; Diego Didona

cs.LGSep 25, 2026

Evaluating the accuracy of KV cache reuse techniques

Authors: Samuel Cestola, Tianxiang Xia, Pengfei Zheng, Weiyan Zheng, Bo Wang, Yi Zhao, Diego Didona

Organizations: Huawei Technologies Ltd. · Department of Computer Science, ETH Zurich

Abstract

Position-independent KV cache reuse aims to reduce latency in retrieval-augmented generation by reusing chunk-level KV caches across prompts. We show that current evaluations of KV cache reuse techniques rely on measurements that fail to faithfully capture the loss of accuracy attributable to reuse, often artificially inflating the reported effectiveness. We also show that existing datasets do not exhibit the reuse dynamics needed to thoroughly evaluate such techniques. To address these issues, we propose an evaluation methodology that measures this accuracy loss without ambiguity and we introduce Boxoffice, a tool that programmatically generates evaluation datasets that exercise challenging KV cache reuse patterns.

Figures & tables

Figure 1: KV cache reuse visualization for a query over $3$ chunks. Each gray box is the KV cache of one chunk, the rectangles inside it are its tokens; each chunk is associated with a shape; the shapes inside a token denote the chunks towards which it stores attention; dashed outlines mark tokens whose attention is recomputed. (1) KV caches prefilled in isolation : each cache stores intra-chunk attention only. (2) Full prefill : each cache additionally stores cross-chunk attention to other previous chunks in the prompt. (3) Cold reuse : start from (1) and recover cross-chunk attention through selective recomputation. (4) Warm reuse : start from caches that already store cross-chunk attention to other chunks, some of which coincide with the prompt’s chunks and some of which do not (e.g. moon, stars); additionally recover the cross-chunk attention to the prompt’s chunks through selective recomputation.

		$\rho_{F_{1}}=\mathrm{normF1}(\mathrm{filter})/\mathrm{normF1}(\mathrm{ALL})$									B0 ablation (200 points per dataset)
		qwen			llama			mistral			qwen			llama			mistral
	method / filter	$B_{0}$	$+\text{NC}$	$+\text{LI}$	$B_{0}$	$+\text{NC}$	$+\text{LI}$	$B_{0}$	$+\text{NC}$	$+\text{LI}$	$B_{0}M_{0}$	$B_{0}M_{+}$	$\%F_{1}$	$B_{0}M_{0}$	$B_{0}M_{+}$	$\%F_{1}$	$B_{0}M_{0}$	$B_{0}M_{+}$	$\%F_{1}$
musique	CB	0.76	0.69	0.69	0.91	0.87	0.87	0.70	0.80	0.80	95	26	24.1	108	13	9.5	96	36	29.8
	CB+Q	0.72	0.58	0.58	0.82	0.83	0.83	0.68	0.71	0.71	93	28	27.6	104	17	18.1	91	41	32.5
	FR	0.74	0.64	0.64	0.83	0.70	0.70	0.58	0.67	0.67	98	23	26.4	103	18	16.9	121	11	42.4
	FR+Q	0.73	0.65	0.65	0.75	0.70	0.70	0.63	0.79	0.79	95	26	26.6	97	24	25.4	123	9	36.8

Table 1: Left:

\rho

under the progressive filters of Section 4.1 ;

\rho<1

means measuring

F1

on the unfiltered dataset inflates the reported accuracy. Right:

B0

ablation.

B_{0}M_{0}

counts queries on which baseline and method both fail;

B_{0}M_{+}

counts queries on which the baseline fails but the method succeeds;

\%F_{1}

is the share of the unfiltered aggregate

F_{1}

contributed by

B_{0}M_{+}

	qwen			llama			mistral
method	2wiki	hotpot	musique	2wiki	hotpot	musique	2wiki	hotpot	musique
CacheBlend (no infusion)	0.13	0.21	0.10	0.38	0.35	0.19	0.14	0.19	0.13
Random	0.39	0.52	0.35	0.48	0.55	0.43	0.32	0.48	0.27
Similarity-no-ctx	0.35	0.47	0.37	0.44	0.56	0.41	0.28	0.44	0.19
Context-nogold	0.44	0.56	0.48	0.53	0.66	0.53	0.41	0.62	0.32
Golden-only	0.63	0.67	0.52	0.57	0.66	0.56	0.52	0.59	0.34

Table 2: Mean norm-F1 of different offline cross-chunk attention infusion modes. The provided cross-chunk attention helps significantly over a random baseline only if it aligns with the query’s context or golden chunks.

Dataset	$CC$	$GC$	$g\to g$	$g\to c$	$g\to o$	$c\to g$	$c\to c$	$c\to o$	$C$	$G$	$F$
MuSiQue	0.54	0.69	0.77	0.73	0.70	0.73	0.74	0.70	$12.6\%$	$26.6\%$	$0\%$
2WikiMQA	0.56	0.81	0.78	0.75	0.71	0.75	0.75	0.70	$11.9\%$	$\phantom{0}0.7\%$	$0\%$
HotpotQA	0.84	0.88	0.81	0.78	0.70	0.78	0.78	0.70	$\phantom{0}0.7\%$	$16.7\%$	$0\%$

Table 3: Left:

CC

(

GC

) is the percentage of a query’s context (golden) chunks identifiable by cosine similarity alone, without knowledge of the query. High values indicate that warm offline retrieval can identify the right cross-attention targets purely from chunk embeddings. Middle: mean cosine similarity between chunk pairs

g

: goldens of the query;

c

: non-golden context chunks of the query;

o

: chunks outside the query. Golden chunks and context chunks are clustered and easily separable from chunks of other queries. Right: reuse statistics.

C

: % of unique chunks that appear in

\geq 2

queries.

G

: among the reused chunks, the % that are golden in some query.

F

: % of queries whose entire context has already appeared in earlier queries.

	Llama										Qwen										Mistral
Config	CB	CB+Q	FR	FR+Q	LM	LM+Q	CC-M2	CC-M2+Q	CC-M4	CC-M4+Q	CB	CB+Q	FR	FR+Q	LM	LM+Q	CC-M2	CC-M2+Q	CC-M4	CC-M4+Q	CB	CB+Q	FR	FR+Q	LM	LM+Q	CC-M2	CC-M2+Q	CC-M4	CC-M4+Q
aligned	.10	.31	.11	.37	.98	.95	.17	.55	.19	.51	.39	.09	.44	.40	.97	.94	.17	.38	.18	.25	.20	.18	.33	.53	.98	.94	.14	.42	.21	.36
all-W	.06	.21	.05	.28	.14	.29	.13	.42	.24	.50	.30	.12	.28	.29	.11	.25	.13	.27	.24	.32	.24	.15	.15	.26	.17	.26	.14	.26	.31	.34
all-N	.06	.25	.13	.43	.07	.55	.16	.57	.23	.57	.39	.20	.30	.39	.05	.11	.18	.36	.24	.34	.29	.16	.24	.57	.09	.29	.16	.34	.32	.39
flipped	.09	.25	.06	.35	.00	.09	.11	.41	.23	.54	.41	.18	.23	.28	.00	.06	.15	.29	.28	.34	.25	.11	.18	.34	.00	.02	.13	.23	.30	.30
average	.08	.26	.09	.36	.30	.47	.14	.49	.22	.53	.37	.15	.31	.34	.28	.34	.16	.33	.24	.31	.24	.15	.23	.42	.31	.38	.14	.31	.28	.35

Table 4: norm-F1 on Boxoffice . Within each (model, configuration), the winning method is in bold.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Paper	HotpotQA	2WikiMQA	MuSiQue
CacheBlend ( Yao et al., 2025 )	✓	✓	✓
Cache-Craft ( Agarwal et al., 2025 )	$\times$	✓	✓
CacheClip ( Yang et al., 2025b )	✓	✓	✓
EPIC ( Hu et al., 2024 )	✓	✓	✓
SamKV ( Cao et al., 2025 )	✓	✓	✓
ContextPilot ( Jiang et al., 2025 )	$\times$	$\times$	$\times$

Appendix

Table 5: Use of LongBench-QA datasets in state-of-the-art papers on KV cache reuse.

Configuration	Winner was	Nonwinners were	Description
aligned	$W$	$N$	Every chunk’s cached state matches its role in the eval query. The winner’s cache encodes “I am the maximum”; each nonwinner’s cache encodes “I am not”. Easy case.
all-W	$W$	$W$	The winner’s cache is aligned, but each nonwinner’s cache also encodes “I am the maximum”. The model must override the “winner” signal carried by nonwinner caches. Medium case.
all-N	$N$	$N$	Every nonwinner’s cache is aligned, but the winner’s encodes “I am not the maximum”. The model must identify the winner despite every cache claiming nonwinner status. Medium case.
flipped	$N$	$W$	The signals are fully inverted relative to the eval roles. The model must recover the correct ranking from a reversed set of cached signals. Hard case.

Appendix

Table 6: Configurations of a Boxoffice query, by the role each chunk played during warmup (

W

inner vs

N

onwinner).

Figure 2: Top : similarity between a golden target chunk in query Q of LongBench-QA w.r.t. other golden chunks Q, other non-golden chunks in Q, and other chunks in the dataset. Middle : same as top but done for target non-golden chunks. Bottom : similarity between a chunk in Boxoffice in Q w.r.t. other chunks in Q and other chunks in the dataset.

	Config	CB	CB+Q	FR	FR+Q	LM	LM+Q	CC-M2	CC-M2+Q	CC-M4	CC-M4+Q
Qwen	aligned	$.39{\scriptscriptstyle\,\pm\,.07}$	$.09{\scriptscriptstyle\,\pm\,.05}$	$.44{\scriptscriptstyle\,\pm\,.06}$	$.40{\scriptscriptstyle\,\pm\,.10}$	$\mathbf{.97{\scriptscriptstyle\,\pm\,.03}}$	$.94{\scriptscriptstyle\,\pm\,.05}$	$.17{\scriptscriptstyle\,\pm\,.13}$	$.38{\scriptscriptstyle\,\pm\,.11}$	$.18{\scriptscriptstyle\,\pm\,.09}$	$.25{\scriptscriptstyle\,\pm\,.10}$
	all-W	$.30{\scriptscriptstyle\,\pm\,.06}$	$.12{\scriptscriptstyle\,\pm\,.06}$	$.28{\scriptscriptstyle\,\pm\,.07}$	$.29{\scriptscriptstyle\,\pm\,.09}$	$.11{\scriptscriptstyle\,\pm\,.06}$	$.25{\scriptscriptstyle\,\pm\,.05}$	$.13{\scriptscriptstyle\,\pm\,.10}$	$.27{\scriptscriptstyle\,\pm\,.08}$	$.24{\scriptscriptstyle\,\pm\,.11}$	$\mathbf{.32{\scriptscriptstyle\,\pm\,.07}}$
	all-N	$.39{\scriptscriptstyle\,\pm\,.06}$	$.20{\scriptscriptstyle\,\pm\,.07}$	$.30{\scriptscriptstyle\,\pm\,.06}$	$\mathbf{.39{\scriptscriptstyle\,\pm\,.11}}$	$.05{\scriptscriptstyle\,\pm\,.03}$	$.11{\scriptscriptstyle\,\pm\,.03}$	$.18{\scriptscriptstyle\,\pm\,.09}$	$.36{\scriptscriptstyle\,\pm\,.11}$	$.24{\scriptscriptstyle\,\pm\,.09}$	$.34{\scriptscriptstyle\,\pm\,.07}$
	flipped	$\mathbf{.41{\scriptscriptstyle\,\pm\,.12}}$	$.18{\scriptscriptstyle\,\pm\,.07}$	$.23{\scriptscriptstyle\,\pm\,.07}$	$.28{\scriptscriptstyle\,\pm\,.06}$	$.00{\scriptscriptstyle\,\pm\,.00}$	$.06{\scriptscriptstyle\,\pm\,.04}$	$.15{\scriptscriptstyle\,\pm\,.08}$	$.29{\scriptscriptstyle\,\pm\,.09}$	$.28{\scriptscriptstyle\,\pm\,.11}$	$.34{\scriptscriptstyle\,\pm\,.10}$
	average	$\mathbf{.37{\scriptscriptstyle\,\pm\,.05}}$	$.15{\scriptscriptstyle\,\pm\,.03}$	$.31{\scriptscriptstyle\,\pm\,.04}$	$.34{\scriptscriptstyle\,\pm\,.04}$	$.28{\scriptscriptstyle\,\pm\,.02}$	$.34{\scriptscriptstyle\,\pm\,.02}$	$.16{\scriptscriptstyle\,\pm\,.05}$	$.33{\scriptscriptstyle\,\pm\,.05}$	$.24{\scriptscriptstyle\,\pm\,.04}$	$.31{\scriptscriptstyle\,\pm\,.04}$
Llama	aligned	$.10{\scriptscriptstyle\,\pm\,.03}$	$.31{\scriptscriptstyle\,\pm\,.09}$	$.11{\scriptscriptstyle\,\pm\,.05}$	$.37{\scriptscriptstyle\,\pm\,.07}$	$\mathbf{.98{\scriptscriptstyle\,\pm\,.02}}$	$.95{\scriptscriptstyle\,\pm\,.03}$	$.17{\scriptscriptstyle\,\pm\,.11}$	$.55{\scriptscriptstyle\,\pm\,.11}$	$.19{\scriptscriptstyle\,\pm\,.06}$	$.51{\scriptscriptstyle\,\pm\,.08}$

Appendix

Table 7: Norm-F1 on Boxoffice at

R{=}0.15

, mean

{\pm}

sample std across 10 generated datasets (seeds

7,11,13,17,19,23,29,31,47,73

). Within each (model, configuration), the winning method is in bold.

	Config	CB	CB+Q	FR	FR+Q	LM	LM+Q	CC-M2	CC-M2+Q	CC-M4	CC-M4+Q
Qwen	aligned	$.03{\scriptscriptstyle\,\pm\,.02}$	$.07{\scriptscriptstyle\,\pm\,.04}$	$.36{\scriptscriptstyle\,\pm\,.05}$	$.34{\scriptscriptstyle\,\pm\,.06}$	$.95{\scriptscriptstyle\,\pm\,.02}$	$\mathbf{.97{\scriptscriptstyle\,\pm\,.02}}$	$.13{\scriptscriptstyle\,\pm\,.13}$	$.18{\scriptscriptstyle\,\pm\,.12}$	$.12{\scriptscriptstyle\,\pm\,.07}$	$.10{\scriptscriptstyle\,\pm\,.05}$
	all-W	$.04{\scriptscriptstyle\,\pm\,.03}$	$.07{\scriptscriptstyle\,\pm\,.03}$	$\mathbf{.23{\scriptscriptstyle\,\pm\,.05}}$	$.23{\scriptscriptstyle\,\pm\,.04}$	$.11{\scriptscriptstyle\,\pm\,.05}$	$.18{\scriptscriptstyle\,\pm\,.05}$	$.10{\scriptscriptstyle\,\pm\,.07}$	$.12{\scriptscriptstyle\,\pm\,.08}$	$.21{\scriptscriptstyle\,\pm\,.08}$	$.20{\scriptscriptstyle\,\pm\,.06}$
	all-N	$.07{\scriptscriptstyle\,\pm\,.04}$	$.09{\scriptscriptstyle\,\pm\,.04}$	$\mathbf{.28{\scriptscriptstyle\,\pm\,.04}}$	$.28{\scriptscriptstyle\,\pm\,.06}$	$.04{\scriptscriptstyle\,\pm\,.03}$	$.08{\scriptscriptstyle\,\pm\,.03}$	$.14{\scriptscriptstyle\,\pm\,.08}$	$.15{\scriptscriptstyle\,\pm\,.07}$	$.15{\scriptscriptstyle\,\pm\,.07}$	$.18{\scriptscriptstyle\,\pm\,.09}$
	flipped	$.11{\scriptscriptstyle\,\pm\,.06}$	$.11{\scriptscriptstyle\,\pm\,.06}$	$.19{\scriptscriptstyle\,\pm\,.05}$	$\mathbf{.19{\scriptscriptstyle\,\pm\,.05}}$	$.00{\scriptscriptstyle\,\pm\,.00}$	$.00{\scriptscriptstyle\,\pm\,.00}$	$.12{\scriptscriptstyle\,\pm\,.08}$	$.14{\scriptscriptstyle\,\pm\,.07}$	$.19{\scriptscriptstyle\,\pm\,.09}$	$.19{\scriptscriptstyle\,\pm\,.08}$
	average	$.07{\scriptscriptstyle\,\pm\,.02}$	$.09{\scriptscriptstyle\,\pm\,.02}$	$.27{\scriptscriptstyle\,\pm\,.03}$	$.26{\scriptscriptstyle\,\pm\,.02}$	$.28{\scriptscriptstyle\,\pm\,.02}$	$\mathbf{.31{\scriptscriptstyle\,\pm\,.01}}$	$.12{\scriptscriptstyle\,\pm\,.04}$	$.15{\scriptscriptstyle\,\pm\,.04}$	$.17{\scriptscriptstyle\,\pm\,.03}$	$.17{\scriptscriptstyle\,\pm\,.03}$
Llama	aligned	$.11{\scriptscriptstyle\,\pm\,.06}$	$.12{\scriptscriptstyle\,\pm\,.06}$	$.09{\scriptscriptstyle\,\pm\,.03}$	$.09{\scriptscriptstyle\,\pm\,.05}$	$\mathbf{.98{\scriptscriptstyle\,\pm\,.02}}$	$.96{\scriptscriptstyle\,\pm\,.03}$	$.13{\scriptscriptstyle\,\pm\,.11}$	$.21{\scriptscriptstyle\,\pm\,.11}$	$.10{\scriptscriptstyle\,\pm\,.04}$	$.16{\scriptscriptstyle\,\pm\,.07}$

Appendix

Table 8: norm-F1 on Boxoffice at

R{=}0.05

, mean

{\pm}

sample std across 10 generated datasets (seeds

7,11,13,17,19,23,29,31,47,73

). Within each (model, configuration), the winning method is in bold.

Figure 3: LMCache (

M=1

) and CacheCraft (

M=2,4)

performance and query-level orientation breakdown on the ten Boxoffice seeds at recompute ratio

R{=}0.15

. The top row shows mean F1 across all three models for the aligned and flipped configurations as the number of stored variants grows from

M{=}1

M{=}4

. The bottom row classifies each evaluation query by the orientation of the selected versions across all 10 chunks: fully aligned means every selected winner/nonwinner version has the role-consistent orientation, mixed means the query contains both properly oriented (aligned) and misoriented (flipped) selected versions, and fully flipped means every selected version is misoriented (flipped).

Figure 4: Evaluation queries chunk reuse structure on ten Boxoffice seeds. The left panel shows the distribution of eval query frequencies per unique chunk, split into winner chunks and nonwinner chunks. The right panel shows the winner’s position among the 10 candidate chunks.

Figure 5: Warm offline retrieval conditioning composition on the ten seeds Boxoffice data as done in

FR

. The stacked bars show the average composition of the retrieved warm neighbors for

k{=}5

, split into smaller versus larger box-office movies relative to the target chunk. Winner chunks by construction have relatively high box office value, so they retrieve mostly smaller neighbors in every configuration. Nonwinner chunks have a more balanced neighborhood. In both cases, the KV cache encodes mixed information about the ranking of a chunk with respect to its neighbors.

$\tau$	CB	CB+Q	FR	FR+Q	CB	CB+Q	FR	FR+Q
	$+$ NC				$+$ LI (meaningful subset)
0.00	0.81	0.74	0.76	0.78	0.80	0.72	0.73	0.77
0.05	0.81	0.74	0.76	0.78	0.80	0.72	0.73	0.77
0.10	0.82	0.74	0.77	0.78	0.80	0.73	0.73	0.77
0.15	0.82	0.75	0.76	0.78	0.81	0.73	0.74	0.77
0.20 (paper)	0.82	0.75	0.77	0.78	0.81	0.73	0.74	0.77

Appendix

Table 9: Sensitivity of

\rho_{F_{1}}

(mean over the

9

dataset

\times

model cells,

R{=}0.15

) to the no-context threshold

\tau

that defines the NC filter. Left: after the NC filter; right: after NC and LI, i.e., on the meaningful subset. The row for

B_{0}

alone does not depend on

\tau

: CB 0.87, CB+Q 0.85, FR 0.83, FR+Q 0.83. The paper uses

\tau{=}0.2

				$\tau{=}0.2$		$+$ LI at other $\tau$
dataset	model	all	$B_{0}$	$+$ NC	$+$ LI	$0.1$	$0.3$	$0.5$
MuSiQue	Qwen	200	79	47	47	44	52	69
	Llama	200	79	57	57	52	60	70
	Mistral	200	68	55	55	55	55	64
2WikiMQA	Qwen	200	96	42	38	37	41	51
	Llama	200	114	65	46	43	50	57

Appendix

Table 10: Number of queries that survive each filter on LongBench-QA (

200

queries per dataset).

B_{0}

is independent of

\tau

;

+

NC and

+

LI are reported at the paper’s

\tau{=}0.2

, and the size of the meaningful subset (

+

LI) is additionally reported for

\tau\in\{0.1,0.3,0.5\}

method	$B_{0}$	$+$ NC	$+$ LI	$+$ NC	$+$ LI	$B_{0}M_{0}$	$B_{0}M_{+}$	$\%F_{1}$
	$\tau{=}0.2$			$\tau{=}0.5$		$B_{0}$ ablation
CB	0.86	0.82	0.82	0.82	0.82	82	18	13.8
CB+Q	0.76	0.72	0.72	0.71	0.71	73	27	23.6
FR	0.91	0.86	0.86	0.89	0.89	81	19	9.3
FR+Q	0.88	0.81	0.81	0.85	0.85	76	24	11.7

Appendix

Table 11: Qwen3-32B on MuSiQue at

R{=}0.15

(

200

queries;

100

survive

B_{0}

68

survive

+

NC and

+

LI at

\tau{=}0.2

). Left:

\rho_{F_{1}}

under the progressive filters at

\tau{=}0.2

and at

\tau{=}0.5

. Right:

B_{0}

ablation, as in Table 1 .

Config	CB	CB+Q	LM	LM+Q	CC-M2	CC-M2+Q	CC-M4	CC-M4+Q
aligned	.26	.08	1.03	1.04	.10	.13	.19	.27
all-W	.28	.07	.11	.14	.21	.21	.39	.42
all-N	.35	.05	.08	.09	.16	.19	.37	.33
flipped	.33	.05	.00	.00	.14	.15	.41	.34
average	.31	.06	.31	.32	.15	.17	.34	.34

Appendix

Table 12: norm-F1 of Qwen3-32B on the Boxoffice argmax template at

R{=}0.15

, per warmup configuration. CB and LM are averaged over 6 generated datasets (seeds

7,11,13,17,19,23

); the remaining methods over the 3 datasets on which they were run (seeds

7,11,13

). FR is not reported because its offline pool was not built for this model. The datasets were filtered with the three

8

B models, so the Qwen3-32B full-prefill baseline is not exactly

1

(mean

F_{1}

between

0.94

and

0.98

per dataset) and norm-F1 can slightly exceed

1

	Llama										Qwen										Mistral
Config	CB	CB+Q	FR	FR+Q	LM	LM+Q	CC-M2	CC-M2+Q	CC-M4	CC-M4+Q	CB	CB+Q	FR	FR+Q	LM	LM+Q	CC-M2	CC-M2+Q	CC-M4	CC-M4+Q	CB	CB+Q	FR	FR+Q	LM	LM+Q	CC-M2	CC-M2+Q	CC-M4	CC-M4+Q
aligned	.69	.40	.08	.31	.51	.68	.35	.65	.33	.72	.27	.09	.11	.34	.56	.65	.21	.36	.30	.35	.23	.07	.10	.14	.66	.72	.29	.42	.35	.43
all-W	.72	.49	.10	.31	.15	.60	.19	.65	.31	.61	.25	.12	.15	.37	.11	.15	.18	.29	.32	.28	.26	.09	.14	.19	.20	.31	.22	.28	.36	.42
all-N	.68	.38	.08	.23	.13	.66	.28	.66	.35	.70	.34	.17	.12	.42	.11	.39	.26	.34	.31	.39	.24	.08	.12	.18	.18	.27	.26	.36	.40	.45
flipped	.64	.44	.07	.24	.05	.52	.19	.65	.30	.59	.25	.16	.13	.30	.00	.09	.18	.31	.32	.32	.26	.08	.09	.17	.04	.08	.26	.33	.36	.37
average	.68	.43	.08	.27	.21	.61	.25	.65	.33	.66	.28	.13	.13	.36	.20	.32	.21	.33	.31	.33	.24	.08	.11	.17	.27	.35	.26	.35	.37	.42

Appendix

Table 13: norm-F1 on the Boxoffice genre-uniqueness template at

R{=}0.15

, mean over 4 generated datasets (seeds

7,11,13,17

;

40

evaluation queries per configuration per seed). Within each (model, configuration), the winning method is in bold.

	Llama										Qwen										Mistral
Config	CB	CB+Q	FR	FR+Q	LM	LM+Q	CC-M2	CC-M2+Q	CC-M4	CC-M4+Q	CB	CB+Q	FR	FR+Q	LM	LM+Q	CC-M2	CC-M2+Q	CC-M4	CC-M4+Q	CB	CB+Q	FR	FR+Q	LM	LM+Q	CC-M2	CC-M2+Q	CC-M4	CC-M4+Q
aligned	.62	.59	.09	.21	.24	.52	.15	.40	.23	.46	.50	.29	.14	.29	.42	.56	.27	.47	.29	.46	.39	.08	.19	.25	.28	.56	.24	.56	.23	.49
all-W	.60	.56	.12	.20	.15	.39	.18	.45	.23	.53	.52	.26	.15	.23	.14	.21	.28	.49	.31	.43	.42	.08	.14	.26	.09	.31	.31	.54	.36	.55
all-N	.67	.68	.14	.23	.10	.38	.14	.49	.14	.51	.48	.25	.06	.27	.10	.44	.22	.42	.29	.44	.43	.08	.16	.31	.13	.46	.22	.48	.35	.48
flipped	.63	.59	.22	.23	.06	.34	.19	.52	.21	.54	.47	.31	.11	.21	.01	.08	.13	.32	.22	.50	.46	.08	.16	.31	.03	.32	.19	.52	.39	.56
average	.63	.61	.14	.22	.14	.40	.17	.46	.21	.51	.49	.28	.11	.25	.17	.32	.22	.43	.28	.46	.42	.08	.17	.28	.13	.41	.24	.52	.33	.52

Appendix

Table 14: norm-F1 on the Boxoffice pivot-join template at

R{=}0.15

, mean over 4 generated datasets (seeds

7,11,13,17

;

40

evaluation queries per configuration per seed). Within each (model, configuration), the winning method is in bold.

KV-Cache Reuse

Key-value (KV) cache reuse lets language models reuse previously computed attention states to cut repeated prefill work and serving latency.

See all 73 papers

Sep 9, 2026

KVShareArena: KV-Cache Reuse Across Contexts and Model Checkpoints

Reusing key-value (KV) caches speeds up LLM inference by avoiding repeated computation on shared text. Standard prefix caching reuses a KV cache only when the LLM is the same and all preceding text is identical, but real workloads often break both conditions: RAG systems place different documents before the same one, agents with different system prompts read the same file or tool output, multi-agent workflows use specialized LLMs on shared material, and an updated model reads documents cached by its previous version. Because KV caches depend on both the preceding text and the model weights, direct reuse can reduce answer quality. Many methods repair or compress the reused cache, but each paper uses its own tasks, models, and cost measures, and existing benchmarks mainly test long-context processing or reuse of an unchanged prefix. We introduce KVShareArena, a benchmark and open evaluation framework for comparing them under the same conditions. KVShareArena has (1) reuse tests on 2,150 questions from three QA datasets, where the preceding text, the cache-writing LLM, or both change while the answering LLM and input stay fixed; (2) five dense and mixture-of-experts LLMs (4B-30B) and six LLM pairs where one version of an LLM reads caches written by another, for 33 model-dataset settings; (3) 11 repair and compression methods from six method classes; (4) four evaluation perspectives: answer quality, prefill computation, KV-cache memory, and latency; and (5) a common interface for adding new methods and an interactive leaderboard. Experiments yield two findings. First, both the quality loss from reuse and which repairs help depend on the LLM, even between two 8B models. Second, most repairs keep their quality when another LLM version wrote the cache, but a trained repair adapter loses quality in 12 of 18 pair-dataset tests. Code and data: https://github.com/xishi404/KVShare-Arena

Xi Shi, Mengxin Zheng, Qian Lou

University of Central Florida

KV Caching LLM Inference Acceleration Prefix Caching KV-Cache Reuse

Ask PDF

Mar 3, 2026

An experimental study of KV cache reuse strategies in chunk-level caching systems

Retrieval-augmented generation improves large language models' accuracy by adding relevant retrieved text to the prompt. Chunk level caching (CLC) accelerates inference by precomputing KV caches for these retrieved chunks and reusing them. However, these caches miss cross-attention dependencies between chunks, which can reduce output quality. Several methods try to improve CLC accuracy using different techniques. We make two main contributions. First, we show that existing CLC approaches have fundamental limitations that limit their accuracy or their applicability. We back this conclusion with an extensive CLC system experimental evaluation. Second, we observe that existing CLC techniques are complementary. We leverage this insight to propose a new CLC design that carefully combines them and achieves better accuracy.

Samuel Cestola, Tianxiang Xia, Zheng Weiyan +2

Huawei, Switzerland. · Huawei, China.

KV-Cache Reuse LLM Inference Acceleration Retrieval-Augmented Generation

Ask PDF

Oct 9, 2026

KV-Repurpose: A Testbed for Evaluating Non-Prefix KV-Cache Reuse

KV-cache reuse in large language model serving is commonly limited to exact prompt prefixes, preventing reuse when the same retrieved document appears after different system prompts, conversation histories, or dynamic metadata. We present KV-Repurpose, an open-source, vLLM-based system for developing and evaluating segment-wise KV-cache reuse beyond exact prefixes. The system combines cache-aware segment scheduling, prompt construction, capacity management, workload generation, and runtime monitoring. Experiments on long-context benchmarks show that naive segment reuse largely preserves output quality for single-document question answering and summarization, while exposing clear limitations in cross-document reasoning. Under a Zipfian RAG-style workload in which 20% of requests share a fixed prefix, KV-Repurpose achieves a substantially higher token-level cache hit rate than prefix caching at the same 30 GB cache capacity, 63.9% versus 3.3%, resulting in lower TTFT. Even with 1 TB of cache capacity, prefix caching reaches only a 17.9% hit rate in this workload. Source code is available under the Apache-2.0 license at https://github.com/taiga333/eval-kv-repurpose.

Taiga Ikeda, Daisuke Miyashita, Jun Deguchi

Kioxia Corporation, Japan

KV-Cache Reuse KV-Cache Management LLM Serving

Ask PDF

Abstract

KV-Cache Reuse

Explore similar work

KVShareArena: KV-Cache Reuse Across Contexts and Model Checkpoints

An experimental study of KV cache reuse strategies in chunk-level caching systems

KV-Repurpose: A Testbed for Evaluating Non-Prefix KV-Cache Reuse