Rehearse Everything, Remember Nothing: Attic-KV Rehearses What Will Be Read

Zhiyun Shi

cs.CLOct 8, 2026

Rehearse Everything, Remember Nothing: Attic-KV Rehearses What Will Be Read

Authors: Zhiyun Shi

Organizations: Nanyang Technological University Singapore

Abstract

Many key-value (KV) caches are compressed before anyone knows what will be asked of them: a document cached for retrieval, a prompt prefix shared across requests, the memory of a long conversation. The prevailing approach scores KV entries by rehearsal: the model rereads the context and keeps the entries it attends to, assuming that the more completely a cache rehearses its context, the better it remembers it. We show that under tight budgets this assumption backfires: rehearse everything, remember nothing. At a 3% keep ratio, rereading the whole context keeps 31.5 of 96.5 points on RULER, and on LongBench's natural-text tasks it falls below methods that rehearse nothing at all. The cause is that a cache keeps what it rehearses: rereading spreads the budget across the whole context, so the answer's own entries survive at little more than chance. Like a student before an exam, a cache remembers more by testing itself than by rereading. Two principles follow: rehearse what will be read, and rehearse as much as there is. We instantiate them as Attic-KV (Attic for short), a training-free rehearsal in which the model quizzes itself with question-answer pairs that quote the context, alongside anchor tokens in a content-adaptive amount. Changing only the rehearsal lifts three hosts that score it in three different ways: Attic alone is the best training-free method in all eight settings we test on RULER and LongBench's natural-text tasks, and plugged into the gradient-based KVgrad and the trained RestoreKV+, it raises them by up to 17.1 and 28.1 points. Its advantage grows as the budget shrinks, reaching 41.9 points over full rereading at a 3% keep ratio, and it compresses faster than rereading the whole context.

Figures & tables

Figure 1. Rehearse everything, remember nothing. A document, here a Wikipedia page, is compressed once, to 5% of its KV cache, before any question arrives. Each method feeds a rehearsal $R$ after the page and keeps the 5% of entries that $R$ attends to most, so a compressed cache, like a small attic, keeps what it rehearses. KVzip+ rehearses the whole page and keeps mostly prose; KV 2 rehearses a fixed share of isolated anchors and keeps only some facts; Attic rehearses self-generated question–answer pairs and anchor sentences, as much as the page holds, and keeps the facts readers later ask about. Answers are verbatim outputs of Qwen3-8B on this LooGLE page. ( Right ) RULER-4K score versus keep ratio on Qwen3-8B: the gap opens at tight budgets and reaches 41.9 points over KVzip+ at 3%.

Figure 2. Rehearse everything, remember nothing. Share of each answer’s KV entries, over all layers and heads, that the cache keeps (Qwen3-8B, RULER-4K development split, the eight needle tasks, keep ratio 5%). Whole-document rehearsal keeps the answer no more often than any other token (dashed line); Attic keeps it on the tasks it answers, and an oracle that rehearses the gold answer shows how much a 5% cache can hold. Below each task: its score with whole-document rehearsal

\rightarrow

with Attic .

Figure 3. A cache keeps what it rehearses (Qwen3-8B, RULER-4K development split, keep ratio 5%). ( Left ) Share of the document that a rehearsal covers (light) and share of the kept KV entries, over all layers and heads, that fall on those positions (dark), for a semantic rehearsal (fact list) and a geometric one (KV 2 anchors). ( Right ) Rehearsing only the fact list: needle coverage of the list against the score. MK/MQ/MV: multi-key/-query/-value needles; S: single needle; CWE: common words; QA: question answering.

Figure 4. Overview of Attic . ( Left ) Attic builds the rehearsal $R$ , the only part of compression it changes: ❶ the keys that stand out from the chunk’s average key are the anchors; ❷ the anchors are expanded to $m_{j}$ tokens, the length of the sentences that hold them, so a prose chunk is rehearsed briefly and a table of facts at length (Principle 2); ❸ the model writes the questions a reader is likely to ask and answers them by quoting the page (Principle 1). ( Right ) The host’s scorer $\phi$ reads the rehearsal back over the cached chunk, scores every KV entry by the attention it receives (heat map illustrative), and keeps the top $r$ . Text is from the LooGLE page of Figure 1 ; the Q&A pair is the first one the model wrote for it. The same rehearsal plugs into KV 2 , KVgrad and RestoreKV+, and nothing is trained.

			Qwen3-8B				Llama-3.1-8B		Qwen3-4B	Qwen3-14B
Group	Method	Reh.	R-4K	R-16K	R-4K 3%	LB	R-4K	LB	R-4K	R-4K	Avg. Rank $\downarrow$
	Full cache (reference)	–	96.5	93.5	96.5	47.1	95.8	45.1	94.6	96.8	–
	StreamingLLM ( Xiao et al., 2024 )	0.00	11.4	12.9	6.1	28.1	15.2	29.3	10.4	20.8	10.62
	SnapKV ( Li et al., 2024a )	0.00	14.9	17.8	8.9	32.2	15.2	29.0	13.9	19.1	9.88
	TOVA ( Oren et al., 2024 )	0.00	6.9	28.8	3.4	31.6	14.3	28.7	4.8	10.5	11.75
	Expected Attention ( Devoto et al., 2025 )	0.00	9.2	8.6	7.8	36.5	17.1	31.8	10.0	12.8	10.00

Table 1. Main results on the test splits at a 5% keep ratio (R-4K 3% : 3%). R: RULER; LB: LongBench (14 natural-text tasks). Reh.: rehearsal length, including QA pairs, as a fraction of the context (Qwen3-8B, RULER-4K test). Attic alone uses KV 2 ’s scorer; a shaded row below a host adds Attic ’s rehearsal to it;

\uparrow

x : gain over KV 2 or the host. KV 2 uses the

\rho

selected on the development split of each model, benchmark and keep ratio; Attic uses one configuration throughout. † Restore tokens released by the RestoreKV authors (none for Qwen3-4B/14B); all other rows are training-free. Bold : above every baseline; underlined : the strongest baseline. Avg. Rank: mean rank over the columns each row has.

	Short dependency		Long dependency
Method	ROUGE-L	METEOR	ROUGE-L	METEOR
Full cache	65.7	34.2	31.9	12.3
KVzip+	28.1	14.0	19.6	6.4
KV 2	34.3	17.2	23.6	7.8
Attic	36.2 $\uparrow$ 1.9	18.6 $\uparrow$ 1.4	22.9 $\downarrow$ 0.7	7.8
RestoreKV+ †	39.0	19.3	25.3	7.2

Table 2. Compress once, answer many on LooGLE (Qwen3-8B, keep ratio 5%): each document is compressed once, then answers all of its questions (ROUGE-L / METEOR,

\times

100). The shaded row below RestoreKV+ adds Attic ’s rehearsal to it;

\uparrow

x /

\downarrow

x : change over KV 2 or the host. † Released restore tokens. Bold : above every baseline; underlined : the strongest baseline.

Figure 5. Tuning a fixed rehearsal ratio, even on the test split, cannot catch up with Attic (keep ratio 5%). Each line is KV 2 ’s score minus Attic ’s on the same benchmark, so Attic is the zero line. Every point lies below it: no single

\rho

gets close to Attic on both benchmarks at once.

Native KV pairs
	Qwen3-8B			Llama	Qwen3-4B
Rehearsal	4K	16K	4K 3%	4K	4K
KV 2 : anchors, tuned fixed $\rho$	76.8	74.5	67.0	74.8	74.1
adaptive amount	78.8 $\uparrow$ 2.0	73.0 $\downarrow$ 1.5	67.8 $\uparrow$ 0.8	75.6 $\uparrow$ 0.8	73.8 $\downarrow$ 0.3
+ self-gen. QA (= Attic )	82.1 $\uparrow$ 3.3	75.1 $\uparrow$ 2.1	73.4 $\uparrow$ 5.6	77.2 $\uparrow$ 1.6	76.5 $\uparrow$ 2.7
With released restore tokens

Table 3. Ablation on RULER test splits (keep ratio 5% unless marked). Each block starts from a baseline and changes one thing per row;

\uparrow

x /

\downarrow

x : change from the row above. “–”: no released restore tokens.

Qwen3-8B (full cache: avg. 47.1)
Method	SD	MD	Sum.	FS	Code	Avg.
KVzip+	23.8	16.3	18.5	46.3	42.9	28.6
KV 2	35.4	35.1	22.7	55.6	50.5	39.1
Attic	37.2	39.4	23.3	54.6	52.3	40.6
KVgrad	31.0	41.7	22.0	57.6	53.4	40.3
+ Attic	32.1	41.5	24.0	60.4	56.4	41.9

Table 4. LongBench test results by official category (keep ratio 5%). SD/MD: single-/multi-document QA; Sum.: summarization; FS: few-shot; Avg.: the 14 natural-text tasks, as in Table 1 . Shaded rows below a host add Attic ’s rehearsal to it. † Released restore tokens. Per model, bold : above every baseline; underlined : the strongest baseline.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Method	S1	S2	S3	MK1	MK2	MK3	MV	MQ	VT	CWE	FWE	QA1	QA2	Avg.
Full cache	100.0	100.0	100.0	100.0	100.0	100.0	100.0	100.0	100.0	99.2	96.8	84.0	75.0	96.5
KVzip+	100.0	63.8	61.4	42.0	60.5	9.4	42.1	56.3	100.0	58.1	68.8	48.0	51.9	58.6
KV 2	100.0	95.7	100.0	94.0	34.9	1.9	95.8	96.6	100.0	85.2	91.4	58.0	44.2	76.8
Attic	100.0	100.0	100.0	98.0	51.2	32.1	97.1	98.9	100.0	92.9	93.0	58.0	46.2	82.1
KVgrad	100.0	85.1	95.5	74.0	41.9	20.8	90.0	85.2	99.3	81.3	74.7	34.0	53.9	72.0
+ Attic	100.0	100.0	100.0	100.0	74.4	54.7	98.3	98.9	100.0	79.4	89.8	46.0	46.2	83.7

Appendix

Table 5. Per-subtask results, Qwen3-8B, RULER-4K (test split).

Method	S1	S2	S3	MK1	MK2	MK3	MV	MQ	VT	CWE	FWE	QA1	QA2	Avg.
Full cache	100.0	100.0	100.0	98.0	100.0	100.0	100.0	100.0	100.0	84.2	92.5	72.0	69.2	93.5
KVzip+	100.0	80.9	88.6	46.0	23.3	7.6	57.9	69.9	100.0	20.6	89.3	42.0	55.8	60.1
KV 2	100.0	100.0	100.0	90.0	34.9	9.4	94.6	98.9	100.0	50.0	93.0	42.0	55.8	74.5
Attic	100.0	100.0	100.0	96.0	27.9	7.6	96.3	99.4	100.0	67.9	91.4	38.0	51.9	75.1
KVgrad	100.0	91.5	97.7	96.0	51.2	20.8	87.5	93.8	100.0	55.2	88.2	38.0	51.9	74.7
+ Attic	100.0	100.0	100.0	100.0	65.1	34.0	95.4	100.0	100.0	77.3	90.3	32.0	53.9	80.6

Appendix

Table 6. Per-subtask results, Qwen3-8B, RULER-16K (test split).

Method	S1	S2	S3	MK1	MK2	MK3	MV	MQ	VT	CWE	FWE	QA1	QA2	Avg.
Full cache	100.0	100.0	100.0	100.0	100.0	100.0	100.0	100.0	100.0	99.2	96.8	84.0	75.0	96.5
KVzip+	100.0	31.9	25.0	6.0	7.0	0.0	23.8	11.4	100.0	6.3	19.4	38.0	40.4	31.5
KV 2	100.0	83.0	100.0	82.0	14.0	0.0	90.0	88.1	100.0	59.4	82.3	38.0	34.6	67.0
Attic	100.0	100.0	100.0	94.0	14.0	1.9	92.9	96.0	100.0	86.7	87.6	44.0	36.5	73.4
KVgrad	100.0	48.9	54.6	64.0	18.6	3.8	66.7	59.1	97.0	57.5	53.2	28.0	44.2	53.5
+ Attic	100.0	100.0	100.0	94.0	18.6	0.0	94.6	97.2	99.6	54.2	85.5	38.0	36.5	70.6

Appendix

Table 7. Per-subtask results, Qwen3-8B, RULER-4K, keep ratio 3% (test split).

Method	S1	S2	S3	MK1	MK2	MK3	MV	MQ	VT	CWE	FWE	QA1	QA2	Avg.
Full cache	100.0	100.0	100.0	100.0	100.0	100.0	100.0	100.0	99.3	99.6	96.8	84.0	65.4	95.8
KVzip+	100.0	93.6	72.7	76.0	93.0	11.3	57.5	83.5	97.0	8.1	76.3	40.0	51.9	66.2
KV 2	100.0	97.9	100.0	98.0	76.7	5.7	80.0	93.2	98.1	29.8	89.3	54.0	50.0	74.8
Attic	100.0	100.0	97.7	98.0	51.2	9.4	90.4	98.9	98.5	68.1	94.6	50.0	46.2	77.2
KVgrad	100.0	95.7	93.2	98.0	97.7	54.7	85.8	94.9	98.9	29.4	87.1	24.0	42.3	77.1
+ Attic	100.0	100.0	100.0	100.0	60.5	56.6	99.2	100.0	98.9	44.2	88.2	48.0	48.1	80.3

Appendix

Table 8. Per-subtask results, Llama-3.1-8B, RULER-4K (test split).

Method	S1	S2	S3	MK1	MK2	MK3	MV	MQ	VT	CWE	FWE	QA1	QA2	Avg.
Full cache	100.0	100.0	100.0	100.0	100.0	100.0	100.0	100.0	100.0	94.0	88.2	76.0	71.2	94.6
KVzip+	100.0	72.3	65.9	42.0	72.1	5.7	32.1	51.1	100.0	36.7	68.8	46.0	42.3	56.5
KV 2	100.0	97.9	100.0	94.0	58.1	13.2	84.2	78.4	100.0	56.0	78.0	50.0	53.9	74.1
Attic	100.0	100.0	100.0	100.0	55.8	7.6	97.1	84.1	100.0	74.4	81.2	48.0	46.2	76.5
KVgrad	100.0	87.2	100.0	78.0	55.8	34.0	72.9	71.6	99.6	75.4	72.6	32.0	57.7	72.1
+ Attic	100.0	100.0	100.0	100.0	83.7	45.3	95.0	94.3	100.0	72.7	79.6	42.0	46.2	81.4

Appendix

Table 9. Per-subtask results, Qwen3-4B, RULER-4K (test split).

Method	S1	S2	S3	MK1	MK2	MK3	MV	MQ	VT	CWE	FWE	QA1	QA2	Avg.
Full cache	100.0	100.0	100.0	100.0	100.0	100.0	100.0	100.0	100.0	99.6	96.2	92.0	71.2	96.8
KVzip+	100.0	87.2	88.6	86.0	95.4	34.0	64.2	86.4	99.6	60.6	81.2	66.0	57.7	77.4
KV 2	100.0	91.5	100.0	96.0	76.7	9.4	95.8	99.4	100.0	84.2	91.9	66.0	42.3	81.0
Attic	100.0	100.0	100.0	100.0	79.1	62.3	100.0	100.0	100.0	87.9	91.9	68.0	50.0	87.6
KVgrad	100.0	95.7	97.7	88.0	74.4	45.3	92.1	77.8	100.0	90.4	80.1	34.0	57.7	79.5
+ Attic	100.0	100.0	100.0	98.0	83.7	77.4	100.0	100.0	100.0	91.7	90.3	48.0	57.7	88.2

Appendix

Table 10. Per-subtask results, Qwen3-14B, RULER-4K (test split).

Method	NQA	Qsp	MFQA	HQA	2WQA	Musq	GovR	QMS	MNews	TREC	TQA	SSum	LCC	RB-P	PCnt	PRet	Avg 14	Avg 16
Full cache	31.9	45.6	54.7	68.1	56.2	34.6	32.6	24.5	26.4	38.7	85.5	39.9	63.9	57.8	9.3	94.6	47.1	47.7
KVzip+	21.3	17.2	33.0	25.8	13.7	9.4	22.3	20.5	12.7	24.0	79.9	35.1	23.4	62.4	2.7	32.0	28.6	27.2
KV 2	33.0	32.8	40.5	52.9	35.8	16.7	26.7	21.2	20.3	45.3	85.2	36.2	37.5	63.6	4.0	50.0	39.1	37.6
Attic	27.7	35.5	48.5	56.4	41.9	19.7	28.7	21.8	19.6	42.7	85.8	35.4	44.1	60.4	2.7	45.8	40.6	38.5
KVgrad	24.9	29.5	38.6	55.8	45.8	23.5	25.5	21.2	19.3	50.7	85.3	36.8	48.3	58.6	9.3	94.1	40.3	41.7
+ Attic	24.1	26.9	45.4	57.2	43.1	24.2	27.9	22.2	22.0	59.3	84.5	37.3	56.0	56.7	2.7	79.3	41.9	41.8

Appendix

Table 11. Per-task LongBench results, Qwen3-8B (test split, keep ratio 5%). Avg 14 : the 14 natural-text tasks used in Table 1 ; Avg 16 : all 16 tasks.

Method	NQA	Qsp	MFQA	HQA	2WQA	Musq	GovR	QMS	MNews	TREC	TQA	SSum	LCC	RB-P	PCnt	PRet	Avg 14	Avg 16
Full cache	40.5	49.4	57.6	56.7	59.0	32.3	33.4	24.8	27.6	30.0	88.3	40.9	49.5	41.7	10.0	100.0	45.1	46.4
KVzip+	33.8	19.4	33.8	25.2	26.5	11.3	17.9	20.3	13.5	10.0	86.2	33.2	27.5	55.5	0.0	34.0	29.6	28.0
KV 2	31.5	36.0	44.2	46.6	43.7	24.1	28.4	22.5	23.6	31.0	86.3	39.9	35.4	54.8	6.0	88.0	39.1	40.1
Attic	33.3	35.3	45.8	45.5	49.2	25.4	27.7	22.4	23.9	26.0	86.9	39.0	36.0	53.7	2.0	50.0	39.3	37.6
KVgrad	32.0	32.4	35.1	39.4	38.0	25.3	25.9	22.5	22.8	44.0	85.3	40.9	40.8	46.8	8.0	98.0	37.9	39.8
+ Attic	29.6	39.3	48.1	47.7	42.4	20.5	27.4	23.1	24.7	38.0	82.9	40.7	40.2	45.7	10.0	82.0	39.3	40.1

Appendix

Table 12. Per-task LongBench results, Llama-3.1-8B (test split, keep ratio 5%). Avg 14 : the 14 natural-text tasks used in Table 1 ; Avg 16 : all 16 tasks.

	Qwen3-8B			Llama-3.1-8B
(a) Rehearsal	Amt.	Label	Score	Amt.	Label	Score
KV 2 (default)	5%	$\sim$ 25%	77	5%	$\sim$ 20%	100
KV 2 ( $\rho{=}0.35$ )	35%	$\sim$ 7%	50	35%	$\sim$ 4.5%	70
Content-adaptive	53%	$\sim$ 5%	40	40%	$\sim$ 4%	30

Appendix

Table 13. What coherent rehearsal reads (development split, keep ratio 5%). (a) Passage retrieval, first 10 questions: as rehearsal grows into coherent text, the kept fraction of “Paragraph

N

” label tokens falls to the 5% base rate and the score follows. (b) Qwen3-8B: breaking the same rehearsal into sparse passes recovers labels but loses RULER needles. Amt.: rehearsal amount (fraction of context); Label: kept fraction of label tokens; content-adaptive: Attic ’s amount without QA rehearsal. PR: passage retrieval; MK2/MK3: multi-key 2/3; MV: multi-value.

Setting	$\boldsymbol{\rho}$ : development score
Qwen3-8B, 5%	0.05: 68.0; 0.25: 74.0; 0.35: 74.6 ; 0.5: 73.9
Llama-3.1-8B, 5%	0.05: 68.1; 0.1: 70.6; 0.25: 73.7; 0.35: 73.0; 0.5: 73.9
Llama-3.1-8B, LB	0.05: 37.2; 0.15: 39.7 ; 0.5: 37.1
Qwen3-8B, LB	0.05: 37.9; 0.15: 38.5; 0.35: 38.0; 0.5: 38.6
Qwen3-4B, 5%	0.25: 70.85; 0.35: 70.34; 0.5: 70.87
Qwen3-14B, 5%	0.25: 76.5; 0.35: 79.4; 0.5: 80.1

Appendix

Table 14. KV 2 development results for each candidate

\rho

(RULER-4K development split; LB: LongBench development split, 14 natural-text tasks); the selected value is in bold .

Setting	0.05	0.15	0.25	0.35	0.5	Amount	Attic
	KV 2 , $\boldsymbol{\rho=}$
Qwen3-8B R-4K	70.3	72.6	–	76.8	75.9	78.8	82.1
Qwen3-8B LB	37.1	38.7	–	38.6	39.1	38.6	40.6
Llama R-4K	70.2	72.5	73.9	–	74.8	75.6	77.2
Llama LB	38.5	39.1	38.7	–	35.0	38.3	39.3

Appendix

Table 15. KV 2

\rho

scan on the test splits (keep ratio 5%; LongBench: 14 natural-text tasks), the data of Figure 5 . Amount: Attic ’s content-adaptive amount without the QA rehearsal. “–”: not run.

Method	3%	5%	10%	20%	30%
Full cache	96.5	96.5	96.5	96.5	96.5
KVzip+	31.5	58.6	93.0	96.0	96.2
KV 2 (dev-tuned $\rho$ )	67.0	76.8	92.3	95.9	95.8
RestoreKV+ †	47.6	78.2	94.9	96.5	96.4
Attic	73.4	82.1	94.1	96.1	96.0
RestoreKV+ + Attic †	75.7	88.9	95.2	96.4	96.2

Appendix

Table 16. RULER-4K test score of Qwen3-8B across keep ratios. † Uses released restore tokens. Shaded rows use Attic ’s rehearsal. Bold : above every baseline; underlined : the strongest baseline.

Explore similar work

Sep 9, 2026cs.CL

KVShareArena: KV-Cache Reuse Across Contexts and Model Checkpoints

Reusing key-value (KV) caches speeds up LLM inference by avoiding repeated computation on shared text. Standard prefix caching reuses a KV cache only when the LLM is the same and all preceding text is identical, but real workloads often break both conditions: RAG systems place different documents before the same one, agents with different system prompts read the same file or tool output, multi-agent workflows use specialized LLMs on shared material, and an updated model reads documents cached by its previous version. Because KV caches depend on both the preceding text and the model weights, direct reuse can reduce answer quality. Many methods repair or compress the reused cache, but each paper uses its own tasks, models, and cost measures, and existing benchmarks mainly test long-context processing or reuse of an unchanged prefix. We introduce KVShareArena, a benchmark and open evaluation framework for comparing them under the same conditions. KVShareArena has (1) reuse tests on 2,150 questions from three QA datasets, where the preceding text, the cache-writing LLM, or both change while the answering LLM and input stay fixed; (2) five dense and mixture-of-experts LLMs (4B-30B) and six LLM pairs where one version of an LLM reads caches written by another, for 33 model-dataset settings; (3) 11 repair and compression methods from six method classes; (4) four evaluation perspectives: answer quality, prefill computation, KV-cache memory, and latency; and (5) a common interface for adding new methods and an interactive leaderboard. Experiments yield two findings. First, both the quality loss from reuse and which repairs help depend on the LLM, even between two 8B models. Second, most repairs keep their quality when another LLM version wrote the cache, but a trained repair adapter loses quality in 12 of 18 pair-dataset tests. Code and data: https://github.com/xishi404/KVShare-Arena

Xi Shi, Mengxin Zheng, Qian Lou

University of Central Florida

KV Caching LLM Inference Acceleration Prefix Caching KV-Cache Reuse

Ask PDF

Sep 27, 2026cs.AR

Budgeted Cache Repair for Cross-Context KV-Cache Reuse

Cross-context KV-cache reuse predicts a shared segment's keys and values under a new prefix instead of recomputing them, and has been reported to do so without quality loss. We find otherwise, and identify two problems. (1) A hidden cost: on MMLU and GSM8K, reuse costs substantial accuracy. (2) A decision at the wrong unit: no rule for deciding whether to reuse a cache removes that cost. What does help is choosing which parts of the cache to recompute, and the value of choosing well falls as the unit of choice grows: informed selection removes 49.5% of the cache error beyond chance at single rows (one token's keys and values), 10.6% at 64-token chunks, and nothing at the level of whole calls. Budgeted Cache Repair (BCR) acts at the unit where selection still pays. It drafts two tokens from the assembled cache, ranks cache rows by the attention those tokens pay them, and recomputes a fixed budget of rows exactly, in one of three layouts. The cost is paid rather than predicted away, and the draft that fails as a gate succeeds as a selector. BCR restores GSM8K to dense-prefill accuracy while still serving most calls from cache, and its best layout outperforms every reuse baseline's mean in the reference grid. The draft also beats a coin-flip selector at the same budget - a control prior evaluations lack.

Haeyong Kang, Chang D. Yoo

Duksung Women’s University · KAIST

KV-Cache Management KV-Cache Reuse Memory-Efficient Inference

Ask PDF

Sep 30, 2026cs.LG

EchoPress: Query-Agnostic KV Cache Pruning via Virtual Context Reconstruction

KV cache pruning reduces long-context inference memory usage by evicting less important key-value pairs. KVzip estimates importance through context reconstruction: prompting a model to repeat the context chunk by chunk. This achieves strong compression quality at the cost of additional forward passes. Learned approximations reduce this cost but require model-specific training. We analyze how KVzip identifies important cached information and show how to approximate its reconstruction scores using information already computed during prefill. These findings motivate EchoPress, a training-free method that approximates reconstruction attention using queries and keys from standard prefill. For each request, it reconstructs only the first chunk to calibrate importance scores for the remaining context. Experiments on LongBench and RULER with Qwen3-8B and Llama-3.1-8B-Instruct show that EchoPress matches KVzip in task accuracy across eviction ratios from 50% to 90%, while reducing compression overhead by a factor of 1.7-19.6 and total prefill time by a factor of up to 2.9. Code is available at https://github.com/ljwljwljwljw/kvpress/tree/echo-press.

Jiawei Lin, Saibo Geng, Thomas Bourgeat

EPFL

KV-Cache Management LLM Inference Acceleration KV-Cache Compression KV-Cache Pruning

Ask PDF