KV$^2$: A Self-Refining KV Cache

Johannes Wesch; Danni Liu; Jan Niehues

cs.AIOct 2, 2026

KV $^2$ : A Self-Refining KV Cache

Authors: Johannes Wesch, Danni Liu, Jan Niehues

Organizations: Karlsruhe Institute of Technology, Germany

Abstract

The memory footprint of the key-value (KV) cache constrains the practical use of long-context models, and it dominates cost when one prefilled context must later serve many different queries. In this reusable setting, query-agnostic compression trades cost against quality: lightweight estimators are cheap but less accurate, whereas full-context reconstruction scoring is more accurate yet reprocesses the entire prompt. We introduce KV $^2$ , a query-agnostic KV-cache compression method based on selective reconstruction. KV $^2$ first uses a lightweight proxy scorer to identify informative in-context tokens, then reprocesses only this subset to compute final eviction scores. On RULER, Needle-in-a-Haystack, and LongBench, KV $^2$ 's margin over baselines widens as the budget tightens: on RULER 16K at a 2% KV-cache budget it improves the average score over the next-best baseline by more than 40 percentage points, and on LongBench it attains the highest average across 2%-10% budgets at lower compression-stage runtime and peak memory than full-context reconstruction. Reusable KV-cache compression thus does not require reprocessing the full context. Our code is available at https://anonymous.4open.science/r/KVsquared-0B97.

Figures & tables

Figure 1: Overview of KV². After prefilling the context cache

\mathrm{KV}_{c}

, the method computes proxy scores as a lightweight estimate of which in-context tokens are most informative. KV² then selects the query tokens

\mathrm{KV}_{c,\mathrm{selected\ query\ tokens}}

, runs Selective Reconstruction (SR) on these selected queries to obtain cache scores, and then evicts low-scoring KV entries before later decoding reuses the compressed cache.

Figure 2: Chunked processing in KV 2 . The context is split into chunks. Each chunk proposes its own reconstruction queries, and the resulting local scores are merged into one global eviction decision.

Figure 3: Detailed view of KV² inside one chunk. Proxy scores first select the ordered reconstruction set

Q_{r}

. These reconstruction queries are then encoded against the shared cache, and the final score for a cached position is the maximum attention it receives during selective reconstruction.

Llama-3.1-8B-Instruct
Method	Single-doc QA			Multi-doc QA			Summ.			Few-shot / Cls.			Synth. Count		Code		Avg.
Method	NrtvQA	Qasper	MFQA-en	Hotpot	2Wiki	MuSiQ	GovRep	QMSum	MultiN	TREC	TrivQA	SAMSum	PCount	PR-en	LCC	RB-P	Avg.
Full	30.21	45.53	55.01	56.01	46.65	31.28	35.13	25.28	27.25	73.00	91.64	43.80	8.91	99.50	63.38	56.64	49.33
KV Size = 2%
KeyDiff	17.24	12.11	16.71	26.07	20.44	5.06	21.65	18.36	10.99	17.00	91.34	36.60	6.00	25.50	12.66	48.75	24.16
KVzip	12.54	11.10	18.98	24.80	21.78	8.69	11.39	17.40	7.36	5.00	87.47	32.31	2.50	4.00	18.20	51.75	20.95

Table 1: LongBench results across KV-cache budgets for Llama-3.1-8B-Instruct and Qwen3-8B . For both, KV² gives the strongest average performance at tighter budgets, with KVzip becoming closer at 10% retention.

Figure 4: RULER 16K score versus compression ratio for Qwen3-8B . The largest gap appears in the severe-budget regime (compression ratio

>

90%), where KV 2 remains much more robust than the compared baselines.

Figure 5: Needle-in-a-Haystack results at a 5% KV-cache budget on Llama-3.1-8B-Instruct across context lengths and needle depths.

Figure 6: Compression-stage runtime (left) and peak memory (right) versus chunk size for single-pass KV 2 compared with KVzip. For KV 2 , the value in parentheses in the legend denotes the reconstruction percentage retained from the proxy-selected query set in the first stage. Missing bars indicate out-of-memory configurations.

Figure 7: RULER 4K score versus compression ratio for Llama-3.1-8B-Instruct when KV 2 is paired with different first-stage proxy scorers.

Figure 8: RULER 4K score versus reconstruction percentage for Llama-3.1-8B-Instruct (left) and Qwen3-8B (right). In the legend, KV Size denotes the final retained KV-cache budget.

Figure 9: RULER 4K score with refinement iterations. Most gains appear in the first few passes, after which improvements diminish. The legend lists the first-stage proxy scorer and the retained reconstruction percentage.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 10: Compression-stage runtime (left) and peak memory (right) versus chunk size for iterative KV 2 . Additional refinement passes increase runtime roughly in proportion to the number of passes, while peak memory is driven mainly by chunk size and changes much less with the number of iterations. All curves use a fixed 2% reconstruction percentage; the number in parentheses in the legend denotes the number of refinement iterations.

Figure 11: RULER 4K score versus KV retention ratio for Llama-3.1-8B-Instruct (left) and Qwen3-8B (right). KV 2 is more robust than lightweight heuristics at small budgets, while KVzip becomes more competitive as more KV entries are kept.

KV-Cache Compression

Key-value cache compression reduces the memory and bandwidth needed to retain attention states during long-context inference.

See all 269 papers

AnchorKV: Anchor-Residual KV Cache Compression
Aug 3, 2026Malik Khalaf, Yara Shamshoum, Nitzan Hodos +2KV Caching Long-Context Language Model Inference
Department of Computer Science, Technion - Israel Institute of Technology
NestedKV: Nested Memory Routing for Long-Context KV Cache Compression
May 26, 2026Hong Chen, Xiang Liu, Yubo Gao +5KV Caching Long-Context Language Modeling
The Hong Kong University of Science and Technology (Guangzhou) · Jimei University
CompressKV: Semantic-Retrieval-Guided KV-Cache Compression for Resource-Efficient Long-Context LLM Inference
Jun 23, 2026Xiaolin Lin, Jingcun Wang, Olga Kondrateva +3LLM Inference Efficiency KV Caching
Technical University of Darmstadt, Darmstadt, Germany · University of Notre Dame, Notre Dame, IN, USA · Technical University of Ilmenau, Ilmenau, Germany

Aug 3, 2026J/K move · Enter open · S save

AnchorKV: Anchor-Residual KV Cache Compression

Malik Khalaf, Yara Shamshoum, Nitzan Hodos, Yuval Sieradzki+1

Department of Computer Science, Technion - Israel Institute of Technology

KV Caching Long-Context Language Model Inference Memory-Efficient Optimization Memory-Efficient Inference KV-Cache Compression

Open paper AskPDF arXiv

The key-value (KV) cache is the primary memory bottleneck in long-context LLM inference. Existing approaches attack it from opposite ends: eviction methods permanently discard tokens, degrading performance whenever a discarded token later proves essential, while quantization methods retain all tokens at low precision but offer limited compression. We propose AnchorKV, a compression scheme that shrinks the cache by

20\times

without discarding a single token. AnchorKV represents the cache using a small set of anchors stored exactly, expresses every other token through its most similar anchor, and refines only those whose approximation most affects the model's output. AnchorKV consistently preserves accuracy across models and datasets, retaining 99% of the full-cache score at the 70B scale, while keeping the entire context at a fraction of its cost.

Abstract

KV-Cache Compression

Explore similar work

AnchorKV: Anchor-Residual KV Cache Compression

NestedKV: Nested Memory Routing for Long-Context KV Cache Compression

CompressKV: Semantic-Retrieval-Guided KV-Cache Compression for Resource-Efficient Long-Context LLM Inference