Fast KV Compaction via Attention Matching

Adam Zweiger; Xinghong Fu; Han Guo; Yoon Kim

cs.LGFeb 18, 2026

Fast KV Compaction via Attention Matching

Authors: Adam Zweiger, Xinghong Fu, Han Guo, Yoon Kim

Organizations: Massachusetts Institute of Technology.

Abstract

Scaling language models to long contexts is often bottlenecked by the size of the key-value (KV) cache. In deployed settings, long contexts are typically managed through compaction in token space via summarization. However, summarization can be highly lossy, substantially harming downstream performance. Recent work on Cartridges has shown that it is possible to train highly compact KV caches in latent space that closely match full-context performance, but at the cost of slow and expensive end-to-end optimization. This work describes an approach for fast context compaction in latent space through Attention Matching, which constructs compact keys and values to reproduce attention outputs and preserve attention mass at a per-KV-head level. We show that this formulation naturally decomposes into simple subproblems, some of which admit efficient closed-form solutions. Within this framework, we develop a family of methods that significantly push the Pareto frontier of compaction time versus quality, achieving up to 50x compaction in seconds on some datasets with little quality loss.

Figures & tables

Figure 1: Accuracy vs. Compaction Time Trade-off (Qwen3-4B; QuALITY). We compare downstream QA accuracy (

n=894

) after compaction, plotted against the average wall-clock time required to compact a context (seconds, log-scale) using a single H100 GPU at a fixed 50

\times

compaction ratio. Our attention-matching (AM) methods trace a speed–quality tradeoff and form the Pareto frontier, outperforming prior token-selection baselines and exceeding the performance of Cartridges ( Eyuboglu et al., 2025 ) while being 2 orders of magnitude faster; additional Cartridges training may further improve its results.

Figure 2: KV compaction via Attention Matching. We replace the full KV cache $(\bm{K},\bm{V})$ with a smaller cache $(\bm{C}_{k},\bm{C}_{v})$ chosen so that attention outputs are approximately preserved when the compacted prefix is concatenated with arbitrary fixed or future tokens $(\bm{K}_{\text{fixed}},\bm{V}_{\text{fixed}})$ .

Figure 3: Head sensitivity curves in Qwen3-4B. We fix all KV heads to a baseline compaction ratio of

0.05\times

and vary the compaction ratio of a single head. We report the change in loss relative to the baseline (lower is better) as a function of the varied head’s compaction ratio. Curves are averaged over

10

QuALITY articles; shaded regions denote

\pm

1 standard error of the mean across articles. Some heads (e.g., L0H0 ) are largely insensitive to additional capacity, whereas others (e.g., L15H2 ) benefit substantially from storing more KV pairs.

Figure 4: Accuracy vs. compaction ratio across methods. We compare AM-OMP and AM-HighestAttnKeys against Cartridges, summarization, and four prior methods. Evaluations are conducted on QuALITY and LongHealth using Qwen3-4B, Llama3.1-8B, and Gemma3-12B. Attention Matching (AM) consistently outperforms other approaches across compaction ratios, while matching Cartridges’ performance at ultra-high compaction.

Method	0%	1%	2%	5%	10%	20%	100%
Qwen3-4B
Baseline (no ctx / orig.)	0.104						0.444
Summarization				0.219	0.283
H2O+		0.188	0.243	0.310	0.350	0.390
KVMerger		0.171	0.215	0.260	0.303	0.356
KVzip		0.127	0.124	0.220	0.411	0.438

Table 1: QASPER, average token-level F1 (

562

contexts,

2{,}010

questions).

Method	0%	1%	2%	5%	10%	20%	100%
Baseline (no ctx / orig.)	17.0						72.3
Summarization			23.4	29.8	44.7
H2O+		10.6	21.3	36.2	44.7	61.7
KVMerger		19.2	21.3	34.0	44.7	48.9
KVzip		17.0	21.3	36.2	48.9	66.0
AM-HighestAttnKeys		61.7	59.6	66.0	70.2	68.1

Table 2: LongBench v2 accuracy (%) , Qwen3-4B-Instruct-2507,

47

-context subset (correct with full context, incorrect without; max

97

k, avg

42

k tokens).

Stage	Method	Time (s)
	context-prefill	7
Query generation	repeat-prefill	8
	self-study	139
	Highest attention	3
Key selection	OMP	565
	OMP-fast ( $k{=}4$ , refit interval=2)	104

Table 3: Wall-clock compaction time breakdown on a 60k-token LongHealth context with Gemma-3-12B on a single H200 GPU .

Method	Cache Size (%)	Accuracy (%)
Full Context	100.00	71.1
Summarize	4.80	55.2
Summarize × (0.2× AM-OMP)	0.92	55.7
Summarize × (0.1× AM-OMP)	0.46	55.0
Summarize × (0.05× AM-OMP)	0.21	49.2

Table 4: Summarization plus Attention Matching. Applying AM-OMP on top of a summary enables up to

\sim

200

\times

total compaction with accuracy comparable to summarization alone.

Figure 5: Leave-one-out experiments. We ablate our main AM-OMP method and measure average log-perplexity of generations decoded from the original cache, since this is lower in variance and well-correlated with downstream performance (Appendix F.2 ). In “no biases,” we first compute OMP as usual and then zero out

\bm{\beta}

, keeping the keys selected by OMP.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Context	Self-Study Tokens	H100-hours
LongHealth-5	60k	100–150M	$\sim\!15$
Quality	7k	25–30M	$\sim\!5$
LongHealth-10 (Original)	120k	300M	-

Appendix

Table 5: Training compute for Cartridges. We sample approximately the same number of self-study tokens per context token as in the original Cartridges implementation.

Method	Queries	Key selection	$\bm{\bm{\beta}}$	Values	Nonuniform
H2O+	context-prefill	HighestAttnKeys	none	direct	none
SnapKV (agnostic)	mock-question	HighestAttnKeys	none	direct	none
PyramidKV (agnostic)	mock-question	HighestAttnKeys	none	direct	linearly decreasing
KVzip-uniform	repeat-prefill	HighestAttnKeys	none	direct	none
KVzip	repeat-prefill	GlobalHighestAttnKeys	none	direct	global top- $k$
KVMerger	none	cosine-similarity merge	none	merged	none

Appendix

Table 6: Baseline configurations. “Mock-question” denotes a single synthetic question generated via self-study to instantiate originally question-aware methods in a question-agnostic setting. “Direct” values indicate that retained keys keep their original values (i.e., no value fitting with least squares); “merged” values denote Gaussian-kernel weighted averaging of clustered tokens (KVMerger).

Name	Prompt
summarize	Summarize the following text: {article_text} Summary:
summarize_indepth	Summarize the following text in depth: {article_text} Summary:
summarize_keypoints_questions	Summarize the following text, providing all key points that might be necessary to answer comprehension questions: {article_text} Summary:
summarize_concise	Summarize the following text concisely: {article_text} Summary:
summarize_very_concise	Summarize the following text very concisely: {article_text} Summary:

Appendix

Table 7: Summarization prompts.

Name	Prompt
3-question	Write 3 questions that test understanding of different parts of the context. Answer with just the 3 questions and options (do not say the correct answer), each one separated with 2 newlines.
summarize	Summarize the main points of the context.
structure_json	Structure the information in JSON form and include all important details like dates, times, names, and numerical values.
aggregate	Aggregate all the key facts mentioned in the context.

Appendix

Table 8: Prompt templates used in self-study.

Figure 6: Reference query sampling comparison. We compare eight variants for sampling queries. Self-study-based methods perform best, especially at the greatest compaction ratios, with repeat and context-prefill close behind. Subsampling reference queries preserves performance.

Figure 7: Head proportions induced by global highest-attention selection. We plot the fraction of total selected keys attributed to each of the

288

KV heads (

36

layers,

8

heads per layer) in Qwen3-4B. Top: proportions across

10

QuALITY articles at a fixed global compaction ratio of

0.05

. Bottom: average proportions across multiple global compaction ratios. The low variance across articles and target ratios indicates stable head importance patterns.

Figure 8: More Head sensitivity curves. Analogous to Figure 3 , showing loss change as a function of per-head compaction ratio for four selected heads. Left: Qwen3-4B with LongHealth exhibits similar patterns to Figure 3 . Right: Llama-3.1-8B-Instruct also shows head-dependent sensitivity patterns: some heads are largely insensitive to KV capacity, while others benefit substantially from retaining more keys.

g_{h}^{+}\leftarrow\begin{cases}J_{h}(\rho_{h})-J_{h}(\rho_{h}+\eta^{\prime}),&\text{if }\rho_{h}+\eta^{\prime}\leq 1,\\ -\infty,&\text{otherwise,}\end{cases}\qquad g_{h}^{-}\leftarrow\begin{cases}J_{h}(\rho_{h}-\eta^{\prime})-J_{h}(\rho_{h}),&\text{if }\rho_{h}\geq\eta^{\prime},\\ +\infty,&\text{otherwise,}\end{cases}

Appendix

Algorithm 4 Nonuniform KV Allocation via Greedy Swaps

Figure 9: Optimized head budgets. We compare the head proportions learned by our nonuniform allocation algorithm to those induced by global highest-attention key selection. The optimized schedule has similar structure as the global highest attention scores schedule. The optimized schedule tends to assign more budget to later layers and less to earlier layers, contrary to PyramidKV ( Cai et al., 2025 ) .

Figure 10: Effect of aggregation function on key selection. We plot accuracy versus compaction ratio for four baseline methods and our Highest Attention Keys method on a subset of

20

QuALITY articles using Qwen3-4B. All methods select the top-

t

keys based on aggregated attention scores over a shared set of reference queries.

Figure 11: Reconstruction loss vs. downstream accuracy. Left: Loss-based analogue of Figure 1 , instead reporting log-perplexity on original-cache generations. Right: Downstream multiple-choice accuracy plotted against this loss across various methods and compaction ratios.

Phys Len	Eff Len	AIME (/30)	Notes
2048	2048	1.25
4096	4096	7.75
8192	8192	13
With online compaction
2048	4096	8	$\leq$ 2 compactions
2048	8192	13	$\leq$ 6 compactions

Appendix

Table 9: AIME performance (pass@1, averaged over 4 runs) with and without mid-trajectory compaction. Effective sequence length is the total number of decoded tokens, while physical sequence length is the maximum physical size of the KV cache. Each compaction reduces the size of the entire KV cache (including previously compacted portions) by

50\%

, so

\leq

2 and

\leq

6 compactions correspond to approximately

2\times

and

4\times

more decoded tokens, respectively. For implementation simplicity, we do not enable non-uniform compaction or on-policy queries. We use AM-HighestAttnKeys-fast. We believe more tuned methods (e.g. with higher compaction ratios, compacting only the most recently generated tokens, etc.) may improve results.

Method	0%	1%	2%	5%	10%	20%	100%
Baseline (no ctx / orig.)	36.8						71.1
KVMerger		43.4	49.4	55.1	60.9	65.5
AM-HighestAttnKeys		60.4	65.0	70.1	71.3	71.0

Appendix

Table 10: QuALITY accuracy (%) with KVMerger , Qwen3-4B,

50

contexts. KVMerger tracks our token-eviction baselines and is well below Attention Matching.

Llama-3.1-8B-Instruct
Method	0%	5%	12.5%	100%
Baseline (no ctx / orig.)	2.6			95.7
PyramidKV			27.2
TOVA			46.4
KeyDiff			55.7
Finch (query-aware)			66.7

Appendix

Table 11: RULER accuracy (%) over

6{,}500

kvpress tasks. Even our fastest variant outperforms KVzip and all leaderboard methods at every evaluated ratio.

Explore similar work

Still: Amortized KV Cache Compaction in a Single Forward Pass
Jun 5, 2026Charles O'Neill, Alex Sandomirsky, Harry Partridge +2KV Caching Long-Context Language Model Inference
Baseten San Francisco · Baseten London
Latent Compaction with Test-Time Training
Oct 14, 2026Yingfa Chen, Xingyu Shen, Chaojun Xiao +1KV-Cache Compression Long-Context Retrieval
NLP Group, DCST, IAI, BNRIST, Tsinghua University, Beijing, China
ARC-KV: Amortizing Anchor Search for Reconstruction-Based KV Cache Compaction
Sep 29, 2026Zheyu Shen, Guanhua Wang, Dezhan Tu +5Long-Context Language Modeling KV-Cache Management
University of Maryland, College Park · Meta

Jun 5, 2026J/K move · Enter open · S save

Still: Amortized KV Cache Compaction in a Single Forward Pass

Charles O'Neill, Alex Sandomirsky, Harry Partridge, Mudith Jayasekara+1

Baseten San Francisco · Baseten London

KV Caching Long-Context Language Model Inference KV-Cache Management LLM Inference Acceleration KV-Cache Compression

Open paper AskPDF arXiv

The KV cache is the memory bottleneck of long-horizon language model deployment. Practically, a deployable compactor must be lightweight enough to call during inference, expressive enough to preserve context under constraint, and reusable across a trajectory. Existing compaction methods satisfy only part of this requirement: selection methods are lightweight but subset-bound, while synthesis methods are expressive but rely on per-context optimization. Here we introduce Still, a small per-layer Perceiver trained once against a frozen base model that produces compact keys and values in a single forward pass. On Qwen and Gemma models, Still occupies the favorable side of the speed--quality frontier across compression ratios from

8\times

200\times

and context lengths from

8

k to

128

k. On the long-context RULER grid, Still exceeds the strongest baseline by 8--22 points. The same compact cache also supports free-form summarization, preserving most of the full-context gain on HELMET and winning a pairwise LongBench summarization comparison against KV-Distill. Because compaction is a forward pass, Still can be applied iteratively, entering a long-horizon regime unavailable to per-context methods. We show that amortization makes long-context cache compaction tractable, and synthesis makes its compact state useful at extreme compression.

Abstract

Explore similar work

Still: Amortized KV Cache Compaction in a Single Forward Pass

Latent Compaction with Test-Time Training

ARC-KV: Amortizing Anchor Search for Reconstruction-Based KV Cache Compaction