The KV-Surgeon: Compensation-Aware Offline KV-Cache Compaction

Moritz Wagner; Christophe Roux; Max Zimmer; Sebastian Pokutta

cs.LGOct 19, 2026

The KV-Surgeon: Compensation-Aware Offline KV-Cache Compaction

Authors: Moritz Wagner, Christophe Roux, Max Zimmer, Sebastian Pokutta

Organizations: Department for AI in Society, Science, and Technology, Zuse Institute Berlin, Germany · Institute of Mathematics, Technische Universität Berlin, Germany

Abstract

Offline compaction of the key--value cache (KV-cache) reduces the cost of repeatedly querying a fixed context by constructing a smaller cache once and reusing it subsequently. The central challenge of compaction is deciding which cache entries to delete and which to retain. Existing methods score each candidate for deletion on its own, with the rest of the cache being held fixed. However, softmax renormalization of the attention operation couples these decisions in a nonlinear manner: removing one entry changes the attention weights of every remaining one. In this work, we introduce KV-Surgeon, which uses calibration data to take interactions into account in a tractable way. We derive a quadratic surrogate that admits closed-form expressions for jointly making the optimal greedy removal decision as well as compensation within the remaining entries. On QuALITY, LongHealth, and Long Code Arena, and at every evaluated compaction ratio, KV-Surgeon substantially advances the cost--accuracy Pareto frontier of dense and mixture-of-experts models from 4B to 30B parameters. At $50\times$ compaction, KV-Surgeon improves question-answering accuracy by up to four points over attention matching (Zweiger et al., 2026), the state of the art, at comparable cost. Lastly, we introduce an even faster variant, which also exceeds the state of the art while requiring only two fifths of its cost.

Figures & tables

Panel of Figure 1 — Figure 1: Cost–accuracy frontier at $50\times$ KV-cache compaction on QuALITY with Qwen3-30B-A3B. KV-Surgeon reaches $72.5\%$ accuracy in $159$ seconds and KV-Surgeon -fast reaches $71.4\%$ in $54$ seconds. Pipeline time per context includes calibration-query generation and compaction. OBCache, KVMerger, and KVzip are adapted to our pipeline ( Appendix G ). Points are five-seed means. The solid and dash-dotted curves are the Pareto frontiers with and without our methods.

Panel of Figure 2 — Figure 2: Mean accuracy over the eight pairs of Table 1 relative to KV-Surgeon . Error bars are one standard error of the difference. AM-OMP and AM-OMP-CS are shown only at the ratios their solver reaches on every model and benchmark.

Method	Time (s)	Qwen3-4B		Llama-3.1-8B		Gemma-3-12B		Qwen3-30B		Mean
Method	Time (s)	QuAL.	LongH.	QuAL.	LongH.	QuAL.	LongH.	QuAL.	LongH.	Mean
Repeat-pass
KV-Surgeon -fast (ours)	54	65.9 $\pm$ 0.4	66.9 $\pm$ 0.5	61.0 $\pm$ 0.4	61.5 $\pm$ 0.6	68.1 $\pm$ 0.5	52.1 $\pm$ 0.4	71.4 $\pm$ 0.3	64.1 $\pm$ 0.2	63.9 $\pm$ 0.15
OBCache (adapted)	58	62.4 $\pm$ 0.7	62.8 $\pm$ 0.9	59.8 $\pm$ 0.3	52.4 $\pm$ 0.2	66.6 $\pm$ 0.4	50.2 $\pm$ 0.8	67.3 $\pm$ 0.4	62.8 $\pm$ 0.4	60.5 $\pm$ 0.20
AM-HAK-fast	54	58.1 $\pm$ 0.5	53.5 $\pm$ 1.3	58.7 $\pm$ 0.6	53.5 $\pm$ 1.1	62.2 $\pm$ 0.4	45.6 $\pm$ 1.2	65.8 $\pm$ 0.5	59.8 $\pm$ 0.6	57.1 $\pm$ 0.30
Self-study

Table 1: Accuracy (%) at

50\times

compaction. Entries are means over five seeds with standard error over seeds in grey . Within each group and column, bold denotes the highest mean and underlining denotes results that trail the highest mean by at most two standard errors of the difference. The Qwen3-4B column uses Qwen3-4B on QuALITY and Qwen3-4B-Instruct-2507 on LongHealth. Summarization uses the prompt whose summary length is closest to

50\times

compaction ( Appendix D ). The Mean column contains the means over the eight model–benchmark pairs, and Time is the geometric mean of pipeline time per context in seconds.

Method	Selection rule	Compensation	Accuracy (%)
KV-Surgeon	Equation 11	✓	66.9
KV-Surgeon w/o compensation	Equation 11	✗	63.9
LOO	Equation 13	✗	64.7

Table 2: Qwen3-4B on QuALITY (first

25

articles) at

50\times

compaction. We compare the impact of compensation during selection. We vary the selection rule, i.e., the saliency by which entries are ranked for deletion, and whether the surviving entries are updated after each deletion.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Model, benchmark	(s)	$50\times$	$5\times$	$50\times$	$5\times$	$50\times$	$5\times$
	Query gen.	LOO		KV-Surgeon		AM-HAK
Qwen3-4B, QuALITY	33	7.7	29	10.5	53	7.0	28
Llama-3.1-8B, LongHealth	274	66	284	93	578	57	277
Gemma-3-12B, QuALITY	29	1.2	4.8	1.7	11	1.1	4.8
Qwen3-30B-A3B, QuALITY	162	5.5	20	–	–	4.6	20
Qwen3-30B-A3B, LongHealth	1062	47	–	65	–	–	–

Appendix

Table 3: Where self-study pipeline time goes. Query generation is the sampled self-study pass, in seconds per context; it does not depend on the method or on the compaction factor. The remaining columns are the compaction step alone, in seconds per context, at

50\times

and

5\times

for the three methods that consume the same queries. Means over all seeded runs; the sums of query generation and compaction differ from the pipeline times of Appendix D by up to

15\%

, which is run-to-run timing variation. ‘–’ marks cells not measured.

	Accuracy (%)					Pipeline time (s)
Method	$100\times$	$50\times$	$20\times$	$10\times$	$5\times$	$100\times$	$50\times$	$20\times$	$10\times$	$5\times$
$L_{2}$ key-norm	37.0 $\pm$ 0.3	37.9 $\pm$ 0.4	41.2 $\pm$ 0.6	51.1 $\pm$ 0.3	64.2 $\pm$ 0.3	0.7	0.6	0.5	0.6	0.6
Q-Filters	47.1 $\pm$ 0.7	52.4 $\pm$ 0.5	62.0 $\pm$ 0.2	66.5 $\pm$ 0.3	69.9 $\pm$ 0.2	0.5	0.6	0.5	0.5	0.6
StreamingLLM	40.4 $\pm$ 0.6	44.7 $\pm$ 0.4	52.0 $\pm$ 0.4	58.4 $\pm$ 0.4	64.2 $\pm$ 0.2	0.7	0.5	0.5	0.8	0.7
KVzip	48.0 $\pm$ 0.4	49.8 $\pm$ 0.4	61.6 $\pm$ 0.4	69.1 $\pm$ 0.5	70.5 $\pm$ 0.2	16	17	21	26	40
H2O	60.0 $\pm$ 0.5	62.3 $\pm$ 0.4	68.2 $\pm$ 0.4	70.3 $\pm$ 0.4	71.3 $\pm$ 0.2	16	17	20	27	40

Appendix

Table 4: QuALITY with Qwen3-4B. Full-context ceiling

72.1\pm 0.3\%

	Accuracy (%)					Pipeline time (s)
Method	$100\times$	$50\times$	$20\times$	$10\times$	$5\times$	$100\times$	$50\times$	$20\times$	$10\times$	$5\times$
$L_{2}$ key-norm	38.2 $\pm$ 0.8	37.5 $\pm$ 0.4	36.0 $\pm$ 0.6	51.8 $\pm$ 0.5	64.8 $\pm$ 0.8	7.0	4.9	4.9	6.0	5.3
Q-Filters	37.6 $\pm$ 0.4	43.6 $\pm$ 0.4	56.2 $\pm$ 0.6	69.1 $\pm$ 0.5	76.5 $\pm$ 0.4	4.8	4.8	5.1	5.4	5.2
StreamingLLM	34.6 $\pm$ 0.5	39.4 $\pm$ 0.7	42.9 $\pm$ 0.8	55.4 $\pm$ 0.3	64.2 $\pm$ 0.4	5.3	5.6	5.2	4.6	5.6
KVzip	31.8 $\pm$ 1.0	39.9 $\pm$ 0.8	64.6 $\pm$ 0.6	75.9 $\pm$ 0.3	77.7 $\pm$ 0.4	206	222	278	363	530
H2O	49.6 $\pm$ 0.6	58.4 $\pm$ 0.4	71.2 $\pm$ 0.4	76.2 $\pm$ 0.9	78.5 $\pm$ 0.2	203	222	277	370	529

Appendix

Table 5: LongHealth with Qwen3-4B-Instruct-2507. Full-context ceiling

80.5\pm 0.8\%

	Accuracy (%)					Pipeline time (s)
Method	$100\times$	$50\times$	$20\times$	$10\times$	$5\times$	$100\times$	$50\times$	$20\times$	$10\times$	$5\times$
$L_{2}$ key-norm	38.0 $\pm$ 0.4	40.2 $\pm$ 0.5	48.1 $\pm$ 0.5	56.4 $\pm$ 0.5	60.6 $\pm$ 0.1	0.5	0.5	0.5	0.5	0.6
Q-Filters	40.8 $\pm$ 0.3	47.7 $\pm$ 0.8	56.7 $\pm$ 1.0	60.9 $\pm$ 0.4	62.5 $\pm$ 0.3	0.6	0.6	0.6	0.6	0.6
StreamingLLM	38.7 $\pm$ 0.5	43.2 $\pm$ 0.4	49.0 $\pm$ 0.7	54.1 $\pm$ 0.4	59.2 $\pm$ 0.2	0.5	0.5	0.6	0.6	0.6
KVzip	46.0 $\pm$ 0.6	50.5 $\pm$ 0.6	58.9 $\pm$ 0.1	64.1 $\pm$ 0.6	63.4 $\pm$ 0.5	14	15	18	24	34
H2O	55.7 $\pm$ 0.5	59.1 $\pm$ 0.4	63.1 $\pm$ 0.3	65.5 $\pm$ 0.3	64.2 $\pm$ 0.4	14	15	18	25	34

Appendix

Table 6: QuALITY with Llama-3.1-8B-Instruct. Full-context ceiling

62.7\pm 0.5\%

	Accuracy (%)					Pipeline time (s)
Method	$100\times$	$50\times$	$20\times$	$10\times$	$5\times$	$100\times$	$50\times$	$20\times$	$10\times$	$5\times$
$L_{2}$ key-norm	33.0 $\pm$ 1.2	34.1 $\pm$ 1.6	37.6 $\pm$ 0.7	50.5 $\pm$ 0.3	62.5 $\pm$ 0.6	5.6	5.9	5.6	5.6	6.2
Q-Filters	36.0 $\pm$ 0.8	36.1 $\pm$ 1.1	39.9 $\pm$ 0.6	49.5 $\pm$ 0.8	61.4 $\pm$ 0.2	7.0	5.7	6.0	6.6	5.7
StreamingLLM	37.1 $\pm$ 1.1	37.2 $\pm$ 0.8	42.2 $\pm$ 1.1	45.4 $\pm$ 0.5	56.3 $\pm$ 0.8	5.9	5.2	5.7	5.6	5.7
KVzip	23.3 $\pm$ 1.1	41.0 $\pm$ 1.4	63.5 $\pm$ 0.7	71.3 $\pm$ 0.4	72.7 $\pm$ 0.5	173	179	229	302	437
H2O	43.5 $\pm$ 0.7	50.6 $\pm$ 0.9	66.2 $\pm$ 0.6	70.2 $\pm$ 0.5	73.2 $\pm$ 0.6	173	189	220	307	437

Appendix

Table 7: LongHealth with Llama-3.1-8B-Instruct. Full-context ceiling

73.0\pm 0.8\%

	Accuracy (%)					Pipeline time (s)
Method	$100\times$	$50\times$	$20\times$	$10\times$	$5\times$	$100\times$	$50\times$	$20\times$	$10\times$	$5\times$
$L_{2}$ key-norm	41.3 $\pm$ 0.4	42.5 $\pm$ 0.6	45.9 $\pm$ 0.4	52.4 $\pm$ 0.2	68.3 $\pm$ 0.3	0.3	0.3	0.3	0.3	0.4
Q-Filters	46.3 $\pm$ 0.6	50.9 $\pm$ 0.3	59.9 $\pm$ 0.7	70.6 $\pm$ 0.4	77.3 $\pm$ 0.1	0.3	0.4	0.4	0.4	0.3
StreamingLLM	44.7 $\pm$ 0.5	47.5 $\pm$ 0.3	53.8 $\pm$ 0.1	60.9 $\pm$ 0.4	70.2 $\pm$ 0.3	0.3	0.3	0.3	0.3	0.3
KVzip	48.3 $\pm$ 0.2	53.0 $\pm$ 0.4	60.4 $\pm$ 0.3	70.3 $\pm$ 0.7	78.7 $\pm$ 0.2	56	63	66	96	78
H2O	60.6 $\pm$ 0.4	66.5 $\pm$ 0.3	73.7 $\pm$ 0.7	78.3 $\pm$ 0.2	80.5 $\pm$ 0.4	53	53	59	64	76

Appendix

Table 8: QuALITY with Qwen3-30B-A3B. Full-context ceiling

80.7\pm 0.4\%

	Accuracy (%)					Pipeline time (s)
Method	$100\times$	$50\times$	$20\times$	$10\times$	$5\times$	$100\times$	$50\times$	$20\times$	$10\times$	$5\times$
Q-Filters	–	43.2 $\pm$ 0.5	–	–	–	–	–	–	–	–
OBCache	52.7 $\pm$ 0.4	62.8 $\pm$ 0.4	68.6 $\pm$ 0.5	74.4 $\pm$ 0.1	76.7 $\pm$ 0.3	391	395	402	409	424
AM-HAK-fast	50.4 $\pm$ 1.1	59.8 $\pm$ 0.6	66.9 $\pm$ 0.4	74.3 $\pm$ 0.5	77.2 $\pm$ 0.2	358	361	362	372	389
KV-Surgeon -fast	54.7 $\pm$ 0.3	64.1 $\pm$ 0.2	70.2 $\pm$ 0.5	75.0 $\pm$ 0.3	77.1 $\pm$ 0.2	363	366	373	386	413
AM-HAK	51.4 $\pm$ 0.4	61.4 $\pm$ 0.4	69.8 $\pm$ 0.5	73.7 $\pm$ 0.1	76.8 $\pm$ 0.3	1121	1129	1130	1159	1162

Appendix

Table 9: LongHealth with Qwen3-30B-A3B. Full-context ceiling

78.3\pm 0.5\%

. Of the query-free methods, only Q-Filters at

50\times

was run on this combination; KVzip, H2O, and KVMerger were not run.

	Accuracy (%)					Pipeline time (s)
Method	$100\times$	$50\times$	$20\times$	$10\times$	$5\times$	$100\times$	$50\times$	$20\times$	$10\times$	$5\times$
$L_{2}$ key-norm	47.4 $\pm$ 0.5	48.5 $\pm$ 0.3	49.3 $\pm$ 0.2	50.8 $\pm$ 0.1	55.3 $\pm$ 0.5	0.3	0.3	0.3	0.3	0.3
Q-Filters	53.2 $\pm$ 0.5	55.9 $\pm$ 0.3	61.7 $\pm$ 0.4	65.4 $\pm$ 0.4	68.2 $\pm$ 0.4	0.3	0.3	0.3	0.3	0.3
StreamingLLM	49.3 $\pm$ 0.6	50.2 $\pm$ 0.3	52.4 $\pm$ 0.5	55.7 $\pm$ 0.3	61.4 $\pm$ 0.4	0.3	0.3	0.3	0.3	0.3
KVzip	56.0 $\pm$ 0.2	57.6 $\pm$ 0.8	63.1 $\pm$ 0.4	67.2 $\pm$ 0.5	69.1 $\pm$ 0.4	6.0	6.2	6.7	7.4	9.1
H2O	61.5 $\pm$ 0.3	66.2 $\pm$ 0.2	68.1 $\pm$ 0.2	71.1 $\pm$ 0.4	71.3 $\pm$ 0.4	6.0	6.1	6.6	7.4	9.0

Appendix

Table 10: QuALITY with Gemma-3-12B-it. Full-context ceiling

70.6\pm 0.4\%

	Accuracy (%)					Pipeline time (s)
Method	$100\times$	$50\times$	$20\times$	$10\times$	$5\times$	$100\times$	$50\times$	$20\times$	$10\times$	$5\times$
$L_{2}$ key-norm	37.6 $\pm$ 0.5	38.9 $\pm$ 1.2	37.8 $\pm$ 0.6	40.4 $\pm$ 0.4	41.0 $\pm$ 0.6	6.2	6.2	6.3	6.1	6.0
Q-Filters	38.4 $\pm$ 0.2	39.0 $\pm$ 0.7	39.4 $\pm$ 0.9	44.5 $\pm$ 0.8	53.9 $\pm$ 0.7	6.0	6.1	6.3	6.1	6.1
StreamingLLM	40.9 $\pm$ 0.3	39.5 $\pm$ 0.9	39.3 $\pm$ 0.6	42.0 $\pm$ 0.3	44.4 $\pm$ 0.6	6.1	5.9	6.0	6.1	6.3
KVzip	38.1 $\pm$ 0.5	39.8 $\pm$ 0.7	44.5 $\pm$ 0.6	52.9 $\pm$ 0.8	60.7 $\pm$ 0.6	55	59	64	74	95
H2O	38.5 $\pm$ 0.6	47.3 $\pm$ 0.5	55.1 $\pm$ 0.2	63.5 $\pm$ 0.6	68.8 $\pm$ 0.7	54	60	64	74	91

Appendix

Table 11: LongHealth with Gemma-3-12B-it. Full-context ceiling

74.5\pm 0.4\%

	Accuracy (%)					Time (s)
Method	$100\times$	$50\times$	$20\times$	$10\times$	$5\times$	$100\times$	$50\times$	$20\times$	$10\times$	$5\times$
StreamingLLM	22.6 $\pm$ 0.2	23.3 $\pm$ 0.2	22.0 $\pm$ 0.1	24.5 $\pm$ 0.1	32.5 $\pm$ 0.1	7.62	8.07	7.78	8.18	8.39
$L_{2}$ key-norm	23.2 $\pm$ 0.1	23.4 $\pm$ 0.1	23.4 $\pm$ 0.2	23.4 $\pm$ 0.1	27.6 $\pm$ 0.2	7.45	8.55	8.77	7.70	8.12
Q-Filters	22.8 $\pm$ 0.1	22.4 $\pm$ 0.2	23.8 $\pm$ 0.1	27.4 $\pm$ 0.1	34.7 $\pm$ 0.2	7.11	7.41	7.50	8.02	7.82
KVzip	20.5 $\pm$ 0.2	22.4 $\pm$ 0.2	28.6 $\pm$ 0.4	36.2 $\pm$ 0.5	38.3 $\pm$ 0.2	217	232	292	379	596
H2O	22.5 $\pm$ 0.1	24.1 $\pm$ 0.3	30.9 $\pm$ 0.2	36.4 $\pm$ 0.2	39.4 $\pm$ 0.2	221	230	307	385	572

Appendix

Table 12: Code completion accuracy (%) and pipeline time vs. compaction ratio on Long Code Arena project-level code completion with Qwen3-4B-Instruct-2507 (20 Python repository snapshots,

n=650

next-line completions, exact match of the first generated line). Each entry is the mean over five seeds; the grey number is the standard error over seeds. Time is average query-generation plus compaction wall-clock per repository, in seconds. Full-context ceiling:

40.6\%

. The best accuracy per ratio is bold; underlined entries are within

2\sqrt{\mathrm{se}^{2}+\mathrm{se}_{\mathrm{best}}^{2}}

of the best. ‘–’ marks ratios the AM-OMP solvers do not reach.

Final refit off (plain eviction on the selected support)
		Default		Ablated
Ablation	Method	Acc. (%)	Time (s)	Acc. (%)	Time (s)
	LOO	65.6 $\pm$ 0.2	38	60.2 $\pm$ 0.4	35
Mass term off ( $\gamma_{0}=0$ )
	KV-Surgeon	66.8 $\pm$ 0.3	41	67.0 $\pm$ 0.4	43
Mild-compaction fallback off (threshold $0.3\to 1.0$ )

Appendix

Table 13: One-component ablations on Qwen3-4B with QuALITY at

50\times

, all

50

articles, five seeds, one standard error over seeds. Time is pipeline time per context (calibration-query generation plus compaction). The self-study controls are the runs of Table 1 ; the fast-tier control is the fast configuration in use at the time of the ablation and is read only against its own row. The head-budget ablation is in Table 15 .

		uniform budgets		optimized budgets
Method	Queries	no refit	refit	+ refit
AM-HAK	self-study	$54.1\pm 0.2$	$57.2\pm 0.5$	$64.1\pm 0.9$
LOO	self-study	$53.2\pm 0.5$	$57.8\pm 0.6$	$65.6\pm 0.2$
KV-Surgeon	self-study	$58.2\pm 0.4$	$60.2\pm 0.3$	$66.8\pm 0.3$
AM-OMP	self-study	–	$59.0\pm 0.5$	$66.4\pm 0.2$
AM-HAK-fast	repeat	$48.1\pm 0.5$	$53.2\pm 0.3$	$58.1\pm 0.5$

Appendix

Table 15: The two shared pipeline components switched independently, at

50\times

on Qwen3-4B / QuALITY,

50

articles, five seeds: the support-fixed refit (values and multipliers) and the optimized per-head budgets. All methods use our calibration queries; the AM-OMP value reconstruction cannot be disabled. LOO with optimized budgets and no refit is the plain-eviction row of Table 13 ; the ported eviction methods are in Table 16 .

Method	as published	calib. queries	+ refit	+ budgets
H2O	$47.4\pm 0.5$	$48.7\pm 0.4$	$56.1\pm 0.4$	$62.3\pm 0.4$
OBCache	$43.2\pm 0.2$	$53.0\pm 0.4$	$58.1\pm 0.3$	$62.4\pm 0.7$
KVMerger	$47.7\pm 0.5$	$47.6\pm 0.2$	$51.5\pm 0.5$	$57.2\pm 0.5$
KVzip	$36.4\pm 0.3$	$42.6\pm 0.3$	$43.5\pm 0.6$	$49.8\pm 0.4$

Appendix

Table 16: Each external baseline from its published configuration to our pipeline, one component at a time, at

50\times

on Qwen3-4B / QuALITY. As published: own queries (H2O and KVMerger use context prefill, OBCache its last-64 observation window, KVzip its repeat prompt with a global cross-head threshold), uniform or own head allocation, no refit. The remaining columns use repeat-pass calibration queries.

	$100\times$			$50\times$
	All	Mixed	Mixed	All	Mixed	Mixed
Multipliers	compacted	compacted	uncompacted	compacted	compacted	uncompacted
Refit (default)	57.5 $\pm$ 0.8	58.0 $\pm$ 0.7	78.2 $\pm$ 0.6	66.9 $\pm$ 0.5	67.9 $\pm$ 0.6	79.4 $\pm$ 0.5
From the elimination	56.7 $\pm$ 0.4	56.5 $\pm$ 0.7	79.2 $\pm$ 0.5	68.5 $\pm$ 0.7	67.8 $\pm$ 0.3	78.7 $\pm$ 0.5
Unit	55.9 $\pm$ 0.6	45.8 $\pm$ 0.8	81.3 $\pm$ 0.2	65.2 $\pm$ 0.6	59.0 $\pm$ 0.9	80.6 $\pm$ 0.5

Appendix

Table 17: Accuracy (%) by served multipliers, KV-Surgeon -fast on LongHealth with Qwen3-4B-Instruct-2507, five seeds, standard error over seeds in grey . All compacted : all five records compacted. Mixed : two or three records served uncompacted; accuracy on questions about the compacted and the uncompacted records is reported separately. Each column covers all

400

questions.

Ratio	$\tilde{w}<0$	median head	heads w/ any	rel. move	corr.	at bounds
$100\times$	$0.42\%$	$0.00\%$	$13.6\%$	$0.81$	$0.39$	$1.3\%$ / $14.7\%$
$50\times$	$0.33\%$	$0.00\%$	$19.4\%$	$0.70$	$0.51$	$1.3\%$ / $9.4\%$
$20\times$	$0.24\%$	$0.00\%$	$24.6\%$	$0.57$	$0.62$	$1.8\%$ / $5.9\%$
$10\times$	$0.24\%$	$0.00\%$	$28.0\%$	$0.54$	$0.68$	$2.9\%$ / $5.5\%$
$5\times$	$0.28\%$	$0.00\%$	$33.0\%$	$0.58$	$0.70$	$4.7\%$ / $4.1\%$

Appendix

Table 18: Diagnostics of the unconstrained multiplier coefficients

\tilde{w}\triangleq B_{:,d_{v}+1}/\gamma

that the elimination leaves on the final support and of the served multipliers

w

after the box-constrained refit of 22 , per ratio, on Qwen3-4B / QuALITY (seed 0, all 14,400 head instances; heads served by fallback or zero budget excluded). Columns: fraction of kept entries with

\tilde{w}_{j}<0

(pooled over heads, and the median head); fraction of heads with at least one negative coefficient; median per-head relative movement

\lVert w-\tilde{w}\rVert_{2}/\lVert\tilde{w}\rVert_{2}

and correlation of

w

against

\tilde{w}

; fraction of served multipliers pinned at the bounds

e^{\pm\beta}

\beta=3

KV-Cache Management

Key-value (KV) cache management reduces and coordinates Transformer attention state to make long-context inference faster and more memory-efficient.

See all 201 papers

Fast KV Compaction via Attention Matching
Feb 18, 2026Adam Zweiger, Xinghong Fu, Han Guo +1KV-Cache Management LLM Context Compression
Massachusetts Institute of Technology.
Practical Online KV Cache Compaction for LLM Agents: An Empirical Study
Aug 2, 2026Yujian Liu, Jiabao Ji, Li An +4KV Caching KV-Cache Eviction
UC Santa Barbara · LinkedIn
ARC-KV: Amortizing Anchor Search for Reconstruction-Based KV Cache Compaction
Sep 29, 2026Zheyu Shen, Guanhua Wang, Dezhan Tu +5Long-Context Language Modeling KV-Cache Management
University of Maryland, College Park · Meta

Feb 18, 2026J/K move · Enter open · S save

Fast KV Compaction via Attention Matching

Adam Zweiger, Xinghong Fu, Han Guo, Yoon Kim

Massachusetts Institute of Technology.

KV-Cache Management LLM Context Compression KV-Cache Compression Long-Context Language Modeling LLM Inference Acceleration

Open paper AskPDF arXiv

Scaling language models to long contexts is often bottlenecked by the size of the key-value (KV) cache. In deployed settings, long contexts are typically managed through compaction in token space via summarization. However, summarization can be highly lossy, substantially harming downstream performance. Recent work on Cartridges has shown that it is possible to train highly compact KV caches in latent space that closely match full-context performance, but at the cost of slow and expensive end-to-end optimization. This work describes an approach for fast context compaction in latent space through Attention Matching, which constructs compact keys and values to reproduce attention outputs and preserve attention mass at a per-KV-head level. We show that this formulation naturally decomposes into simple subproblems, some of which admit efficient closed-form solutions. Within this framework, we develop a family of methods that significantly push the Pareto frontier of compaction time versus quality, achieving up to 50x compaction in seconds on some datasets with little quality loss.

Abstract

KV-Cache Management

Explore similar work

Fast KV Compaction via Attention Matching

Practical Online KV Cache Compaction for LLM Agents: An Empirical Study

ARC-KV: Amortizing Anchor Search for Reconstruction-Based KV Cache Compaction