KV-Kaizen: Learning Context-Adaptive Cache Compression Choices

Joao Monteiro; Louis Béthune; Anastasiia Filippova; Sonia Laguna; David Grangier; Marco Cuturi

cs.AISep 29, 2026

KV-Kaizen: Learning Context-Adaptive Cache Compression Choices

Authors: Joao Monteiro, Louis Béthune, Anastasiia Filippova, Sonia Laguna, David Grangier, Marco Cuturi

Organizations: Apple

Abstract

As the context size of text processed with an LLM grows, the size of KV caches can outstrip the memory allocated for the original model weights. This impacts LLM throughput negatively, since decoding is memory-bound and decode cost grows with cache size. Recent work alleviates this bottleneck by discarding the least relevant tokens. Eviction introduces a tension, since a one-off decision to discard content may prove detrimental later. Instead, we focus on alternative choices that can lead to cache compression without evicting tokens. We achieve this by learning a selector that is able to produce, based on context, a per-layer cache configuration towards an overall compression budget. The selector operates along three axes: sharing one cache across layers (depth), caching at fewer bits (precision), or truncating the low-rank latent cache representations (rank). We call the resulting method KV-Kaizen, for the many small per-layer choices it compounds. We observe that these interventions taken independently and uniformly over all layers limit achievable compression because they degrade accuracy. Crucially, composing them locally and adaptively to the context can instead preserve accuracy while achieving large memory savings. At inference, the selector runs once, before pre-fill. In evaluations on instruction following and reasoning tasks, our selectors reach the Pareto frontier of accuracy against cache size, against learning-free and post-hoc baselines. On long-context tasks, KV-Kaizen improves on eviction and can be composed with it, reaching a 32x smaller decode-time cache on a 14B model while preserving accuracy. A 4x cache size reduction incurs no accuracy degradation from 7B parameters up, and a compressed model is more accurate than a smaller uncompressed one with the same cache size. Together, these findings support pre-training large models and compressing them only afterwards.

Figures & tables

Figure 1 : Per-layer cache selection and composition of compression axes. The selector chooses among $16$ width–precision pairs or, alternatively, a inherit/reuse action per layer. Producing a cache costs $(d+d_{R})b$ bits per token; inheritance costs zero. The configuration is fixed during generation, including at prefill time. In theory, the maximal compression factor that could be attained for a 48 layers model such as Qwen2.5-14B could be in practice $8\times$ (precision) times $10.7$ (lowest rank), and depth $48$ , by reusing the first KV layer at all subsequent layers ( Table 5 ). This would yield a maximal compression of $4108\times$ , though all of our experiments target far lower factors.

Figure 2 : Selector and pre-fill latency versus input length. Measurements use Qwen2.5-7B at

4\times

compression, batch size

1

, on one H100. The selector takes about

3

ms, or

3.3\%

–

0.20\%

of pre-fill latency.

Figure 3 : Accuracy versus cache compression on Qwen2.5-7B . Accuracy is the mean of IFEval and GSM8K; compression is relative to the full

7

B bf16 cache. The starred curve shows uncompressed models at four sizes. The black curve is the best KVK run per budget; the pale band gives the Pareto frontier of all methods shown. Static and random configurations are fine-tuned. KIVI: grouped, bf16 residual.

Panel of Table 1 — Table 1: Mean IFEval and GSM8K accuracy differences at $4\times$ cache reduction. The $3$ B, $7$ B, and $14$ B columns use Qwen2.5 ; Ref. identifies the size-matched control (MLA for rank, non-MLA otherwise). The first row gives absolute accuracies; subsequent rows give differences from Ref., with higher values better. Bold marks differences $\geq-0.053$ , ie the largest observed seed range ( Table 16 ).

Figure 4 : IFEval and GSM8K accuracy versus realized cache compression on Qwen2.5-14B . Each curve uses a different subset of compression axes: hue marks rank, shade and marker mark precision, and a dashed line marks depth. Horizontal lines mark the GQA and MLA controls.

Panel of Table 2 — Table 2: RULER accuracy at $\bm{16}$ k on Qwen2.5-14B . NIAH multi-key and QA report retrieval and question-answering accuracy over $275$ documents each. $\rho$ is the decode-time cache reduction relative to the uncompressed GQA model; keep is the retained token fraction, and $G$ the quantization group size. Parentheses give QA differences from the SFT control. Single-needle accuracy is $1.000$ throughout and therefore omitted from the table.

Panel of Table 3 — Table 3: Mean IFEval and GSM8K accuracy with budget-conditioned selectors on Qwen2.5-7B . $\rho_{\star}$ is the requested cache reduction factor. Against $\{1,2,4\}$ , the uncompressed control scores $0.679$ at $1\times$ , and single-target selectors $0.690$ at $2\times$ (precision only) and $0.679$ at $4\times$ . With all three approaches the MLA control scores $0.574$ ; requests are met within $2\%$ except $1.94\times$ , $1.78\times$ and $1.90\times$ at $2\times$ , in row order, and $3.81\times$ for the $540$ k selector at $4\times$ .

Figure 5 : Pre-fill time, decode throughput and peak decode memory on an Apple M3 Ultra. Qwen2.5-14B with

4

-bit weights at batch size

1

. Hue marks GQA or an MLA latent of rank

128

(rank), shade and marker a bf16 or

4

-bit cache (precision), and a dashed line

12

groups of

4

layers sharing a cache (depth). Pre-fill time is reported relative to the uncompressed cache in bf16 precision.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Backbone	Layers $L$	KV heads $n_{\text{kv}}$	Scalars/token/layer $2n_{\text{kv}}d_{h}$	MLA width $r$
Qwen2.5-1.5B-Instruct	28	2	512	256
Qwen2.5-3B-Instruct	36	2	512	256
Qwen2.5-7B-Instruct	28	4	1024	512
Qwen2.5-14B-Instruct	48	8	2048	1024
Qwen2.5-32B-Instruct	64	8	2048	1024
Mistral-7B-Instruct-v0.3	32	8	2048	1024

Appendix

Table 4: Backbone dimensions used to compute cache size.

L

is the number of layers,

n_{\text{kv}}

the number of KV heads per layer, and

2n_{\text{kv}}d_{h}

the number of cached scalars per token per layer. All models have head dimension

d_{h}=128

. The converted MLA latent has width

r=n_{\text{kv}}d_{h}

, excluding the

64

-dimensional positional key.

			Qwen2.5-14B		Qwen2.5-3B
Approaches	Backbone	Actions	Plans	$[\rho_{\min},\rho_{\max}]$	Plans	$[\rho_{\min},\rho_{\max}]$
Precision	GQA	4	$10^{29}$	$1.00$ – $8.0\times$	$10^{22}$	$1.00$ – $8.0\times$
Depth	GQA	2	$10^{14}$	$1.00$ – $48.0\times$	$10^{11}$	$1.00$ – $36.0\times$
Depth + precision	GQA	5	$10^{34}$	$1.00$ – $384\times$	$10^{25}$	$1.00$ – $288\times$
Rank	MLA	4	$10^{29}$	$1.88$ – $10.7\times$	$10^{22}$	$1.60$ – $5.3\times$
Rank + precision	MLA	16	$10^{58}$	$1.88$ – $85.3\times$	$10^{43}$	$1.60$ – $42.7\times$

Appendix

Table 5: Action counts and feasible cache reduction factors. Approaches lists the enabled compression axes; backbone identifies the attention architecture. Actions counts choices per layer, Plans counts assignments across

L

layers before budget and first-layer constraints, and

[\rho_{\min},\rho_{\max}]

gives minimum and maximum cache reduction relative to the original bf16 GQA cache. The two models have

(L,2n_{\text{kv}}d_{h},r)=(48,2048,1024)

and

(36,512,256)

. Rank uses MLA with positional-key width

d_{R}=64

. Bit-widths are

\{2,4,8,16\}

and latent widths

\{r/8,r/4,r/2,r\}

Scheme	Bit-width $b$	Bits/element $\downarrow$	$\rho\uparrow$
Our quantizer	8	8.125	1.97 $\times$
Our quantizer	4	4.125	3.88 $\times$
Our quantizer	2	2.125	7.53 $\times$
KIVI, $g$ =32	4	5.688	2.81 $\times$
KIVI, $g$ =64	4	5.219	3.07 $\times$
KIVI, $g$ =128	4	4.984	3.21 $\times$

Appendix

Table 6: Storage cost and cache reduction with quantization metadata.

b

is the quantization bit-width; bits/element includes metadata and any full-precision residual window.

\rho

16

divided by this storage cost, relative to bf16 . Our quantizer stores one bf16 scale per

128

-element token-head row. KIVI stores a scale and zero-point per group of

g

elements and a

128

-token bf16 window in a

2048

-token context. Bold highlights the factors used in the text.

Approaches	Target $\rho_{\star}$	Adaptive	Swept	Accuracy difference $\uparrow$
		Mean accuracy $\uparrow$
All three, $7$ B	$8\times$	$0.541$	$0.534$	$-0.007$
Precision, $7$ B	$4\times$	$0.670$	$0.668$	$-0.002$
All three, $14$ B	$16\times$	$0.610$	$0.571$	$-0.039$
All three, $7$ B	$16\times$	$0.429$	$0.448$	$+0.019$
All three, $14$ B	$8\times$	$0.622$	$0.644$	$+0.022$

Appendix

Table 7: Mean IFEval and GSM8K accuracy with adaptive and swept constant rate multipliers. Rows identify the compression axes and Qwen2.5 model size;

\rho_{\star}

is the target cache reduction factor. Adaptive and Swept give accuracies; the last column subtracts adaptive from swept accuracy. Pairs differ by at most

2\%

in realized compression. The upper six accuracy differences are within

0.053

, the largest observed seed range.

Figure 6 : Compression factor (left) and rate multiplier (right) during training. Ten runs use all three axes on Qwen2.5 at

1.5

3

7

, and

14

B, with targets

4\times

8\times

, and

16\times

;

1.5

B is evaluated only at

4\times

. Color denotes model size and line style denotes target. Training progress is normalized by

270{,}000

steps.

Approaches	Loss range	Per-group loss gain $\uparrow$	Mean regret $\downarrow$
Depth only	$1.4826$	$+0.0000$	$-0.4988$
All three	$0.3014$	$+0.0048$	$-0.0143$

Appendix

Table 8: Loss differences between learned and alternative cache configurations. We score

24

alternatives on

128

prompts with trained

7

B weights fixed and cache costs approximately matched. Negative regret favors the selector. All differences are in nats.

	Measured $\downarrow$		Predicted
Configuration	$4\times$	$8\times$	$8\times$
Searched	$0.005$	$0.030$	$0.022$
Random	$0.125$	$1.253$	–
Uniform	–	$4.00$	$0.264$

Appendix

Table 9: Loss increase from compressing the uncompressed MLA control of Qwen2.5-7B without training. Measured is the rise in teacher-forced loss on calibration data; predicted is the sum of per-layer rises that the search minimizes. Random is the best of

64

random configurations at the same budget. No uniform configuration meets the

4\times

budget exactly, and dashes mark values not measured. All values are in nats.

		Accuracy $\uparrow$
Configuration	$\rho\uparrow$	IFEval	GSM8K	Mean
Qwen2.5-7B-Instruct , base	1.00 $\times$	0.706	0.701	0.704
Our SFT, uncompressed (control)	1.00 $\times$	0.719	0.639	0.679
+ post-hoc 4-bit, plain	3.88 $\times$	0.113	0.001	0.057 ( $-0.622$ )
+ trained under KIVI, 4-bit $G$ =32	2.81 $\times$	0.701	0.680	0.690 ( $+0.011$ )
+ post-hoc KIVI, 4-bit $G$ =32	2.81 $\times$	0.721	0.635	0.678 ( $-0.001$ )

Appendix

Table 10: Cache reduction and task accuracy with quantization, trained compression, and smaller models.

\rho

is the cache reduction factor relative to the full Qwen2.5-7B bf16 cache. IFEval and GSM8K report task accuracies on

[0,1]

; Mean is their arithmetic mean. Parentheses give mean-accuracy differences from the SFT control; bold marks means within

0.053

of that control or higher.

G

is the quantization group size. Post-hoc factors include metadata; trained precision gives

4.00\times

before metadata and

3.88\times

after it.

		Accuracy $\uparrow$
Approaches	Ref.	IFEval	GSM8K	Mean
Uncompressed (GQA)	–	0.719	0.639	0.679
MLA conversion only	GQA	0.636	0.513	0.574 ( $-0.105$ )
Precision	GQA	0.723	0.617	0.670 ( $-0.009$ )
Depth	GQA	0.547	0.245	0.396 ( $-0.283$ )
Precision + depth	GQA	0.721	0.637	0.679 ( $-0.000$ )

Appendix

Table 11: Task accuracy by compression axis on Qwen2.5-7B at

4\times

cache reduction. IFEval and GSM8K report task accuracies on

[0,1]

; Mean is their arithmetic mean. Ref. identifies the control used for parenthesized mean-accuracy differences: MLA for configurations using rank, GQA otherwise. The conversion-only row is compared with GQA. Bold marks means within

0.053

of the corresponding control or higher

		Accuracy $\uparrow$
Configuration	$\rho\uparrow$	NIAH single	NIAH multi-key	QA (SQuAD)
No compression (control)	$1.0$	1.000	0.985	0.540
KVK , $4\times$ (precision + depth)	$4.0$	0.996	0.956	0.510 ( $-0.030$ )
+ SnapKV , keep $0.25$	$16.0$	0.985	0.887	0.472 ( $-0.068$ )
KVK , $3.9\times$ (precision only)	$3.9$	0.978	0.913	0.508 ( $-0.032$ )
KVK , $4\times$ (depth only)	$4.0$	0.964	0.324	0.107 ( $-0.434$ )

Appendix

Table 12: RULER accuracy at

\bm{16}

k on Qwen2.5-7B .

\rho

is the decode-time cache reduction factor relative to the uncompressed GQA model. NIAH single and multi-key report retrieval accuracy; QA reports SQuAD question-answering accuracy. Parentheses give QA differences from the control. Keep is the retained token fraction, and

G

the quantization group size. Rank uses MLA; other rows use GQA. Bold identifies KVK configurations and their combinations with eviction. The last four policies select tokens before the query arrives.

$32\times$ in total
Configuration	$\rho\uparrow$	QA (SQuAD) $\uparrow$
Uncompressed control	$1.0$	$0.543$
KVK $4\times$ + SnapKV, keep $0.125$	$32.0$	$0.531\,(-0.012)$
KVK $8\times$ + SnapKV, keep $0.25$	$32.0$	$0.491\,(-0.052)$
Without eviction
KVK $16\times$	$16.0$	$0.258\,(-0.285)$

Appendix

Table 13: Question answering at

16

k on Qwen2.5-14B when a larger cache reduction is split between our compression and eviction.

\rho

is the decode-time cache reduction relative to the uncompressed GQA model, and keep the fraction of positions SnapKV retains. KVK configurations use precision and depth. Parentheses give differences from the uncompressed control; the

32\times

configuration with keep

0.125

0.012

below it. In the lower block, KVK and four-bit KIVI with

G=128

share the evictor, and KVK is ahead by

0.098

0.074

and

0.067

Backbone	Untuned	After our SFT	Accuracy change
	Mean accuracy $\uparrow$
Qwen2.5-0.5B	0.297	0.436	$+0.139$
Qwen2.5-1.5B	0.488	0.577	$+0.090$
Qwen2.5-3B	0.581	0.673	$+0.093$
Qwen2.5-7B	0.704	0.679	$-0.025$
Qwen2.5-14B	0.774	0.723	$-0.052$

Appendix

Table 14: Mean IFEval and GSM8K accuracy before and after SFT without cache compression. Untuned and After our SFT report arithmetic means of the two task accuracies. The Accuracy change is the score after SFT minus the untuned score, computed before rounding.

Which layers hold the anchors: only depth available, GQA backbone, Qwen2.5-3B
		Mean accuracy $\uparrow$ at target $\rho_{\star}$
Sharing configuration	Inheriting fraction	$4\times$	$8\times$
Uncompressed control	0.000	0.673	–
Learned	0.750	0.320 ( $-0.354$ )	–
Spaced evenly	0.750	0.339 ( $-0.334$ )	–
Drawn at random	0.750	0.360 ( $-0.313$ )	–

Appendix

Table 15: Task accuracy with learned and fixed cross-layer sharing. Inheriting fraction is the fraction of layers reading another layer’s cache;

4\times

and

8\times

columns give arithmetic mean IFEval and GSM8K accuracy at those cache reduction targets. Control rows give uncompressed accuracy. Parentheses subtract the block’s control; dashes indicate unavailable comparisons. Bold marks scores within

0.053

of the best compressed configuration in each block and column, excluding the starred entry. In the third block, learned, fixed-sharing, and no-sharing runs realize

4.00

3.94

, and

4.00\times

, respectively, and

8.00\times

at the higher target. ∗ Constant rate multiplier; other compressed rows use the adaptive rule.

Configuration	Target $\rho_{\star}$	Seeds	Mean accuracy $\uparrow$
Uncompressed, $7$ B	$1\times$	3	0.679, 0.635, 0.626
Uncompressed, $14$ B	$1\times$	2	0.723, 0.681
Precision + depth, $7$ B	$4\times$	3	0.679, 0.679, 0.652
Precision + depth, $7$ B	$8\times$	3	0.255, 0.262, 0.288
Precision + depth, $14$ B	$16\times$	2	0.322, 0.295
All three, $3$ B	$4\times$	3	0.327, 0.334, 0.347

Appendix

Table 16: Mean IFEval and GSM8K accuracy across training seeds. Rows identify the Qwen2.5 model size and compression axes.

\rho_{\star}

is the target cache reduction factor; Seeds counts training runs. Mean accuracy lists the arithmetic mean accuracy for each seed. Bold marks the largest accuracy range,

0.053

. Other tables use the first listed seed.

At a matched realized $4\times$
	Mean accuracy / difference $\uparrow$
Approaches	1.5B	3B	7B	14B	32B
Uncompressed control	0.577	0.673	0.679	0.723	0.793
Untuned model	0.488	0.581	0.704	0.774	0.779
Precision	$-0.034$	$-0.041$	$\mathbf{-0.009}$	$\mathbf{+0.040}$	$\mathbf{+0.016}$
Precision + depth	–	$-0.063$	$\mathbf{-0.000}$	$\mathbf{+0.031}$	$\mathbf{+0.030}$

Appendix

Table 17: Mean IFEval and GSM8K accuracy differences across Qwen2.5 model sizes. Columns give parameter counts; blocks give realized cache reduction factors. The first two rows report absolute arithmetic mean accuracies after and before SFT. Other entries subtract the size-matched MLA control for configurations using rank, or GQA control otherwise. Bold marks differences

\geq-0.053

; dashes indicate unavailable runs.

	Mean accuracy $\uparrow$ at requested $\rho_{\star}$
Training targets	$1\times$	$2\times$	$4\times$	$8\times$	$16\times$	$32\times$	Final $\beta$
Precision and depth
$\{1,2,4\}$	$0.703$	$0.659$	$0.612$	–	–	–	$355$
$\{1,2,4,8\}$	$0.672$	$0.395\,(2.31\times)$	$0.355$	$0.353$	–	–	$886$
$\{1,2,4,8,16\}$	$0.655$	$0.471\,(2.33\times)$	$0.269\,(5.47\times)$	$0.206\,(9.00\times)$	$0.213$	–	$2251$
$\{1,2,4,8,16,32\}$	$0.585$	$0.263$	$0.100\,(6.78\times)$	$0.107\,(20.4\times)$	$0.106\,(32.0\times)$	$0.106$	$3722$

Appendix

Table 18: Mean IFEval and GSM8K accuracy with selectors trained on wider sets of targets, on Qwen2.5-7B .

\rho_{\star}

is the requested cache reduction factor, and blocks follow Table 3 . Parentheses give the realized factor where a selector misses the request by more than

2\%

. Final

\beta

is the rate multiplier at the end of training. Dashes mark targets outside the training set or values not recorded.

		Inheriting fraction at target $\rho_{\star}$
Approaches	Backbone	$4\times$	$8\times$	$16\times$
Precision + depth	Non-MLA	$0.000$	$0.021$ – $0.107$	$0.500$ – $0.528$
Depth + rank	MLA	–	–	$0.333$ – $0.667$
All three	MLA	$0.000$	$0.000$	$0.000$

Appendix

Table 19: Fraction of layers that inherit another layer’s cache in the selector’s configuration. Ranges span the evaluated model sizes, and the all-three row covers eleven fixed-target configurations. Past

8\times

on a non-MLA backbone, seven of

73

trained configurations reach the factor, all with

50\%

94\%

of layers inheriting. One budget-conditioned configuration with all three approaches inherits at

3.66\times

, for

21.5\%

of its memory saving. Dashes mark values not recorded.

One configuration
		Distinct configurations / prompts
Model size	Target $\rho_{\star}$	IFEval	GSM8K	RULER	All three
$3$ B	$4\times$	$1/64$	$1/64$	$1/96$	$1/224$
$3$ B	$16\times$	$1/64$	$1/64$	$1/96$	$1/224$
$7$ B	$4\times$	$1/64$	$1/64$	$1/96$	$1/224$
$7$ B	$8\times$	$1/64$	$1/64$	$1/96$	$1/224$

Appendix

Table 20: Number of distinct per-layer cache configurations returned by selectors on Qwen2.5 . Rows give the model size and the target cache reduction factor

\rho_{\star}

. Each entry is the number of distinct configurations a selector returns over the prompts sampled from a dataset, out of the number of prompts:

64

from IFEval,

64

from GSM8K, and

96

from RULER,

32

at each of

4

8

k, and

16

k tokens. The last column pools all

224

prompts. Selectors in the upper block use all three approaches. † Rate multiplier capped at

50

. ‡ Constant rate multiplier; this selector realizes

7.73\times

		Mean accuracy $\uparrow$
Selector	Input	Precision	All three
Transformer (ours)	Token embeddings	$0.670$	$0.547$
Two-layer MLP	Mean-pooled embeddings	$0.660$	$0.510$
Linear heads	Last-layer hidden states	$0.674$	$0.601$

Appendix

Table 21: Mean IFEval and GSM8K accuracy with three selector architectures on Qwen2.5-7B at

4\times

. Every selector realizes

4.00\times

except the hidden-state variant with all three approaches, which realizes

3.86\times

Figure 9 : Qwen2.5-14B (4 bits weights) with bf16 KV-cache.

Explore similar work

Apr 22, 2026cs.LG

LKV: End-to-End Learning of Head-wise Budgets and Token Selection for LLM KV Cache Eviction

Long-context inference in Large Language Models (LLMs) is bottlenecked by the linear growth of Key-Value (KV) cache memory. Existing KV cache compression paradigms are fundamentally limited by heuristics: heuristic budgeting relies on statistical priors rather than task objectives, causing resource misallocation, while heuristic selection relies on coupled query-key interactions or static inductive biases (e.g., attention sinks). To address this limitation, we introduce LKV (Learned KV Eviction), which formulates KV compression as an end-to-end differentiable optimization problem. LKV integrates LKV-H to learn task-optimized global budgets, and LKV-T to derive intrinsic KV importance without materializing attention matrices. This design bypasses heuristic proxies, strictly aligning compression with task objectives. Extensive evaluations demonstrate that LKV achieves state-of-the-art performance on both LongBench and RULER benchmarks at high compression rates. In particular, on LongBench, LKV achieves near-lossless performance with only 15% KV cache retention. Crucially, our analysis identifies learned budgeting as the dominant driver of fidelity, demonstrating that data-driven allocation is essential to overcome the limitations of hand-crafted heuristics.

Enshuai Zhou, Yifan Hao, Chao Wang +7

University of Science and Technology of China, Hefei, China · State Key Lab of Processors, Institute of Computing Technology, CAS, Beijing, China · University of Chinese Academy of Sciences, Beijing, China

Key-Value Cache Compression Large Language Model Compression Question-Answering Benchmarks Token Budget Allocation Inductive Bias

Ask PDF

Jun 30, 2026cs.CL

SeKV: Resolution-Adaptive KV Cache with Hierarchical Semantic Memory for Long-Context LLM Inference

Large language models increasingly operate over long contexts, where the KV cache becomes a dominant memory bottleneck: its size grows linearly with sequence length and must be retained throughout decoding, making full GPU caching prohibitively expensive without compression. Existing KV cache compression methods struggle to balance efficiency with faithful context preservation. Token eviction discards information, while semantic grouping fixes compression decisions at prefill time; neither can recover token-level detail from a compressed span once it becomes relevant during generation. As a solution, we propose SeKV, a resolution-adaptive semantic KV cache that organizes context into entropy-guided semantic spans and stores them across a GPU-CPU memory hierarchy without discarding information. Each span keeps a lightweight summary vector on GPU for coarse routing and a low-rank SVD basis on CPU for on-demand token-level reconstruction. A trained zoom-in mechanism selectively expands query-relevant spans during decoding, enabling precise retrieval without materializing the full KV cache on GPU. SeKV enables adaptive token-level reconstruction while keeping the base LLM fully frozen and adding fewer than 0.05% trainable parameters. Across four benchmarks, SeKV improves over the strongest semantic compression baseline by 5.9% on average while reducing GPU memory by 53.3% versus full KV caching at 128K context. Code is available on https://github.com/AmirAbaskohi/SeKV.

Amirhossein Abaskohi, Giuseppe Carenini, Peter West +1

University of British Columbia · Microsoft Research

Key-Value Cache Compression Key-Value Cache Efficient Long-Context Inference Graphics Processing Unit Memory LLM Inference Optimization Long-Term Memory+1

Ask PDF

Aug 3, 2026cs.LG

AnchorKV: Anchor-Residual KV Cache Compression

The key-value (KV) cache is the primary memory bottleneck in long-context LLM inference. Existing approaches attack it from opposite ends: eviction methods permanently discard tokens, degrading performance whenever a discarded token later proves essential, while quantization methods retain all tokens at low precision but offer limited compression. We propose AnchorKV, a compression scheme that shrinks the cache by

20\times

without discarding a single token. AnchorKV represents the cache using a small set of anchors stored exactly, expresses every other token through its most similar anchor, and refines only those whose approximation most affects the model's output. AnchorKV consistently preserves accuracy across models and datasets, retaining 99% of the full-cache score at the 70B scale, while keeping the entire context at a fraction of its cost.

Malik Khalaf, Yara Shamshoum, Nitzan Hodos +2

Department of Computer Science, Technion - Israel Institute of Technology

Key-Value Cache Compression Key-Value Cache Cache Large Language Model Compression LLM Inference Optimization Data Compression Methods+1

Ask PDF