BellowsKV: One Latent for Many Layers, Expanded on Demand

Jiamu Zhang; Hanjie Chen

cs.LGOct 26, 2026

BellowsKV: One Latent for Many Layers, Expanded on Demand

Authors: Jiamu Zhang, Hanjie Chen

Organizations: Rice University

Abstract

Large language models (LLMs) now serve as the backbone of agents that reason over many turns, call tools, and pursue long-horizon tasks. To avoid recomputation, a model stores the attention keys and values of all past tokens in a key-value (KV) cache. Each layer maintains its own cache, so KV-cache memory can become the main bottleneck for GPU serving. Most compression methods operate within a single layer, by quantizing the cache entries, reducing the channel dimension, or evicting less important tokens. All three lose substantial accuracy at high compression ratios. Cross-layer methods share one compressed cache across adjacent layers to go beyond this limit. However, these methods also degrade sharply at high compression ratios. We find that the KV-cache representations of adjacent layers are quite different, whereas the input hidden states of adjacent layers are highly similar. We also find that, under the same memory budget, reducing the dimension of this shared latent loses far more accuracy than quantizing it. Based on these findings, we propose BellowsKV, a training-free method that maps the hidden states of n adjacent layers to one shared latent per token. At inference time, each layer reads its own keys and values from this latent through a layer-specific projection. We keep the latent at full dimension and compress it in two other ways: BellowsKV-Quant stores the latent at low precision, which is nearly lossless because the latent has no dominant outlier channels; BellowsKV-Evict makes one shared eviction decision per block, because the layers rank token importance very similarly. Across Llama-3.1-8B, Mistral-7B and Qwen2.5-7B, BellowsKV-Quant retains about 99% of full-cache accuracy at 3.6x compression and 97-98% at 13-14x, where the strongest cross-layer method retains at most 84% at 16x. Code: https://github.com/MorrisZJ/BellowsKV

Figures & tables

Figure 1: Comparison of different cross-layer KV-Cache compression methods : (a) Trained-to-share architectures reach one cache only by retraining. (b) Training-free methods factorize the caches (c) BellowsKV folds the block’s hidden states into one latent, buying the compression ratio from the numbers of shared layers, bits and tokens instead.

Figure 2: (a) Cosine similarity between adjacent layers. The keys and values are close to zero, while the hidden states reach

0.87

on Llama and

0.82

on Qwen. (b) Per-channel dynamic range of the keys. The raw cache concentrates the range in a few channels (top). A data-free Hadamard rotation and our block-level projection are both flat (bottom), and the relative INT4 error falls from

0.156

to about

0.10

. (c) Spearman correlation between the token importance scores of adjacent layers, between

0.76

and

0.80

Llama-3.1-8B-Instruct
Method	Comp.	Qsp	MFQ	HQA	MSQ	NQA	Gov	QMS	PsR	LCC	RBP	2Wk	Avg
MiniCache fp16	$1.31\times$	72.1	72.3	74.7	86.8	55.0	82.7	89.5	70.7	80.4	82.6	55.6	74.8
KVSharer fp16	$1.33\times$	91.5	96.9	93.9	98.1	78.2	73.7	97.9	99.5	85.4	84.5	70.2	88.2
BellowsQ $n{=}2$ INT8	$3.64\times$	95.6	97.6	99.6	99.4	95.9	98.5	102.5	100.0	98.7	99.0	101.0	98.9
CommonKV	$3.95\times$	92.2	90.6	98.0	100.1	89.0	89.7	96.5	99.5	98.4	97.8	91.1	94.8
xKV	$4.00\times$	96.6	98.6	96.0	102.1	97.9	97.0	100.8	100.0	99.4	99.4	92.1	98.2

Table 1: Protocol B: per-task retention of full-cache accuracy (%) for every operating point of every training-free cross-layer method. Rows are grouped by measured compression tier. Within each tier, the highest compression ratio is always that of BellowsQ, so a baseline with a higher average also stores more memory. best and second average per tier are marked. Task codes and settings: Appendix C.1 .

Model	Tier	Dense	BellowsQ	CommonKV	xKV	OTT	KIVI	PolarQuant
			cross-layer latent			per-layer quantized
Qwen3-32B	$\sim$ 8 $\times$	85.54	84.22	81.08	26.51	77.95	75.42	51.93
Qwen3-32B	$\sim$ 16 $\times$	85.54	86.02	66.02	0 8.43	–	–	–
Qwen3-30B-A3B	$\sim$ 8 $\times$	86.14	85.30	49.88	0 5.30	81.45	77.59	0 4.22
Qwen3-30B-A3B	$\sim$ 16 $\times$	86.14	85.78	–	–	–	–	–

Table 2: Protocol C: single-turn retrieve-then-read on BrowseComp-Plus on all 830 queries, with the oracle prompt at a 32,768-token budget. Results are scored by the benchmark’s own judge. The notation “–” means that the method cannot reach that compression rate. See full table in Appendix C.3

Arm	Comp.	Llama	Mistral	Qwen
xKV, fp16, rank $p$	$4.0\times$	98.2	99.4	99.1
xKV, fp16, rank $p/2$	$8.0\times$	–	–	89.8
xKV, fp16, rank $p/4$	$16.0\times$	83.5	79.5	54.4
xKV-hold : the same basis, near-full rank, our INT4 leg	$15.1\times$	97.6	99.2	23.4
BellowsQ $n{=}4$ INT4 (the frozen consensus)	$13.1$ – $13.8\times$	98.0	98.1	97.3
our construction, per-sequence SVD oracle ( $2.2\times$ prefill)	$15.1\times$	97.3	–	–

Table 3: Cutting rank is the cliff; the rival that holds its rank and cuts bits instead comes back. (See full table in Appendix 19 )

Method	Comp.	Llama	Mistral	Qwen
Dense fp16	$1.00\times$	53.29	52.78	53.23
BellowsS $n{=}2$	$1.9\times$	53.07	53.04	51.88
BellowsS $n{=}3$	$2.9$ – $2.9\times$	52.67	52.87	51.92
BellowsQ $n{=}2$ INT8	$3.6$ – $3.7\times$	53.05	52.92	52.00
PolarQuant r4t4v4	$3.7\times$	53.66	–	53.90
KIVI K4V4	$3.8\times$	53.92	52.80	–

Table 4: Protocol A: LongBench average over eight tasks under the official harness. The table reports every official configuration of KIVI, OTT, RotateKV, TurboQuant and PolarQuant, and every BellowsKV point, grouped by measured compression tier. Within every tier, the highest compression ratio is that of BellowsKV, so a per-layer quantizer with a higher average also stores more memory. No per-layer point exists above

8\times

, because two bits at full dimension already give

8\times

. Per-task results: Appendix C.2 . Equal bit-width comparison: Table 18 .

Budget	Route	Keep	Comp.	Llama	Mistral	Qwen
$\sim$ 8 $\times$	BellowsE	1/2	7.7–7.8	97.6	97.9	99.2
$\sim$ 8 $\times$	Per-layer eviction	1/8	8.0	94.7	94.9	98.3
$\sim$ 16 $\times$	BellowsE	1/4	15.2–15.4	96.1	96.4	97.9
$\sim$ 16 $\times$	Per-layer eviction	1/16	16.0	89.5	90.7	92.6
$\sim$ 30 $\times$	BellowsQ $\times$ BellowsE	1/2	26.3–27.7	97.5	97.9	99.2
	BellowsE	1/8	29.6–30.0	92.5	93.8	95.0

Table 5: Protocol B retention (%) of full-cache accuracy. All routes use the same scoring rule. The per-layer baseline applies this rule independently in each layer, on the fp16 cache of that layer. The comparison at equal keep ratio is given in Table 15 .

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

decode speed
Design	Read path	Relation predicts	Measured / reported
BellowsKV	$R{=}d_{kv}$ , fp16	1.00 $\times$	0.995 $\times$ (32.5 tok/s)
Palu (rank-cutting)	$R{=}d_{kv}/2$ , fp16	2.00 $\times$	1.89 $\times$ (reported)
capacity from the freed bytes
HuggingFace Transformers	max batch	–	2 $\times$
vLLM paged allocator	KV pool	–	14.76 $\times$ (6.02M tokens)

Appendix

Table 6: BellowsKV decodes at parity, and the same relation prices Palu’s speedup within 6%. Decode speed and capacity (H100 SXM, HuggingFace Transformers, 32k context, no custom kernel): the reconstruct-at-read relation of Section 4.4 on both designs it covers, and what the freed bytes buy under two engines.

Model	Qsp	MFQ	HQA	MSQ	NQA	Gov	QMS	PsR	LCC	RBP	2Wk
Llama-3.1-8B	122	110	195	200	115	177	199	200	93	442	175
Mistral-7B	143	117	195	200	85	182	190	200	150	470	183
Qwen2.5-7B	125	111	195	200	115	180	199	200	96	444	178

Appendix

Table 7: Protocol B qualifying sample per task (inputs longer than 4k tokens and within the 32k window; the count differs per model because tokenizers differ). Every Protocol-B number in the paper uses all qualifying samples.

Axis	Tier	Measured ratio
Protocol A (quantizers)	$<$ 4 $\times$	$<$ 4 $\times$
	$\sim$ 5 $\times$	4–6.6 $\times$
	$\sim$ 8 $\times$	6.6–9.5 $\times$
	$\geq$ 12 $\times$	$\geq$ 9.5 $\times$ (no per-layer point exists)
Protocol B (sharing)	$\leq$ 4 $\times$	1–3.9 $\times$ (as-shipped fp16 baselines)
	4–11 $\times$	3.9–10.9 $\times$

Appendix

Table 8: Compression tiers. Bins on the measured ratio (all metadata counted) that every comparison table uses; a row belongs to the tier its own measured ratio falls in, never to a nominal one. The edges are chosen so that the highest ratio inside a tier is always a BellowsKV point, which puts every within-tier comparison on the baselines’ side of the byte budget, and so that no tier holds our rows alone. Protocol A’s top tier is the one exception: no per-layer quantizer exists past

8\times

Method	Configurations run	Llama	Mistral	Qwen
KIVI	K4V4, K2V2	✓	✓	–
OTT	K4V4, K2V2	✓	✓	–
RotateKV	4-bit † , 3-bit, 2-bit	✓	✓	–
TurboQuant	K4V4, K4V2, K2V2	✓	✓	✓ ‡
PolarQuant	r4t4v4, r3t3v4, r2t2v4, r2t2v2	✓	–	✓ ‡

Appendix

Table 9: Every Protocol-A baseline configuration, and the models it exists for. Each method runs its own official code at its published settings, with group size

128

and the method’s own residual-window convention.

\checkmark

marks a model the method ships modelling for; coverage on Qwen is thinner than on the other two, which is stated wherever it matters. † RotateKV’s 4-bit configuration exists only in the Mistral config set. ‡ Neither PolarQuant nor TurboQuant ships a Qwen path, so the Qwen adapters are ours and quantize bias-free keys (see Compatibility fixes ).

		measured compression
Method	Setting	Llama	Mistral	Qwen
xKV	rank $p$ , fp16	$4.00\times$	$4.00\times$	$4.00\times$
xKV	rank $p/2$ , fp16	–	–	$8.00\times$
xKV	rank $p/4$ , fp16	$16.00\times$	$16.00\times$	$16.00\times$
CommonKV	$r{=}3328$ ( $r{=}1664$ ), INT4	$3.95\times$	$3.95\times$	$3.95\times$
CommonKV	$r{=}1664$ ( $r{=}832$ ), INT4	$7.89\times$	$7.90\times$	$7.89\times$

Appendix

Table 10: The setting behind every Protocol-B baseline row of Table 1 , with the ratio it measures. All of them run inside the xKV harness on the same qualifying samples against the same dense run; the four training-free methods other than xKV were ported into it for this paper and the harness patches are released with the code. Where a setting differs on Qwen, whose KV width is half and which has

28

layers, the Qwen value is in parentheses; where all three differ the values are listed Llama, Mistral, Qwen.

p

is xKV’s full per-sequence rank. Storage bits are the method’s own composition, not ours.

Figure 3: On Llama the cross-layer axis stalls at

16\times

because rank is cut, and holding the rank recovers it. Protocol B retention of dense accuracy against measured compression, one line per training-free cross-layer method over all of its registered operating points, one harness, full sample. Filled markers are the latent methods and hollow grey markers the raw-cache methods; the shaded bands are the

\sim

\times

and

\sim

\times

tiers of Table 1 . The star is xKV-hold, xKV’s own basis kept at near-full rank and stored with our INT4 leg instead of truncated (Section 4.2 ); the hollow blue circle is BellowsQ composed with a keep-

1/2

verdict (Section 4.3 ). † 10 tasks.

Figure 4: Mistral repeats the pattern: every latent method works at

\sim

\times

, only the full-rank latent survives past

12\times

, and the transplant recovers xKV to within a point of the consensus. Axes, series and markers as in Figure 3 .

Figure 5: Qwen is where the rivals fall furthest and where the transplant collapses. xKV loses more than forty points between

4\times

and

16\times

, and xKV-hold, which recovers on the other two models, drops to

23.4\%

here, for a cause we have not isolated (Section 4.2 ). Axes, series and markers as in Figure 3 .

Method	Comp.	Qasper	QMSum	MNews	TREC	TQA	SAMSum	LCC	RB-P	Avg
Dense fp16	$1.00\times$	46.57	25.19	27.03	72.50	91.65	43.47	63.38	56.55	53.29
BellowsS $n{=}2$	$1.95\times$	46.76	25.32	26.08	72.50	91.28	43.90	62.38	56.32	53.07
BellowsS $n{=}3$	$2.90\times$	45.25	25.13	25.90	72.50	91.44	43.69	62.28	55.15	52.67
PolarQuant r4t4v4	$3.66\times$	46.01	25.56	26.69	72.50	91.86	42.88	65.38	58.44	53.66
BellowsQ $n{=}2$ INT8	$3.67\times$	46.47	25.39	26.19	72.50	91.28	43.77	62.52	56.28	53.05
KIVI K4V4	$3.76\times$	46.11	25.14	27.11	72.50	92.42	44.10	65.05	58.91	53.92

Appendix

Table 11: Protocol A, Llama-3.1-8B. Absolute LongBench scores, full test sets, every BellowsKV point and every official baseline configuration, grouped by measured compression tier (rules separate tiers; bins in Table 8 ); within a tier rows are sorted by ratio and the best and second Avg are marked. Baselines run each method’s official code; PolarQuant runs its Mixed K+V path, and on Qwen it and TurboQuant quantize bias-free keys (Appendix C.1 ).

Method	Comp.	Qasper	QMSum	MNews	TREC	TQA	SAMSum	LCC	RB-P	Avg
Dense fp16	$1.00\times$	38.15	25.57	26.43	76.00	88.59	47.46	59.46	60.56	52.78
BellowsS $n{=}2$	$1.96\times$	38.75	25.62	26.06	76.00	88.81	46.78	60.36	61.97	53.04
BellowsS $n{=}3$	$2.91\times$	38.20	25.04	25.75	76.00	89.57	47.17	59.84	61.41	52.87
BellowsQ $n{=}2$ INT8	$3.69\times$	38.49	25.34	25.81	76.00	88.56	46.74	60.42	61.99	52.92
OTT K4V4	$3.76\times$	37.95	25.80	26.50	76.00	88.89	47.08	60.40	61.30	52.99
KIVI K4V4	$3.76\times$	36.88	25.42	26.58	76.00	88.89	47.09	60.53	60.99	52.80

Appendix

Table 12: Protocol A, Mistral-7B. Absolute LongBench scores, full test sets, every BellowsKV point and every official baseline configuration, grouped by measured compression tier (rules separate tiers; bins in Table 8 ); within a tier rows are sorted by ratio and the best and second Avg are marked. Baselines run each method’s official code; PolarQuant runs its Mixed K+V path, and on Qwen it and TurboQuant quantize bias-free keys (Appendix C.1 ).

Method	Comp.	Qasper	QMSum	MNews	TREC	TQA	SAMSum	LCC	RB-P	Avg
Dense fp16	$1.00\times$	43.52	23.61	23.98	71.50	89.64	45.98	60.77	66.82	53.23
BellowsS $n{=}2$	$1.92\times$	42.82	24.16	23.68	71.00	88.67	45.05	56.66	63.01	51.88
BellowsS $n{=}3$	$2.86\times$	42.99	23.50	23.48	71.00	89.00	45.12	58.00	62.25	51.92
BellowsQ $n{=}2$ INT8	$3.57\times$	43.22	24.06	23.58	71.00	88.60	44.91	57.27	63.34	52.00
PolarQuant r4t4v4	$3.66\times$	43.94	24.42	25.52	72.00	89.46	45.76	63.68	66.41	53.90
BellowsS $n{=}4$	$3.78\times$	42.20	23.80	22.94	71.00	88.15	44.65	55.44	61.51	51.21

Appendix

Table 13: Protocol A, Qwen2.5-7B. Absolute LongBench scores, full test sets, every BellowsKV point and every official baseline configuration, grouped by measured compression tier (rules separate tiers; bins in Table 8 ); within a tier rows are sorted by ratio and the best and second Avg are marked. Baselines run each method’s official code; PolarQuant runs its Mixed K+V path, and on Qwen it and TurboQuant quantize bias-free keys (Appendix C.1 ).

Method	Comp.	Score
Qwen3-32B
Dense fp16	$1.00\times$	85.54
PolarQuant †	$3.66\times$	83.49
KIVI †	$3.76\times$	84.94
xKV †	$4.00\times$	78.07
PolarQuant †	$6.74\times$	51.93

Appendix

Table 14: Protocol C Full Table Single-turn retrieve-then-read on BrowseComp-Plus: the benchmark’s oracle prompt filled to

32{,}768

tokens with its own mined hard negatives, all

830

queries, greedy, the identical context for every method so dense is the ceiling. Score is the benchmark’s own judge and Comp. is measured compression with all metadata counted.

Budget	Route	Keep	Comp.	Llama	Mistral	Qwen
$\sim$ 2 $\times$	Per-layer eviction	1/2	2.0	98.4	99.4	100.9
$\sim$ 4 $\times$	Per-layer eviction	1/4	4.0	97.3	97.7	99.5
$\sim$ 8 $\times$	BellowsE	1/2	7.7–7.8	97.6	97.9	99.2
$\sim$ 8 $\times$	Per-layer eviction	1/8	8.0	94.7	94.9	98.3
$\sim$ 16 $\times$	BellowsE	1/4	15.2–15.4	96.1	96.4	97.9
$\sim$ 16 $\times$	Per-layer eviction	1/16	16.0	89.5	90.7	92.6

Appendix

Table 15: The eviction axis by budget tier (Protocol B, full sample; retention %). Every route runs the identical verdict; rows are grouped by measured compression so that each tier compares like with like: the per-layer control prunes its own fp16 cache, BellowsE prunes the shared latent, and at

\sim

\times

the composed point meets the naive stack and deep eviction. Comp. spans the three models (Qwen’s KV width differs). Best per tier in bold.

From keep	Model	prune harder	share
1/2	Llama-3.1-8B	3.74	0.82
	Mistral-7B	4.47	1.45
	Qwen2.5-7B	2.63	1.74
1/4	Llama-3.1-8B	7.80	1.24
	Mistral-7B	7.02	1.28
	Qwen2.5-7B	6.87	1.57

Appendix

Table 16: The exchange rate between the two currencies. Retention points paid to buy

\sim

\times

by pruning four times harder, against points paid to buy

\sim

3.9

\times

by sharing across four layers at a fixed keep ratio.

Route	Comp.	Llama	Mistral	Qwen
BellowsQ $\times$ BellowsE (evict 1/2)	27.7/26.3	97.5	97.9	99.2
naive stack: xKV $+$ INT4 $+$ pooled SnapKV	30.1	96.1	98.4	23.8
BellowsE alone (fp16, evict 1/8)	30.0/29.6	92.5	93.8	95.0

Appendix

Table 17: The

\sim

\times

budget class, all three models (Protocol B, full sample; retention %). The composed point stores slightly more than the stack’s nominal ratio (27.7/26.3

\times

against 30.1

\times

, a 9–13% byte gap in its favour); the stack’s verdict is pooled per group because per-layer verdicts cannot share a basis. Setup in Appendix C.1 .

Llama-3.1-8B-Instruct (dense 53.29)
Bits	Method	Comp.	Avg	vs best baseline
INT4	KIVI	3.76	53.92	–
	OTT	3.76	53.83	–
	PolarQuant	3.66	53.66	–
	TurboQuant	4.00	52.84	–
	BellowsQ $n{=}2$	6.75	52.92	-1.00 at 1.8 $\times$ its compression

Appendix

Table 18: Equal bit-width (Protocol A, full sample). At a fixed bit-width the only difference is what is quantized: each layer’s own cache, or one latent shared by

n

layers. LongBench average and measured compression. Qwen’s per-layer baselines are PolarQuant and TurboQuant, both with bias-free keys (Appendix C.1 ).

Llama-3.1-8B-Instruct
Arm	Comp.	Qsp	MFQ	HQA	MSQ	NQA	Gov	QMS	PsR	LCC	RBP	2Wk	Avg
xKV (fp16, rank $p/4$ )	$16.00\times$	91.4	80.9	82.4	87.3	73.3	75.6	94.4	95.5	92.8	86.8	58.5	83.5
xKV-hold	$15.06\times$	97.5	95.4	96.0	100.9	96.5	96.4	99.2	99.5	101.6	97.4	92.9	97.6
BellowsQ $n{=}4$ INT4	$13.84\times$	94.6	99.0	99.1	96.4	96.2	93.5	98.7	100.0	99.7	99.0	101.5	98.0
Mistral-7B-Instruct-v0.3
xKV (fp16, rank $p/4$ )	$16.00\times$	90.4	88.5	47.8	42.1	85.2	93.0	91.8	81.7	92.7	90.0	71.1	79.5

Appendix

Table 19: The transplant, per task. Protocol B retention (%) of xKV at

16\times

(fp16, rank

p/4

), xKV-hold (the same per-sequence basis at near-full rank with our INT4 leg,

15.1\times

) and BellowsQ

n{=}4

INT4, all three models. The recovery lands on the tasks truncation had cost; on Qwen the arm collapses on every task.

Figure 6: The rank cliff on the rival’s own ladder. Protocol-B retention of xKV’s fp16 basis against the rank kept (

p

p/2

p/4

, i.e.

4\times

8\times

16\times

), three models, with xKV-hold (the same basis at rank

p

stored at INT4,

15.1\times

) and BellowsQ

n{=}4

INT4 at full rank. Cutting rank falls; cutting bits at full rank does not, except on Qwen, where every arm that stores xKV’s basis at four bits collapses (Section 4.2 ).

Variant	Comp.	Qsp	MFQ	HQA	MSQ	NQA	Gov	QMS	PsR	LCC	RBP	2Wk	Avg
BellowsQ (final)	13.84	94.6	99.0	99.1	96.4	96.2	93.5	98.7	100.0	99.7	99.0	101.5	98.0
+ V-whisper	13.13	94.9	99.1	98.2	98.5	99.3	95.2	100.7	99.5	101.0	98.9	100.7	98.7
$-$ whisper	13.90	93.3	97.5	99.0	97.1	97.5	95.4	100.4	100.0	100.7	98.6	97.5	97.9
+ per-sequence gauge	13.90	93.2	94.8	99.5	98.5	93.5	92.3	98.6	99.0	99.1	97.8	97.8	96.7
per-seq SVD oracle	15.06	91.6	92.8	98.8	99.5	95.3	99.9	99.5	99.5	96.3	96.4	101.2	97.3
xKV-hold (+ our INT4)	15.06	97.5	95.4	96.0	100.9	96.5	96.4	99.2	99.5	101.6	97.4	92.9	97.6

Appendix

Table 20: Adapting to the sequence buys nothing the frozen consensus lacks, and the whisper’s value is the worst case, not the mean. Adaptivity and whisper ablation, per task (Llama-3.1-8B, Protocol B, full sample), the rows behind Section 4.2 ; the oracle re-decides the basis per sequence at 2.2

\times

prefill cost. Setup in Appendix C.1 .

KV-Cache Compression

Key-value cache compression reduces the memory and bandwidth needed to retain attention states during long-context inference.

See all 269 papers

VQKV: High-Fidelity and High-Ratio Cache Compression via Vector-Quantization
Mar 17, 2026Yixuan Wang, Qingyu Shi, Jiayu Zhou +3KV-Cache Compression KV-Cache Quantization
LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University · Shanghai Innovation Institute · Shanghai Artificial Intelligence Laboratory +2
BELT : BEhavior-Aligned Low-Rank KV Cache Compression with Learned Top-k Rank Allocation
Oct 13, 2026Yaoyang Liu, Shuwei Shen, Jincheng Tang +5KV-Cache Compression Low-Rank Compression
Peking University · Tencent · BeiHang University
AnchorKV: Anchor-Residual KV Cache Compression
Aug 3, 2026Malik Khalaf, Yara Shamshoum, Nitzan Hodos +2KV Caching Long-Context Language Model Inference
Department of Computer Science, Technion - Israel Institute of Technology

Mar 17, 2026J/K move · Enter open · S save

VQKV: High-Fidelity and High-Ratio Cache Compression via Vector-Quantization

Yixuan Wang, Qingyu Shi, Jiayu Zhou, Dianbo Liu+2

LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University · Shanghai Innovation Institute · Shanghai Artificial Intelligence Laboratory+2

KV-Cache Compression KV-Cache Quantization Memory Compression LLM Inference Acceleration Vector Quantization

Open paper AskPDF arXiv

The growing context length of Large Language Models (LLMs) enlarges the Key-Value (KV) cache, limiting deployment in resource-limited environments. Prior training-free approaches for KV cache compression typically rely on low-rank approximation or scalar quantization, which fail to simultaneously achieve high compression ratios and high reconstruction fidelity. We propose VQKV, a novel, training-free method introducing vector quantization (VQ) to obtain highly compressed KV representations while preserving high model fidelity, allowing for the representation of thousands of floating-point values with just a few integer indices. As a result, VQKV achieves an 82.8% compression ratio on LLaMA3.1-8B while retaining 98.6% of the baseline performance on LongBench and enabling 4.3x longer generation length on the same memory footprint.

Abstract

KV-Cache Compression

Explore similar work

VQKV: High-Fidelity and High-Ratio Cache Compression via Vector-Quantization

BELT : BEhavior-Aligned Low-Rank KV Cache Compression with Learned Top-k Rank Allocation

AnchorKV: Anchor-Residual KV Cache Compression