Large language models (LLMs) now serve as the backbone of agents that reason over many turns, call tools, and pursue long-horizon tasks. To avoid recomputation, a model stores the attention keys and values of all past tokens in a key-value (KV) cache. Each layer maintains its own cache, so KV-cache memory can become the main bottleneck for GPU serving. Most compression methods operate within a single layer, by quantizing the cache entries, reducing the channel dimension, or evicting less important tokens. All three lose substantial accuracy at high compression ratios. Cross-layer methods share one compressed cache across adjacent layers to go beyond this limit. However, these methods also degrade sharply at high compression ratios. We find that the KV-cache representations of adjacent layers are quite different, whereas the input hidden states of adjacent layers are highly similar. We also find that, under the same memory budget, reducing the dimension of this shared latent loses far more accuracy than quantizing it. Based on these findings, we propose BellowsKV, a training-free method that maps the hidden states of n adjacent layers to one shared latent per token. At inference time, each layer reads its own keys and values from this latent through a layer-specific projection. We keep the latent at full dimension and compress it in two other ways: BellowsKV-Quant stores the latent at low precision, which is nearly lossless because the latent has no dominant outlier channels; BellowsKV-Evict makes one shared eviction decision per block, because the layers rank token importance very similarly. Across Llama-3.1-8B, Mistral-7B and Qwen2.5-7B, BellowsKV-Quant retains about 99% of full-cache accuracy at 3.6x compression and 97-98% at 13-14x, where the strongest cross-layer method retains at most 84% at 16x. Code: https://github.com/MorrisZJ/BellowsKV
Figures & tables
Figure 1: Comparison of different cross-layer KV-Cache compression methods : (a) Trained-to-share architectures reach one cache only by retraining. (b) Training-free methods factorize the caches (c) BellowsKV folds the block’s hidden states into one latent, buying the compression ratio from the numbers of shared layers, bits and tokens instead.
Figure 2: (a) Cosine similarity between adjacent layers. The keys and values are close to zero, while the hidden states reach 0.87 on Llama and 0.82 on Qwen. (b) Per-channel dynamic range of the keys. The raw cache concentrates the range in a few channels (top). A data-free Hadamard rotation and our block-level projection are both flat (bottom), and the relative INT4 error falls from 0.156 to about 0.10 . (c) Spearman correlation between the token importance scores of adjacent layers, between 0.76 and 0.80 .
Method
Comp.
Qsp
MFQ
HQA
MSQ
NQA
Gov
QMS
PsR
LCC
RBP
2Wk
Avg
Llama-3.1-8B-Instruct
MiniCache fp16
1.31×
72.1
72.3
74.7
86.8
55.0
82.7
89.5
70.7
80.4
82.6
55.6
74.8
KVSharer fp16
1.33×
91.5
96.9
93.9
98.1
78.2
73.7
97.9
99.5
85.4
84.5
70.2
88.2
BellowsQ n=2 INT8
3.64×
95.6
97.6
99.6
99.4
95.9
98.5
102.5
100.0
98.7
99.0
101.0
98.9
CommonKV
3.95×
92.2
90.6
98.0
100.1
89.0
89.7
96.5
99.5
98.4
97.8
91.1
94.8
xKV
4.00×
96.6
98.6
96.0
102.1
97.9
97.0
100.8
100.0
99.4
99.4
92.1
98.2
Table 1: Protocol B: per-task retention of full-cache accuracy (%) for every operating point of every training-free cross-layer method. Rows are grouped by measured compression tier. Within each tier, the highest compression ratio is always that of BellowsQ, so a baseline with a higher average also stores more memory. best and second average per tier are marked. Task codes and settings: Appendix C.1 .
cross-layer latent
per-layer quantized
Model
Tier
Dense
BellowsQ
CommonKV
xKV
OTT
KIVI
PolarQuant
Qwen3-32B
∼ 8 ×
85.54
84.22
81.08
26.51
77.95
75.42
51.93
∼ 16 ×
85.54
86.02
66.02
0 8.43
–
–
–
Qwen3-30B-A3B
∼ 8 ×
86.14
85.30
49.88
0 5.30
81.45
77.59
0 4.22
∼ 16 ×
86.14
85.78
–
–
–
–
–
Table 2: Protocol C: single-turn retrieve-then-read on BrowseComp-Plus on all 830 queries, with the oracle prompt at a 32,768-token budget. Results are scored by the benchmark’s own judge. The notation “–” means that the method cannot reach that compression rate. See full table in Appendix C.3
Arm
Comp.
Llama
Mistral
Qwen
xKV, fp16, rank p
4.0×
98.2
99.4
99.1
xKV, fp16, rank p/2
8.0×
–
–
89.8
xKV, fp16, rank p/4
16.0×
83.5
79.5
54.4
xKV-hold : the same basis, near-full rank, our INT4 leg
Table 3: Cutting rank is the cliff; the rival that holds its rank and cuts bits instead comes back. (See full table in Appendix 19 )
Method
Comp.
Llama
Mistral
Qwen
Dense fp16
1.00×
53.29
52.78
53.23
BellowsS n=2
1.9×
53.07
53.04
51.88
BellowsS n=3
2.9 – 2.9×
52.67
52.87
51.92
BellowsQ n=2 INT8
3.6 – 3.7×
53.05
52.92
52.00
PolarQuant r4t4v4
3.7×
53.66
–
53.90
KIVI K4V4
3.8×
53.92
52.80
–
Table 4: Protocol A: LongBench average over eight tasks under the official harness. The table reports every official configuration of KIVI, OTT, RotateKV, TurboQuant and PolarQuant, and every BellowsKV point, grouped by measured compression tier. Within every tier, the highest compression ratio is that of BellowsKV, so a per-layer quantizer with a higher average also stores more memory. No per-layer point exists above 8× , because two bits at full dimension already give 8× . Per-task results: Appendix C.2 . Equal bit-width comparison: Table 18 .
Budget
Route
Keep
Comp.
Llama
Mistral
Qwen
∼ 8 ×
BellowsE
1/2
7.7–7.8
97.6
97.9
99.2
Per-layer eviction
1/8
8.0
94.7
94.9
98.3
∼ 16 ×
BellowsE
1/4
15.2–15.4
96.1
96.4
97.9
Per-layer eviction
1/16
16.0
89.5
90.7
92.6
∼ 30 ×
BellowsQ × BellowsE
1/2
26.3–27.7
97.5
97.9
99.2
BellowsE
1/8
29.6–30.0
92.5
93.8
95.0
Table 5: Protocol B retention (%) of full-cache accuracy. All routes use the same scoring rule. The per-layer baseline applies this rule independently in each layer, on the fp16 cache of that layer. The comparison at equal keep ratio is given in Table 15 .
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Design
Read path
Relation predicts
Measured / reported
decode speed
BellowsKV
R=dkv , fp16
1.00 ×
0.995 × (32.5 tok/s)
Palu (rank-cutting)
R=dkv/2 , fp16
2.00 ×
1.89 × (reported)
capacity from the freed bytes
HuggingFace Transformers
max batch
–
2 ×
vLLM paged allocator
KV pool
–
14.76 × (6.02M tokens)
Appendix
Table 6: BellowsKV decodes at parity, and the same relation prices Palu’s speedup within 6%. Decode speed and capacity (H100 SXM, HuggingFace Transformers, 32k context, no custom kernel): the reconstruct-at-read relation of Section 4.4 on both designs it covers, and what the freed bytes buy under two engines.
Model
Qsp
MFQ
HQA
MSQ
NQA
Gov
QMS
PsR
LCC
RBP
2Wk
Llama-3.1-8B
122
110
195
200
115
177
199
200
93
442
175
Mistral-7B
143
117
195
200
85
182
190
200
150
470
183
Qwen2.5-7B
125
111
195
200
115
180
199
200
96
444
178
Appendix
Table 7: Protocol B qualifying sample per task (inputs longer than 4k tokens and within the 32k window; the count differs per model because tokenizers differ). Every Protocol-B number in the paper uses all qualifying samples.
Axis
Tier
Measured ratio
Protocol A (quantizers)
< 4 ×
< 4 ×
∼ 5 ×
4–6.6 ×
∼ 8 ×
6.6–9.5 ×
≥ 12 ×
≥ 9.5 × (no per-layer point exists)
Protocol B (sharing)
≤ 4 ×
1–3.9 × (as-shipped fp16 baselines)
4–11 ×
3.9–10.9 ×
Appendix
Table 8: Compression tiers. Bins on the measured ratio (all metadata counted) that every comparison table uses; a row belongs to the tier its own measured ratio falls in, never to a nominal one. The edges are chosen so that the highest ratio inside a tier is always a BellowsKV point, which puts every within-tier comparison on the baselines’ side of the byte budget, and so that no tier holds our rows alone. Protocol A’s top tier is the one exception: no per-layer quantizer exists past 8× .
Method
Configurations run
Llama
Mistral
Qwen
KIVI
K4V4, K2V2
✓
✓
–
OTT
K4V4, K2V2
✓
✓
–
RotateKV
4-bit † , 3-bit, 2-bit
✓
✓
–
TurboQuant
K4V4, K4V2, K2V2
✓
✓
✓ ‡
PolarQuant
r4t4v4, r3t3v4, r2t2v4, r2t2v2
✓
–
✓ ‡
Appendix
Table 9: Every Protocol-A baseline configuration, and the models it exists for. Each method runs its own official code at its published settings, with group size 128 and the method’s own residual-window convention. ✓ marks a model the method ships modelling for; coverage on Qwen is thinner than on the other two, which is stated wherever it matters. † RotateKV’s 4-bit configuration exists only in the Mistral config set. ‡ Neither PolarQuant nor TurboQuant ships a Qwen path, so the Qwen adapters are ours and quantize bias-free keys (see Compatibility fixes ).
measured compression
Method
Setting
Llama
Mistral
Qwen
xKV
rank p , fp16
4.00×
4.00×
4.00×
xKV
rank p/2 , fp16
–
–
8.00×
xKV
rank p/4 , fp16
16.00×
16.00×
16.00×
CommonKV
r=3328 ( r=1664 ), INT4
3.95×
3.95×
3.95×
CommonKV
r=1664 ( r=832 ), INT4
7.89×
7.90×
7.89×
Appendix
Table 10: The setting behind every Protocol-B baseline row of Table 1 , with the ratio it measures. All of them run inside the xKV harness on the same qualifying samples against the same dense run; the four training-free methods other than xKV were ported into it for this paper and the harness patches are released with the code. Where a setting differs on Qwen, whose KV width is half and which has 28 layers, the Qwen value is in parentheses; where all three differ the values are listed Llama, Mistral, Qwen. p is xKV’s full per-sequence rank. Storage bits are the method’s own composition, not ours.
Figure 3: On Llama the cross-layer axis stalls at 16× because rank is cut, and holding the rank recovers it. Protocol B retention of dense accuracy against measured compression, one line per training-free cross-layer method over all of its registered operating points, one harness, full sample. Filled markers are the latent methods and hollow grey markers the raw-cache methods; the shaded bands are the ∼ 4 × and ∼ 16 × tiers of Table 1 . The star is xKV-hold, xKV’s own basis kept at near-full rank and stored with our INT4 leg instead of truncated (Section 4.2 ); the hollow blue circle is BellowsQ composed with a keep- 1/2 verdict (Section 4.3 ). † 10 tasks.
Figure 4: Mistral repeats the pattern: every latent method works at ∼ 4 × , only the full-rank latent survives past 12× , and the transplant recovers xKV to within a point of the consensus. Axes, series and markers as in Figure 3 .
Figure 5: Qwen is where the rivals fall furthest and where the transplant collapses. xKV loses more than forty points between 4× and 16× , and xKV-hold, which recovers on the other two models, drops to 23.4% here, for a cause we have not isolated (Section 4.2 ). Axes, series and markers as in Figure 3 .
Method
Comp.
Qasper
QMSum
MNews
TREC
TQA
SAMSum
LCC
RB-P
Avg
Dense fp16
1.00×
46.57
25.19
27.03
72.50
91.65
43.47
63.38
56.55
53.29
BellowsS n=2
1.95×
46.76
25.32
26.08
72.50
91.28
43.90
62.38
56.32
53.07
BellowsS n=3
2.90×
45.25
25.13
25.90
72.50
91.44
43.69
62.28
55.15
52.67
PolarQuant r4t4v4
3.66×
46.01
25.56
26.69
72.50
91.86
42.88
65.38
58.44
53.66
BellowsQ n=2 INT8
3.67×
46.47
25.39
26.19
72.50
91.28
43.77
62.52
56.28
53.05
KIVI K4V4
3.76×
46.11
25.14
27.11
72.50
92.42
44.10
65.05
58.91
53.92
Appendix
Table 11: Protocol A, Llama-3.1-8B. Absolute LongBench scores, full test sets, every BellowsKV point and every official baseline configuration, grouped by measured compression tier (rules separate tiers; bins in Table 8 ); within a tier rows are sorted by ratio and the best and second Avg are marked. Baselines run each method’s official code; PolarQuant runs its Mixed K+V path, and on Qwen it and TurboQuant quantize bias-free keys (Appendix C.1 ).
Method
Comp.
Qasper
QMSum
MNews
TREC
TQA
SAMSum
LCC
RB-P
Avg
Dense fp16
1.00×
38.15
25.57
26.43
76.00
88.59
47.46
59.46
60.56
52.78
BellowsS n=2
1.96×
38.75
25.62
26.06
76.00
88.81
46.78
60.36
61.97
53.04
BellowsS n=3
2.91×
38.20
25.04
25.75
76.00
89.57
47.17
59.84
61.41
52.87
BellowsQ n=2 INT8
3.69×
38.49
25.34
25.81
76.00
88.56
46.74
60.42
61.99
52.92
OTT K4V4
3.76×
37.95
25.80
26.50
76.00
88.89
47.08
60.40
61.30
52.99
KIVI K4V4
3.76×
36.88
25.42
26.58
76.00
88.89
47.09
60.53
60.99
52.80
Appendix
Table 12: Protocol A, Mistral-7B. Absolute LongBench scores, full test sets, every BellowsKV point and every official baseline configuration, grouped by measured compression tier (rules separate tiers; bins in Table 8 ); within a tier rows are sorted by ratio and the best and second Avg are marked. Baselines run each method’s official code; PolarQuant runs its Mixed K+V path, and on Qwen it and TurboQuant quantize bias-free keys (Appendix C.1 ).
Method
Comp.
Qasper
QMSum
MNews
TREC
TQA
SAMSum
LCC
RB-P
Avg
Dense fp16
1.00×
43.52
23.61
23.98
71.50
89.64
45.98
60.77
66.82
53.23
BellowsS n=2
1.92×
42.82
24.16
23.68
71.00
88.67
45.05
56.66
63.01
51.88
BellowsS n=3
2.86×
42.99
23.50
23.48
71.00
89.00
45.12
58.00
62.25
51.92
BellowsQ n=2 INT8
3.57×
43.22
24.06
23.58
71.00
88.60
44.91
57.27
63.34
52.00
PolarQuant r4t4v4
3.66×
43.94
24.42
25.52
72.00
89.46
45.76
63.68
66.41
53.90
BellowsS n=4
3.78×
42.20
23.80
22.94
71.00
88.15
44.65
55.44
61.51
51.21
Appendix
Table 13: Protocol A, Qwen2.5-7B. Absolute LongBench scores, full test sets, every BellowsKV point and every official baseline configuration, grouped by measured compression tier (rules separate tiers; bins in Table 8 ); within a tier rows are sorted by ratio and the best and second Avg are marked. Baselines run each method’s official code; PolarQuant runs its Mixed K+V path, and on Qwen it and TurboQuant quantize bias-free keys (Appendix C.1 ).
Method
Comp.
Score
Qwen3-32B
Dense fp16
1.00×
85.54
PolarQuant †
3.66×
83.49
KIVI †
3.76×
84.94
xKV †
4.00×
78.07
PolarQuant †
6.74×
51.93
Appendix
Table 14: Protocol C Full Table Single-turn retrieve-then-read on BrowseComp-Plus: the benchmark’s oracle prompt filled to 32,768 tokens with its own mined hard negatives, all 830 queries, greedy, the identical context for every method so dense is the ceiling. Score is the benchmark’s own judge and Comp. is measured compression with all metadata counted.
Budget
Route
Keep
Comp.
Llama
Mistral
Qwen
∼ 2 ×
Per-layer eviction
1/2
2.0
98.4
99.4
100.9
∼ 4 ×
Per-layer eviction
1/4
4.0
97.3
97.7
99.5
∼ 8 ×
BellowsE
1/2
7.7–7.8
97.6
97.9
99.2
Per-layer eviction
1/8
8.0
94.7
94.9
98.3
∼ 16 ×
BellowsE
1/4
15.2–15.4
96.1
96.4
97.9
Per-layer eviction
1/16
16.0
89.5
90.7
92.6
Appendix
Table 15: The eviction axis by budget tier (Protocol B, full sample; retention %). Every route runs the identical verdict; rows are grouped by measured compression so that each tier compares like with like: the per-layer control prunes its own fp16 cache, BellowsE prunes the shared latent, and at ∼ 30 × the composed point meets the naive stack and deep eviction. Comp. spans the three models (Qwen’s KV width differs). Best per tier in bold.
From keep
Model
prune harder
share
1/2
Llama-3.1-8B
3.74
0.82
Mistral-7B
4.47
1.45
Qwen2.5-7B
2.63
1.74
1/4
Llama-3.1-8B
7.80
1.24
Mistral-7B
7.02
1.28
Qwen2.5-7B
6.87
1.57
Appendix
Table 16: The exchange rate between the two currencies. Retention points paid to buy ∼ 4 × by pruning four times harder, against points paid to buy ∼ 3.9 × by sharing across four layers at a fixed keep ratio.
Route
Comp.
Llama
Mistral
Qwen
BellowsQ × BellowsE (evict 1/2)
27.7/26.3
97.5
97.9
99.2
naive stack: xKV + INT4 + pooled SnapKV
30.1
96.1
98.4
23.8
BellowsE alone (fp16, evict 1/8)
30.0/29.6
92.5
93.8
95.0
Appendix
Table 17: The ∼ 30 × budget class, all three models (Protocol B, full sample; retention %). The composed point stores slightly more than the stack’s nominal ratio (27.7/26.3 × against 30.1 × , a 9–13% byte gap in its favour); the stack’s verdict is pooled per group because per-layer verdicts cannot share a basis. Setup in Appendix C.1 .
Bits
Method
Comp.
Avg
vs best baseline
Llama-3.1-8B-Instruct (dense 53.29)
INT4
KIVI
3.76
53.92
–
OTT
3.76
53.83
–
PolarQuant
3.66
53.66
–
TurboQuant
4.00
52.84
–
BellowsQ n=2
6.75
52.92
-1.00 at 1.8 × its compression
Appendix
Table 18: Equal bit-width (Protocol A, full sample). At a fixed bit-width the only difference is what is quantized: each layer’s own cache, or one latent shared by n layers. LongBench average and measured compression. Qwen’s per-layer baselines are PolarQuant and TurboQuant, both with bias-free keys (Appendix C.1 ).
Arm
Comp.
Qsp
MFQ
HQA
MSQ
NQA
Gov
QMS
PsR
LCC
RBP
2Wk
Avg
Llama-3.1-8B-Instruct
xKV (fp16, rank p/4 )
16.00×
91.4
80.9
82.4
87.3
73.3
75.6
94.4
95.5
92.8
86.8
58.5
83.5
xKV-hold
15.06×
97.5
95.4
96.0
100.9
96.5
96.4
99.2
99.5
101.6
97.4
92.9
97.6
BellowsQ n=4 INT4
13.84×
94.6
99.0
99.1
96.4
96.2
93.5
98.7
100.0
99.7
99.0
101.5
98.0
Mistral-7B-Instruct-v0.3
xKV (fp16, rank p/4 )
16.00×
90.4
88.5
47.8
42.1
85.2
93.0
91.8
81.7
92.7
90.0
71.1
79.5
Appendix
Table 19: The transplant, per task. Protocol B retention (%) of xKV at 16× (fp16, rank p/4 ), xKV-hold (the same per-sequence basis at near-full rank with our INT4 leg, 15.1× ) and BellowsQ n=4 INT4, all three models. The recovery lands on the tasks truncation had cost; on Qwen the arm collapses on every task.
Figure 6: The rank cliff on the rival’s own ladder. Protocol-B retention of xKV’s fp16 basis against the rank kept ( p , p/2 , p/4 , i.e. 4× , 8× , 16× ), three models, with xKV-hold (the same basis at rank p stored at INT4, 15.1× ) and BellowsQ n=4 INT4 at full rank. Cutting rank falls; cutting bits at full rank does not, except on Qwen, where every arm that stores xKV’s basis at four bits collapses (Section 4.2 ).
Variant
Comp.
Qsp
MFQ
HQA
MSQ
NQA
Gov
QMS
PsR
LCC
RBP
2Wk
Avg
BellowsQ (final)
13.84
94.6
99.0
99.1
96.4
96.2
93.5
98.7
100.0
99.7
99.0
101.5
98.0
+ V-whisper
13.13
94.9
99.1
98.2
98.5
99.3
95.2
100.7
99.5
101.0
98.9
100.7
98.7
− whisper
13.90
93.3
97.5
99.0
97.1
97.5
95.4
100.4
100.0
100.7
98.6
97.5
97.9
+ per-sequence gauge
13.90
93.2
94.8
99.5
98.5
93.5
92.3
98.6
99.0
99.1
97.8
97.8
96.7
per-seq SVD oracle
15.06
91.6
92.8
98.8
99.5
95.3
99.9
99.5
99.5
96.3
96.4
101.2
97.3
xKV-hold (+ our INT4)
15.06
97.5
95.4
96.0
100.9
96.5
96.4
99.2
99.5
101.6
97.4
92.9
97.6
Appendix
Table 20: Adapting to the sequence buys nothing the frozen consensus lacks, and the whisper’s value is the worst case, not the mean. Adaptivity and whisper ablation, per task (Llama-3.1-8B, Protocol B, full sample), the rows behind Section 4.2 ; the oracle re-decides the basis per sequence at 2.2 × prefill cost. Setup in Appendix C.1 .
The growing context length of Large Language Models (LLMs) enlarges the Key-Value (KV) cache, limiting deployment in resource-limited environments. Prior training-free approaches for KV cache compression typically rely on low-rank approximation or scalar quantization, which fail to simultaneously achieve high compression ratios and high reconstruction fidelity. We propose VQKV, a novel, training-free method introducing vector quantization (VQ) to obtain highly compressed KV representations while preserving high model fidelity, allowing for the representation of thousands of floating-point values with just a few integer indices. As a result, VQKV achieves an 82.8% compression ratio on LLaMA3.1-8B while retaining 98.6% of the baseline performance on LongBench and enabling 4.3x longer generation length on the same memory footprint.
Yixuan Wang, Qingyu Shi, Jiayu Zhou +3
LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University · Shanghai Innovation Institute · Shanghai Artificial Intelligence Laboratory +2
Long-context inference makes the key-value (KV) cache a major memory bottleneck for large language models. We propose BELT, a low-rank KV cache compression framework. Its key-side objective targets attention distributions rather than raw keys or pre-softmax logits, yielding a quadratic surrogate with a closed-form low-rank solution. A learned global rank allocator uses differentiable soft selection during training and hard selection at deployment to meet a user-specified component budget. Experiments demonstrate improved model quality under aggressive compression and increased decoding throughput at comparable accuracy. Evaluated across three LLMs and multiple benchmarks, BELT achieves up to 70% low-rank KV cache compression while outperforming the strongest baseline by up to 23.3 points on LongBench, and up to 17.8x overall KV cache reduction when combined with quantization. With Triton-based GPU kernels and a quantized cache, BELT delivers up to 2.7x speedup for the attention module and 3.3x end-to-end decoding throughput.
The key-value (KV) cache is the primary memory bottleneck in long-context LLM inference. Existing approaches attack it from opposite ends: eviction methods permanently discard tokens, degrading performance whenever a discarded token later proves essential, while quantization methods retain all tokens at low precision but offer limited compression. We propose AnchorKV, a compression scheme that shrinks the cache by 20× without discarding a single token. AnchorKV represents the cache using a small set of anchors stored exactly, expresses every other token through its most similar anchor, and refines only those whose approximation most affects the model's output. AnchorKV consistently preserves accuracy across models and datasets, retaining 99% of the full-cache score at the 70B scale, while keeping the entire context at a fraction of its cost.
Malik Khalaf, Yara Shamshoum, Nitzan Hodos +2
Department of Computer Science, Technion - Israel Institute of Technology