Autoregressive decoding relies on the KV cache, whose memory footprint grows linearly with context length and caps batch size and throughput. Existing post-training methods exploit redundancy across tokens, layers, and feature dimensions, or lower numerical precision. We identify a complementary axis: within each attention head, values are largely predictable from their corresponding keys. We exploit this with value cache amortisation, which fits a lightweight per-head predictor at test time and stores residuals only for the tokens it reconstructs worst, giving 4× near-lossless value compression. The predictor can be initialised by meta-learning for substantially reduced prefill overhead, or fitted from scratch in settings that allow a one-off cost, such as multi-query inference. We pair value amortisation with cross-layer low-rank key compression to produce LowRAM, an asymmetric compression pipeline that stays close to lossless across three model families. By reconstructing only a selected subset of KV pairs at each decoding step, LowRAM can keep decoding overhead minimal and achieve up to 5× higher peak decode throughput at 4× compression. LowRAM further composes with quantisation and eviction, reaching 26.9× compression at 94.5% of baseline performance on long contexts. These results show that treating keys and values as structurally distinct enables efficient, aggressive, and composable KV cache compression.
Figures & tables
Figure 1: Mistral-7B compression on LongBench contexts longer than 10K tokens across different KV cache compression methods: eviction (adapted from Yang et al. (2025b) ), quantisation ( Zandieh et al., 2026 ) , LowRAM (our proposed method), and LowRAM-EQ (LowRAM combined with eviction and quantisation). Left : recovered performance against compression ratio. Right : maximum evaluated compression retaining at least 90% of baseline performance. LowRAM-EQ combines LowRAM, eviction, and quantisation. See Appendix C for a description of how compression is computed.
Figure 2: At inference, a per-head MLP learns to reconstruct values from keys, then is cached for decoding alongside residuals for the worst-reconstructed values.
Figure 3: Source-target analysis for predictive KV cache compression at a fixed 3× compression ratio for Mistral-7B. The left and centre panels show recovered LongBench performance when reconstructing current-layer keys KL and values VL , respectively, from alternative current- or previous-layer representations. The dotted line indicates uncompressed baseline performance. The right panel plots recovered performance against reconstruction mean-squared error for each source-target pair, showing that lower reconstruction error does not necessarily correspond to better downstream performance.
RULER 4K (%)
LongBench (%)
Model
2x
3x
4x
2x
3x
4x
Mistral-7B
99.4
98.8
96.0
100.7
99.4
97.9
Llama3.1-8B
99.6
97.9
93.9
98.7
98.5
99.6
Qwen3.5-4B
100.0
100.0
100.1
97.8
95.0
94.4
Table 1: Recovered average performance of LowRAM on RULER 4K and LongBench. Scores are averaged across each benchmark and reported as a percentage of the corresponding uncompressed baseline. The compression ratio reported for Qwen3.5-4B concerns only the full attention layers.
RULER (%)
LongBench (%)
SR
Meta-init
16K
32K
64K
> 10K
×
×
96.9
100.0
95.8
95.3
✓
×
97.7
99.0
96.8
95.3
✓
✓
96.3
96.8
92.4
92.9
Table 2: Recovered LongBench ( > 10K) and RULER 16-64K performance of LowRAM on Llama-3.1-8B-Instruct at 4 × compression, with and without selective reconstruction (SR) and a meta-learned initialisation (Meta-init). Rows without Meta-init use 300 gradient updates per context (vs. 150 in Table 1 ); Meta-init uses 2.
Figure 4: Decode throughput for Llama-3.1-8B at different batch sizes and sequence lengths at 4× compression, comparing exact KV attention against LowRAM-SR (LowRAM with selective reconstruction). Appendix F reports the full selective-reconstruction configuration.
RULER (4K)
LongBench
Model
Method
CR
Score (%)
CR
Score (%)
Mistral-7B
Eviction
2×
99.9
2×
99.4
Quantisation
5.1×
99.5
5.1×
98.9
LowRAM
4×
96.0
4×
97.9
LowRAM-E
8×
91.8
8×
96.6
LowRAM-Q
4.8×
99.6
6.4×
99.4
Table 3: Recovered performance of LowRAM and LowRAM combined with eviction (LowRAM-E), quantisation (LowRAM-Q), and both (LowRAM-EQ). The achieved compression ratio varies across benchmarks because it depends on sequence length. CR refers to compression ratio.
Eviction
Quantisation
LowRAM
LowRAM-EQ
CR
10.0×
7.5×
9.0×
26.9×
Score (%)
91.6
92.3
90.7
94.5
Table 4: Maximum evaluated compression retaining at least 90% of Mistral-7B performance on LongBench contexts longer than 10K tokens. CR uses the byte-level accounting in Appendix C .
RULER 4K (%)
LongBench (%)
Method
2×
3×
4×
2×
3×
4×
Amortisation
100.0
99.6
98.2
100.4
100.4
99.3
xKV
99.9
99.7
99.3
95.0
95.3
95.8
Table 5: Recovered performance of amortisation and xKV applied to the value cache in isolation on Llama-3.1-8B, reported as a percentage of the uncompressed-cache baseline. The displayed configuration uses a two-layer per-head MLP with 300 per-context gradient updates, without warm-starting or Wlin . Table 8 provides the complete optimisation-budget and initialisation sweep.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Setting
Number of MLP layers
2
Hidden-width factor γ
1
Activation
ReLU
Loss
MSE
Optimiser
Adam
Learning rate
0.001
Appendix
Table 6: Hyperparameters used to train the value-reconstruction networks used for value cache amortisation.
Method
LongBench (%)
Δ
Amortisation
99.0
w/o error-residual store
64.9
-34.1
w/o slim initialisation
97.2
-1.8
w/o warm start
97.1
-1.9
w per-layer MLP
79.5
-19.5
w linear predictor
57.5
-41.5
Appendix
Table 7: Ablation of the main components of the amortised value cache method for Mistral-7B at 3× value cache compression. LongBench scores are reported as percentages of the uncompressed KV cache baseline, and Δ denotes the absolute change in recovered performance relative to the full amortisation method.
Figure 5: Residual selection strategies at equal residual budgets on RULER NIAH-Multikey-1 for Llama-3.1-8B-Instruct, using the full LowRAM pipeline at 4 × compression with the meta-learned initialisation (2 gradient updates). Left: mean squared error between attention outputs computed with the compressed and exact KV caches ( n=16 ). Right: NIAH accuracy ( n=200 ). + 8 sinks forces residuals for the first 8 tokens within the same budget. All strategies share the same compressed key cache and are evaluated on the same sequences.
RULER 4K (%)
LongBench (%)
Predictor
Updates
2×
3×
4×
2×
3×
4×
MLP
10
90.8
78.0
61.5
76.2
69.0
59.0
MLP +Wlin
10
98.4
94.4
79.0
84.9
82.3
78.1
MLP
100
99.4
97.6
88.9
93.6
91.7
88.9
MLP +Wlin
100
99.9
99.1
95.1
94.4
93.8
91.5
MLP
200
99.9
99.2
97.1
98.1
98.7
97.9
Appendix
Table 8: Recovered performance of value cache amortisation on Llama-3.1-8B as a function of the per-context optimisation budget and Slim Attention initialisation. Keys are retained exactly, and results are reported relative to the uncompressed-cache baseline. All predictors are two-layer per-head MLPs trained without warm-starting. The 300-update configuration is the high-fidelity result summarised in Table 5 .
Model
Value Predictor
Wlin shape
CR
LongBench (%)
Mistral-7B
Joint Wlin
(HKV,dh,HKV,dh)
10 ×
39.7
Per-head Wlin
(HKV,dh,dh)
79 ×
9.7
fθ + joint residual Wlin
(HKV,dh,HKV,dh)
3 ×
9.5
fθ + per-head residual Wlin
(HKV,dh,dh)
3 ×
84.7
Llama-3.1-8B
Joint Wlin
(HKV,dh,HKV,dh)
11 ×
67.8
Per-head Wlin
(HKV,dh,dh)
89 ×
20.3
Appendix
Table 9: Ablation of joint and per-head Wlin initialisation for value cache amortisation on LongBench. Joint Wlin permits mixing across KV heads, whereas per-head Wlin applies an independent dh×dh map to each KV head. LongBench scores are percentages of the uncompressed baseline. CR refers to compression ratio. Differences in achieved compression ratio between Mistral and Llama reflect model-specific effective context lengths induced by their different tokenizers.
RULER (4K)
LongBench
Model
Method
CR
%
CR
%
Mistral-7B
Eviction
2×
99.9
2×
99.4
3×
99.3
3×
97.6
4×
98.0
4×
96.3
Quantisation
5.1×
99.5
5.1×
98.9
LowRAM
2×
99.4
2×
100.7
Appendix
Table 10: Recovered performance of LowRAM, used alone or combined with eviction and quantisation, across a range of compression settings for Mistral-7B and Llama-3.1-8B. Results are reported as percentages of the corresponding uncompressed baseline on RULER 4K and LongBench. Grey rows show eviction and quantisation applied independently, blue rows show LowRAM alone, and unshaded rows show LowRAM combined with eviction (LowRAM-E), quantisation (LowRAM-Q), or both (LowRAM-EQ). CR denotes the total compression ratio achieved on each benchmark; for combined methods, it accounts for the composition of the individual compression components and may differ between benchmarks because of sequence-length variation and across models due to model-specific effective context lengths induced by their different tokenizers.
LongBench ( > 10K)
Method
CR
Score (%)
Eviction
3.0×
98.1
4.0×
97.8
8.0×
93.9
10.0×
91.6
Quantisation
3.9×
100.0
Appendix
Table 11: Compression ratio and recovered performance for Mistral-7B using different KV cache compression methods, evaluated on the subset of LongBench examples with sequence lengths exceeding 10k tokens. Scores are reported as percentages of the uncompressed baseline. CR refers to compression ratio and is calculated from the byte-level key and value footprints described in Appendix C
Component
Hyperparameter
Setting
Meta-optimisation
Algorithm
FOMAML
Inner-loop updates, K
2
Inner optimiser
Adam
Inner learning rate, α
10−3
Outer optimiser
AdamW
Outer learning rate, β
3×10−4
Appendix
Table 12: Hyperparameters used to meta-learn the value-predictor initialisation. The inner and outer objectives use the same key-value pairs because the desired behaviour is rapid memorisation of the observed context.
RULER 4K (%)
LongBench (%)
2 ×
3 ×
4 ×
2 ×
3 ×
4 ×
98.4 ( − 1.2)
96.1 ( − 1.8)
91.2 ( − 2.7)
93.6 ( − 5.1)
93.0 ( − 5.5)
91.1 ( − 8.5)
Appendix
Table 13: Recovered performance of LowRAM when the amortised value cache is initialised from meta-learned weights and adapted with only 2 gradient steps for Llama-3.1-8B. Scores are reported as percentages of the uncompressed baseline; values in parentheses give the drop in percentage points relative to vanilla LowRAM (Table 1 ), which uses 150 gradient steps per context.
Parameter
Value
Chunk size
8 tokens
Token budget
2048 tokens
Local window
32 tokens
Outlier chunks
48 chunks (384 tokens)
Max. tokens attended per step
2464 tokens
Appendix
Table 14: Selective reconstruction configuration.
Figure 6: TTFT overhead of value cache amortisation alone versus the number of gradient updates used to fit the MLP, at a 4K-token sequence length with batch size 4. Keys are left uncompressed, so this isolates the cost of fitting the value predictor. Bars show raw TTFT; annotations give the overhead relative to the uncompressed baseline. See Figure 7 for the full LowRAM pipeline, including key factorisation.
Figure 7: TTFT of the uncompressed baseline (exact KV) and the full LowRAM pipeline at 4 × compression across context lengths and batch sizes. LowRAM timings include both value cache amortisation (2 gradient updates) and low-rank factorisation of the keys via randomised Cholesky-QR, so overheads exceed those of the value cache alone (Figure 6 ). Annotations give the overhead relative to the baseline; OOM marks configurations where the baseline runs out of memory.
Figure 8: Analytical memory footprint of the amortised value cache as a function of sequence length. Left: total value cache memory for the uncompressed baseline against value cache amortisation at 4 × compression. Right: decomposition of our footprint on a logarithmic scale. The MLP parameters and the linear residual Wlin are constant in sequence length, while only the stored residuals and their indices scale with T . The fixed parameter cost dominates only at short contexts (below ∼2 – 3 k tokens), and is quickly amortised as the context grows.
Long-context inference in large language models (LLMs) is increasingly limited by the memory required for the key-value (KV) cache. KV cache compression addresses this problem by reducing the storage cost of previous tokens. Among existing approaches, low-rank compression is particularly attractive because it represents every token in reduced dimensions. Previous low-rank methods typically derive fixed projection spaces from model weights, construct fixed spaces from calibration activations, or construct a shared basis over a broad cache region. Such representations may not capture detailed but important information. We partition each per-head KV cache into fixed-length logical pages and observe substantial low-rank structure within individual pages. Based on this observation, we propose PuzzleKV, a training- and calibration-free method that treats each completed page as an independent compression unit. PuzzleKV decomposes pages within each layer and KV head, computes attention directly over dense and factorized pages, and incrementally compresses newly eligible pages during autoregressive decoding. Experiments across models, context lengths, and benchmarks demonstrate the effectiveness of PuzzleKV under matched storage budgets. At approximately 60% of the original KV cache storage, PuzzleKV achieves more than 96% of Full KV performance across both evaluated models and all benchmark settings, with substantial gains over Global SVD on RULER and competitive performance on LongBench. To achieve a more aggressive compression ratio, PuzzleKV can be further combined with quantization while retaining more than 93% of Full KV performance using only 18.7% of the original storage.
Long-context inference makes the key-value (KV) cache a major memory bottleneck for large language models. We propose BELT, a low-rank KV cache compression framework. Its key-side objective targets attention distributions rather than raw keys or pre-softmax logits, yielding a quadratic surrogate with a closed-form low-rank solution. A learned global rank allocator uses differentiable soft selection during training and hard selection at deployment to meet a user-specified component budget. Experiments demonstrate improved model quality under aggressive compression and increased decoding throughput at comparable accuracy. Evaluated across three LLMs and multiple benchmarks, BELT achieves up to 70% low-rank KV cache compression while outperforming the strongest baseline by up to 23.3 points on LongBench, and up to 17.8x overall KV cache reduction when combined with quantization. With Triton-based GPU kernels and a quantized cache, BELT delivers up to 2.7x speedup for the attention module and 3.3x end-to-end decoding throughput.
Large language models (LLMs) have shown strong performance across diverse tasks, but their inference with long input contexts is bottlenecked by memory size and bandwidth. The Key-Value (KV) cache size grows linearly with sequence length and needs to be re-read from off-chip high-bandwidth memory (HBM) to on-chip memory at every decoding step, resulting in memory-bound inference. Existing methods reduce the cache by either eviction or quantization, but typically treat the two in isolation. In this paper, we cast KV cache compression as a rate-distortion problem, under which eviction and quantization are two end-points of the same bit allocation scheme. This exposes the need to optimize them jointly, motivating our method, RDKV (Rate-Distortion KV cache compression). RDKV derives the weight of each token or channel from the distortion that compression induces on the attention computation. Based on these weights, it assigns each token or channel a bit-width ranging from full precision down to zero bits guided by reverse water-filling, applied once after the prefilling stage. Experiments on LongBench, RULER, and InfiniteBench show that RDKV outperforms the best evaluated baseline by 9.1% on average. On LongBench it recovers 97.81% of full-cache accuracy with only 2.48% cache retention. Compared with full-cache FlashAttention-2 decoding, it achieves 4.5x decode speedup and 1.9x peak memory reduction with 128K context length, while maintaining comparable performance.