Autoregressive decoding relies on the KV cache, whose memory footprint grows linearly with context length and caps batch size and throughput. Existing post-training methods exploit redundancy across tokens, layers, and feature dimensions, or lower numerical precision. We identify a complementary axis: within each attention head, values are largely predictable from their corresponding keys. We exploit this with value cache amortisation, which fits a lightweight per-head predictor at test time and stores residuals only for the tokens it reconstructs worst, giving 4× near-lossless value compression. The predictor can be initialised by meta-learning for substantially reduced prefill overhead, or fitted from scratch in settings that allow a one-off cost, such as multi-query inference. We pair value amortisation with cross-layer low-rank key compression to produce LowRAM, an asymmetric compression pipeline that stays close to lossless across three model families. By reconstructing only a selected subset of KV pairs at each decoding step, LowRAM can keep decoding overhead minimal and achieve up to 5× higher peak decode throughput at 4× compression. LowRAM further composes with quantisation and eviction, reaching 26.9× compression at 94.5% of baseline performance on long contexts. These results show that treating keys and values as structurally distinct enables efficient, aggressive, and composable KV cache compression.
Figures & tables
Figure 1: Mistral-7B compression on LongBench contexts longer than 10K tokens across different KV cache compression methods: eviction (adapted from Yang et al. (2025b) ), quantisation ( Zandieh et al., 2026 ) , LowRAM (our proposed method), and LowRAM-EQ (LowRAM combined with eviction and quantisation). Left : recovered performance against compression ratio. Right : maximum evaluated compression retaining at least 90% of baseline performance. LowRAM-EQ combines LowRAM, eviction, and quantisation. See Appendix C for a description of how compression is computed.
Figure 2: At inference, a per-head MLP learns to reconstruct values from keys, then is cached for decoding alongside residuals for the worst-reconstructed values.
Figure 3: Source-target analysis for predictive KV cache compression at a fixed 3× compression ratio for Mistral-7B. The left and centre panels show recovered LongBench performance when reconstructing current-layer keys KL and values VL , respectively, from alternative current- or previous-layer representations. The dotted line indicates uncompressed baseline performance. The right panel plots recovered performance against reconstruction mean-squared error for each source-target pair, showing that lower reconstruction error does not necessarily correspond to better downstream performance.
RULER 4K (%)
LongBench (%)
Model
2x
3x
4x
2x
3x
4x
Mistral-7B
99.4
98.8
96.0
100.7
99.4
97.9
Llama3.1-8B
99.6
97.9
93.9
98.7
98.5
99.6
Qwen3.5-4B
100.0
100.0
100.1
97.8
95.0
94.4
Table 1: Recovered average performance of LowRAM on RULER 4K and LongBench. Scores are averaged across each benchmark and reported as a percentage of the corresponding uncompressed baseline. The compression ratio reported for Qwen3.5-4B concerns only the full attention layers.
RULER (%)
LongBench (%)
SR
Meta-init
16K
32K
64K
> 10K
×
×
96.9
100.0
95.8
95.3
✓
×
97.7
99.0
96.8
95.3
✓
✓
96.3
96.8
92.4
92.9
Table 2: Recovered LongBench ( > 10K) and RULER 16-64K performance of LowRAM on Llama-3.1-8B-Instruct at 4 × compression, with and without selective reconstruction (SR) and a meta-learned initialisation (Meta-init). Rows without Meta-init use 300 gradient updates per context (vs. 150 in Table 1 ); Meta-init uses 2.
Figure 4: Decode throughput for Llama-3.1-8B at different batch sizes and sequence lengths at 4× compression, comparing exact KV attention against LowRAM-SR (LowRAM with selective reconstruction). Appendix F reports the full selective-reconstruction configuration.
RULER (4K)
LongBench
Model
Method
CR
Score (%)
CR
Score (%)
Mistral-7B
Eviction
2×
99.9
2×
99.4
Quantisation
5.1×
99.5
5.1×
98.9
LowRAM
4×
96.0
4×
97.9
LowRAM-E
8×
91.8
8×
96.6
LowRAM-Q
4.8×
99.6
6.4×
99.4
Table 3: Recovered performance of LowRAM and LowRAM combined with eviction (LowRAM-E), quantisation (LowRAM-Q), and both (LowRAM-EQ). The achieved compression ratio varies across benchmarks because it depends on sequence length. CR refers to compression ratio.
Eviction
Quantisation
LowRAM
LowRAM-EQ
CR
10.0×
7.5×
9.0×
26.9×
Score (%)
91.6
92.3
90.7
94.5
Table 4: Maximum evaluated compression retaining at least 90% of Mistral-7B performance on LongBench contexts longer than 10K tokens. CR uses the byte-level accounting in Appendix C .
RULER 4K (%)
LongBench (%)
Method
2×
3×
4×
2×
3×
4×
Amortisation
100.0
99.6
98.2
100.4
100.4
99.3
xKV
99.9
99.7
99.3
95.0
95.3
95.8
Table 5: Recovered performance of amortisation and xKV applied to the value cache in isolation on Llama-3.1-8B, reported as a percentage of the uncompressed-cache baseline. The displayed configuration uses a two-layer per-head MLP with 300 per-context gradient updates, without warm-starting or Wlin . Table 8 provides the complete optimisation-budget and initialisation sweep.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Setting
Number of MLP layers
2
Hidden-width factor γ
1
Activation
ReLU
Loss
MSE
Optimiser
Adam
Learning rate
0.001
Appendix
Table 6: Hyperparameters used to train the value-reconstruction networks used for value cache amortisation.
Method
LongBench (%)
Δ
Amortisation
99.0
w/o error-residual store
64.9
-34.1
w/o slim initialisation
97.2
-1.8
w/o warm start
97.1
-1.9
w per-layer MLP
79.5
-19.5
w linear predictor
57.5
-41.5
Appendix
Table 7: Ablation of the main components of the amortised value cache method for Mistral-7B at 3× value cache compression. LongBench scores are reported as percentages of the uncompressed KV cache baseline, and Δ denotes the absolute change in recovered performance relative to the full amortisation method.
Figure 5: Residual selection strategies at equal residual budgets on RULER NIAH-Multikey-1 for Llama-3.1-8B-Instruct, using the full LowRAM pipeline at 4 × compression with the meta-learned initialisation (2 gradient updates). Left: mean squared error between attention outputs computed with the compressed and exact KV caches ( n=16 ). Right: NIAH accuracy ( n=200 ). + 8 sinks forces residuals for the first 8 tokens within the same budget. All strategies share the same compressed key cache and are evaluated on the same sequences.
RULER 4K (%)
LongBench (%)
Predictor
Updates
2×
3×
4×
2×
3×
4×
MLP
10
90.8
78.0
61.5
76.2
69.0
59.0
MLP +Wlin
10
98.4
94.4
79.0
84.9
82.3
78.1
MLP
100
99.4
97.6
88.9
93.6
91.7
88.9
MLP +Wlin
100
99.9
99.1
95.1
94.4
93.8
91.5
MLP
200
99.9
99.2
97.1
98.1
98.7
97.9
Appendix
Table 8: Recovered performance of value cache amortisation on Llama-3.1-8B as a function of the per-context optimisation budget and Slim Attention initialisation. Keys are retained exactly, and results are reported relative to the uncompressed-cache baseline. All predictors are two-layer per-head MLPs trained without warm-starting. The 300-update configuration is the high-fidelity result summarised in Table 5 .
Model
Value Predictor
Wlin shape
CR
LongBench (%)
Mistral-7B
Joint Wlin
(HKV,dh,HKV,dh)
10 ×
39.7
Per-head Wlin
(HKV,dh,dh)
79 ×
9.7
fθ + joint residual Wlin
(HKV,dh,HKV,dh)
3 ×
9.5
fθ + per-head residual Wlin
(HKV,dh,dh)
3 ×
84.7
Llama-3.1-8B
Joint Wlin
(HKV,dh,HKV,dh)
11 ×
67.8
Per-head Wlin
(HKV,dh,dh)
89 ×
20.3
Appendix
Table 9: Ablation of joint and per-head Wlin initialisation for value cache amortisation on LongBench. Joint Wlin permits mixing across KV heads, whereas per-head Wlin applies an independent dh×dh map to each KV head. LongBench scores are percentages of the uncompressed baseline. CR refers to compression ratio. Differences in achieved compression ratio between Mistral and Llama reflect model-specific effective context lengths induced by their different tokenizers.
RULER (4K)
LongBench
Model
Method
CR
%
CR
%
Mistral-7B
Eviction
2×
99.9
2×
99.4
3×
99.3
3×
97.6
4×
98.0
4×
96.3
Quantisation
5.1×
99.5
5.1×
98.9
LowRAM
2×
99.4
2×
100.7
Appendix
Table 10: Recovered performance of LowRAM, used alone or combined with eviction and quantisation, across a range of compression settings for Mistral-7B and Llama-3.1-8B. Results are reported as percentages of the corresponding uncompressed baseline on RULER 4K and LongBench. Grey rows show eviction and quantisation applied independently, blue rows show LowRAM alone, and unshaded rows show LowRAM combined with eviction (LowRAM-E), quantisation (LowRAM-Q), or both (LowRAM-EQ). CR denotes the total compression ratio achieved on each benchmark; for combined methods, it accounts for the composition of the individual compression components and may differ between benchmarks because of sequence-length variation and across models due to model-specific effective context lengths induced by their different tokenizers.
LongBench ( > 10K)
Method
CR
Score (%)
Eviction
3.0×
98.1
4.0×
97.8
8.0×
93.9
10.0×
91.6
Quantisation
3.9×
100.0
Appendix
Table 11: Compression ratio and recovered performance for Mistral-7B using different KV cache compression methods, evaluated on the subset of LongBench examples with sequence lengths exceeding 10k tokens. Scores are reported as percentages of the uncompressed baseline. CR refers to compression ratio and is calculated from the byte-level key and value footprints described in Appendix C
Component
Hyperparameter
Setting
Meta-optimisation
Algorithm
FOMAML
Inner-loop updates, K
2
Inner optimiser
Adam
Inner learning rate, α
10−3
Outer optimiser
AdamW
Outer learning rate, β
3×10−4
Appendix
Table 12: Hyperparameters used to meta-learn the value-predictor initialisation. The inner and outer objectives use the same key-value pairs because the desired behaviour is rapid memorisation of the observed context.
RULER 4K (%)
LongBench (%)
2 ×
3 ×
4 ×
2 ×
3 ×
4 ×
98.4 ( − 1.2)
96.1 ( − 1.8)
91.2 ( − 2.7)
93.6 ( − 5.1)
93.0 ( − 5.5)
91.1 ( − 8.5)
Appendix
Table 13: Recovered performance of LowRAM when the amortised value cache is initialised from meta-learned weights and adapted with only 2 gradient steps for Llama-3.1-8B. Scores are reported as percentages of the uncompressed baseline; values in parentheses give the drop in percentage points relative to vanilla LowRAM (Table 1 ), which uses 150 gradient steps per context.
Parameter
Value
Chunk size
8 tokens
Token budget
2048 tokens
Local window
32 tokens
Outlier chunks
48 chunks (384 tokens)
Max. tokens attended per step
2464 tokens
Appendix
Table 14: Selective reconstruction configuration.
Figure 6: TTFT overhead of value cache amortisation alone versus the number of gradient updates used to fit the MLP, at a 4K-token sequence length with batch size 4. Keys are left uncompressed, so this isolates the cost of fitting the value predictor. Bars show raw TTFT; annotations give the overhead relative to the uncompressed baseline. See Figure 7 for the full LowRAM pipeline, including key factorisation.
Figure 7: TTFT of the uncompressed baseline (exact KV) and the full LowRAM pipeline at 4 × compression across context lengths and batch sizes. LowRAM timings include both value cache amortisation (2 gradient updates) and low-rank factorisation of the keys via randomised Cholesky-QR, so overheads exceed those of the value cache alone (Figure 6 ). Annotations give the overhead relative to the baseline; OOM marks configurations where the baseline runs out of memory.
Figure 8: Analytical memory footprint of the amortised value cache as a function of sequence length. Left: total value cache memory for the uncompressed baseline against value cache amortisation at 4 × compression. Right: decomposition of our footprint on a logarithmic scale. The MLP parameters and the linear residual Wlin are constant in sequence length, while only the stored residuals and their indices scale with T . The fixed parameter cost dominates only at short contexts (below ∼2 – 3 k tokens), and is quickly amortised as the context grows.