KV Cache Compression Through the Lens of Transform Coding

Hannah Laus; Claudio Mayrink Verdun; Hao Wang; Flavio du Pin Calmon; Felix Krahmer

cs.LGAug 14, 2026

KV Cache Compression Through the Lens of Transform Coding

Authors: Hannah Laus, Claudio Mayrink Verdun, Hao Wang, Flavio du Pin Calmon, Felix Krahmer

Organizations: Technical University of Darmstadt, the Technical University of Munich and the Munich Center for Machine Learning · Massachusetts Institute of Technology · Red Hat AI Innovation Team and the MIT-IBM Watson AI Lab · John A. Paulson School Of Engineering And Applied Sciences at Harvard University

Abstract

The key-value (KV) cache stores information from past tokens and is a major memory bottleneck in long-context inference. Existing quantization methods address this bottleneck by representing the KV cache uniformly with lower-precision data types and designing quantization schemes to minimize reconstruction error in the cache itself, without accounting for how that error propagates through attention mechanisms. We prove that, under a white-noise quantization model, the expected attention-aware distortion decomposes into additive key and value contributions that factor across tokens and channels. Building on transform coding and reverse water-filling, which are classical tools from signal processing and rate-distortion theory, we introduce Attention-Aware Transform Coding (AATC), which allocates bits over a calibration set to minimize attention-aware distortion. On Llama-3.1-8B-Instruct and Qwen-2.5-7B-Instruct, evaluated across LongBench, RULER, GSM8K, MMLU-Pro, and MATH-500, our method achieves near-lossless accuracy at approximately $5.8\times$ compression, whereas each baseline degrades in at least some settings.

Figures & tables

Factor	Design axis	Mechanism	Representative methods
$a_{i}^{2}$ (token)	token eviction	drop tokens	H2O [ 9 ] , SnapKV [ 22 ] , StreamingLLM [ 10 ] , VATP [ 23 ] , CriticalKV [ 24 ] , Expected Attention [ 26 ]
$\\|v_{i}\mathbf{W}_{O}-o\\|^{2}$ (token)	output-relevance	drop tokens	CAOTE [ 25 ] (pre- $\mathbf{W}_{O}$ )
$q_{c}^{2}$ , $C_{Q}$ (K channel)	query-aware key precision	VQ/MP/rotation/ TC+bit-alloc.	A 2 ATS [ 12 ] , MixKVQ [ 21 ] , OSCAR [ 19 ] , SQuat [ 20 ] , Ours (Section V )
$\mathbf{W}_{O_{c}}$ (V channel)	output-projection- aware value	TC+bit-alloc	(implicit in KVQuant [ 6 ] ), Ours (Section V )
$\mathbb{E}[(\delta\textnormal{k}_{ic})^{2}]$ , $\mathbb{E}[(\delta\textnormal{v}_{ic})^{2}]$	per-channel sensitivity	scaling/codebook/ SVD/MP/ TC+bit-alloc	KIVI [ 5 ] , KVQuant [ 6 ] , KVTC [ 8 ] , PALU [ 7 ] , RateQuant [ 11 ] , Ours (Section V )

TABLE I: Distortion factors in

D=D_{K}+D_{V}

and the design axes they motivate. Each row corresponds to one factor in the per-token, per-channel decomposition of attention-output error under KV quantization. Methods are listed by the factor they primarily target. Here, VQ

=

Vector quantization, MP

=

mixed precision and TC

=

transform coding.

\{b_{c}^{(K,\ell)}\}\;\leftarrow\;\operatorname*{arg\,min}_{\sum_{\ell,c}b_{c}^{(K)}\leq\mathcal{B}_{K}}\;\sum_{\ell,c}\Bigl(w_{\ell,c}^{(K)}\cdot(\sigma_{\ell,c}^{(K)})^{2}2^{-2b_{\ell,c}}\Bigr)

Algorithm 1 Attention-Aware Transform Coding (AATC) for KV cache Compression

Fig. 1: RULER score versus context length for AATC and AATC var-only at 2.5-bit (solid) and 2-bit (dashed), on Llama-3.1-8B-Instruct and Qwen-2.5-7B-Instruct .

		LongBench	RULER				Reasoning
Method	Comp.	avg.	4k	8k	16k	32k	GSM8K	MMLU -m	MMLU -cs	MATH-500
Llama-3.1-8B-Instruct
Baseline	1.00	0.570	0.859	0.856	0.828	0.775	0.789	0.438	0.456	0.454
KIVI	5.00	0.564	0.846	0.840	0.810	0.763	0.730	0.390	0.420	0.452
PALU	5.39	0.550	0.851	0.849	0.83	0.769	0.617	0.274	0.320	0.314
KVQuant	5.91	0.549	0.715	0.729	0.676	0.654	0.754	0.376	0.402	0.392

TABLE II: Comp. is the KV cache compression ratio relative to the FP16 baseline. LongBench is the task-averaged score over seven English subsets (per-subtask breakdown in Table III ). Here

\sigma

is the standard error of each score, the standard deviation of the per-example scores divided by

\sqrt{N}

over the

N

evaluated samples. A method is called statistically indistinguishable from FP16 when its score lies within

2\sigma

of the FP16 score.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

	Method	TriviaQA	Qasper	TREC	SAMSum	LCC	RepoB-P	QMSum	avg.
Llama-3.1-8B-Instruct	Baseline	0.917	0.455	0.725	0.437	0.634	0.567	0.254	0.570
	KIVI	0.913	0.440	0.725	0.448	0.619	0.552	0.250	0.564
	PALU	0.892	0.449	0.710	0.438	0.539	0.576	0.248	0.550
	KVQuant	0.921	0.413	0.675	0.415	0.633	0.546	0.241	0.549
	AATC var	0.922	0.452	0.715	0.430	0.634	0.559	0.251	0.566
	AATC (ours)	0.919	0.444	0.690	0.434	0.623	0.546	0.252	0.558

Appendix

TABLE III: Per-subtask LongBench scores at the main-table operating point: F1 for QA (TriviaQA, Qasper), ROUGE-L for summarization (SAMSum, QMSum) and query summarization, accuracy for classification (TREC), and edit-similarity for code (LCC, RepoBench-P). Bold = best compressed method per column, and the FP16 baseline is a reference.

			LongBench	RULER				Reasoning
	Method	Comp.	avg.	4k	8k	16k	32k	GSM8K	MMLU-m	MMLU-cs	MATH
Llama	Baseline (FP16)	1.00	0.570	0.859	0.856	0.828	0.775	0.789	0.438	0.456	0.454
	AATC var	7.0	0.543	0.818	0.822	0.775	0.677	0.767	0.402	0.427	0.484
	AATC (ours)	7.0	0.568	0.851	0.861	0.809	0.735	0.756	0.416	0.434	0.448
Qwen	Baseline (FP16)	1.00	0.574	0.720	0.716	0.718	0.720	0.818	0.708	0.567	0.680
	AATC var	6.9	0.518	0.688	0.653	0.640	0.566	0.795	0.644	0.510	0.664

Appendix

TABLE IV: AATC at the aggressive 2-bit operating point (

\approx

\times

compression), against the FP16 reference. Baseline compression methods (KIVI, PALU, KVQuant), which run at 2-bit and

\approx

5 to 6

\times

, are reported in the main results table ( II ).

Fig. 2: LongBench average and GSM8K versus KV averages bits on Llama-3.1-8B-Instruct .

	Config	Comp	TriviaQA	Qasper	TREC	SAMSum	LCC	RepoB-P	QMSum	LB avg.	GSM8K
Llama-3.1-8B–Instruct	Baseline FP16	1.0	0.917	0.455	0.725	0.437	0.634	0.567	0.254	0.570	0.789
	KIVI 4-bit	3.1	0.917	0.454	0.725	0.440	0.634	0.568	0.250	0.570	0.795
	KVQuant 4-bit	3.4	0.914	0.447	0.725	0.426	0.631	0.560	0.256	0.566	0.779
	AATC 4-bit	3.8	0.916	0.446	0.725	0.436	0.624	0.565	0.253	0.566	0.783
	KIVI 3-bit	3.8	0.914	0.450	0.725	0.435	0.634	0.564	0.254	0.568	0.792
	KVQuant 3-bit	4.3	0.921	0.452	0.725	0.433	0.631	0.559	0.250	0.567	0.776

Appendix

TABLE V: LongBench per-subtask scores and GSM8K across KV multiple bit-budgets. Metrics: F1 (TriviaQA, Qasper), accuracy (TREC), ROUGE-L (SAMSum, QMSum), edit-similarity (LCC, RepoBench-P). LB avg. is the seven-task mean.

AATC variant (2-bit, Llama-3.1-8B-Instruct )	TriviaQA	Qasper	TREC	SAMSum	LCC	RepoB-P	QMSum	LB avg.	GSM8K
AATC 2-bit (standard)	0.917	0.456	0.720	0.438	0.630	0.570	0.248	0.568	0.751
Allocation
no $q_{c}^{2}$ in allocation	0.910	0.410	0.685	0.444	0.545	0.473	0.248	0.531	0.752
no $W_{O}$ in allocation	0.916	0.452	0.720	0.430	0.602	0.551	0.251	0.560	0.763
per-layer allocation	0.913	0.430	0.715	0.445	0.583	0.533	0.249	0.553	0.760
global allocation	0.847	0.337	0.595	0.432	0.475	0.462	0.231	0.483	0.740

Appendix

TABLE VI: Ablation of AATC’s bit-allocation and calibration choices (2-bit, Llama-3.1-8B-Instruct , with LongBench per-subtask, LB avg. = seven-task mean and GSM8K ).

KV-Cache Compression

Key-value cache compression reduces the memory and bandwidth needed to retain attention states during long-context inference.

See all 257 papers

Spend Bits Where Queries Look: KV Cache Vector Quantization with Attention-Preserving Transforms
Aug 4, 2026Samuel Fernández-Menduiña, Amir Ziashahabi, Eduardo Pavez +2KV Caching Vector Quantization
Department of Electrical and Computer Engineering, University of Southern California
The risk of KV cache compression
Jul 1, 2026Lukas Haverbeck, Carmen Amo Alonso, Andres Felipe Posada-Moreno +2Transformer Inference KV Caching
RWTH Aachen University · Stanford University
A JoLT for the KV cache: Near-Lossless KV Cache Compression via Joint Rank-bit Allocation
Jul 14, 2026Rahul Krishnan, Volker SchulzTucker Decomposition KV Caching
Universität Trier · Fachbereich IV, Mathematik, Universität Trier

Aug 4, 2026J/K move · Enter open · S save

Spend Bits Where Queries Look: KV Cache Vector Quantization with Attention-Preserving Transforms

Samuel Fernández-Menduiña, Amir Ziashahabi, Eduardo Pavez, Antonio Ortega+1

Department of Electrical and Computer Engineering, University of Southern California

KV Caching Vector Quantization KV-Cache Quantization Low-Bit Quantization

Open paper AskPDF arXiv

Long-context LLM decoding reads the key-value (KV) cache at every step. Loading it takes longer than computing attention over it, so throughput is bandwidth-bound. Hence, reducing the cache size can raise both decoding speed and serving capacity. The challenge is to reduce cache size while preserving the attention products, keeping reconstruction cheap, and using a fixed per-token bit count. At two bits per element, the most competitive methods rely on orthogonal transforms. However, existing techniques are either data-oblivious or use the query statistics without deriving the transform from a distortion criterion. Moreover, they rely on transforms built on top of random or Hadamard rotations, which equalize variances across entries rather than compacting energy, and fixed-width scalar quantizers, which are suboptimal at low rates. In this paper, we formulate KV cache quantization as a transform coding problem in which distortion is the error in the attention products. We derive closed-form optimal transforms for keys and values from calibration statistics, under a high-resolution model. We show that the optimal key transform is not orthogonal and satisfies a generalized Parseval relation: the attention-aware distortion becomes mean-squared error (MSE) in the transform domain. Thus, we can use MSE-optimal vector quantizers applied directly to the transformed key coefficients. To meet the fixed-width layout requirement, we show that grouping coefficients into equal-volume partitions makes equal-size codebooks attain the variable-rate optimum under the same high-resolution model. At two bits per element, our method, termed NOVA-KV, recovers most of the long-context retrieval accuracy lost by scalar quantization methods at comparable throughput.

Abstract

KV-Cache Compression

Explore similar work

Spend Bits Where Queries Look: KV Cache Vector Quantization with Attention-Preserving Transforms

The risk of KV cache compression

A JoLT for the KV cache: Near-Lossless KV Cache Compression via Joint Rank-bit Allocation