WUSH-KV: KV Cache Quantization with Data-Adaptive Transforms

Jiale Chen; Vage Egiazarian; Eldar Kurtić; Torsten Hoefler; Dan Alistarh

cs.LGSep 29, 2026

WUSH-KV: KV Cache Quantization with Data-Adaptive Transforms

Authors: Jiale Chen, Vage Egiazarian, Eldar Kurtić, Torsten Hoefler, Dan Alistarh

Organizations: Institute of Science and Technology Austria (ISTA) · Red Hat AI · ETH Zürich

Abstract

KV cache memory and bandwidth costs grow with context length and batch size, which limits efficient long-context inference. To address this bottleneck, we introduce WUSH-KV for low-bit KV-cache quantization. It adapts WUSH, which constructs a data-aware transform from the second-order statistics of both factors in a matrix product to reduce quantization error. WUSH-KV uses calibration data to construct separate key and value transforms, with the value transform folded into the model weights and the key transform applied after RoPE. The transforms can be paired with clipped quantizers. For one such quantizer, QuEST INT, we show that, under mild assumptions, the WUSH transform is near-optimal. With this quantizer, WUSH-KV reduces layerwise reconstruction error and achieves the lowest end-to-end perplexity among other tested transforms. For end-to-end evaluation, we integrate WUSH-KV into SGLang using OSCAR-style percentile-clipped affine quantization. At 2-bit, WUSH-KV performs comparably to or outperforms the OSCAR transform across all evaluated models and downstream tasks.

Figures & tables

Figure 1: Relative (squared)

\mathrm{L}_{2}

module-output errors for I, R, H, OSCAR, WUSH, and WUSH-A with both keys and values quantized at 2-bit.

Transform	4-Bit	3-Bit	2-Bit
I	11.36	15.29	38.08
R	9.77	10.18	14.22
H	9.77	10.06	258.29
OSCAR	9.77	10.19	13.74
WUSH	9.73	9.82	10.51

Table 1: Qwen3-8B WikiText-2 perplexity. The full-precision baseline is

9.72

Model	Method	AIME 2025	MATH-500	GPQA Diamond	LiveCodeBench v6
Qwen3-4B- Thinking-2507	BF16	77.8 $\pm$ 5.1 [-1pt] 21.8k 0%	89.5 $\pm$ 1.5 [-1pt] 5.8k 0%	69.9 $\pm$ 2.0 [-1pt] 9.0k 0%	57.1 $\pm$ 0.6 [-1pt] 19.2k 0%
	TurboQuant-Style H Transform	0.0 $\pm$ 0.0 [-1pt] 91.8k 100%	18.7 $\pm$ 1.0 [-1pt] 52.9k 76%	16.2 $\pm$ 1.8 [-1pt] 56.4k 68%	0.0 $\pm$ 0.0 [-1pt] 91.7k 99%
	OSCAR	65.6 $\pm$ 1.9 [-1pt] 24.2k 0%	89.3 $\pm$ 0.5 [-1pt] 6.6k 0%	65.3 $\pm$ 2.8 [-1pt] 10.1k 1%	50.3 $\pm$ 4.5 [-1pt] 25.0k 2%
	WUSH-KV WUSH Transform	71.1 $\pm$ 6.9 [-1pt] 24.3k 0%	90.1 $\pm$ 0.5 [-1pt] 6.6k 0%	66.3 $\pm$ 2.8 [-1pt] 10.0k 0%	52.4 $\pm$ 0.3 [-1pt] 23.0k 0%
Qwen3-8B	BF16	67.8 $\pm$ 1.9 [-1pt] 45.4k 11%	95.1 $\pm$ 0.3 [-1pt] 10.8k 1%	57.4 $\pm$ 3.8 [-1pt] 16.5k 1%	47.8 $\pm$ 2.2 [-1pt] 45.7k 11%
	TurboQuant-Style H Transform	0.0 $\pm$ 0.0 [-1pt] 92.1k 98%	21.9 $\pm$ 0.1 [-1pt] 52.3k 75%	30.1 $\pm$ 2.8 [-1pt] 28.6k 29%	1.0 $\pm$ 0.3 [-1pt] 87.8k 89%

Table 2: Downstream results with 2-bit KV caches. Each cell gives the task score (%) with its standard deviation over three seeds. The gray second line gives the mean generated-token count in thousands (k) and the budget-exhaustion rate. Bold marks the best quantized mean score and the shortest mean output for each model and task.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 2 — Figure 2: Relative (squared) $\mathrm{L}_{2}$ errors at 2-bit for I, R, H, OSCAR, WUSH, and WUSH-A. The top row shows query-key dot-product and softmax errors with only keys quantized. The bottom row shows module-output errors with only keys or only values quantized.

Figure 3: Relative (squared)

\mathrm{L}_{2}

module-output errors at 3-bit and 4-bit for I, R, H, OSCAR, WUSH, and WUSH-A.

Calibration, Quantizer	AIME 2025	MATH-500	GPQA Diamond	LiveCodeBench v6
WUSH-KV
FineWeb-Edu, OSCAR	46.7 $\pm$ 3.3	92.9 $\pm$ 0.1	54.5 $\pm$ 2.3	35.2 $\pm$ 0.7
FineWeb-Edu, OSCAR	53.4k 12%	12.2k 2%	16.3k 1%	53.9k 26%
FineWeb-Edu, QuEST	28.9 $\pm$ 6.9	88.7 $\pm$ 1.0	50.5 $\pm$ 2.5	33.1 $\pm$ 0.6
FineWeb-Edu, QuEST	63.4k 42%	11.7k 4%	22.1k 4%	63.7k 46%
OSCAR

Appendix

Table 3: Qwen3-8B downstream scores (%) over three seeds for the main configurations and quantizer or calibration alternatives. The gray second line gives the mean generated-token count in thousands (k) and the budget-exhaustion rate. Main configurations are marked in bold.

Method	4k	8k	16k	32k	64k	128k	Overall
BF16	99.71	97.79	92.85	94.29	85.09	79.17	91.48
WUSH-KV	97.44	91.92	80.75	76.87	68.36	57.26	78.77
OSCAR	96.63	89.79	80.00	74.00	64.36	25.69	71.74

Appendix

Table 4: Qwen3-8B RULER NIAH accuracy (%) over eight subtasks by context length.

Method	2-Needle	4-Needle	8-Needle	Overall	AUC
BF16	36.89 $\pm$ 1.96	22.66 $\pm$ 1.82	16.38 $\pm$ 0.44	25.31	21.95
WUSH-KV	10.55 $\pm$ 0.24	9.66 $\pm$ 0.37	8.45 $\pm$ 0.24	9.55	8.35
OSCAR	7.27 $\pm$ 0.61	6.98 $\pm$ 0.30	6.11 $\pm$ 0.05	6.79	4.72

Appendix

Table 5: Qwen3-8B MRCR summary accuracy (%).

Method	0k-8k	8k-16k	16k-32k	32k-64k	64k-128k
2-Needle
BF16	54.72 $\pm$ 2.40	41.14 $\pm$ 1.01	37.69 $\pm$ 3.47	27.48 $\pm$ 2.17	23.16 $\pm$ 3.31
WUSH-KV	18.05 $\pm$ 0.26	10.82 $\pm$ 0.92	10.14 $\pm$ 0.18	8.14 $\pm$ 0.79	5.49 $\pm$ 0.14
OSCAR	15.36 $\pm$ 1.30	8.88 $\pm$ 0.56	7.05 $\pm$ 0.40	4.69 $\pm$ 0.69	0.29 $\pm$ 0.11
4-Needle
BF16	29.64 $\pm$ 1.71	22.73 $\pm$ 1.16	23.59 $\pm$ 1.26	19.48 $\pm$ 3.17	17.14 $\pm$ 2.10

Appendix

Table 6: Qwen3-8B MRCR accuracy (%) by needle count and context-length bucket.

Method	0k-8k	8k-16k	16k-32k	32k-64k	64k-128k	Overall	AUC
2-Needle
BF16	70.98	60.14	50.88	38.27	36.87	51.49 $\pm$ 2.03	43.70
WUSH-KV	16.39	10.79	8.93	10.48	6.18	10.57 $\pm$ 0.69	9.25
OSCAR	24.83	14.39	9.01	10.87	3.92	12.64 $\pm$ 0.79	9.46
4-Needle
BF16	47.80	27.65	23.84	28.62	24.22	30.75 $\pm$ 1.76	27.03

Appendix

Table 7: Qwen3-32B MRCR accuracy (%) by needle count, including context-length buckets, overall scores, and AUC.

KV-Cache Quantization

Key-value cache quantization compresses transformer attention state to reduce memory use and bandwidth during long-context inference.

See all 100 papers

Spend Bits Where Queries Look: KV Cache Vector Quantization with Attention-Preserving Transforms
Aug 4, 2026Samuel Fernández-Menduiña, Amir Ziashahabi, Eduardo Pavez +2KV Caching Vector Quantization
Department of Electrical and Computer Engineering, University of Southern California
KV Cache Compression Through the Lens of Transform Coding
Aug 14, 2026Hannah Laus, Claudio Mayrink Verdun, Hao Wang +2KV-Cache Compression KV-Cache Quantization
Technical University of Darmstadt, the Technical University of Munich and the Munich Center for Machine Learning · Massachusetts Institute of Technology · Red Hat AI Innovation Team and the MIT-IBM Watson AI Lab +1
InnerQ: Hardware-Aware Tuning-Free Quantization of KV Cache for Large Language Models
Feb 26, 2026Sayed Mohammadreza Tayaranian Hosseini, Amir Ardakani, Warren J. GrossHardware-Aware Quantization KV-Cache Quantization
Department of Electrical and Computer Engineering McGill University Montreal, QC, Canada

Aug 4, 2026J/K move · Enter open · S save

Spend Bits Where Queries Look: KV Cache Vector Quantization with Attention-Preserving Transforms

Samuel Fernández-Menduiña, Amir Ziashahabi, Eduardo Pavez, Antonio Ortega+1

Department of Electrical and Computer Engineering, University of Southern California

KV Caching Vector Quantization KV-Cache Quantization Low-Bit Quantization

Open paper AskPDF arXiv

Long-context LLM decoding reads the key-value (KV) cache at every step. Loading it takes longer than computing attention over it, so throughput is bandwidth-bound. Hence, reducing the cache size can raise both decoding speed and serving capacity. The challenge is to reduce cache size while preserving the attention products, keeping reconstruction cheap, and using a fixed per-token bit count. At two bits per element, the most competitive methods rely on orthogonal transforms. However, existing techniques are either data-oblivious or use the query statistics without deriving the transform from a distortion criterion. Moreover, they rely on transforms built on top of random or Hadamard rotations, which equalize variances across entries rather than compacting energy, and fixed-width scalar quantizers, which are suboptimal at low rates. In this paper, we formulate KV cache quantization as a transform coding problem in which distortion is the error in the attention products. We derive closed-form optimal transforms for keys and values from calibration statistics, under a high-resolution model. We show that the optimal key transform is not orthogonal and satisfies a generalized Parseval relation: the attention-aware distortion becomes mean-squared error (MSE) in the transform domain. Thus, we can use MSE-optimal vector quantizers applied directly to the transformed key coefficients. To meet the fixed-width layout requirement, we show that grouping coefficients into equal-volume partitions makes equal-size codebooks attain the variable-rate optimum under the same high-resolution model. At two bits per element, our method, termed NOVA-KV, recovers most of the long-context retrieval accuracy lost by scalar quantization methods at comparable throughput.

Abstract

KV-Cache Quantization

Explore similar work

Spend Bits Where Queries Look: KV Cache Vector Quantization with Attention-Preserving Transforms

KV Cache Compression Through the Lens of Transform Coding

InnerQ: Hardware-Aware Tuning-Free Quantization of KV Cache for Large Language Models