Tailoring the Quantization Space for 1-Bit KV Cache Compression

Minsoo Cheong; Donghyun Son; Sungjoo Yoo

cs.LGOct 2, 2026

Tailoring the Quantization Space for 1-Bit KV Cache Compression

Authors: Minsoo Cheong, Donghyun Son, Sungjoo Yoo

Organizations: Stanford University · Seoul National University

Abstract

The key-value (KV) cache becomes a major memory bottleneck in long-context LLM inference, placing substantial pressure on memory capacity and bandwidth. To mitigate this bottleneck, vector quantization (VQ) has emerged as a promising approach for aggressive KV cache compression. However, existing VQ methods degrade substantially in the 1-bit regime. At such extreme compression, each codebook must represent a larger group of channels with a limited set of centroids, making effective use of its capacity increasingly challenging. To address this, we introduce $\textbf{TaSQ}$ , which tailors the VQ target space by combining query-guided channel weighting, cross-head normalization, and covariance-aware channel grouping to better reflect the error sensitivity and statistical structure of cached activations. Since these transforms are RoPE-compatible and can be easily merged into projection weights and codebooks, TaSQ preserves the conventional VQ lookup structure and adds negligible serving overhead. Across general, long-chain-of-thought reasoning, and long-context retrieval benchmarks, TaSQ consistently outperforms existing low-bit KV cache VQ baselines while preserving reasoning stability. On a single RTX 6000 Ada GPU, its SGLang implementation supports up to $14\times$ larger batch sizes and achieves $1.87\times$ higher peak throughput compared to the BF16 baseline.

Figures & tables

Figure 1: (a) Per-channel query medians and P10–P90 intervals. (b) Absolute inter-channel key correlations, with self-correlations masked. (c) Key distributions before and after RoPE for channels 44/45, using identical sampled tokens. (d) K-means reconstruction MSE on the fitting samples across all 32 layers, with matched group layouts and codebook sizes. All panels use a single 2,048-token GPQA window from Llama-3.1-8B; (a–c) show layer 15, KV head 0.

Figure 2: Overview of TaSQ. It tailors the VQ target space through query-guided channel weighting, cross-head shared-scale normalization, and covariance-aware grouping that preserves RoPE pairs, thereby reducing quantization error.

Figure 3: (a)

QK^{\top}

MSE (top) and attention KL divergence (bottom) with and without query-guided weighting. (b) Relationship between the group cost

c_{h}(G)

and VQ reconstruction MSE at layer 15, KV head 0. (c) Pearson correlation between group cost and reconstruction MSE across layers, averaged over 8 KV heads. (d) Total group cost (top) and VQ reconstruction MSE (bottom) under contiguous and covariance-aware grouping. Results are obtained with Llama-3.1-8B using disjoint GPQA samples for calibration and evaluation.

Model	Method	Bits (K/V)	GSM8K	MATH500	MBPP	HumanEval	BBH	MMLU	Avg.
Llama-3.1-8B-Instruct	BF16	16.000/16.000	83.62	42.20	59.60	62.20	73.38	62.64	63.94
	CQ	1.250/1.250	68.01	19.60	52.00	54.27	40.08	54.21	48.03
	NovaKV	1.375/1.250	76.95	29.40	55.00	53.66	48.46	56.43	53.32
	NSNQuant	1.238/1.238	73.39	31.40	50.20	57.93	48.57	58.33	53.30
	TaSQ	1.266/1.250	81.73	34.00	58.40	58.54	64.28	58.33	59.21
Qwen3-4B	BF16	16.000/16.000	86.28	72.60	64.00	81.71	78.03	74.72	76.22

Table 1: General performance under low-bit KV cache quantization. All results use greedy decoding. Bold denotes the best quantized result; higher is better.

Model	Method	Bits (K/V)	AIME’24	AIME’25	LCB-v6	SciBench	Avg.
Qwen3-4B-Thinking-2507	BF16	16.000/16.000	74.44 $\pm$ 1.92	74.44 $\pm$ 5.09	45.85 $\pm$ 0.62	73.99 $\pm$ 0.43	67.18 $\pm$ 1.37
	CQ	1.250/1.250	26.67 $\pm$ 3.33	14.44 $\pm$ 5.09	18.99 $\pm$ 0.44	54.00 $\pm$ 0.94	28.52 $\pm$ 1.54
	NovaKV	1.375/1.250	7.78 $\pm$ 1.92	6.67 $\pm$ 3.33	9.16 $\pm$ 0.38	27.94 $\pm$ 2.08	12.89 $\pm$ 1.10
	NSNQuant	1.238/1.238	44.44 $\pm$ 6.94	38.89 $\pm$ 1.92	34.66 $\pm$ 0.33	68.69 $\pm$ 0.55	46.67 $\pm$ 1.81
	TaSQ	1.266/1.250	68.89 $\pm$ 1.92	56.67 $\pm$ 3.33	43.00 $\pm$ 0.47	73.65 $\pm$ 0.60	60.55 $\pm$ 0.98
DeepSeek-R1-Distill-Llama-8B	BF16	16.000/16.000	53.33 $\pm$ 3.33	31.11 $\pm$ 1.92	39.68 $\pm$ 0.56	38.49 $\pm$ 0.30	40.65 $\pm$ 0.98

Table 2: Long-CoT reasoning performance under low-bit KV cache quantization. We report the mean and standard deviation over three seeds. Bold denotes the best quantized result; higher is better.

Model	Method	Bits (K/V)	4k	8k	16k	32k	64k
Llama-3.1-8B-Instruct	BF16	16.000/16.000	98.50 $\pm$ 1.73	98.83 $\pm$ 0.76	98.67 $\pm$ 0.29	98.67 $\pm$ 0.76	98.17 $\pm$ 1.04
	CQ	1.250/1.250	19.33 $\pm$ 1.61	15.50 $\pm$ 3.61	12.67 $\pm$ 1.26	12.83 $\pm$ 3.18	6.33 $\pm$ 1.26
	NovaKV	1.375/1.250	84.83 $\pm$ 0.29	81.67 $\pm$ 3.18	64.00 $\pm$ 2.18	21.33 $\pm$ 2.25	1.83 $\pm$ 0.29
	NSNQuant	1.238/1.238	89.17 $\pm$ 1.61	86.00 $\pm$ 1.32	86.83 $\pm$ 4.25	88.17 $\pm$ 1.15	82.17 $\pm$ 0.58
	TaSQ	1.266/1.250	93.33 $\pm$ 2.08	93.17 $\pm$ 0.76	92.00 $\pm$ 1.32	93.67 $\pm$ 1.04	93.17 $\pm$ 0.76
Qwen3-4B-Thinking-2507	BF16	16.000/16.000	100.00 $\pm$ 0.00	100.00 $\pm$ 0.00	99.83 $\pm$ 0.29	99.67 $\pm$ 0.29	95.33 $\pm$ 2.36

Table 3: Needle-in-a-haystack retrieval performance at increasing context lengths. We report the mean and standard deviation over three seeds. Bold denotes the best quantized result; higher is better.

Figure 4: Serving efficiency of Qwen3-4B-Thinking-2507 model measured on one RTX 6000 Ada GPU. (a) End-to-end throughput vs. batch size (2048 in / 32768 out); dashed line = BF16 peak, dotted line = BF16’s largest feasible batch. (b) Batch-1 prefill latency at 8k/16k/32k. (c) Generated tokens per problem on LiveCodeBench-v6, with the 32k cap shaded and the cap-hit rate per method.

Method	PPL $\downarrow$
FP16	6.2374
Full TaSQ	8.4559
w/o query-guided weighting	8.5172
w/o cross-head normalization	8.5103
w/o covariance-aware grouping	10.1731

Table 4: Component ablation of TaSQ on base Llama-3.1-8B. We report WikiText-2 perplexity at sequence length 2,048. Bold denotes the best quantized result.

Figure 5: GSM8K and MBPP performance versus average key bits for Llama-3.1-8B-Instruct (a, c) and Qwen3-4B (b, d). Scores are normalized to each model’s BF16 baseline (dashed, 100%); values remain in BF16.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Channel	Representation	Rate
CQ	K	1,024-centroid VQ	$10/8=1.250$
	V	1,024-centroid VQ	$10/8=1.250$
NSNQuant	K	256-centroid VQ + normalization metadata	$8/8+0.238=1.238$
	V	256-centroid VQ + normalization metadata	$8/8+0.238=1.238$
NovaKV	K	1,024-centroid VQ + one 16-bit head-wise scale	$10/8+16/128=1.375$
	V	1-bit SQ + two 16-bit head-wise parameters	$1+32/128=1.250$

Appendix

Table 5: Effective cache rate in bits per element. Per-token metadata costs are included in the reported rates.

Figure 6: Key–value channel characteristics across layers of Llama-3.1-8B on the same 2,048-token GPQA segment. (a) Mean coefficient of variation (CV) of channel sensitivity; lower values indicate flatter sensitivity. (b) Mean absolute inter-channel Pearson correlation with the diagonal excluded. Both metrics are averaged over the 8 KV heads.

Value quantization	PPL $\downarrow$
TaSQ	8.45594
TaSQ + value-side transformations	8.42058

Appendix

Table 6: Value-side ablation on Llama-3.1-8B: WikiText-2 perplexity at sequence length 2,048 (lower is better). The TaSQ key path is identical in both rows.

		Llama-3.1-8B			Qwen3-4B
scale / codebook	bits/elem	mean	min	max	mean	min	max
per-head, $K=1024$	1.3750	0.004483	0.002882	0.006953	0.019794	0.006732	0.030261
pooled, $K=1024$	1.2656	0.004564	0.002965	0.007037	0.019688	0.006754	0.030014
pooled, $K=2048$	1.3906	0.003460	0.002245	0.005313	0.014763	0.005045	0.022639

Appendix

Table 7: Relative SSE with separately fitted codebooks, across all layers. Bits per channel include fp16 scale metadata with

g=8

d=128

, and

H=8

Figure 7: Attention-logit NMSE (log scale) versus key position for Llama-3.1-8B-Instruct (a) and Qwen3-4B (b). Dotted lines mark the 2048 calibration length.

Context	Latency (ms/step)		Overhead
(tokens)	Attention	Dense Restoration	(%)
1 024	1.55	0.55	$+36\%$
4 096	2.11	1.23	$+58\%$
16 384	4.14	4.27	$+103\%$

Appendix

Table 8: Cost of dense key restoration for Qwen3-4B-Thinking-2507 on one RTX 3090 at batch size 1. Restoration is measured as a standalone BF16 GEMM over reconstructed keys, excluding VQ decoding; overhead is relative to attention latency.

Key Transform	Relative Attention-Logit Error
	Calibration Corpus	GSM8K
$\operatorname{diag}(\tilde{A}_{h})^{1/2}$ (TaSQ)	$0.00287$	$\mathbf{0.00234}$
$\tilde{A}_{h}^{1/2}$	$\mathbf{0.00264}$	$0.00933$

Appendix

Table 9: Relative attention-logit error for Llama-3.1-8B-Instruct, averaged over

32

layers. Both variants use the same calibration data and

K=1024

codebooks.

Binary Quantization

See all 9 papers

VQKV: High-Fidelity and High-Ratio Cache Compression via Vector-Quantization
Mar 17, 2026Yixuan Wang, Qingyu Shi, Jiayu Zhou +3KV-Cache Compression KV-Cache Quantization
LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University · Shanghai Innovation Institute · Shanghai Artificial Intelligence Laboratory +2
OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyond
May 19, 2026Zunhai Su, Rui Yang, Chao Zhang +12KV Caching Rotation-Based Quantization
Tsinghua University · Meituan LongCat Team · The University of Hong Kong +3
KV-COBRA: KV Cache Compression via Co-Optimized Bit-Rank Allocation
Sep 21, 2026Sihyeon Ha, Jaeho Lee, Yo-Seb JeonLow-Rank Compression KV-Cache Compression
Pohang University of Science and Technology (POSTECH)

Mar 17, 2026J/K move · Enter open · S save

VQKV: High-Fidelity and High-Ratio Cache Compression via Vector-Quantization

Yixuan Wang, Qingyu Shi, Jiayu Zhou, Dianbo Liu+2

LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University · Shanghai Innovation Institute · Shanghai Artificial Intelligence Laboratory+2

KV-Cache Compression KV-Cache Quantization Memory Compression LLM Inference Acceleration Vector Quantization

Open paper AskPDF arXiv

The growing context length of Large Language Models (LLMs) enlarges the Key-Value (KV) cache, limiting deployment in resource-limited environments. Prior training-free approaches for KV cache compression typically rely on low-rank approximation or scalar quantization, which fail to simultaneously achieve high compression ratios and high reconstruction fidelity. We propose VQKV, a novel, training-free method introducing vector quantization (VQ) to obtain highly compressed KV representations while preserving high model fidelity, allowing for the representation of thousands of floating-point values with just a few integer indices. As a result, VQKV achieves an 82.8% compression ratio on LLaMA3.1-8B while retaining 98.6% of the baseline performance on LongBench and enabling 4.3x longer generation length on the same memory footprint.

Abstract

Binary Quantization

Explore similar work

VQKV: High-Fidelity and High-Ratio Cache Compression via Vector-Quantization

OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyond

KV-COBRA: KV Cache Compression via Co-Optimized Bit-Rank Allocation