VQKV: High-Fidelity and High-Ratio Cache Compression via Vector-Quantization

Yixuan Wang; Qingyu Shi; Jiayu Zhou; Dianbo Liu; Ziwei He; Zhouhan Lin

cs.CLMar 17, 2026

VQKV: High-Fidelity and High-Ratio Cache Compression via Vector-Quantization

Authors: Yixuan Wang, Qingyu Shi, Jiayu Zhou, Dianbo Liu, Ziwei He, Zhouhan Lin

Organizations: LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University · Shanghai Innovation Institute · Shanghai Artificial Intelligence Laboratory · University of Cambridge ·

Abstract

The growing context length of Large Language Models (LLMs) enlarges the Key-Value (KV) cache, limiting deployment in resource-limited environments. Prior training-free approaches for KV cache compression typically rely on low-rank approximation or scalar quantization, which fail to simultaneously achieve high compression ratios and high reconstruction fidelity. We propose VQKV, a novel, training-free method introducing vector quantization (VQ) to obtain highly compressed KV representations while preserving high model fidelity, allowing for the representation of thousands of floating-point values with just a few integer indices. As a result, VQKV achieves an 82.8% compression ratio on LLaMA3.1-8B while retaining 98.6% of the baseline performance on LongBench and enabling 4.3x longer generation length on the same memory footprint.

Figures & tables

Figure 1: Overview of our VQKV. The left part shows the detailed process of our VQKV on prefilling stage and decoding stage. The right part shows the overview of our VQKV.

	Ratio	Single-Doc			Multi-Doc			Summary			Few-shot			Synthetic		Code		Avg.
NQ	Ratio	Qsp	MF	HQ	WQ	Msq	GR	QS	MN	TR	TQ	SS	PC	PR	LCC	Re-P		Avg.
LLaMA3.1-8B	0	13.0	20.4	32.1	12.0	14.0	8.7	29.7	25.2	1.0	73.5	91.0	47.2	0.8	26.8	72.0	69.2	33.5
+ SnapKV	-	12.7	19.5	31.2	11.5	13.9	8.9	23.7	24.1	0.9	70.0	91.3	46.5	0.9	27.8	71.7	68.5	31.7
+ KIVI	75.0%	19.2	20.8	31.2	15.2	17.7	8.2	17.6	12.1	3.9	72.5	89.6	33.9	2.2	15.9	67.6	64.4	30.7
+ ASVD	80.0%	4.6	9.9	16.1	9.7	7.4	5.2	9.0	16.6	12.8	60.0	78.7	33.7	2.8	4.9	30.4	36.9	21.2
+ Palu	80.0%	6.4	16.6	23.1	9.7	12.3	6.6	16.5	21.5	10.7	72.5	84.6	37.1	1.3	14.3	64.7	59.1	28.6

Table 1: Results of LLaMA3.1-8B and LLaMA3.2-3B on LongBench . The Ratio means discarding ratios. Our VQKV achieves the closest performance to the full cache models on comparable compression ratio against other methods.

Panel of Figure 2 — Figure 2: Results of LLaMA3.1-8B on NIAH.

Panel of Figure 3 — Figure 3: Results of LLaMA3.2-3B on NIAH.

Panel of Figure 4 — Figure 4: Detailed results of LLaMA3.1-8B on RULER in different context length. The results of full cache model are taken as the 100% reference.

Panel of Figure 5 — Figure 5: Detailed results of LLaMA3.2-3B on RULER in different context length. The results of full cache model are taken as the 100% reference.

	4K	8K	16K	32K	Avg.
LLaMA3.1-8B	94.74	92.72	93.51	90.16	92.78
+ ASVD	39.75	32.94	25.27	19.29	29.31
+ SnapKV	91.18	77.12	69.80	58.52	74.16
+ Palu	74.70	66.01	59.80	52.44	63.24
+ KIVI	57.91	52.81	47.60	49.73	52.01
+ VQKV(Ours)	92.88	89.61	87.02	79.71	87.31

Table 2: Results of LLaMA3.1-8B and LLaMA3.2-3B on RULER. Our VQKV outperforms other methods on 4K to 32K length.

Ratio	Method	Single-Doc			Multi-Doc			Summary			Few-shot			Synthetic		Code		Avg.
Ratio	Method	NQ	Qsp	MF	HQ	WQ	Msq	GR	QS	MN	TR	TQ	SS	PC	PR	LCC	Re-P	Avg.
0	LLaMA3.1-8B	13.0	20.4	32.1	12.0	14.0	8.7	29.7	25.2	1.0	73.5	91.0	47.2	0.8	26.8	72.0	69.2	33.5
50.0%	ASVD	11.5	20.3	32.0	10.6	11.8	8.5	28.4	24.3	0.1	73.5	90.7	46.0	2.0	40.1	70.1	67.8	33.6
	Palu	16.3	20.3	31.9	12.1	11.8	7.7	18.1	11.6	1.3	76.0	90.2	35.3	1.5	8.0	67.8	65.4	29.7
	VQKV	12.6	20.1	32.4	12.1	14.1	8.7	29.6	24.9	0.9	73.5	91.0	47.3	0.9	26.8	71.7	69.4	33.5
60.0%	ASVD	12.7	20.0	34.0	10.6	12.0	6.9	28.5	24.6	0.4	70.5	89.9	44.2	2.1	40.7	69.6	67.8	33.4

Table 3: Results of LLaMA3.1-8B on LongBench with different compression ratios. The Ratio means discarding ratios. Our VQKV achieves the best performance on comparable compression ratio against other methods.

Figure 6: The results of LLaMA3.1-8B on LongBench with different compression ratio.

Panel of Figure 7 — Figure 7: Results of VQKV on LongBench with different combinations of codebooks on LLaMA3.1-8B.

Figure 8: Generation length and corresponding peak memory usage between full-cache LLaMA3.1-8B and our VQKV. The prompt length is 65536 and FlashAttention is used on both.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

$N^{k}$	$S^{k}$	$N^{v}$	$S^{v}$	Ratio	LongBench	NIAH	RULER	RULER_4K	RULER_8K	RULER_16K	RULER_32K
56	1024	16	128	83.6%	32.43	99.86	82.70	90.58	86.28	80.58	73.35
56	1024	16	512	82.8%	33.05	100.00	87.31	92.88	89.61	87.02	79.71
56	1024	16	2048	82.0%	33.40	100.00	87.49	93.96	89.55	87.02	79.42
56	1024	16	8192	81.3%	33.47	100.00	90.33	94.08	91.70	90.09	85.45
56	1024	16	32768	80.5%	33.47	100.00	90.38	94.27	91.56	90.31	85.39
56	1024	8	512	84.6%	31.39	95.94	61.16	74.43	65.60	56.85	47.78

Appendix

Table 4: Detailed results of different combinations of codebooks of VQKV on LLaMA3.1-8B [ Dubey et al., 2024 ] . Ratio means discarding ratios.

KV-Cache Compression

Key-value cache compression reduces the memory and bandwidth needed to retain attention states during long-context inference.

See all 256 papers

EchoKV: Efficient KV Cache Compression via Similarity-Based Reconstruction
Mar 24, 2026Shiyu Ji, Yixuan Wang, Yijun Liu +2KV-Cache Compression LLM Inference Acceleration
Research Center for Social Computing and Interactive Robotics, Harbin Institute of Technology, China
PuzzleKV: Page-Wise Low-Rank Decomposition for KV Cache Compression
Aug 24, 2026Zizhong Wang, Jieying Wang, Zhao Zhang +1KV-Cache Compression Low-Rank Compression
NestedKV: Nested Memory Routing for Long-Context KV Cache Compression
May 26, 2026Hong Chen, Xiang Liu, Yubo Gao +5KV Caching Long-Context Language Modeling
The Hong Kong University of Science and Technology (Guangzhou) · Jimei University

Mar 24, 2026J/K move · Enter open · S save

EchoKV: Efficient KV Cache Compression via Similarity-Based Reconstruction

Shiyu Ji, Yixuan Wang, Yijun Liu, Qingfu Zhu+1

Research Center for Social Computing and Interactive Robotics, Harbin Institute of Technology, China

KV-Cache Compression LLM Inference Acceleration

Open paper AskPDF arXiv

The increasing memory demand of the Key-Value (KV) cache poses a significant bottleneck for Large Language Models (LLMs) in long-context applications. Existing low-rank KV compression methods reduce this footprint by modifying model projections, limiting the flexibility to switch back to standard full-cache inference when sufficient memory is available. In this paper, we propose EchoKV, a flexible KV cache compression framework that supports on-demand transitions from full KV caching to compressed caching. Unlike traditional compression-decompression paradigms, EchoKV utilizes a lightweight network to reconstruct the discarded KV components from a partial subset, exploiting intrinsic inter-layer and intra-layer similarities among attention heads. We further introduce a lightweight two-stage fine-tuning strategy, requiring only a few minutes on a single A100 GPU for a 7B model. Experimental results on LongBench and RULER demonstrate that EchoKV consistently outperforms existing methods across multiple compression ratios and backbone models while preserving the throughput of full-cache inference in short-context scenarios.

Abstract

KV-Cache Compression

Explore similar work

EchoKV: Efficient KV Cache Compression via Similarity-Based Reconstruction

PuzzleKV: Page-Wise Low-Rank Decomposition for KV Cache Compression

NestedKV: Nested Memory Routing for Long-Context KV Cache Compression