EchoKV: Efficient KV Cache Compression via Similarity-Based Reconstruction

Shiyu Ji; Yixuan Wang; Yijun Liu; Qingfu Zhu; Wanxiang Che

cs.CLMar 24, 2026

EchoKV: Efficient KV Cache Compression via Similarity-Based Reconstruction

Authors: Shiyu Ji, Yixuan Wang, Yijun Liu, Qingfu Zhu, Wanxiang Che

Organizations: Research Center for Social Computing and Interactive Robotics, Harbin Institute of Technology, China

Abstract

The increasing memory demand of the Key-Value (KV) cache poses a significant bottleneck for Large Language Models (LLMs) in long-context applications. Existing low-rank KV compression methods reduce this footprint by modifying model projections, limiting the flexibility to switch back to standard full-cache inference when sufficient memory is available. In this paper, we propose EchoKV, a flexible KV cache compression framework that supports on-demand transitions from full KV caching to compressed caching. Unlike traditional compression-decompression paradigms, EchoKV utilizes a lightweight network to reconstruct the discarded KV components from a partial subset, exploiting intrinsic inter-layer and intra-layer similarities among attention heads. We further introduce a lightweight two-stage fine-tuning strategy, requiring only a few minutes on a single A100 GPU for a 7B model. Experimental results on LongBench and RULER demonstrate that EchoKV consistently outperforms existing methods across multiple compression ratios and backbone models while preserving the throughput of full-cache inference in short-context scenarios.

Figures & tables

Figure 1: Illustration of the differences between existing low-rank sharing approaches and EchoKV. Unlike the compression-decompression paradigm, EchoKV employs a lightweight network to reconstruct the discarded KV components from retained ones.

Figure 2: Schematic illustration of the training and inference workflows for EchoKV compared to the standard KV cache. The figure presents a schematic illustration for a single token, where distinct cache blocks correspond to different segments of the flattened KV representation.

	Methods	Single-Document QA			Multi-Document QA			Summarization			Few-shot Learning			Synthetic		Code		Avg.
	Methods	Nrtv	Qasp	MF	Hotpot	2Wiki	Musiq	Gov	QMS	MN	TREC	TQA	SAM	PC	PR	LCC	RB	Avg.
Llama3.1-8B-Instruct	Full KV	31.38	46.60	56.85	58.10	50.01	32.52	34.66	25.19	27.11	73.00	92.11	43.70	6.56	100.0	65.52	54.75	49.88
	Compression Ratio = 0.7
	MiniCache	17.87	22.26	29.51	26.72	11.47	9.71	24.61	21.85	23.29	65.00	75.05	31.46	5.83	60.50	41.34	37.19	31.48
	ThinK	32.77	45.44	58.83	58.30	51.09	32.65	32.63	24.96	26.19	73.00	91.72	42.10	6.96	99.50	65.94	55.64	49.86
	Palu	28.96	46.73	53.26	53.08	45.35	29.91	32.51	24.69	25.42	73.00	89.10	43.77	2.50	96.50	57.90	56.78	47.47

Table 1: Experimental results of different models and methods on LongBench. ∗ Since ThinK only compresses Keys, we double the compression rate calculation to account for Value storage. We report results at 0.6 because the 0.5 ratio is infeasible. Compression Ratio = Size of Compressed Cache / Size of Full Cache.

Llama3.1-8B-Instruct
Method	Needle In A Haystack (NIAH)								Synthetic			QA		Avg.
Method	S1	S2	S3	MK1	MK2	MK3	MV	MQ	VT	CWE	FWE	QA1	QA2
Full KV	100.0	100.0	100.0	99.00	99.00	99.00	98.75	98.50	99.20	17.80	87.00	87.00	54.00		87.63
ThinK*	1.00	2.00	0.00	2.00	0.00	0.00	1.75	3.25	0.60	0.00	48.00	63.00	40.00	12.43
Palu	100.0	98.00	75.00	97.00	78.00	30.00	73.00	64.25	86.00	15.30	78.33	53.00	38.00	68.14
CommonKV	92.00	94.00	83.00	97.00	67.00	1.00	87.00	94.25	79.80	0.10	85.67	75.00	50.00	69.68

Table 2: Experimental results of different methods on RULER at a compression ratio of 0.5. ∗ Indicates the actual compression ratio is 0.6, consistent with Table 1 .

Model	Ratio	Converted	FT	EchoKV
Llama-3.1-8B	0.5	27.65	36.89	48.53
Llama-3.1-8B	0.3	11.76	32.31	45.27
Mistral-7B	0.5	21.15	29.88	46.36
Mistral-7B	0.3	12.36	25.00	42.78

Table 3: LongBench average scores compared with TransMLA. “Converted” denotes direct architecture conversion, while “FT” denotes additional training on the same 1.6K Long Alpaca samples used by EchoKV.

Loss	Stage	Time	LB Avg.
KV-MSE	I	12.5 min	48.99
O-MSE	I	15.4 min	48.76
QK-KL	II	39.7 min	49.11
O-MSE	II	14.5 min	49.26

Table 4: Training time and performance across different loss functions.

Panel of Figure 3 — Figure 3: Analysis experiments on EchoKV. All evaluations are conducted using Llama3.1-8B-Instruct [ 31 ] on the LongBench [ 13 ] benchmark.

Methods	Single-Document QA			Multi-Document QA			Summarization			Few-shot Learning			Synthetic		Code
Methods	Nrtv	Qasp	MF	Hotpot	2Wiki	Musiq	Gov	QMS	MN	TREC	TQA	SAM	PC	PR	LCC	RB
Full KV	31.38	46.60	56.85	58.10	50.01	32.52	34.66	25.19	27.11	73.00	92.11	43.70	6.56	100.00	65.52	54.75
Local Input	31.88	46.45	57.32	58.02	49.49	32.68	30.80	24.98	26.16	73.00	92.50	42.45	6.88	98.50	64.72	55.53
Global Input	31.67	46.10	57.85	58.19	49.54	32.81	32.27	24.63	26.32	73.00	92.41	41.77	7.00	88.00	64.34	55.32
Combined Input	31.60	46.48	57.43	58.27	49.84	33.02	32.34	25.09	26.22	73.00	92.58	42.49	7.00	94.50	64.46	55.73

Table 5: Results of the ablation study on different input features. Evaluations are conducted on LongBench using Llama-3.1-8B-Instruct with a compression ratio of 0.75. Bolded entries represent datasets where the two features exhibit differences.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Target Ratio	Group Size ( $S$ )	Local Dim ( $D_{\text{local}}$ )	Predictor Input Dim	Predictor Output Dim	Total Params
0.7	2	384	1408	640	28.8M
0.5	2	0	1024	1024	33.6M
0.3	4	64	1088	960	50.2M

Appendix

Table 6: Hyperparameter configurations for EchoKV across different target compression ratios on Llama-3.1-8B-Instruct, Mistral-7B-Instruct-v0.3, and Mistral-Small-24B-Instruct-2501.

Stage 1
Configuration	Value
Optimizer	AdamW
Learning rate	$5\mathrm{e}{-4}$
Scheduler	Cosine Decay
Warmup steps	0
Batch size	1

Appendix

Table 7: Hyperparameter details of the EchoKV training process. In Stage 2, only the parameters that differ from those in Stage 1 are described.

Llama3.1-8B-Instruct
Method	Needle In A Haystack (NIAH)								Synthetic			QA		Avg.
Method	S1	S2	S3	MK1	MK2	MK3	MV	MQ	VT	CWE	FWE	QA1	QA2
Compression Ratio = 0.5
Full KV	100.0	100.0	100.0	99.00	99.00	99.00	98.75	98.50	99.20	17.80	87.00	87.00	54.00		87.63
ThinK*	1.00	2.00	0.00	2.00	0.00	0.00	1.75	3.25	0.60	0.00	48.00	63.00	40.00		12.43
Palu	100.0	98.00	75.00	97.00	78.00	30.00	73.00	64.25	86.00	15.30	78.33	53.00	38.00	68.14

Appendix

Table 8: Experimental results of different methods on RULER at compression ratios of 0.5 and 0.3. ∗ Indicates the actual compression ratio is 0.6.

Figure 4: LongBench average as a function of training data size on Llama-3.1-8B at a compression ratio of 0.3. EchoKV reaches strong performance with substantially fewer training samples than TransMLA.

	Methods	Single-Document QA			Multi-Document QA			Summarization			Few-shot Learning			Synthetic		Code		Avg.
	Methods	Nrtv	Qasp	MF	Hotpot	2Wiki	Musiq	Gov	QMS	MN	TREC	TQA	SAM	PC	PR	LCC	RB	Avg.
Llama3.1-8B-Instruct	Full KV	31.38	46.60	56.85	58.10	50.01	32.52	34.66	25.19	27.11	73.00	92.11	43.70	6.56	100.0	65.52	54.75	49.88
	Compression Ratio = 0.7
	MiniCache	17.87	22.26	29.51	26.72	11.47	9.71	24.61	21.85	23.29	65.00	75.05	31.46	5.83	60.50	41.34	37.19	31.48
	ThinK	32.77	45.44	58.83	58.30	51.09	32.65	32.63	24.96	26.19	73.00	91.72	42.10	6.96	99.50	65.94	55.64	49.86
	Palu	28.96	46.73	53.26	53.08	45.35	29.91	32.51	24.69	25.42	73.00	89.10	43.77	2.50	96.50	57.90	56.78	47.47

Appendix

Table 9: Experimental results on LongBench including the exploratory EchoKV-Hybrid variant. ∗ Since ThinK only compresses Keys, we double the compression rate calculation to account for Value storage. We report results at 0.6 because the 0.5 ratio is infeasible. Compression Ratio = Size of Compressed Cache / Size of Full Cache.

Figure 5: RULER average for different group sizes

S

and local dimensions

m

on Llama-3.1-8B-Instruct. The gray dashed line denotes the full-KV baseline, and each point is annotated with its retained local dimension

m

. Smaller

S

corresponds to a shorter prediction distance to the retained full layer.

Panel of Figure 6 — Figure 6: Visualization of NIAH results on Llama-3.1-8B-Instruct with a compression ratio of 0.3.

Panel of Figure 7 — Figure 7: Visualization of NIAH results on Mistral-7B-Instruct-v0.3 with a compression ratio of 0.3.

Ratio	Benchmark	Prefix	Random	Per-head	Oracle
0.7 $(S=2,m=384)$	LongBench	49.075	49.509	49.177	49.553
0.7 $(S=2,m=384)$	RULER	86.386	84.765	82.805	85.440
0.3 $(S=4,m=64)$	LongBench	45.267	45.916	46.038	45.205
0.3 $(S=4,m=64)$	RULER	60.302	48.446	43.125	60.159

Appendix

Table 10: Comparison of local-dimension selection strategies on Llama-3.1-8B-Instruct. Prefix denotes the default contiguous selection used by EchoKV.

Architecture	LongBench Avg.	RULER Avg.
Linear	48.53	83.52
MLP (param-matched)	48.80	82.70
MLP (hidden = input dim)	48.84	83.94

Appendix

Table 11: Comparison of prediction network architectures on Llama-3.1-8B-Instruct at a compression ratio of 0.5.

KV-Cache Compression

Key-value cache compression reduces the memory and bandwidth needed to retain attention states during long-context inference.

See all 257 papers

PuzzleKV: Page-Wise Low-Rank Decomposition for KV Cache Compression
Aug 24, 2026Zizhong Wang, Jieying Wang, Zhao Zhang +1KV-Cache Compression Low-Rank Compression
VQKV: High-Fidelity and High-Ratio Cache Compression via Vector-Quantization
Mar 17, 2026Yixuan Wang, Qingyu Shi, Jiayu Zhou +3KV-Cache Compression KV-Cache Quantization
LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University · Shanghai Innovation Institute · Shanghai Artificial Intelligence Laboratory +2
CompressKV: Semantic-Retrieval-Guided KV-Cache Compression for Resource-Efficient Long-Context LLM Inference
Jun 23, 2026Xiaolin Lin, Jingcun Wang, Olga Kondrateva +3LLM Inference Efficiency KV Caching
Technical University of Darmstadt, Darmstadt, Germany · University of Notre Dame, Notre Dame, IN, USA · Technical University of Ilmenau, Ilmenau, Germany

Aug 24, 2026J/K move · Enter open · S save

PuzzleKV: Page-Wise Low-Rank Decomposition for KV Cache Compression

Zizhong Wang, Jieying Wang, Zhao Zhang, Jiajia Li

KV-Cache Compression Low-Rank Compression Long-Context Language Model Inference

Open paper AskPDF arXiv

Long-context inference in large language models (LLMs) is increasingly limited by the memory required for the key-value (KV) cache. KV cache compression addresses this problem by reducing the storage cost of previous tokens. Among existing approaches, low-rank compression is particularly attractive because it represents every token in reduced dimensions. Previous low-rank methods typically derive fixed projection spaces from model weights, construct fixed spaces from calibration activations, or construct a shared basis over a broad cache region. Such representations may not capture detailed but important information. We partition each per-head KV cache into fixed-length logical pages and observe substantial low-rank structure within individual pages. Based on this observation, we propose PuzzleKV, a training- and calibration-free method that treats each completed page as an independent compression unit. PuzzleKV decomposes pages within each layer and KV head, computes attention directly over dense and factorized pages, and incrementally compresses newly eligible pages during autoregressive decoding. Experiments across models, context lengths, and benchmarks demonstrate the effectiveness of PuzzleKV under matched storage budgets. At approximately 60% of the original KV cache storage, PuzzleKV achieves more than 96% of Full KV performance across both evaluated models and all benchmark settings, with substantial gains over Global SVD on RULER and competitive performance on LongBench. To achieve a more aggressive compression ratio, PuzzleKV can be further combined with quantization while retaining more than 93% of Full KV performance using only 18.7% of the original storage.

Abstract

KV-Cache Compression

Explore similar work

PuzzleKV: Page-Wise Low-Rank Decomposition for KV Cache Compression

VQKV: High-Fidelity and High-Ratio Cache Compression via Vector-Quantization

CompressKV: Semantic-Retrieval-Guided KV-Cache Compression for Resource-Efficient Long-Context LLM Inference