KVSlimmer: Theoretical Insights and Practical Optimizations for Asymmetric KV Merging

Lianjun Liu; Hongli An; Weiqi Yan; Xin Du; Shengchuan Zhang; Huazhong Liu; Yunshan Zhong

cs.CLMar 1, 2026

KVSlimmer: Theoretical Insights and Practical Optimizations for Asymmetric KV Merging

Authors: Lianjun Liu, Hongli An, Weiqi Yan, Xin Du, Shengchuan Zhang, Huazhong Liu, Yunshan Zhong

Organizations: School of Information and Communication Engineering, Hainan University, Haikou, China · School of Cyberspace Security, Hainan University, Haikou, China · MAC Lab, Department of Artificial Intelligence, School of Informatics, Xiamen University, Xiamen, China · School of Computer Science and Technology, Hainan University, Haikou, China

Abstract

The growing computational and memory demands of the Key-Value (KV) cache significantly limit the ability of Large Language Models (LLMs). While KV merging has emerged as a promising solution, existing methods that rely on empirical observations of KV asymmetry and gradient-based Hessian approximations lack a theoretical foundation and incur suboptimal compression and inference overhead. To bridge these gaps, we establish a theoretical framework that characterizes this asymmetry through the spectral energy distribution of projection weights, demonstrating that concentrated spectra in Query/Key weights induce feature homogeneity, whereas dispersed spectra in Value weights preserve heterogeneity. Then, we introduce KVSlimmer, an efficient algorithm that captures exact Hessian information through a mathematically exact formulation, and derives a closed-form solution utilizing only forward-pass variables, resulting in a gradient-free approach that is both memory- and time-efficient. Extensive experiments across various models and benchmarks demonstrate that KVSlimmer consistently outperforms SOTA methods. For instance, on Llama3.1-8B-Instruct, it improves the LongBench average score by 0.92 while reducing memory costs and latency by 29% and 28%, respectively.Code is available at https://github.com/lianjunl13-sudo/KVSlimmer.

Figures & tables

Figure 1: Comparison between AsymKV and KVSlimmer for KV cache merging.

Figure 2: Layer-wise QKV similarity and spectral analysis. Left column: Mean adjacent-token cosine similarity for Query (Q), Key (K), and Value (V), averaged over attention heads. Middle column: Eigenvalue distributions of the projection matrices $\mathbf{W}_{Q}$ , $\mathbf{W}_{K}$ , and $\mathbf{W}_{V}$ , sorted in descending order. Right column: Mode-wise contribution coefficients $c_{i}$ (Eq. 8 ), plotted according to the eigenvalue index. The first two rows show results from Llama-3.1-8B-Instruct , while the last two rows show results from Mistral-7B-Instruct-v0.3 .

Panel of Figure 3 — Figure 3: Head-level mean alignment relationships at Layer 2 of Llama-3.1-8B-Instruct on 2WikiMQA. Each point corresponds to one attention head, positioned by its global mean cosine alignment.

	Single-Doc	Multi-Doc	Sum	Few-shot	Synthetic	Code	Avg.
Llama3.1-8B-Instruct
Full Context	43.73	44.49	29.12	69.36	53.56	50.95	48.07
StreamingLLM	28.15	27.19	25.15	63.17	16.33	52.15	35.50
LongCache	28.98	27.84	25.35	64.73	19.68	51.61	36.46
H 2 O	33.30	34.43	26.60	66.23	14.75	52.65	38.53
LLMLingua-2	32.02	32.24	24.99	27.87	17.67	50.07	30.43

Table 1: Performance on LongBench. KVSlimmer outperforms its baselines on most settings.

Model	Overall	Easy	Hard	Short	Medium	Long
Full Context	30.02	30.73	29.58	35.00	27.91	25.93
StreamingLLM	27.04	27.60	26.69	32.78	23.26	25.00
LongCache	28.43	28.13	28.62	32.78	25.58	26.85
H 2 O	28.23	28.12	28.29	31.67	26.98	25.00
CaM	28.23	28.64	27.97	31.67	26.98	25.00
AsymKV	30.02	30.23	29.90	32.78	27.44	28.85

Table 2: Performance on LongBenchV2.

Figure 4: Relative runtime of KVSlimmer compared to AsymKV across LongBench datasets.

Figure 5: Inference efficiency of decoder stage.

Figure 6: Peak GPU memory of KVSlimmer and AsymKV across different chunk sizes.

Panel of Figure 7 — Figure 7: Results on different compression ratios.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Layer-wise QKV similarity and spectral analysis for Llama-3.1-8B-Instruct . Left column: Mean adjacent-token cosine similarity for Query (Q), Key (K), and Value (V), averaged over attention heads. Middle column: Eigenvalue distributions of the projection matrices $\mathbf{W}_{Q}$ , $\mathbf{W}_{K}$ , and $\mathbf{W}_{V}$ , sorted in descending order. Right column: Mode-wise contribution coefficients $c_{i}$ (Eq. 8 ), plotted according to the eigenvalue index.

Figure 9: Layer-wise QKV similarity and spectral analysis for Mistral-7B-Instruct-v0.3 . Left column: Mean adjacent-token cosine similarity for Query (Q), Key (K), and Value (V), averaged over attention heads. Middle column: Eigenvalue distributions of the projection matrices $\mathbf{W}_{Q}$ , $\mathbf{W}_{K}$ , and $\mathbf{W}_{V}$ , sorted in descending order. Right column: Mode-wise contribution coefficients $c_{i}$ (Eq. 8 ), plotted according to the eigenvalue index.

Panel of Figure 10 — Figure 10: Head-level mean alignment relationships at Layer 5 of Llama-3.1-8B-Instruct on 2WikiMQA. Each point corresponds to one attention head, positioned by its global mean cosine alignment.

Panel of Figure 11 — Figure 11: Head-level mean alignment relationships at Layer 22 of Llama-3.1-8B-Instruct on 2WikiMQA. Each point corresponds to one attention head, positioned by its global mean cosine alignment.

Panel of Figure 12 — Figure 12: Head-level mean alignment relationships at Layer 9 of Mistral-7B-Instruct-v0.3 on 2WikiMQA. Each point corresponds to one attention head, positioned by its global mean cosine alignment.

Panel of Figure 13 — Figure 13: Head-level mean alignment relationships at Layer 15 of Mistral-7B-Instruct-v0.3 on 2WikiMQA. Each point corresponds to one attention head, positioned by its global mean cosine alignment.

Panel of Figure 14 — Figure 14: Head-level mean alignment relationships at Layer 20 of Mistral-7B-Instruct-v0.3 on 2WikiMQA. Each point corresponds to one attention head, positioned by its global mean cosine alignment.

KV-Cache Compression

Key-value cache compression reduces the memory and bandwidth needed to retain attention states during long-context inference.

See all 267 papers

GRKV: Global Regression for Training-Free KV Cache Compression in Long-Context LLMs
May 29, 2026Junjie Peng, You Wu, Haoyi Wu +4KV Caching Long-Context Language Modeling
School of Computer Science and Engineering, Sun Yat-sen University · School of Information Science and Technology, ShanghaiTech University · Shanghai Engineering Research Center of Intelligent Vision and Imaging +2
LKV: End-to-End Learning of Head-wise Budgets and Token Selection for LLM KV Cache Eviction
Apr 22, 2026Enshuai Zhou, Yifan Hao, Chao Wang +7KV Caching Long-Context Language Model Inference
University of Science and Technology of China, Hefei, China · State Key Lab of Processors, Institute of Computing Technology, CAS, Beijing, China · University of Chinese Academy of Sciences, Beijing, China
Fast KVzip: Efficient and Accurate LLM Inference with Gated KV Eviction
Jan 25, 2026Jang-Hyun Kim, Dongyoon Han, Sangdoo YunKV-Cache Eviction KV-Cache Compression
NAVER AI Lab.

May 29, 2026J/K move · Enter open · S save

GRKV: Global Regression for Training-Free KV Cache Compression in Long-Context LLMs

Junjie Peng, You Wu, Haoyi Wu, Jialong Han+3

School of Computer Science and Engineering, Sun Yat-sen University · School of Information Science and Technology, ShanghaiTech University · Shanghai Engineering Research Center of Intelligent Vision and Imaging+2

KV Caching Long-Context Language Modeling KV-Cache Compression Ridge Regression

Open paper AskPDF arXiv

Large language models (LLMs) with extended context lengths rely on the key-value (KV) cache to support attention over prior tokens. However, maintaining the KV cache incurs substantial memory overhead, motivating KV-cache compression methods that enforce a fixed budget through eviction and merging. Modern eviction methods increasingly adopt span-based retention because preserving contiguous spans is empirically effective and better preserves semantic coherence. Yet, when combined with post-eviction merging, span-based retention concentrates merges onto a small set of span-boundary carrier tokens, producing a highly imbalanced merge pattern that exacerbates over-merging and increases information loss. To address this imbalance, we propose GRKV (Global Regression for KV Cache), a training-free KV-cache merging method that directly minimizes the discrepancy between compressed-cache and full-cache attention outputs. GRKV uses ridge-regression-based merge steps to distribute information from evicted tokens across retained tokens, while regularizing the updates to prevent over-smoothing. Across the LongBench and RULER long-context benchmarks, GRKV is the only merging method that improves overall performance with minimal overhead. Our code is available at https://github.com/pjunjie/GRKV.

Abstract

KV-Cache Compression

Explore similar work

GRKV: Global Regression for Training-Free KV Cache Compression in Long-Context LLMs

LKV: End-to-End Learning of Head-wise Budgets and Token Selection for LLM KV Cache Eviction

Fast KVzip: Efficient and Accurate LLM Inference with Gated KV Eviction