KV-Cache Compression

Ask about this topic

Overview

Key-value cache compression reduces the memory and bandwidth cost of storing attention states, enabling longer contexts, larger batches, and inference on constrained hardware. Common approaches include token eviction and merging, quantization, low-rank or latent representations, and adaptive per-layer or per-head budgets; some methods also reconstruct or retrieve compressed states on demand. Work is evaluated on long-context retrieval and question answering benchmarks such as LongBench, RULER, and Needle-in-a-Haystack, as well as reasoning, multimodal tasks, and end-to-end serving. The central tension is that aggressive compression can harm accuracy, evidence use, or safety, while a high compression ratio does not guarantee faster inference or lower peak memory, so workload-aware evaluation remains essential.

Latest papers

CardsList

269 papers49 papers in the last four weeks, up 32% on the four weeks before. 0.3% of all new papers.

  1. DCKV: Learned Transform Coding of the Prompt KV Cache in Hybrid Language Models

    Oct 27, 2026Egor Maximov, Aleksei Merkelov, Vasiliy Alekseev +5KV-Cache CompressionKV-Cache Quantization

    MIL Team · DAIMLD

  2. Folding Attention Biases into Keys Makes Compacted KV Caches Run on Standard Kernels

    Oct 27, 2026Utkarsh KanwatKV-Cache CompressionMemory-Efficient Attention

    Independent Researcher

  3. BellowsKV: One Latent for Many Layers, Expanded on Demand

    Oct 26, 2026Jiamu Zhang, Hanjie ChenKV-Cache CompressionLLM Inference

    Rice University

  4. Attention under Bounded Key-Query Similarity: Space Complexity and Transformer Anisotropy

    Oct 26, 2026Justin Y. Chen, Ying Feng, Piotr Indyk +3Attention ApproximationKV-Cache Compression

    Massachusetts Institute of Technology (MIT). · École Polytechnique Fédérale de Lausanne (EPFL).

  5. Video-MemComp: Taylor-Guided Sensitivity-Aware Memory Management for Streaming Video Understanding

    Oct 23, 2026Yonghui Niu, Yuhao Ji, Chang Nie +6Streaming Video UnderstandingKV-Cache Compression

    NJU · Jiutian AI · CASIA

  6. Drift Says When, Budget Says How Much: Rate-Controlled KV Re-selection for Long-Form Multimodal Generation

    Oct 22, 2026Bin Hu, Yingbo XiaoMultimodal Large Language ModelsKV-Cache Compression

    Tsinghua University · The Chinese University of Hong Kong, Shenzhen

  7. RIFT: KV Cache Compression for Draft Models with Complementary Compressibility Training

    Oct 22, 2026Dor Tsur, Maor Ashkenazi, Maximilian Golub +7KV-Cache CompressionLLM Inference Acceleration

    NVIDIA

  8. SAKURA: Serving Attention Key-Value State Under Rate Adaptation for Private Image Editing on Phones

    Oct 21, 2026Shuncheng Jie, Desen Sun, Ri Hong +3Edge-Cloud Collaborative InferenceDiffusion Model Serving

    University of Waterloo Canada

  9. MoEZip: Routing-Aware KV Cache Compression for Sparse Mixture-of-Experts LLMs

    Oct 21, 2026Minsang Kim, Seungjun BaekKV-Cache CompressionLong-Context Language Model Inference

    Korea University · SK Telecom

  10. RACE: RoPE-Aligned Attention Compression Based on Query-Key Activation Energies

    Oct 20, 2026Irene Tenison, Abdulrahman Alabdulkareem, Daniel Karl I. Weidele +2KV-Cache CompressionTransformer Attention

    MIT CSAIL · MIT-IBM Computing Research Lab

  11. Offline Makes Online Better: Calibrated Cache Compression for Video Generation

    Oct 20, 2026Zhen Wang, Ning Lai, Qingtao YuKV-Cache CompressionEfficient Video Generation

    Homenet Research · CarryAI · Australian National University

  12. TideKV: Delayed KV Eviction via Layer-Wise Chunked Prefill for Memory-Efficient Long-Context LLM Inference

    Oct 17, 2026Yu Wang, Zhu Wang, Ning Hu +4Memory-Efficient InferenceLong-Context Language Model Inference

    Northwestern Polytechnical University, No. 127 West Youyi Road, Xian, 710129, Shaanxi, China · Xi’an Aeronautics Computing Technique Research Institute, AVIC, No. 15 Jinye 2nd Road, Xi’an, 710065, Shaanxi, China · College of Computer Science and Technology, Harbin Engineering University, No. 145 Nantong Street, Harbin, 150001, Heilongjiang, China

  13. Compression Does Not End at Selection: Post-Selection Positional Geometry for Sparse KV Caches

    Oct 15, 2026Wanyang Hong, Zhaoning ZhangKV-Cache ManagementKV-Cache Compression

    National Key Laboratory of Parallel and Distributed Computing College of Computer Science and Technology National University of Defense Technology, Changsha, China.

  14. Latent Compaction with Test-Time Training

    Oct 14, 2026Yingfa Chen, Xingyu Shen, Chaojun Xiao +1LLM Context CompressionKV-Cache Compression

    NLP Group, DCST, IAI, BNRIST, Tsinghua University, Beijing, China

  15. Kwai Summary Attention V2 Technical Report

    Oct 13, 2026OneRec Team, Chenglong Chu, Guohong Mu +30KV-Cache CompressionHybrid Attention

  16. ChampionKV: Head-Wise Adaptive Budget Allocation for KV Cache Compression

    Oct 13, 2026Yuechuan Chen, Yanda Guo, Xiaoxi Liu +4KV-Cache CompressionKV-Cache Management

    Sun Yat-sen University · The Chinese University of Hong Kong · University of Electronic Science and Technology of China +2

  17. AlignPred: Error-Controlled KV Cache Compression via Alignment and Predictive Coding

    Oct 13, 2026Xu Yang, Jiapeng Zhang, Zhiwei Zuo +5KV-Cache CompressionMemory Compression

    Hunan University · Central China Normal University · Tencent

  18. BELT : BEhavior-Aligned Low-Rank KV Cache Compression with Learned Top-k Rank Allocation

    Oct 13, 2026Yaoyang Liu, Shuwei Shen, Jincheng Tang +5KV-Cache CompressionLow-Rank Compression

    Peking University · Tencent · BeiHang University

  19. Virtual Head Attention

    Oct 13, 2026Xibo Ding, Guoxia Wang, Jinle Zeng +4Long-Context Language ModelingGrouped-Query Attention

    Baidu Inc. · Peking University

  20. MLAfication: Efficiently Enabling High-Fidelity MLA for Modern LLMs

    Oct 12, 2026Zile Wang, Zhide Xie, Tao JiMulti-Head Latent AttentionLLM Inference Acceleration

    Fudan University

  21. E8 Lattice Key Quantization for Low-Bit KV-Cache Compression

    Oct 11, 2026Sasho Nedelkoski, Alexander Acker, Soeren Becker +2KV-Cache CompressionKV-Cache Quantization

    Technical University of Berlin Berlin, Germany · logsight.ai Berlin, Germany

  22. GateSight: Recurrent Memory Anticipates Attention Memory for Query-Agnostic KV Eviction in Hybrid LLMs

    Oct 10, 2026Junjie Li, Jiong Lou, Jie LiKV-Cache EvictionKV-Cache Compression

    Shanghai Jiao Tong University

  23. FrameFold: Response-Guided KV Cache Consolidation for Autoregressive Long Video Generation

    Oct 10, 2026Muyu Liu, Mingming Liu, Hong Gu +2Autoregressive Video GenerationLong Video Generation

    ShanghaiTech University · vivo BlueImage Lab, vivo Mobile Communication Co., Ltd., China

  24. EdgeLLM-Nano: Auditing Polar-Guided Key-Cache Compression and Consistency-Anchored Pruning on a 33.6 MB On-Device Language Model

    Oct 10, 2026S M Mahbub Murshed, Hayden Bright, Rezaul Karim +1KV-Cache CompressionKV-Cache Quantization

    Causal Dynamics Lab · Electrical Engineering and Computer Science Department, University of Missouri-Columbia · Computer Science & Engineering, University of Chittagong

  25. Kronecker Product Attention: Memory-Efficient KV Compression via Structured Factorization

    Oct 8, 2026Bahareh Nikpour, Hossein Rajabzadeh, Maryam Dialameh +2Transformer AttentionKV-Cache Compression

    Huawei Technologies, Toronto, Canada · University of Waterloo, Waterloo, Canada · Huawei Noah’s Ark Lab

  26. VFold: Symmetry-Aware Cross-Layer Value Cache Compression

    Oct 8, 2026Neha Verma, Sungwon Kim, Kenton Murray +1KV-Cache Compression

    Johns Hopkins University · George Mason University

  27. Rehearse Everything, Remember Nothing: Attic-KV Rehearses What Will Be Read

    Oct 8, 2026Zhiyun ShiKV-Cache CompressionKV-Cache Pruning

    Nanyang Technological University Singapore

  28. Compile the Table: Query-Calibrated Operator Compression for Tabular In-Context Learning

    Oct 8, 2026Xu Zhao, Jiaming Zhao, Bin Zhao +1Context CompressionTabular ICL

  29. Read What Matters: Query-Adaptive Quantization for KV Caches

    Oct 8, 2026Siddharth Bhandari, Lucas Gretta, Krishna Balasubramanian +1KV-Cache QuantizationKV-Cache Compression

    Amazon · University of California, Berkeley

  30. CORAL-KV: Compact Cross-Layer Prediction and Residual Coding for Autoregressive Video KV Caches

    Oct 7, 2026Albert Ge, Akash Haridas, Utkarsh Saxena +5KV-Cache CompressionKV-Cache Quantization

    Advanced Micro Devices Inc. (AMD) · University of Wisconsin-Madison