KV-Cache Compression

Ask about this topic

Overview

Key-value cache compression reduces the memory and bandwidth cost of storing attention states, enabling longer contexts, larger batches, and inference on constrained hardware. Common approaches include token eviction and merging, quantization, low-rank or latent representations, and adaptive per-layer or per-head budgets; some methods also reconstruct or retrieve compressed states on demand. Work is evaluated on long-context retrieval and question answering benchmarks such as LongBench, RULER, and Needle-in-a-Haystack, as well as reasoning, multimodal tasks, and end-to-end serving. The central tension is that aggressive compression can harm accuracy, evidence use, or safety, while a high compression ratio does not guarantee faster inference or lower peak memory, so workload-aware evaluation remains essential.

Latest papers

CardsList

269 papers49 papers in the last four weeks, up 32% on the four weeks before. 0.3% of all new papers.

  1. SplitZip: Ultra Fast Lossless KV Compression for Disaggregated LLM Serving

    May 3, 2026Yipin Guo, Siddharth JoshiDisaggregated LLM ServingKV-Cache Compression

    University of Notre Dame IN, USA

  2. Make Your LVLM KV Cache More Lightweight

    May 1, 2026Xihao Chen, Yangyang Guo, Roger ZimmermannKV CachingEfficient VLM Inference

    Integrative Sciences and Engineering Programme, National University of Singapore · School of Computing, National University of Singapore

  3. KARA: Efficient Reasoning LLM Serving via Sliding-Window KV Cache Compression

    May 1, 2026Shen Han, Yuyang Wu, Junpu Yu +1KV CachingMemory-Efficient Inference

    Carnegie Mellon University

  4. Statistical Inference and Quality Measures of KV Cache Quantisations Inspired by TurboQuant

    Apr 27, 2026Paolo D'AlbertoKV CachingRate-Distortion Optimization

    AMD

  5. PolyKV: A Shared Asymmetrically-Compressed KV Cache Pool for Multi-Agent LLM Inference

    Apr 27, 2026Ishan Patel, Ishan JoshiLLM Inference EfficiencyMulti-Agent LLM Systems

    Independent Researcher

  6. Sub-Token Routing for KV Cache Compression

    Apr 23, 2026Wei Jiang, Wei WangLLM Inference EfficiencyKV Caching

    Futurewei Technologies San Jose, CA 95131, USA

  7. Sparse Forcing: Native Trainable Sparse Attention for Real-time Autoregressive Diffusion Video Generation

    Apr 23, 2026Boxun Xu, Yuming Du, Zichang Liu +7Autoregressive DiffusionSelf-Attention

    Meta Superintelligence Labs · University of California, Santa Barbara · Work done at Meta

  8. LKV: End-to-End Learning of Head-wise Budgets and Token Selection for LLM KV Cache Eviction

    Apr 22, 2026Enshuai Zhou, Yifan Hao, Chao Wang +7KV CachingLong-Context Language Model Inference

    University of Science and Technology of China, Hefei, China · State Key Lab of Processors, Institute of Computing Technology, CAS, Beijing, China · University of Chinese Academy of Sciences, Beijing, China

  9. DASH-KV: Accelerating Long-Context LLM Inference via Asymmetric KV Cache Hashing

    Apr 21, 2026Jinyu Guo, Zhihan Zhang, Jiehui Xie +7Long-Context Language Model InferenceLLM Inference Acceleration

    School of Information and Software Engineering, University of Electronic Science and Technology of China · School of Computer Science and Engineering, University of Electronic Science and Technology of China · Bangladesh University of Engineering and Technology +2

  10. Neural Garbage Collection: Learning to Forget while Learning to Reason

    Apr 20, 2026Michael Y. Li, Jubayer Ibn Hamid, Emily B. Fox +1LLM CompressionMemory-Efficient Optimization

    Stanford University

  11. How Much Cache Does Reasoning Need? Depth-Cache Tradeoffs in KV-Compressed Transformers

    Apr 20, 2026Xiao WangTransformer InferenceKV Caching

  12. MoE-nD: Per-Layer Mixture-of-Experts Routing for Multi-Axis KV Cache Compression

    Apr 20, 2026Libo Sun, Peixiong He, Po-Wei Harn +1LLM InferenceKV Caching

    Department of Computer Science and Software Engineering, Auburn University, Auburn, AL, USA · Department of Information Management, National Central University, Taoyuan, Taiwan

  13. Reducing Peak Memory Usage for Modern Multimodal Large Language Model Pipelines

    Apr 17, 2026Junwan Kim, Hyunkyung BaeEfficient Multimodal InferenceMultimodal Large Language Models

    New York University

  14. YOCO++: Enhancing YOCO with KV Residual Connections for Efficient LLM Inference

    Apr 15, 2026You Wu, Ziheng Chen, Yizhen Zhang +6LLM Inference AccelerationKV-Cache Compression

    ShanghaiTech University · Alibaba Group · Zhejiang University

  15. When Less Latent Leads to Better Relay: Information-Preserving Compression for Latent Multi-Agent LLM Collaboration

    Apr 14, 2026Yiping Li, Zhiyu An, Wan DuLLM CompressionKV Caching

    Department of Computer Science and Engineering, University of California, Merced, Merced, CA 95343

  16. Latent-Condensed Transformer for Efficient Long Context Modeling

    Apr 14, 2026Zeng You, Yaofo Chen, Qiuwu Chen +5Long-Context Language ModelingLLM Inference Acceleration

    South China University of Technology · Pengcheng Laboratory · AIGCode +2

  17. Attention-Weighted Value Projection for KV-Cache Compression

    Apr 13, 2026Samuel SalfatiGrouped-Query AttentionKV Caching

    fraQtl AI Research

  18. Transactional Attention: Semantic Sponsorship for KV-Cache Retention

    Apr 13, 2026Abhinaba BasuKV-Cache ManagementKV-Cache Eviction

    National Institute of Electronics and Information Technology (NIELIT) · Indian Institute of Information Technology, Allahabad (IIITA)

  19. CASK: Core-Aware Selective KV Compression for Reasoning Traces

    Apr 13, 2026Buseong Kim, Heejun GwonKV-Cache CompressionMemory-Efficient Inference

    d’strict Korea

  20. ZoomR: Memory Efficient Reasoning through Multi-Granularity Key Value Retrieval

    Apr 13, 2026David H. Yang, Yuxuan Zhu, Mohammad Mohammadi Amiri +4Memory-Efficient InferenceEfficient Language Model Reasoning

    Rensselaer Polytechnic Institute · IBM Research

  21. CodeComp: Structural KV Cache Compression for Agentic Coding

    Apr 11, 2026Qiujiang Chen, Jing Xiong, Chenyang Zhao +2KV-Cache CompressionSoftware Engineering

    The University of Hong Kong · LMSYS Org

  22. Sequential KV Cache Compression via Probabilistic Language Tries: Beyond the Per-Vector Shannon Limit

    Apr 10, 2026Gregory MagarshakKV-Cache CompressionMemory Compression

  23. KV Cache Offloading for Context-Intensive Tasks

    Apr 9, 2026Andrey Bocharnikov, Ivan Ermakov, Denis Kuznedelev +2KV-Cache OffloadingKV Caching

    HSE, Yandex · Yandex · Yandex, NSU

  24. StructKV: Preserving the Structural Skeleton for Scalable Long-Context Inference

    Apr 8, 2026Zhirui Chen, Peiyang Liu, Ling ShaoKV-Cache CompressionLong-Context Language Model Inference

    UCAS-Terminus AI Lab, University of Chinese Academy of Sciences, China · National Engineering Research Center for Software Engineering, Peking University

  25. HybridKV: Hybrid KV Cache Compression for Efficient Multimodal Large Language Model Inference

    Apr 7, 2026Bowen Zeng, Feiyang Ren, Jun Zhang +4KV-Cache CompressionEfficient Multimodal Inference

    The State Key Laboratory of Blockchain and Data Security, Zhejiang University · Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security · Hangzhou Dianzi University, Hangzhou, China

  26. TriAttention: Efficient Long Reasoning with Trigonometric KV Compression

    Apr 6, 2026Weian Mao, Xi Lin, Wei Huang +5Long-Context Reasoning in Language ModelsKV-Cache Compression

    MIT · ZJU · NVIDIA

  27. eOptShrinkQ: Near-Lossless KV Cache Compression Through Optimal Spectral Denoising and Quantization

    Apr 6, 2026Pei-Chun SuKV-Cache CompressionKV-Cache Quantization

    Department of Applied and Computational Mathematics, Yale University, New Haven, CT, USA

  28. IsoQuant: Hardware-Aligned SO(4) Isoclinic Rotations for LLM KV Cache Compression

    Mar 30, 2026Zhongping JiRotation-Based QuantizationKV-Cache Compression

  29. KVSculpt: KV Cache Compression as Distillation

    Mar 29, 2026Bo Jiang, Sian JinKV-Cache ManagementKV-Cache Compression

    Temple University

  30. KV Cache Quantization for Self-Forcing Video Generation: A 33-Method Empirical Study

    Mar 29, 2026Suraj Ranganath, Vaishak Menon, Anish PatnaikKV-Cache QuantizationKV-Cache Compression

    University of California, San Diego