KV-Cache Pruning

Ask about this topic

Overview

Key-value cache pruning studies how to shrink the attention states that language and multimodal models retain during generation, enabling longer contexts and more efficient serving. Methods score tokens using signals such as attention and recency, predicted future utility, task or reasoning-step relevance, and uncertainty; others allocate budgets across layers or regions, quantize entries, or combine pruning with specialized hardware. Work spans long-context retrieval and reasoning, multi-turn agents, GUI tasks, and video generation, with both training-free policies and learned pruning mechanisms. A central tension is that useful importance signals and cache budgets vary by task, while aggressive compression must also deliver real gains in latency and memory traffic—not just a smaller cache—without losing answer-critical evidence.

Latest papers

CardsList

36 papers7 papers in the last four weeks, up 133% on the four weeks before. 0.0% of all new papers.

  1. Evaluating the Applicability of LLM Inference Optimizations on Apple Silicon

    Oct 22, 2026Kyrylo Yemets, Andrii Yaroshevych, Victor Muryn +5LLM InferenceOn-Device Language Model Inference

    MacPaw AI&Research, Kyiv, Ukraine · Department of Artificial Intelligence, Lviv Polytechnic National University, Lviv, Ukraine · Institute of Mathematics of NAS of Ukraine, Kyiv, Ukraine

  2. The KV-Surgeon: Compensation-Aware Offline KV-Cache Compaction

    Oct 19, 2026Moritz Wagner, Christophe Roux, Max Zimmer +1KV-Cache ManagementKV-Cache Pruning

    Department for AI in Society, Science, and Technology, Zuse Institute Berlin, Germany · Institute of Mathematics, Technische Universität Berlin, Germany

  3. Rehearse Everything, Remember Nothing: Attic-KV Rehearses What Will Be Read

    Oct 8, 2026Zhiyun ShiKV-Cache CompressionKV-Cache Pruning

    Nanyang Technological University Singapore

  4. Dual-QK: Sharp Queries and Flat Keys for Prunable 2-bit KV Caches

    Oct 7, 2026Sunjoo Whang, Jungjun Oh, Minsung Kim +5KV-Cache CompressionKV-Cache Quantization

    KAIST · GIST · Google Research

  5. A Self-Pruning Transformer: Extreme KV-Cache Compression with Universal Attention

    Oct 6, 2026Davis Wertheimer, Haochen Shen, Ahan Gupta +6KV-Cache CompressionTransformer Attention

    IBM Research, USA · SSAIL Lab, University of Illinois Urbana-Champaign

  6. EchoPress: Query-Agnostic KV Cache Pruning via Virtual Context Reconstruction

    Sep 30, 2026Jiawei Lin, Saibo Geng, Thomas BourgeatKV-Cache ManagementLLM Inference Acceleration

    EPFL

  7. DeCoPrune: Efficient KV-Cache Pruning for Autoregressive Video Diffusion via Denoising Consistency

    Sep 30, 2026Zeqi Xiao, Qingle Liu, Kaiwen Zhang +3KV-Cache ManagementAutoregressive Video Generation

    Nanyang Technological University · Tsinghua University · Princeton University

  8. ActKV: Efficient LLM Agents through Action-Guided KV Cache Management

    Sep 25, 2026Zihan Wang, Cheng Tang, Lei Gong +4KV-Cache ManagementKV-Cache Compression

    University of Science and Technology of China Hefei, China · Suzhou Institute for Advanced Research, University of Science and Technology of China Suzhou, China

  9. Beyond Mean Attention: Diversity-Aware, Layer-Wise Scoring for KV Cache Eviction

    Sep 25, 2026Tianfang Xie, Wei ZhuKV-Cache EvictionKV-Cache Pruning

    Georgia Institute of Technology Atlanta, GA 30332, USA · Zhangjiang Lab Shanghai, China

  10. Pixel Decodability Is Not a Compression Signal: Causally Evaluating Importance Proxies for Visual KV-Cache Eviction

    Sep 14, 2026Chenyu Zhou, Qiliang Jiang, Shuning Wu +1KV CachingKV-Cache Eviction

    School of Engineering, Institute of Science Tokyo, Japan · College of Control Science and Engineering, Zhejiang University, China · Department of Electrical and Computer Engineering, National University of Singapore

  11. Small Frequency Corrections Can Change What Survives KV Cache Compression

    Aug 27, 2026Hong Chen, Yudong Zeng, Yongwei Huang +5KV CachingKV-Cache Eviction

    The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · Bosum Institute of Management Science, Shenzhen, China

  12. StepKV: Step-Aware KV Cache Compression for LLM Agents

    Aug 26, 2026Boyu Feng, Jiahong Liu, Yifan Li +7KV CachingKV-Cache Compression

    The Chinese University of Hong Kong · Huawei Technologies Co., Ltd

  13. Beyond Sparse Weights: When Is Attention Compressible?

    Aug 21, 2026Chiwun Yang, Xiaoyu LiKV-Cache CompressionMemory Compression

    City University of Hong Kong · University of New South Wales

  14. CommitKV: Lifecycle-Aware KV Cache Compression via Commit Transitions for Multi-Turn Agents

    Aug 8, 2026Weizhong Huang, Jinchao Zhang, Xiawu ZhengKV CachingLLM Agent Memory

    Xiamen University · WeChat AI, Tencent Inc., China

  15. QEvict: Recoverable Quantized KV Eviction for Attention-Drift-Robust Long-Context Decoding

    Aug 5, 2026Ayushman Garg, Akshita Gupta, Shaswata Bhattacharya +3KV CachingLong-Context Language Model Inference

    Mehta Family School of Data Science and Artificial Intelligence, Indian Institute of Technology Roorkee · Yardi School of Artificial Intelligence, Indian Institute of Technology Delhi · Department of Electrical Engineering, Indian Institute of Technology Delhi

  16. TaskPress: Query-Agnostic KV Cache Compression via Task-Guided Pruning

    Aug 4, 2026Wonpyo Park, Seung-won HwangLLM PruningKV Caching

    Seoul National University · Google

  17. Back from the Future: Key-Value Cache Management by Counter-Causal Surprise

    Jul 30, 2026Stephen Gould, Anton van den HengelKV CachingKV-Cache Eviction

    Metacognition AI · Australian National University · Adelaide University

  18. HiKV: Hierarchical Importance-Aware KV Cache with Hardware Acceleration for LLM Decoding

    Jul 24, 2026Chao Fang, Jun Yin, Man Shi +1KV CachingMemory-Efficient Optimization

    ESAT-MICAS, KU Leuven, Leuven, Belgium · UC Berkley, CA, USA

  19. Adaptive Filtering of the KV Cache: Diagnosing and Correcting Structural-Role Bias in LLM Inference

    Jul 14, 2026Soumil MandalKV CachingLong-Context Language Model Inference

    ServiceNow

  20. KVpop -- Key-Value Cache Compression with Predictive Online Pruning

    Jul 6, 2026Lukas Hauzenberger, Niklas Schmidinger, Anamaria-Roberta Hartl +5KV CachingKV-Cache Eviction

    1NXAI · 2Johannes Kepler University Linz, Austria

  21. Precomputing the Future-Offset Average in TriAttention

    Jul 1, 2026Amarnath MukherjeeLLM Inference EfficiencyKV Caching

    Hozhoke, Inc.

  22. EpiKV: Epiphany-Aware KV Cache Eviction Without the Attention Matrix

    Jun 25, 2026Steven Kolawole, Virginia SmithKV CachingKV-Cache Eviction

    Language Technologies Institute Carnegie Mellon University · Machine Learning Department Carnegie Mellon University

  23. IntentKV: Cross-Turn Intent-Aware KV Cache Pruning for Agent Inference

    Jun 6, 2026Junjie Li, Jiong Lou, Jie LiKV CachingKV-Cache Management

    Shanghai Jiao Tong University

  24. CONF-KV: Confidence-Aware KV Cache Eviction with Mixed-Precision Storage for Long-Horizon LLM

    May 24, 2026Yubo Li, Yidi MiaoKV CachingLong-Context Language Model Inference

    Carnegie Mellon University

  25. Adaptive Mass-Segmented KV Compression for Long-Context Reasoning

    May 22, 2026Junzhe Yang, Xiaoyu ShenKV CachingLong-Context Language Model Inference

    Shanghai Jiao Tong University · Institute of Digital Twin, Eastern Institute of Technology, Ningbo

  26. HeatKV: Head-tuned KV-cache Compression for Visual Autoregressive Modeling

    May 14, 2026Jonathan Cederlund, Axel Berg, William Isaksson +3KV CachingVisual Autoregressive Models

    Dept. of Automatic Control, Lund University · Arm

  27. Self-Pruned Key-Value Attention: Learning When to Write by Predicting Future Utility

    May 13, 2026Gergely Szilvasy, Manuel Faysse, Maria Lomeli +5Self-AttentionKV Caching

    Meta FAIR · MICS, CentraleSupélec

  28. Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction

    May 10, 2026Ngoc Bui, Hieu Trung Nguyen, Arman Cohan +1Self-AttentionKV Caching

    Department of Computer Science, Yale University · The Chinese University of Hong Kong

  29. ProxyKV: Cross-Model Proxy Pruning for Efficient Long-Context LLM Inference

    May 9, 2026Junjie Li, Jiong Lou, Jie LiLLM PruningKV Caching

    Shanghai Jiao Tong University

  30. When Does Value-Aware KV Eviction Help? A Fixed-Contract Diagnostic for Non-Monotone Cache Compression

    May 7, 2026Ruijie Zhang, Haozhe Liang, Da Chang +4KV CachingLong-Context Language Model Inference