KV-Cache Management

Ask about this topic

Overview

The field studies how to store, reuse, compress, and move the key-value states that Transformers accumulate during generation, which can dominate memory and data-transfer costs in long-context inference. Methods include token eviction and learned importance scoring, quantization and compact representations, prefix or cross-request sharing, and offloading across GPU, host memory, and storage; serving systems also coordinate these choices with scheduling and batching. Evaluations commonly measure quality on long-context retrieval and reasoning benchmarks such as LongBench and RULER alongside memory use, time to first token, and throughput. The central tension is reducing cache cost without losing information needed later: results depend on the task, workload, hardware, and attention implementation, so robust policies and faithful end-to-end evaluation remain important challenges.

Latest papers

CardsList

201 papers36 papers in the last four weeks, up 13% on the four weeks before. 0.2% of all new papers.

  1. Leyline: KV Cache Directives for Agentic Inference

    May 31, 2026Bole Ma, Jan Eitzinger, Harald KoestlerLLM Inference EfficiencyKV Caching

    Erlangen National High Performance Computing Center, Erlangen, Germany

  2. BudgetDraft: Acceptance-Aware Multi-View Training for Sparse-KV Speculative Decoding

    May 29, 2026Liang He, Jingbo Wen, Qishi Zhan +4KV CachingLong-Context Language Model Inference

    Shanghai Institute of Optics and Fine Mechanics · The University of Sydney · Marquette University +4

  3. Probing the Prompt KV Cache: Where It Becomes Dispensable

    May 28, 2026Vinayshekhar Bannihatti Kumar, Manoj Ghuhan Arivazhagan, Disha Makhija +1KV CachingKV-Cache Management

    AWS AI Labs

  4. A Policy-Driven Runtime Layer for Agentic LLM Serving

    May 26, 2026Rui Zhang, Chaeeun Kim, Liting HuLLM Inference EfficiencyMulti-Agent LLM Systems

    University of California, Santa Cruz

  5. Enabling KV Caching of Shared Prefix for Diffusion Language Models

    May 26, 2026Younghun Go, Jaehoon Han, Changyong Shin +2KV CachingKV-Cache Management

    Korea University, South Korea

  6. Meta-Soft: Leveraging Composable Meta-Tokens for Context-Preserving KV Cache Compression

    May 21, 2026Wei Luo, Yi Huang, Songchen Ma +3KV CachingKV-Cache Eviction

    Guangdong Institute of Intelligence Science and Technology · University of Macau · Chinese University of Hong Kong, Shenzhen +1

  7. ArborKV: Structure-Aware KV Cache Management for Scaling Tree-based LLM Reasoning

    May 21, 2026Yeqiu Chen, Ziyan Liu, Zhenxin Huang +3Inference-Time SearchTree of Thoughts

    University of Science and Technology of China

  8. Tensor Cache: Eviction-conditioned Associative Memory for Transformers

    May 21, 2026Kabir Swain, Sijie Han, Daniel Karl I. Weidele +2Memory-Augmented Neural NetworksSelf-Attention

    Massachusetts Institute of Technology, Cambridge, MA, USA · University of Toronto, Toronto, Canada · IBM Research, Cambridge, MA, USA

  9. KVDrive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inference

    May 18, 2026Jian Lin, Jiazhi Mi, Zicong Hong +5KV-Cache OffloadingKV Caching

    Hong Kong University of Science and Technology, China · Xi’an Jiaotong University, China

  10. VeriCache: Turning Lossy KV Cache into Lossless LLM Inference

    May 17, 2026Jiayi Yao, Samuel Shen, Kuntai Du +7KV-Cache OffloadingKV Caching

    University of Chicago · Tensormesh Inc. · Samsung Semiconductor +1

  11. ObjectCache: Layerwise Object-Storage Retrieval for KV Cache Reuse

    May 16, 2026Yu Zhu, Aditya Dhakal, Yunming Xiao +2LLM Inference EfficiencyKV-Cache Offloading

    ETH Zurich Switzerland · HPE Labs United States of America · The Chinese University of Hong Kong, Shenzhen China

  12. CompactAttention: Accelerating Chunked Prefill with Block-Union KV Selection

    May 16, 2026Jiwon Song, Dongwon Jo, Beomseok Kang +1Self-AttentionKV-Cache Management

    Seoul National University

  13. HeatKV: Head-tuned KV-cache Compression for Visual Autoregressive Modeling

    May 14, 2026Jonathan Cederlund, Axel Berg, William Isaksson +3KV CachingVisual Autoregressive Models

    Dept. of Automatic Control, Lund University · Arm

  14. Head Forcing: Long Autoregressive Video Generation via Head Heterogeneity

    May 14, 2026Jiahao Tian, Yiwei Wang, Gang Yu +1Diffusion TransformerAutoregressive Diffusion

    AGI Lab, Westlake University · University of California at Merced · StepFun

  15. Minimal-Intervention KV Retention via Set-Conditioned Diversity

    May 14, 2026Libo Sun, Po-wei Harn, Peixiong He +1KV CachingKV-Cache Eviction

    Department of Computer Science and Software Engineering Auburn University, Auburn, AL, USA · Department of Information Management National Central University, Taoyuan, Taiwan

  16. Self-Pruned Key-Value Attention: Learning When to Write by Predicting Future Utility

    May 13, 2026Gergely Szilvasy, Manuel Faysse, Maria Lomeli +5Self-AttentionKV Caching

    Meta FAIR · MICS, CentraleSupélec

  17. SPHERICAL KV: Angle-Domain Attention and Rate-Distortion Retention for Efficient Long-Context Inference

    May 13, 2026Anay Chauhan, Gurucharan Marthi Krishna Kumar, Arion Das +4KV CachingLong-Context Language Model Inference

    1Synopsys, India · 2McGill University, Canada · 3IIIT Ranchi, India +4

  18. Pyramid Forcing: Head-Aware Pyramid KV Cache Policy for High-Quality Long Video Generation

    May 13, 2026Jiayu Chen, Junbei Tang, Wenbiao Zhao +6Self-AttentionLong-Horizon Video Generation

    Peking University · South China University of Technology · Xinjiang University +2

  19. KV-RM: Regularizing KV-Cache Movement for Static-Graph LLM Serving

    May 10, 2026Zhiqing Zhong, Zhijing Ye, Jian Zhang +3KV CachingKV-Cache Management

    Stevens Institute of Technology Hoboken, New Jersey, USA · The Hong Kong University of Science and Technology (Guangzhou) Guangzhou, Guangdong, China · Argonne National Laboratory Lemont, Illinois, USA +1

  20. Not All Thoughts Need HBM: Semantics-Aware Memory Hierarchy for LLM Reasoning

    May 10, 2026Aojie Yuan, Tianqi Shen, Dajun ZhangKV-Cache OffloadingKV Caching

    University of Southern California · †University of Wisconsin–Madison

  21. PEEK: Predictive Queue-Informed KV Cache Management for LLM Serving

    May 10, 2026Bing Xie, Zhipeng Wang, Masahiro Tanaka +1LLM ServingKV Caching

    †Corresponding author.

  22. HAServe: Horizon-Aware Scheduling and Memory Co-Design for Efficient LLM Serving

    May 9, 2026Xingyu Qu, Tianhao Lin, Yiqi Li +2LLM ServingKV Caching

    School of Computer Science, Wuhan University · Amazon

  23. Memory-Efficient Looped Transformer: Decoupling Compute from Memory in Looped Language Models

    May 8, 2026Victor Conchello Vendrell, Arnau Padres Masdemont, Niccolò Grillo +3KV-Cache ManagementMemory-Efficient Inference

    Qualcomm AI Research is an initiative of Qualcomm Technologies, Inc.

  24. An Efficient Hybrid Sparse Attention with CPU-GPU Parallelism for Long-Context Inference

    May 8, 2026Feiyu Yao, Zhixiong Niu, Xiaqing Li +3GPU AccelerationSelf-Attention

    Beijing University of Technology Beijing, China · Microsoft Research Beijing, China · Beijing Jiaotong University Beijing, China

  25. CSR: Infinite-Horizon Real-Time Policies with Massive Cached State Representations

    May 8, 2026Robin Karlsson, Go SuzuiRobotic ControlLanguage Model-Based Control

    1GODOT Inc, Hyogo, Japan · 2Graduate School of Informatics, Nagoya University, Aichi, Japan.

  26. Sparse Attention as a Range Searching Problem: Towards an Inference-Efficient Index for KV Cache

    May 7, 2026Mohsen Dehghankar, Abolfazl AsudehSelf-AttentionKV Caching

    Department of Computer Science University of Illinois Chicago

  27. Efficient Serving for Dynamic Agent Workflows with Prediction-based KV-Cache Management

    May 7, 2026Haoyu Zheng, Fangcheng Fu, Jia Wu +6LLM Inference EfficiencyKV Caching

    Wuhan University · Shanghai Jiao Tong University · Macquarie University +2

  28. Shallow Prefill, Deep Decoding: Efficient Long-Context Inference via Layer-Asymmetric KV Visibility

    May 7, 2026Jungsuk Oh, Hyeseo Jeon, Hyunjune Ji +2Self-AttentionLong-Context Language Model Inference

    Graduate School of Data Science Seoul National University Seoul, Republic of Korea

  29. Requests of a Feather Must Flock Together: Batch Size vs. Prefix Homogeneity in LLM Inference

    May 7, 2026Saksham Rathi, Preeti, Mythili VutukuruLLM Inference SchedulingKV-Cache Management

    Indian Institute of Technology Bombay, Mumbai, India

  30. QKVShare: Quantized KV-Cache Handoff for Multi-Agent On-Device LLMs

    May 5, 2026Pratik Honavar, Tejpratap GVSLMixed-Precision QuantizationMulti-Agent LLM Systems