KV-Cache Management

Ask about this topic

Overview

The field studies how to store, reuse, compress, and move the key-value states that Transformers accumulate during generation, which can dominate memory and data-transfer costs in long-context inference. Methods include token eviction and learned importance scoring, quantization and compact representations, prefix or cross-request sharing, and offloading across GPU, host memory, and storage; serving systems also coordinate these choices with scheduling and batching. Evaluations commonly measure quality on long-context retrieval and reasoning benchmarks such as LongBench and RULER alongside memory use, time to first token, and throughput. The central tension is reducing cache cost without losing information needed later: results depend on the task, workload, hardware, and attention implementation, so robust policies and faithful end-to-end evaluation remain important challenges.

Latest papers

CardsList

201 papers36 papers in the last four weeks, up 13% on the four weeks before. 0.2% of all new papers.

  1. WAKMan: Workload Aware KVCache Manager

    Oct 26, 2026Suhas Kumar Morisetty, Vaibhavi Lokegaonkar, Sachin Mathew Varghese +1KV-Cache ManagementKV-Cache Eviction

    Columbia University · University of Maryland, College Park · Capital One

  2. LLMs in Cloud: A Discussion on Architecture, Orchestration, and Inference

    Oct 24, 2026Jannatun Noor, Athar Noor Mohammad Rafee, Md. Sadman Haque +3LLM ServingLLM Inference

    Computing for Sustainability and Social Good (C2SG) Research Group, Department of Computer Science and Engineering, United International University, Dhaka, Dhaka 1212, Bangladesh. · Department of Computer Science and Engineering, School of Data & Computational Sciences, BRAC University, Dhaka, Dhaka 1212, Bangladesh.

  3. LSTM-Style Looped Transformers with Shared KV Caches

    Oct 23, 2026Andreas KirschRecurrent TransformersKV-Cache Management

  4. Why Full-Duplex Speech Models Collapse in Long Conversations

    Oct 23, 2026Yeonju Kim, Bella Godiva, Se Jin Park +1Full-Duplex Spoken Dialogue SystemsLong-Context Modeling

    Integrated Vision and Language Lab, KAIST, South Korea · Kyung Hee University, South Korea

  5. Gandalf: Flash Shall Not Die! Increasing Flash Storage Lifetime for KV Cache Offloading in LLM Serving

    Oct 22, 2026Maharani A. P. Irawan, Kuntai Du, Jiayi Yao +4KV-Cache OffloadingKV-Cache Management

    The University of Chicago, USA · TensorMesh Inc., USA · Samsung Semiconductor, USA

  6. Cache Locally, Recompute Selectively for Streaming Video Understanding

    Oct 21, 2026Junhyeok Lee, Sangyoon Lee, Jaeho LeeStreaming Video UnderstandingStreaming Inference

    Pohang University of Science and Technology (POSTECH)

  7. Cache Engineering for Agents

    Oct 20, 2026Tianyu Fu, Ruoxi Wang, Zihan Min +6KV-Cache ManagementMemory Management for LLM Agents

    Tsinghua University · The Chinese University of Hong Kong

  8. Local Inference of LLMs on Long Contexts: A Data Systems Perspective

    Oct 20, 2026Mattias Dürrmeier, Alberto LernerLLM InferenceLong-Context Inference

    University of Fribourg, Fribourg, Switzerland · Computing Flows GmbH, Fribourg, Switzerland

  9. The KV-Surgeon: Compensation-Aware Offline KV-Cache Compaction

    Oct 19, 2026Moritz Wagner, Christophe Roux, Max Zimmer +1KV-Cache ManagementKV-Cache Pruning

    Department for AI in Society, Science, and Technology, Zuse Institute Berlin, Germany · Institute of Mathematics, Technische Universität Berlin, Germany

  10. MOSAIC: Multi-Objective Search for Allocating Inference Cache Across LLM Decoder Layers

    Oct 18, 2026Sravanth Kodavanti, Manjunath ArvetiKV-Cache ManagementLong-Context Language Model Inference

    Samsung Research · Georgia Institute of Technology

  11. DLite: History-free Speculative Decoding for Agentic AI

    Oct 16, 2026Jiaming Xu, Hanzhen Wang, Tianlang Zhao +7Speculative DecodingLLM Inference Acceleration

    Shanghai Jiao Tong University · Shanghai Innovation Institute · Tsinghua University +1

  12. Augment the Cache, Replace the Activation: A 2x2 Study of Internal-State Sharing

    Oct 15, 2026Adam Deng, Shicheng Liu, Yada ZhuMulti-Agent CommunicationMulti-Agent LLM Systems

    MIT ORC · MIT CSAIL · IBM Research

  13. When the KV Cache Is Slower Than the GPU: Scaling Remote KV Tiers for Context-Parallel Prefill

    Oct 15, 2026Muhammad IbrahimContext ParallelismKV-Cache Management

    Multiverse Computing

  14. PriorityFEC: Deadline-Aware Streaming FEC for KV-Cache Handoff in Edge Multi-Agent Systems

    Oct 15, 2026Han Shen, Qifu Tyler Sun, Fanyang Meng +2KV-Cache ManagementLLM Inference Acceleration

  15. From Long Context to Selective Memory:A Survey of Memory-Centric Compression for Large Language Models

    Oct 15, 2026Chaogui GouLLM Context ManagementMemory-Efficient Inference

    USTB

  16. Compression Does Not End at Selection: Post-Selection Positional Geometry for Sparse KV Caches

    Oct 15, 2026Wanyang Hong, Zhaoning ZhangKV-Cache ManagementKV-Cache Compression

    National Key Laboratory of Parallel and Distributed Computing College of Computer Science and Technology National University of Defense Technology, Changsha, China.

  17. ChampionKV: Head-Wise Adaptive Budget Allocation for KV Cache Compression

    Oct 13, 2026Yuechuan Chen, Yanda Guo, Xiaoxi Liu +4KV-Cache CompressionKV-Cache Management

    Sun Yat-sen University · The Chinese University of Hong Kong · University of Electronic Science and Technology of China +2

  18. Budgeted KV-Cache Repair for Evolving Contexts

    Oct 12, 2026Han Yu, Wenhui Zhu, Jincheng Cao +15KV-Cache ManagementLLM Inference Acceleration

    LinkedIn · Clemson University · Arizona State University +1

  19. SAFE-KV: Causal Suffix Repair for Behavior-Preserving Non-Prefix KV Cache Reuse in RAG

    Oct 10, 2026Xingjian ZhangKV-Cache ReuseKV-Cache Management

    Carnegie Mellon University, Pittsburgh, PA, USA

  20. Adaptive DHiSparse: Speculative Decoding over a Load-Adaptive KV Hierarchy

    Oct 9, 2026Naji Alazhar, Wassel AlazharLLM Inference AccelerationKV-Cache Management

    Umans AI

  21. KV-Repurpose: A Testbed for Evaluating Non-Prefix KV-Cache Reuse

    Oct 9, 2026Taiga Ikeda, Daisuke Miyashita, Jun DeguchiKV-Cache ReuseKV-Cache Management

    Kioxia Corporation, Japan

  22. Enabling Dynamic Computation in Looped LMs

    Oct 6, 2026Aayush Mishra, Arnau Padrés Masdemont, Victor Conchello Vendrell +3LLM Inference AccelerationAdaptive Computation

    Qualcomm AI Research · Johns Hopkins University

  23. Request Order Matters: Cache-History Sensitivity in Selective KV-Cache Reuse for Rolling Agents

    Oct 5, 2026Tiffany Gu, Annie Guan, Manshu Huang +3KV-Cache ManagementLLM Inference Acceleration

    Capital One · University of Washington

  24. WakeKV: Reactive, Reversible KV Residency for Heads That Change Their Minds

    Oct 2, 2026Utkarsh RanjanKV-Cache ManagementKV-Cache Eviction

    UC San Diego

  25. CommunityKV: Efficient Long-Context Decoding via Graph Partitioning

    Sep 30, 2026Joe McKenna, Anastasios Alexandridis, Nathan Susanj +1Long-Context Language Model InferenceKV-Cache Management

    Amazon AGI

  26. EchoPress: Query-Agnostic KV Cache Pruning via Virtual Context Reconstruction

    Sep 30, 2026Jiawei Lin, Saibo Geng, Thomas BourgeatKV-Cache ManagementLLM Inference Acceleration

    EPFL

  27. Characterizing High Bandwidth Flash for LLM Serving

    Sep 30, 2026Zack Yu, Chloe Wong, Coleman Hooper +7LLM Inference EfficiencyLLM Serving

    University of California, Berkeley Berkeley, California, USA · FuriosaAI Seoul, South Korea · University of California, Berkeley; ICSI; LBNL Berkeley, California, USA

  28. DeCoPrune: Efficient KV-Cache Pruning for Autoregressive Video Diffusion via Denoising Consistency

    Sep 30, 2026Zeqi Xiao, Qingle Liu, Kaiwen Zhang +3KV-Cache ManagementAutoregressive Video Generation

    Nanyang Technological University · Tsinghua University · Princeton University

  29. ARC-KV: Amortizing Anchor Search for Reconstruction-Based KV Cache Compaction

    Sep 29, 2026Zheyu Shen, Guanhua Wang, Dezhan Tu +5Long-Context Language ModelingKV-Cache Management

    University of Maryland, College Park · Meta

  30. KV-streams for Efficient Compaction in Agentic Reinforcement Learning

    Sep 28, 2026Emiliano Penaloza, Dane Malenfant, Dheeraj Vattikonda +15Recurrent Language ModelsAgentic RL

    Mila · Microsoft · Université de Montréal +9