KV-Cache Management

Ask about this topic

Overview

The field studies how to store, reuse, compress, and move the key-value states that Transformers accumulate during generation, which can dominate memory and data-transfer costs in long-context inference. Methods include token eviction and learned importance scoring, quantization and compact representations, prefix or cross-request sharing, and offloading across GPU, host memory, and storage; serving systems also coordinate these choices with scheduling and batching. Evaluations commonly measure quality on long-context retrieval and reasoning benchmarks such as LongBench and RULER alongside memory use, time to first token, and throughput. The central tension is reducing cache cost without losing information needed later: results depend on the task, workload, hardware, and attention implementation, so robust policies and faithful end-to-end evaluation remain important challenges.

Latest papers

CardsList

201 papers36 papers in the last four weeks, up 13% on the four weeks before. 0.2% of all new papers.

  1. Pixel Decodability Is Not a Compression Signal: Causally Evaluating Importance Proxies for Visual KV-Cache Eviction

    Sep 14, 2026Chenyu Zhou, Qiliang Jiang, Shuning Wu +1KV CachingKV-Cache Eviction

    School of Engineering, Institute of Science Tokyo, Japan · College of Control Science and Engineering, Zhejiang University, China · Department of Electrical and Computer Engineering, National University of Singapore

  2. Physically Partitioned KVCache Format for CPU--GPU Load Balancing in MoE Inference

    Sep 13, 2026Enda Yu, Dezun Dong, Xiangke LiaoKV-Cache ManagementKV-Cache Offloading

    National University of Defense Technology China

  3. Fine-Tuning a KV Cache Concatenation-Aware Model or Recomputing KV Caches? Why Not Both?

    Sep 9, 2026Fumihiko Tachibana, Daisuke Miyashita, Jun DeguchiRetrieval-Augmented GenerationKV Caching

    Kioxia Corporation

  4. BIO-MEMART: Biometric-Aware KV Cache Memory for Multi-User LLM Agents

    Sep 8, 2026Yanhong Qian, Xuanying He, Qingguo Meng +3Agent MemoryPrivacy-Preserving Language Models

    Anhui Provincial Key Laboratory of Secure Artificial Intelligence, School of Artificial Intelligence, Anhui University, Hefei 230093, China

  5. VestigeKV: The NoPE-MLA KV Cache Carries Its Own Sparse-Attention Signal in a Vestigial Branch

    Sep 3, 2026WenJie FanKV CachingKV-Cache Management

    Yotta Labs

  6. GrowPage: On-Demand KV Budgeting for Efficient LLM Reasoning Serving

    Sep 3, 2026Qiankun Ma, Yanjiang Zhou, Zinan Xiong +5LLM InferenceKV Caching

    Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · Pengcheng Laboratory · University of Chinese Academy of Sciences

  7. SGD-KV: Summarization Guided KV Cache Compression

    Sep 3, 2026Zeyu Liu, Woomin Song, Xuandi Fu +5KV CachingLong-Context Language Model Inference

    USC, USA · KAIST, Korea · Amazon AGI, USA

  8. ShallowStream: Index Shallow then Answer Deep for Streaming Video Understanding

    Sep 2, 2026Jitai Hao, Ke Yang, Di Yan +3Streaming Video UnderstandingEfficient Multimodal Inference

    Harbin Institute of Technology (Shenzhen)

  9. Language Models Can Control Their Own Attention

    Sep 2, 2026Namgyu Ho, Huzama Ahmad, Woosung Koh +3Efficient Language Model InferenceLong-Context Language Modeling

    KAIST AI · Google DeepMind

  10. Small Frequency Corrections Can Change What Survives KV Cache Compression

    Aug 27, 2026Hong Chen, Yudong Zeng, Yongwei Huang +5KV CachingKV-Cache Eviction

    The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · Bosum Institute of Management Science, Shenzhen, China

  11. Trust the Mass: Forced Weights in KV-Cache Eviction

    Aug 25, 2026Jack Shi, Jerry GuKV-Cache EvictionKV-Cache Management

    Stanford University

  12. Elastic KV Cache for LLM Serving:A Working Reclamation Mechanism, and Why Chunked Prefill Already Closes the Gap

    Aug 24, 2026Sathishkumar SivashanmugamKV-Cache ManagementLLM Inference Efficiency

    Amazon Web Services

  13. Squeezing the Cache, Preserving the Truth: Monotonic Equipotential Allocation with Geodesia-KV

    Aug 19, 2026Vincenzo Dentamaro, Pancrazio Auteri, Giuseppe PirloKV-Cache CompressionKV-Cache Management

    University of Bari Aldo Moro, Department of Computer Science

  14. GraniKV: Asymmetric Granularity KV-Cache Paging for Multi-Agent Systems with Long Shared Prefix

    Aug 16, 2026Jinhyun Jeon, Sungjoo YooKV-Cache ManagementPrefix Caching

    Seoul National University

  15. vToken: Token-Level Virtualization for Reclaimable KV Caches

    Aug 13, 2026Yuanhang Gao, Xiangrui Yang, Yuanfeng Chen +4KV CachingKV-Cache Eviction

    National University of Defense Technology · Changsha, China · Peking University +1

  16. Scheduling Mixed RL Rollouts Beyond Prefix Locality

    Aug 11, 2026Zetao Hong, Song Yuan, Yuanhao Ding +4LLM Inference SchedulingKV-Cache Management

    State Key Laboratory of Novel Software Technology, Nanjing University · StepFun

  17. ImpactHO: Importance-Aware KV Cache Transfer for Multi-User Edge LLM Handover

    Aug 11, 2026Minwoo Kim, Soochang Song, Namyoon Lee +2KV CachingEdge Computing

    Department of Electrical Engineering, Pohang University of Science and Technology (POSTECH), Pohang 37673, South Korea · Department of Electrical and Computer Engineering, Ajou University, Suwon 16499, South Korea

  18. Governing the KV Cache: Preventing Timing Side-Channel Leakage in Multi-Tenant LLM Inference

    Aug 10, 2026Tejasvi C. AddagadaKV CachingKV-Cache Management

  19. DistillCache: KL-Guided Adaptive KV-Cache Eviction for Memory-Efficient LLM Inference

    Aug 9, 2026Asaad AlthoubiKV CachingLong-Context Language Model Inference

    Oklahoma State University Stillwater, OK USA

  20. RippleKV: Cross-Layer KV Cache Allocation via Perturbation Propagation

    Aug 9, 2026Dongjie Xu, Kai Qian, Julius +6KV CachingKV-Cache Management

    1Soochow University · 2Tencent Inc. · 3The Hong Kong University of Science and Technology +2

  21. Every Cache Entry Earns Its Place: Global Allocation of Resolution and Coverage for KV Cache Compression

    Aug 7, 2026Haolin Tian, Yuzhe Liu, Tonghan WangKV CachingKV-Cache Management

    Tsinghua University Beijing, China

  22. Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving

    Aug 6, 2026Muhammad Adnan, Rohan Mahapatra, Prashant J. Nair +4LLM ServingLLM Inference Scheduling

    The University of British Columbia · Microsoft Azure Research · NVIDIA

  23. QEvict: Recoverable Quantized KV Eviction for Attention-Drift-Robust Long-Context Decoding

    Aug 5, 2026Ayushman Garg, Akshita Gupta, Shaswata Bhattacharya +3KV CachingLong-Context Language Model Inference

    Mehta Family School of Data Science and Artificial Intelligence, Indian Institute of Technology Roorkee · Yardi School of Artificial Intelligence, Indian Institute of Technology Delhi · Department of Electrical Engineering, Indian Institute of Technology Delhi

  24. TaskPress: Query-Agnostic KV Cache Compression via Task-Guided Pruning

    Aug 4, 2026Wonpyo Park, Seung-won HwangLLM PruningKV Caching

    Seoul National University · Google

  25. Practical Online KV Cache Compaction for LLM Agents: An Empirical Study

    Aug 2, 2026Yujian Liu, Jiabao Ji, Li An +4KV CachingKV-Cache Eviction

    UC Santa Barbara · LinkedIn

  26. Stage-Replay Divergence Follows the KV Cache: Fixed-Prefix Precision Controls and Bidirectional Cache Transplantation

    Jul 30, 2026Alexander Boesgaard LorupLLM InferenceKV Caching

    Openhagen

  27. Back from the Future: Key-Value Cache Management by Counter-Causal Surprise

    Jul 30, 2026Stephen Gould, Anton van den HengelKV CachingKV-Cache Eviction

    Metacognition AI · Australian National University · Adelaide University

  28. KAP: Bridging the Knowledge Selection-Runtime Consumption Gap in LLM Systems

    Jul 27, 2026Shuo Wang, Fang Xi, Wenyuan Huang +2LLM ServingKV-Cache Management

    QiYuanLab Beijing, China

  29. HeadCast: Casting Attention Heads for Efficient Autoregressive Video Generation

    Jul 22, 2026Jinliang Shen, Lianghao Su, Zheming Li +4KV-Cache ManagementAutoregressive Video Generation

    1Peking University · 2KlingAI Research

  30. Talaria: Session-Aware Serverless Serving of Hundred-Billion-Parameter LLMs

    Jul 19, 2026Utopia Meng, Unicornt Zhao, Derek Li +2LLM ServingLLM Inference Scheduling