KV-Cache Reuse

Ask about this topic

Overview

KV-cache reuse studies how language models can retain and reuse attention states instead of repeatedly processing the same documents, conversation history, or shared context, reducing latency and memory costs in retrieval-augmented generation, long-context, and multi-agent settings. Approaches include exact-prefix caching, selective recomputation and cache repair, compression, position-invariant reuse, and transferring cache states across models; serving systems add scheduling and storage tiers to make reuse practical. A central challenge is preserving answer quality when cached states are used in a different position, context, model, or request order, so many methods combine reuse with targeted correction or recomputation. Evaluation remains sensitive to workload and measurement choices, and safety, privacy, and timing leakage are additional concerns when caches are shared.

Latest papers

CardsList

73 papers19 papers in the last four weeks, up 19% on the four weeks before. 0.1% of all new papers.

  1. Stateful Inference for Low-Latency Multi-Agent Tool Calling

    May 25, 2026Victor NorgrenLLM Inference EfficiencyMulti-Agent LLM Systems

    LayerScale, Inc.

  2. ObjectCache: Layerwise Object-Storage Retrieval for KV Cache Reuse

    May 16, 2026Yu Zhu, Aditya Dhakal, Yunming Xiao +2LLM Inference EfficiencyKV-Cache Offloading

    ETH Zurich Switzerland · HPE Labs United States of America · The Chinese University of Hong Kong, Shenzhen China

  3. PEEK: Predictive Queue-Informed KV Cache Management for LLM Serving

    May 10, 2026Bing Xie, Zhipeng Wang, Masahiro Tanaka +1LLM ServingKV Caching

    †Corresponding author.

  4. Efficient Serving for Dynamic Agent Workflows with Prediction-based KV-Cache Management

    May 7, 2026Haoyu Zheng, Fangcheng Fu, Jia Wu +6LLM Inference EfficiencyKV Caching

    Wuhan University · Shanghai Jiao Tong University · Macquarie University +2

  5. When Hidden States Drift: Can KV Caches Rescue Long-Range Speculative Decoding?

    Apr 29, 2026Tianyu Liu, Yuhao Shen, Xinyi Hu +8Speculative DecodingLLM Inference Acceleration

    1Qwen Applications Business Group of Alibaba · University of Science and Technology of China · 3Zhejiang University

  6. PolyKV: A Shared Asymmetrically-Compressed KV Cache Pool for Multi-Agent LLM Inference

    Apr 27, 2026Ishan Patel, Ishan JoshiLLM Inference EfficiencyMulti-Agent LLM Systems

    Independent Researcher

  7. MTServe: Efficient Serving for Generative Recommendation Models with Hierarchical Caches

    Apr 24, 2026Xin Wang, Chi Ma, Shaobin Chen +14KV-Cache OffloadingKV Caching

    School of Computer Science, Wuhan University · Meituan · Nvidia

  8. Predictive Multi-Tier Memory Management for KV Cache in Large-Scale GPU Inference

    Apr 19, 2026Sanjeev Rao GanjihalKV-Cache OffloadingKV Caching

    Independent Researcher

  9. KV Packet: Recomputation-Free Context-Independent KV Caching for LLMs

    Apr 14, 2026Chuangtao Chen, Grace Li Zhang, Xunzhao Yin +3KV CachingKV-Cache Reuse

    Technical University of Munich Munich, Germany · Technical University of Darmstadt Darmstadt, Germany · Zhejiang University Hangzhou, China +1

  10. LoSA: Locality Aware Sparse Attention for Block-Wise Diffusion Language Models

    Apr 13, 2026Haocheng Xi, Harman Singh, Yuezhou Hu +9Diffusion Language Model InferenceDiffusion Model Inference Acceleration

    University of California, Berkeley · FuriosaAI · ICSI +2

  11. ForkKV: Scaling Multi-LoRA Agent Serving via Copy-on-Write Disaggregated KV Cache

    Apr 7, 2026Shao Wang, Rui Ren, Lin GuiLLM ServingLLM Inference Acceleration

    Shanghai Jiao Tong University, Shanghai, China

  12. An experimental study of KV cache reuse strategies in chunk-level caching systems

    Mar 3, 2026Samuel Cestola, Tianxiang Xia, Zheng Weiyan +2KV-Cache ReuseLLM Inference Acceleration

    Huawei, Switzerland. · Huawei, China.

  13. RelayCaching: Accelerating LLM Collaboration via Decoding KV Cache Reuse

    Feb 28, 2026Yingsheng Geng, Yuchong Gao, Weihong Wu +2LLM Inference AccelerationKV-Cache Reuse

    Beijing University of Posts and Telecommunications, Beijing, China · Tsinghua University, Beijing, China · University of Electronic Science and Technology of China, Chengdu, China +1