Prefix Caching

Ask about this topic

Overview

Prefix caching stores the key-value (KV) states computed for a prompt prefix and reuses them when later requests share that prefix, reducing repeated prefill work and time-to-first-token. Serving systems use radix-tree matching, cache-aware ordering and routing, and scheduling policies to exploit reuse in settings such as retrieval-augmented generation and multi-agent workloads. Eviction policies and GPU, host-memory, or disk offloading manage limited cache capacity; hybrid recurrent models can use checkpoints or suffix replay to reconstruct reusable state. The main tension is that cache hits improve efficiency only when reused state remains correct and useful, while memory pressure, changing workloads, load imbalance, and privacy risks can limit sharing or affect quality.

Latest papers

CardsList

36 papers6 papers in the last four weeks, down 45% on the four weeks before. 0.0% of all new papers.

  1. WAKMan: Workload Aware KVCache Manager

    Oct 26, 2026Suhas Kumar Morisetty, Vaibhavi Lokegaonkar, Sachin Mathew Varghese +1KV-Cache ManagementKV-Cache Eviction

    Columbia University · University of Maryland, College Park · Capital One

  2. Tandem: Serving Hybrid Linear-Attention GUI Agents on Mobile NPUs

    Oct 19, 2026Zhuomin Chen, Jingchao Ni, Xu Zheng +4On-Device Language Model InferenceMobile GUI Agents

    Florida International University, Miami, FL, USA · University of Houston, Houston, TX, USA · Independent Researcher +2

  3. RadixFlow: Filling Execution Bubbles in Prefix-Cached Hybrid LLM Serving

    Oct 13, 2026Huihao Jing, Wenbin Hu, Shaojin Chen +6LLM ServingLLM Inference Scheduling

    HKUST

  4. CADOC: Cache-Aware Dynamic Object Context for Long-Horizon Agents

    Sep 29, 2026Junjie Yao, Zhangchen Zhou, Zhi-Qin John XuCost-Aware InferencePrefix Caching

    School of Mathematical Sciences, Shanghai Jiao Tong University, Shanghai, China. · Institute of Natural Sciences, Shanghai Jiao Tong University, Shanghai, China.

  5. WavePP: High-Throughput Pipeline Parallel LLM Prefill under Prefix Reuse

    Sep 28, 2026Aaryam SharmaLLM Inference EfficiencyLLM Serving

    Baseten

  6. TempoKV: Timely Staging of LLM KV Caches for Memory-Semantic Flash

    Sep 28, 2026Jay H. Park, Hyungjun Kim, Dong KimKV-Cache OffloadingKV-Cache Management

    Samsung Semiconductor

  7. Dynamic Flow, Static Graph: KV Cache Reuse for Efficient LLM Serving on Mobile NPUs

    Sep 28, 2026Zhengxiang Huang, Shengheng Chen, Chaoyue Niu +6AI Accelerator InferenceOn-Device Language Model Inference

    Shanghai Jiao Tong University Shanghai, China · Alibaba Group Hangzhou, China · Alibaba Group Hangzhou, Zhejiang, China

  8. EfficientAgent: What Makes KV Cache Offloading Work for Concurrent Agents?

    Sep 27, 2026Kunming Shao, Jierun Chen, Jiangnan Yu +7KV-Cache OffloadingKV-Cache Management

    The Hong Kong University of Science and Technology · Huawei Technologies Ltd. · The University of Hong Kong +1

  9. Just Let Linear States Forget the Distant Past: Prefix Caching via Suffix Replay for Hybrid LLMs

    Sep 27, 2026Yirui Liu, Ruoling Qi, Xuaner Wu +6Long-Context Language Model InferenceLLM Inference Acceleration

    Institute of Artificial Intelligence, China Telecom (TeleAI) China · Shanghai Jiao Tong University · Individual Researcher +1

  10. When Fancy Eviction Fails: Rethinking Cache Replacement For LLM Prefix Reuse

    Sep 24, 2026Yiyu Liu, Minlan Yu, Juncheng YangLLM Inference EfficiencyKV Caching

    Harvard University

  11. PrefixBench-H100: Characterizing Prefix Reuse and Time-to-First-Token in H100 LLM Serving

    Sep 17, 2026Omkar Shewale, Deepak Kumar, Divakar Kumar YadavLLM Inference EfficiencyLLM Inference

    Illinois Institute of Technology · University of Wisconsin–Milwaukee

  12. Shared-Prefix KV Reuse Across Standard LoRA Adapters: Quality and Serving Tradeoffs

    Sep 15, 2026Dushyant RajputLLM Inference EfficiencyLLM Serving

    AltSlate Labs LLP

  13. Shared KV Caching for Replicated 27B Inference: Correctness Failures and Performance Boundaries

    Sep 14, 2026Frank LiKV CachingKV-Cache Management

    UNSW Sydney

  14. Prefix Sharing Is a Sorting Problem

    Sep 12, 2026Rong HePrefix CachingLLM Request Scheduling

  15. Building py-kvcache: A Performance Characterization of External KV Caching for vLLM with NVMe SSDs

    Sep 10, 2026Joseph Kanichai, Tiziano De Matteis, Animesh TrivediKV-Cache OffloadingKV Caching

    Vrije Universiteit Amsterdam Amsterdam, the Netherlands · IBM Research Zurich Zurich, Switzerland

  16. KVShareArena: KV-Cache Reuse Across Contexts and Model Checkpoints

    Sep 9, 2026Xi Shi, Mengxin Zheng, Qian LouKV CachingLLM Inference Acceleration

    University of Central Florida

  17. Same Request, Different Answer: Quantization Amplifies Cache-Induced Divergence in LLM Serving

    Sep 4, 2026Aditi PatodiyaLLM InferencePrefix Caching

    Independent Researcher, Milpitas, CA 95035 USA

  18. Tail-Replay: Escaping the Curse of Linear Attention in Prefix Caching for Hybrid LLMs

    Aug 31, 2026Yirui Liu, Ruoling Qi, Xuaner Wu +2LLM Inference AccelerationPrefix Caching

    Institute of Artificial Intelligence, China Telecom (TeleAI) · Shanghai Jiao Tong University

  19. TOPAS: Workflow-Aware Prefix-State Scheduling for Multi-Agent LLM Serving

    Aug 26, 2026Hongqiu Ni, Han Tian, Chi Zhang +2LLM Inference SchedulingLLM Agent Serving

    University of Science and Technology of China, China · Hefei University of Technology, China

  20. CacheRouter: A Dual-Path Tool Routing Architecture with Cache-Preserving Main-Model Isolation for Long-Tail Tool Discovery

    Aug 24, 2026Donghui Zha, Lingwei Xu, Linxiao Wu +2Tool Discovery for LLM AgentsLLM Tool Use

    School of Mathematical Sciences, Beijing University of Posts and Telecommunications, Beijing 100876, China · School of Mathematics and Statistics, Chongqing University, Chongqing 401331, China · School of Science, Beijing Forestry University, Beijing 100083, China

  21. CacheRoute: Planned Prefix-Affinity Routing for Large-Scale LLM Serving

    Aug 20, 2026Huang ChengLLM InferenceLLM Routing

    Meta, Menlo Park, California, USA

  22. GraniKV: Asymmetric Granularity KV-Cache Paging for Multi-Agent Systems with Long Shared Prefix

    Aug 16, 2026Jinhyun Jeon, Sungjoo YooKV-Cache ManagementPrefix Caching

    Seoul National University

  23. Cache-Aware Prompt Compression:A Two-Tier Cost Model for LLM API Caching

    Jul 17, 2026Yan SongLLM ServingLLM Inference

    PayPal Inc.

  24. CacheWeaver: Cache-Aware Evidence Ordering for Efficient Grounded RAG Inference

    Jun 18, 2026Kaizhen Tan, Rong Gu, Mingyuan LiLLM Inference EfficiencyRetrieval-Augmented Generation

    Heinz College of Information Systems and Public Policy, Carnegie Mellon University, USA

  25. Models Take Notes at Prefill: KV Cache Can Be Editable and Composable

    Jun 14, 2026Bojie LiKV CachingKV-Cache Management

    Pine AI

  26. MiniPIC: Flexible Position-Independent Caching in <100LOC

    Jun 11, 2026Nathan Ordonez, Thomas ParnellLLM Inference EfficiencyKV Caching

    IBM Research · IBM Research Zurich, Switzerland

  27. Enabling KV Caching of Shared Prefix for Diffusion Language Models

    May 26, 2026Younghun Go, Jaehoon Han, Changyong Shin +2KV CachingKV-Cache Management

    Korea University, South Korea

  28. ObjectCache: Layerwise Object-Storage Retrieval for KV Cache Reuse

    May 16, 2026Yu Zhu, Aditya Dhakal, Yunming Xiao +2LLM Inference EfficiencyKV-Cache Offloading

    ETH Zurich Switzerland · HPE Labs United States of America · The Chinese University of Hong Kong, Shenzhen China

  29. Not All Tokens Are Worth Caching: Learning Semantic-Aware Eviction for LLM Prefix Caches

    May 12, 2026Shaoke Fang, Ziang Li, Wenfei Wu +3LLM Inference EfficiencyKV Caching

    Peking University · FirestAI · Tsinghua University +2

  30. Agent-X: Full Pipeline Acceleration of On-device AI Agents

    May 11, 2026Jinha Chung, Byeongjun Shin, Jiin Kim +1LLM Inference EfficiencyOn-Device Language Model Inference

    KAIST Republic of Korea, Daejeon