KV-Cache Offloading

Ask about this topic

Overview

KV-cache offloading moves the attention state a language model needs during generation out of limited GPU memory, allowing longer contexts or more concurrent requests. Systems use tiers such as CPU memory, SSDs, and remote storage, with paging, reuse-aware placement, and prefetching to retrieve cached state when needed. Other approaches selectively retrieve or compress cache entries, and schedule transfers to overlap with GPU computation; they are evaluated on long-context tasks, serving latency and throughput, and agent workloads. The central trade-off is between memory capacity and the latency and bandwidth cost of retrieving state, while compression or eviction can also reduce answer quality; which strategy works best depends on hardware and workload.

Latest papers

CardsList

42 papers6 papers in the last four weeks, down 25% on the four weeks before. 0.0% of all new papers.

  1. WorldPager: Zero-Stall Tiered Memory for Long-Horizon World Models on a Single Consumer GPU

    Oct 26, 2026Bingnan Liu, Ruini XueKV-Cache OffloadingInteractive World Models

    School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, China

  2. Gandalf: Flash Shall Not Die! Increasing Flash Storage Lifetime for KV Cache Offloading in LLM Serving

    Oct 22, 2026Maharani A. P. Irawan, Kuntai Du, Jiayi Yao +4KV-Cache OffloadingKV-Cache Management

    The University of Chicago, USA · TensorMesh Inc., USA · Samsung Semiconductor, USA

  3. When the KV Cache Is Slower Than the GPU: Scaling Remote KV Tiers for Context-Parallel Prefill

    Oct 15, 2026Muhammad IbrahimContext ParallelismKV-Cache Management

    Multiverse Computing

  4. Adaptive DHiSparse: Speculative Decoding over a Load-Adaptive KV Hierarchy

    Oct 9, 2026Naji Alazhar, Wassel AlazharLLM Inference AccelerationKV-Cache Management

    Umans AI

  5. WakeKV: Reactive, Reversible KV Residency for Heads That Change Their Minds

    Oct 2, 2026Utkarsh RanjanKV-Cache ManagementKV-Cache Eviction

    UC San Diego

  6. TempoKV: Timely Staging of LLM KV Caches for Memory-Semantic Flash

    Sep 28, 2026Jay H. Park, Hyungjun Kim, Dong KimKV-Cache OffloadingKV-Cache Management

    Samsung Semiconductor

  7. PulseInfer: I/O-Centric Sparse KV Cache Offloading for Efficient Long-Context LLM Decoding

    Sep 28, 2026Qiuyang Zhang, Kai Zhou, Kai Lu +6KV-Cache OffloadingLong-Context Language Model Inference

    Huazhong University of Science and Technology Wuhan, China · UCloud Technology Co., Ltd. Shanghai, China

  8. EfficientAgent: What Makes KV Cache Offloading Work for Concurrent Agents?

    Sep 27, 2026Kunming Shao, Jierun Chen, Jiangnan Yu +7KV-Cache OffloadingKV-Cache Management

    The Hong Kong University of Science and Technology · Huawei Technologies Ltd. · The University of Hong Kong +1

  9. The KV Cache Is the New Memory Wall

    Sep 25, 2026Tejinder SinghKV CachingLong-Context Language Model Inference

    Dell Technologies, Santa Clara, CA 95054, USA

  10. Bridging LLM Serving and CXL-SSDs with Chunk-Aware KV Cache Management

    Sep 20, 2026Hyunsun Chung, Taewan Noh, Minji Kim +3LLM Inference AccelerationKV-Cache Offloading

    Sogang University · Samsung Electronics Co. · ETRI

  11. Fathom: Per-Query Read Depth for Sparse Decoding over Offloaded KV Caches

    Sep 15, 2026Vivek KalyanaranganKV-Cache OffloadingKV Caching

  12. Where Should the KV Cache Live? Placement Policies Across GPU, CPU, and SSD for Long-Lived Sessions

    Sep 14, 2026Srikanta Datta Tumkur, Jay Iyer, Mehar Simhadri +3KV-Cache ManagementKV-Cache Offloading

    Vizuara

  13. Physically Partitioned KVCache Format for CPU--GPU Load Balancing in MoE Inference

    Sep 13, 2026Enda Yu, Dezun Dong, Xiangke LiaoKV-Cache ManagementKV-Cache Offloading

    National University of Defense Technology China

  14. Building py-kvcache: A Performance Characterization of External KV Caching for vLLM with NVMe SSDs

    Sep 10, 2026Joseph Kanichai, Tiziano De Matteis, Animesh TrivediKV-Cache OffloadingKV Caching

    Vrije Universiteit Amsterdam Amsterdam, the Netherlands · IBM Research Zurich Zurich, Switzerland

  15. KVMem: Virtualizing Million-Token Agent Workspaces on a Consumer GPU

    Sep 4, 2026Di Chai, Leye Wang, Zeshen Su +2Memory Management for LLM AgentsLLM Context Management

    Shanghai University of Finance and Economics · Peking University · Northwestern Polytechnical University +1

  16. WnW: Waxing-and-Waning KV Cache for Long-Form Speech LLMs

    Aug 24, 2026Yiming Yao, Chenyang Lyu, Xuanfan Ni +4KV-Cache OffloadingKV-Cache Compression

    School of Informatics, Xiamen University · Alibaba Group

  17. PatchKV: Efficient KV Cache Recovery for Dynamically Edited LLM Contexts

    Aug 18, 2026Guotao Yang, Rui Guo, Siwei He +3KV-Cache ReuseLLM Inference Efficiency

    Tianjin University, China

  18. Long-Context Fine-Tuning with Limited VRAM

    Jul 16, 2026Vladimir Fedosov, Aleksandr Sazhin, Artemiy Grinenko +1Fine-TuningKV-Cache Offloading

    BMW Group

  19. SeKV: Resolution-Adaptive KV Cache with Hierarchical Semantic Memory for Long-Context LLM Inference

    Jun 30, 2026Amirhossein Abaskohi, Giuseppe Carenini, Peter West +1KV-Cache OffloadingKV Caching

    University of British Columbia · Microsoft Research

  20. Hierarchical Global Attention (HGA)

    Jun 29, 2026Woernle Frank, Fedosov Vladimir, Grinenko ArtemiyKV-Cache OffloadingLong-Context Modeling

    BMW Group

  21. HERALD: High-Throughput Block Diffusion LLM Serving via CPU-GPU Cooperative KV Cache Retrieval

    Jun 19, 2026Omin Kwon, Doyeon Kim, Jongseok Park +3KV-Cache OffloadingDiffusion Model Serving

    Seoul National University Seoul, Republic of Korea · UC Berkeley Berkeley, CA, USA

  22. SparDA: Sparse Decoupled Attention for Efficient Long-Context LLM Inference

    Jun 3, 2026Yaosheng Fu, Guangxuan Xiao, Xin Dong +2KV-Cache OffloadingLong-Context Language Model Inference

    1NVIDIA · 2Thinking Machines Lab · 3ByteDance Seed +1

  23. Multi-Segment Attention: Enabling Efficient KV-Cache Management for Faster Large Language Model Serving

    Jun 1, 2026Chunan Shi, Yilei Chen, Yilin Chen +2KV-Cache OffloadingLLM Inference

    Peking University Beijing, China

  24. KVDrive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inference

    May 18, 2026Jian Lin, Jiazhi Mi, Zicong Hong +5KV-Cache OffloadingKV Caching

    Hong Kong University of Science and Technology, China · Xi’an Jiaotong University, China

  25. VeriCache: Turning Lossy KV Cache into Lossless LLM Inference

    May 17, 2026Jiayi Yao, Samuel Shen, Kuntai Du +7KV-Cache OffloadingKV Caching

    University of Chicago · Tensormesh Inc. · Samsung Semiconductor +1

  26. ObjectCache: Layerwise Object-Storage Retrieval for KV Cache Reuse

    May 16, 2026Yu Zhu, Aditya Dhakal, Yunming Xiao +2LLM Inference EfficiencyKV-Cache Offloading

    ETH Zurich Switzerland · HPE Labs United States of America · The Chinese University of Hong Kong, Shenzhen China

  27. Not All Thoughts Need HBM: Semantics-Aware Memory Hierarchy for LLM Reasoning

    May 10, 2026Aojie Yuan, Tianqi Shen, Dajun ZhangKV-Cache OffloadingKV Caching

    University of Southern California · †University of Wisconsin–Madison

  28. Unifying Sparse Attention with Hierarchical Memory for Scalable Long-Context LLM Serving

    Apr 29, 2026Zihan Zhao, Baotong Lu, Shengjie Lin +8LLM ServingKV-Cache Offloading

    University of Virginia · Microsoft Research · Georgia Institute of Technology +1

  29. DUAL-BLADE: Dual-Path NVMe-Direct KV-Cache Offloading for Edge LLM Inference

    Apr 29, 2026Bodon Jeong, Hongsu Byun, Youngjae Kim +4On-Device Language Model InferenceKV-Cache Offloading

    1Sogang University, Seoul, Republic of Korea · 2Florida State University, FL, USA · 3Samsung Electronics Co.

  30. MTServe: Efficient Serving for Generative Recommendation Models with Hierarchical Caches

    Apr 24, 2026Xin Wang, Chi Ma, Shaobin Chen +14KV-Cache OffloadingKV Caching

    School of Computer Science, Wuhan University · Meituan · Nvidia