Overview

KV-cache offloading moves the attention state a language model needs during generation out of limited GPU memory, allowing longer contexts or more concurrent requests. Systems use tiers such as CPU memory, SSDs, and remote storage, with paging, reuse-aware placement, and prefetching to retrieve cached state when needed. Other approaches selectively retrieve or compress cache entries, and schedule transfers to overlap with GPU computation; they are evaluated on long-context tasks, serving latency and throughput, and agent workloads. The central trade-off is between memory capacity and the latency and bandwidth cost of retrieving state, while compression or eviction can also reduce answer quality; which strategy works best depends on hardware and workload.

Latest papers

CardsList

47 papers6 papers in the last four weeks, down 14% on the four weeks before. 0.0% of all new papers.

  1. XM-Sparse: High-Throughput Sparse-Attention LLM Serving over Scale-up Memory Pools

    Nov 17, 2026Yijun Sun, Xudong Liao, Zhuobin Huang +3KV-Cache OffloadingLLM Inference Acceleration

  2. Making Preemption Productive in Long-Output LLM Serving

    Nov 15, 2026Ziji Shi, Xinyi Wan, Chaoyi Ruan +3LLM ServingLLM Inference Scheduling

  3. Verify Before You Fetch: Reliable KV Cache Compression for LLM Inference

    Nov 8, 2026Devanshu Jha, Gaurav Kumar, Agniva Chowdhury +2KV-Cache CompressionLong-Context Language Model Inference

  4. FlashOffloading: Low-Latency KV Cache Offloading for Long-Context LLM Serving

    Nov 5, 2026Xuan Wang, Zhuanhao Wu, Tingzhu Fang +17KV-Cache OffloadingLLM Serving

  5. WorldPager: Zero-Stall Tiered Memory for Long-Horizon World Models on a Single Consumer GPU

    Oct 26, 2026Bingnan Liu, Ruini XueKV-Cache OffloadingInteractive World Models

  6. Gandalf: Flash Shall Not Die! Increasing Flash Storage Lifetime for KV Cache Offloading in LLM Serving

    Oct 22, 2026Maharani A. P. Irawan, Kuntai Du, Jiayi Yao +4KV-Cache OffloadingKV-Cache Management

  7. TempoKV: Timely Staging of LLM KV Caches for Memory-Semantic Flash

    Sep 28, 2026Jay H. Park, Hyungjun Kim, Dong KimKV-Cache OffloadingKV-Cache Management

  8. PulseInfer: I/O-Centric Sparse KV Cache Offloading for Efficient Long-Context LLM Decoding

    Sep 28, 2026Qiuyang Zhang, Kai Zhou, Kai Lu +6KV-Cache OffloadingLong-Context Language Model Inference

  9. EfficientAgent: What Makes KV Cache Offloading Work for Concurrent Agents?

    Sep 27, 2026Kunming Shao, Jierun Chen, Jiangnan Yu +7KV-Cache OffloadingKV-Cache Management

  10. The KV Cache Is the New Memory Wall

    Sep 25, 2026Tejinder SinghKV CachingLong-Context Language Model Inference

  11. Bridging LLM Serving and CXL-SSDs with Chunk-Aware KV Cache Management

    Sep 20, 2026Hyunsun Chung, Taewan Noh, Minji Kim +3LLM Inference AccelerationKV-Cache Offloading

  12. Physically Partitioned KVCache Format for CPU--GPU Load Balancing in MoE Inference

    Sep 13, 2026Enda Yu, Dezun Dong, Xiangke LiaoKV-Cache ManagementKV-Cache Offloading

  13. Building py-kvcache: A Performance Characterization of External KV Caching for vLLM with NVMe SSDs

    Sep 10, 2026Joseph Kanichai, Tiziano De Matteis, Animesh TrivediKV-Cache OffloadingKV Caching

  14. KVMem: Virtualizing Million-Token Agent Workspaces on a Consumer GPU

    Sep 4, 2026Di Chai, Leye Wang, Zeshen Su +2Memory Management for LLM AgentsLLM Context Management

  15. WnW: Waxing-and-Waning KV Cache for Long-Form Speech LLMs

    Aug 24, 2026Yiming Yao, Chenyang Lyu, Xuanfan Ni +4KV-Cache OffloadingKV-Cache Compression

  16. Long-Context Fine-Tuning with Limited VRAM

    Jul 16, 2026Vladimir Fedosov, Aleksandr Sazhin, Artemiy Grinenko +1Fine-TuningKV-Cache Offloading

  17. SeKV: Resolution-Adaptive KV Cache with Hierarchical Semantic Memory for Long-Context LLM Inference

    Jun 30, 2026Amirhossein Abaskohi, Giuseppe Carenini, Peter West +1KV-Cache OffloadingKV Caching

  18. Hierarchical Global Attention (HGA)

    Jun 29, 2026Woernle Frank, Fedosov Vladimir, Grinenko ArtemiyKV-Cache OffloadingLong-Context Modeling

  19. HERALD: High-Throughput Block Diffusion LLM Inference via CPU-GPU Cooperative KV Cache Retrieval

    Jun 19, 2026Omin Kwon, Doyeon Kim, Jongseok Park +4KV-Cache OffloadingDiffusion Model Serving

  20. SparDA: Sparse Decoupled Attention for Efficient Long-Context LLM Inference

    Jun 3, 2026Yaosheng Fu, Guangxuan Xiao, Xin Dong +2KV-Cache OffloadingLong-Context Language Model Inference

  21. KVDrive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inference

    May 18, 2026Jian Lin, Jiazhi Mi, Zicong Hong +5KV-Cache OffloadingKV Caching

  22. VeriCache: Turning Lossy KV Cache into Lossless LLM Inference

    May 17, 2026Jiayi Yao, Samuel Shen, Kuntai Du +7KV-Cache CompressionKV-Cache Offloading