KV-Cache Offloading

Ask about this topic

Overview

KV-cache offloading moves the attention state a language model needs during generation out of limited GPU memory, allowing longer contexts or more concurrent requests. Systems use tiers such as CPU memory, SSDs, and remote storage, with paging, reuse-aware placement, and prefetching to retrieve cached state when needed. Other approaches selectively retrieve or compress cache entries, and schedule transfers to overlap with GPU computation; they are evaluated on long-context tasks, serving latency and throughput, and agent workloads. The central trade-off is between memory capacity and the latency and bandwidth cost of retrieving state, while compression or eviction can also reduce answer quality; which strategy works best depends on hardware and workload.

Latest papers

CardsList

42 papers6 papers in the last four weeks, down 25% on the four weeks before. 0.0% of all new papers.

  1. SparKV: Overhead-Aware KV Cache Loading for Efficient On-Device LLM Inference

    Apr 23, 2026Hongyao Liu, Liuqun Zhai, Junyi Wang +3On-Device Language Model InferenceKV-Cache Offloading

    Department of Computer Science, City University of Hong Kong, Hong Kong SAR.

  2. Predictive Multi-Tier Memory Management for KV Cache in Large-Scale GPU Inference

    Apr 19, 2026Sanjeev Rao GanjihalKV-Cache OffloadingKV Caching

    Independent Researcher

  3. IceCache: Memory-efficient KV-cache Management for Long-Sequence LLMs

    Apr 12, 2026Yuzhen Mao, Qitong Wang, Martin Ester +1KV-Cache ManagementLLM Inference Acceleration

    Simon Fraser University, BC, Canada · Harvard University, Cambridge, USA

  4. KV Cache Offloading for Context-Intensive Tasks

    Apr 9, 2026Andrey Bocharnikov, Ivan Ermakov, Denis Kuznedelev +2KV-Cache OffloadingKV Caching

    HSE, Yandex · Yandex · Yandex, NSU

  5. AsyncTLS: Efficient Generative LLM Inference with Asynchronous Two-level Sparse Attention

    Apr 9, 2026Yuxuan Hu, Jianchao Tan, Jiaqi Zhang +7LLM Inference AccelerationLong-Context Language Model Inference

    School of Information, Renmin University of China, Beijing, China · Meituan, Beijing, China

  6. Comparative Characterization of KV Cache Management Strategies for LLM Inference

    Apr 6, 2026Oteo Mamo, Olga Kogiou, Hyunjin Yi +1KV-Cache ManagementLLM Inference Efficiency

    Florida State University, Tallahassee, FL, USA

  7. ScoutAttention: Efficient KV Cache Offloading via Layer-Ahead CPU Pre-computation for LLM Inference

    Mar 28, 2026Qiuyang Zhang, Kai Zhou, Ding Tang +5KV-Cache OffloadingLLM Inference Acceleration

    Huazhong University of Science and Technology, Wuhan, China · Huawei Technologies, Hefei, China · Research Center for High Efficiency Computing Infrastructure, Zhejiang Lab, Hangzhou, China

  8. TTKV: Temporal-Tiered KV Cache for Long-Context LLM Inference

    Mar 27, 2026Gradwell Dzikanyanga, Weihao Yang, Hao Huang +4KV-Cache ManagementLong-Context Language Model Inference

    Harbin Institute of Technology, Shenzhen · Guangzhou University

  9. KV Cache Optimization Strategies for Scalable and Efficient LLM Inference

    Mar 20, 2026Yichun Xu, Navjot K. Khaira, Tejinder SinghKV-Cache ManagementLLM Inference Efficiency

    Dell Technologies, Hopkinton, MA 01748, USA · Dell Technologies, Santa Clara, CA 95054, USA

  10. Agent Memory Below the Prompt: Persistent Q4 KV Cache for Multi-Agent LLM Inference on Edge Devices

    Feb 17, 2026Yakov Pyotr ShkolnikovMemory Management for LLM AgentsLLM Agent Memory

  11. NOSA: Native and Offloadable Sparse Attention

    Oct 15, 2025Yuxiang Huang, Pengjie Wang, Jicheng Han +9KV-Cache OffloadingGPU Acceleration

    NLP Group, DCST, IAI, BNRIST, Tsinghua University, Beijing, China. · OpenBMB, China. · BUPT, Beijing, China. +1

  12. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving

    Jun 24, 2024Ruoyu Qin, Zheming Li, Weiran He +4LLM ServingDisaggregated LLM Serving

    Moonshot AI · Tsinghua University