Overview

The field studies how to store, reuse, compress, and move the key-value states that Transformers accumulate during generation, which can dominate memory and data-transfer costs in long-context inference. Methods include token eviction and learned importance scoring, quantization and compact representations, prefix or cross-request sharing, and offloading across GPU, host memory, and storage; serving systems also coordinate these choices with scheduling and batching. Evaluations commonly measure quality on long-context retrieval and reasoning benchmarks such as LongBench and RULER alongside memory use, time to first token, and throughput. The central tension is reducing cache cost without losing information needed later: results depend on the task, workload, hardware, and attention implementation, so robust policies and faithful end-to-end evaluation remain important challenges.

Latest papers

CardsList

217 papers23 papers in the last four weeks, down 44% on the four weeks before. 0.2% of all new papers.

  1. Learning to Evict from Key-Value Cache

    Feb 10, 2026Luca Moschella, Laura Manduchi, Ozan SenerKV CachingLong-Context Language Model Inference

  2. Near-Oracle KV Selection via Pre-hoc Sparsity for Long-Context Inference

    Feb 9, 2026Yifei Gao, Lei Wang, Rong-Cheng Tu +3KV Cache SelectionSelf-Attention

  3. NOSA: Native and Offloadable Sparse Attention

    Oct 15, 2025Yuxiang Huang, Pengjie Wang, Jicheng Han +9KV-Cache OffloadingGPU Acceleration

  4. LLM Serving Optimization with Variable Prefill and Decode Lengths

    Aug 8, 2025Meixuan Wang, Yinyu Ye, Zijie ZhouLLM ServingLLM Inference Scheduling

  5. ReFreeKV: Towards Threshold-Free KV Cache Compression

    Feb 24, 2025Xuanfan Ni, Liyan Xu, Chenyang Lyu +6KV CachingMemory-Efficient Optimization

  6. Saving GPU Hours in LLM Inference System Development and Online Workloads with Simulation and DBMS-Inspired Cache Replacement Policies

    Nov 12, 2024Kyoungmin Kim, Jiacheng Li, Kijae Hong +3LLM Inference EfficiencyLLM Inference

  7. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving

    Jun 24, 2024Ruoyu Qin, Zheming Li, Weiran He +4LLM ServingDisaggregated LLM Serving