Overview

Key-value caching stores the keys and values computed for earlier tokens so a Transformer can generate new tokens without recomputing the full history. Because these caches can consume substantial memory and bandwidth in long-context and high-concurrency inference, methods include token eviction, low-bit quantization, low-rank compression, sparse retrieval, and offloading or reusing cached prefixes. Researchers evaluate these approaches on long-context retrieval, question answering, and reasoning benchmarks, as well as serving metrics such as time to first token, decode speed, and throughput. The central tension is reducing cache cost without losing useful evidence or changing model behavior, and the best trade-off often depends on the workload, model, and hardware.

Latest papers

CardsList

285 papers3 papers in the last four weeks, down 40% on the four weeks before. 0.0% of all new papers.

  1. KVPath: Connecting Key-Value Representations Across Transformer Layers

    Nov 11, 2026Xuan Luo, Zhiyuan Wang, Hanzhong Cao +1KV CachingKV-Cache Compression

  2. DepthGraft: Structural Regularization through Hierarchical Cross-Layer KV Reconstruction

    Nov 2, 2026Xiaohan Qin, Xiangdong Zhang, Yu Wang +4KV CachingLanguage Model Pretraining

  3. The KV Cache Is the New Memory Wall

    Sep 25, 2026Tejinder SinghKV CachingLong-Context Language Model Inference

  4. Bridging LLM Serving and CXL-SSDs with Chunk-Aware KV Cache Management

    Sep 20, 2026Hyunsun Chung, Taewan Noh, Minji Kim +3LLM Inference AccelerationKV-Cache Offloading

  5. PrefixBench-H100: Characterizing Prefix Reuse and Time-to-First-Token in H100 LLM Serving

    Sep 17, 2026Omkar Shewale, Deepak Kumar, Divakar Kumar YadavLLM Inference EfficiencyLLM Inference

  6. Ask the Tool, Don't Guess: Agent Tool Calls Hold Their Progress, and the Serving System Should Read It

    Sep 16, 2026Yipeng Liu, Yingqiang Zhang, Feifei Li +1LLM Inference EfficiencyKV Caching

  7. Contiguity, Not Importance: Budgeted Repair of Stale KV Caches After Document Edits

    Sep 16, 2026Mingyang Mao, Wyatt Mackey, Xiaomin LinRetrieval-Augmented GenerationKV Caching

  8. T-LoopFormer: Token-Level Elastic-Depth Looped Transformers with Dynamic Routing

    Sep 14, 2026Mingqian Yu, Wenpeng Zhang, Shaobo Cui +1KV CachingRecurrent Transformers

  9. Pixel Decodability Is Not a Compression Signal: Causally Evaluating Importance Proxies for Visual KV-Cache Eviction

    Sep 14, 2026Chenyu Zhou, Qiliang Jiang, Shuning Wu +1KV CachingKV-Cache Eviction

  10. AgentKV: Phase-Aware KV Eviction for Agentic LLMs

    Sep 14, 2026Taowen Tony Liu, Jeffrey T. H. Wong, Can Xiao +3KV CachingKV-Cache Eviction

  11. Building py-kvcache: A Performance Characterization of External KV Caching for vLLM with NVMe SSDs

    Sep 10, 2026Joseph Kanichai, Tiziano De Matteis, Animesh TrivediKV-Cache OffloadingKV Caching

  12. OmniKVQuant: KV Cache Quantization for Omni-LLMs

    Sep 10, 2026Suho Yoo, Hyunjong Ok, Jongmin Choi +2Efficient Multimodal InferenceKV Caching

  13. Composable CXL Memory as a Kubernetes-Native Shared Memory for LLM Serving

    Sep 9, 2026Hongjian Fan, Kevin Zhang, David Habinsky +1LLM Inference EfficiencyKV Caching

  14. KVShareArena: KV-Cache Reuse Across Contexts and Model Checkpoints

    Sep 9, 2026Xi Shi, Mengxin Zheng, Qian LouKV CachingLLM Inference Acceleration