Overview

The field studies how to store, reuse, compress, and move the key-value states that Transformers accumulate during generation, which can dominate memory and data-transfer costs in long-context inference. Methods include token eviction and learned importance scoring, quantization and compact representations, prefix or cross-request sharing, and offloading across GPU, host memory, and storage; serving systems also coordinate these choices with scheduling and batching. Evaluations commonly measure quality on long-context retrieval and reasoning benchmarks such as LongBench and RULER alongside memory use, time to first token, and throughput. The central tension is reducing cache cost without losing information needed later: results depend on the task, workload, hardware, and attention implementation, so robust policies and faithful end-to-end evaluation remain important challenges.

Latest papers

CardsList

217 papers23 papers in the last four weeks, down 44% on the four weeks before. 0.2% of all new papers.

  1. Keep the Reasoning, Read the Image: On-Demand Visual KV for Multi-Turn Reasoning VLMs

    Nov 18, 2026Ziyi Wang, Xiyue Zhang, Bing Hu +3Efficient VLM InferenceKV-Cache Management

  2. Making Preemption Productive in Long-Output LLM Serving

    Nov 15, 2026Ziji Shi, Xinyi Wan, Chaoyi Ruan +3LLM ServingLLM Inference Scheduling

  3. Architectures and KV Replay for Depth-Recurrent Language Models

    Nov 14, 2026Lucia Licakova, Niklas Schmidinger, Lukas Hauzenberger +2Recurrent TransformersKV-Cache Management

  4. Looping Deeper Without Growing the KV Cache: Recurrent Depth for Causal Encoder-Decoders

    Nov 9, 2026Jiaqi Leng, Minshen Zhang, Zhaofeng Sun +3LLM Inference AccelerationKV-Cache Management

  5. LooM: Loop-Invariant Memory for Causal Encoder-Decoder Language Models

    Nov 4, 2026Lele Yang, Muxi Diao, Zhonghao Yan +7Adaptive ComputationRecurrent Transformers

  6. Where Should a KV-Cache Repair Budget Go? Distractor Structure and Contextual Compatibility in Non-Prefix Reuse

    Nov 4, 2026Yuxing Peng, Zhiqing Tang, Weijia JiaKV-Cache ReuseKV-Cache Management

  7. EchoKV: Self-Corrective KV Cache Eviction via Iterative Query Feedback

    Nov 4, 2026JianKun Zou, Tongtong YuanKV-Cache EvictionKV-Cache Management

  8. BasisKV: Value Latents for Communication and Key Routing

    Nov 2, 2026Lixun Zhang, Zishan Shao, Kangning Cui +3KV Cache SelectionKV-Cache Management

  9. EarlySelectKV: Anticipatory KV Selection via Cross-Layer Query Proxies

    Nov 1, 2026Yuhao Liu, Yuehao Li, Linghe Zhang +4KV Cache SelectionLong-Context Language Model Inference

  10. BitLLM: Efficient Low-Bit, Conversion-Hiding LLM Decoding on Mobile GPUs

    Oct 31, 2026Hangyu Zheng, Zhihao Shu, Kunxiong Zhu +4On-Device Language Model InferenceLLM Inference Acceleration

  11. WAKMan: Workload Aware KVCache Manager

    Oct 26, 2026Suhas Kumar Morisetty, Vaibhavi Lokegaonkar, Sachin Mathew Varghese +1KV-Cache ManagementLanguage Model Inference Caching

  12. LLMs in Cloud: A Discussion on Architecture, Orchestration, and Inference

    Oct 24, 2026Jannatun Noor, Athar Noor Mohammad Rafee, Md. Sadman Haque +3LLM ServingLLM Inference

  13. Why Full-Duplex Speech Models Collapse in Long Conversations

    Oct 23, 2026Yeonju Kim, Bella Godiva, Se Jin Park +1Full-Duplex Spoken Dialogue SystemsLong-Context Modeling

  14. Gandalf: Flash Shall Not Die! Increasing Flash Storage Lifetime for KV Cache Offloading in LLM Serving

    Oct 22, 2026Maharani A. P. Irawan, Kuntai Du, Jiayi Yao +4KV-Cache OffloadingKV-Cache Management

  15. Cache Locally, Recompute Selectively for Streaming Video Understanding

    Oct 21, 2026Junhyeok Lee, Sangyoon Lee, Jaeho LeeStreaming Video UnderstandingStreaming Inference

  16. Cache Engineering for Agents

    Oct 20, 2026Tianyu Fu, Ruoxi Wang, Zihan Min +6KV-Cache ManagementMemory Management for LLM Agents

  17. Local Inference of LLMs on Long Contexts: A Data Systems Perspective

    Oct 20, 2026Mattias Dürrmeier, Alberto LernerLLM InferenceLong-Context Inference

  18. The KV-Surgeon: Compensation-Aware Offline KV-Cache Compaction

    Oct 19, 2026Moritz Wagner, Christophe Roux, Max Zimmer +1KV-Cache ManagementKV-Cache Pruning

  19. MOSAIC: Multi-Objective Search for Allocating Inference Cache Across LLM Decoder Layers

    Oct 18, 2026Sravanth Kodavanti, Manjunath ArvetiLanguage Model Inference CachingKV-Cache Management

  20. DLite: History-free Speculative Decoding for Agentic AI

    Oct 16, 2026Jiaming Xu, Hanzhen Wang, Tianlang Zhao +7Speculative DecodingLLM Inference Acceleration