KV-Cache Management

Follow this topic Ask about this topic

The field studies how to store, reuse, compress, and move the key-value states that Transformers accumulate during generation, which can dominate memory and data-transfer costs in long-context inference. Methods include token eviction and learned importance scoring, quantization and compact representations, prefix or cross-request sharing, and offloading across GPU, host memory, and storage; serving systems also coordinate these choices with scheduling and batching. Evaluations commonly measure quality on long-context retrieval and reasoning benchmarks such as LongBench and RULER alongside memory use, time to first token, and throughput. The central tension is reducing cache cost without losing information needed later: results depend on the task, workload, hardware, and attention implementation, so robust policies and faithful end-to-end evaluation remain important challenges.

Latest papers

201 papers36 papers in the last four weeks, up 13% on the four weeks before. 0.2% of all new papers.

AdapShot: Adaptive Many-Shot In-Context Learning with Semantic-Aware KV Cache Reuse
May 5, 2026Jie Ou, Jinyu Guo, Shiyao Guo +5Long-Context Language Model Inference KV-Cache Management
School of Information and Software Engineering, University of Electronic Science and Technology of China
Stochastic Sparse Attention for Memory-Bound Inference
May 3, 2026Kyle Lee, Corentin Delacour, Kevin Callahan-Coray +5GPU Acceleration Self-Attention
Department of Electrical and Computer Engineering, University of California, Santa Barbara, CA, USA · Department of Electrical and Computer Engineering, University of Michigan, Ann Arbor, MI, USA · Department of Electrical and Computer Engineering, Carnegie Mellon University, Pittsburgh, PA, USA
Decouple and Cache: KV Cache Construction for Streaming Video Understanding
May 3, 2026Zhanzhong Pang, Dibyadip Chatterjee, Fadime Sener +1Streaming Video Understanding KV Caching
National University of Singapore.
Benchmarking KV-Cache Optimizations across Task Quality and System Performance for Long-Context Serving
May 3, 2026Nikita Agrawal, Ruben MayerLLM Serving KV Caching
University of Bayreuth
Motion-Aware Caching for Efficient Autoregressive Video Generation
May 3, 2026Jing Xu, Yuexiao Ma, Xuzhe Zheng +7Video Diffusion Models KV-Cache Management
1Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China · 2ByteDance · 3Max Planck Institute for Intelligent Systems, 4ELLIS Institute Tbingen, 5Tbingen AI Center
Unifying Sparse Attention with Hierarchical Memory for Scalable Long-Context LLM Serving
Apr 29, 2026Zihan Zhao, Baotong Lu, Shengjie Lin +8LLM Serving KV-Cache Offloading
University of Virginia · Microsoft Research · Georgia Institute of Technology +1
DUAL-BLADE: Dual-Path NVMe-Direct KV-Cache Offloading for Edge LLM Inference
Apr 29, 2026Bodon Jeong, Hongsu Byun, Youngjae Kim +4On-Device Language Model Inference KV-Cache Offloading
1Sogang University, Seoul, Republic of Korea · 2Florida State University, FL, USA · 3Samsung Electronics Co.
DepthKV: Layer-Dependent KV Cache Pruning for Long-Context LLM Inference
Apr 27, 2026Zahra Dehghanighobadi, Asja FischerLLM Pruning KV Caching
1Ruhr University Bochum · 2UAR Research Center for Trustworthy Data Science and Security
MTServe: Efficient Serving for Generative Recommendation Models with Hierarchical Caches
Apr 24, 2026Xin Wang, Chi Ma, Shaobin Chen +14KV-Cache Offloading KV Caching
School of Computer Science, Wuhan University · Meituan · Nvidia
SparKV: Overhead-Aware KV Cache Loading for Efficient On-Device LLM Inference
Apr 23, 2026Hongyao Liu, Liuqun Zhai, Junyi Wang +3On-Device Language Model Inference KV-Cache Offloading
Department of Computer Science, City University of Hong Kong, Hong Kong SAR.
LKV: End-to-End Learning of Head-wise Budgets and Token Selection for LLM KV Cache Eviction
Apr 22, 2026Enshuai Zhou, Yifan Hao, Chao Wang +7KV Caching Long-Context Language Model Inference
University of Science and Technology of China, Hefei, China · State Key Lab of Processors, Institute of Computing Technology, CAS, Beijing, China · University of Chinese Academy of Sciences, Beijing, China
River-LLM: Large Language Model Seamless Exit Based on KV Share
Apr 20, 2026Yingtao Shen, An ZouDecoder-Only Language Models KV-Cache Management
Shanghai Jiao Tong University Shanghai, China
How Much Cache Does Reasoning Need? Depth-Cache Tradeoffs in KV-Compressed Transformers
Apr 20, 2026Xiao WangTransformer Inference KV Caching
Predictive Multi-Tier Memory Management for KV Cache in Large-Scale GPU Inference
Apr 19, 2026Sanjeev Rao GanjihalKV-Cache Offloading KV Caching
Independent Researcher
Sparse Prefix Caching for Hybrid and Recurrent LLM Serving
Apr 17, 2026Mikhail Shirokikh, Sergey NikolenkoLLM Serving KV-Cache Management
ART: Attention Run-time Termination for Efficient Large Language Model Decoding
Apr 15, 2026Chen Qiu, Guozhong Li, Cristian McGee +2Long-Context Language Model Inference LLM Inference Acceleration
King Abdullah University of Science and Technology · University of Central Florida
PipeLive: Efficient Live In-place Pipeline Parallelism Reconfiguration for Dynamic LLM Serving
Apr 14, 2026Xu Bai, Muhammed Tawfiqul Islam, Chen Wang +1LLM Inference Efficiency LLM Serving
DisNet Lab University of Melbourne Melbourne, VIC, Australia · IBM Research Yorktown Heights, NY, USA
Transactional Attention: Semantic Sponsorship for KV-Cache Retention
Apr 13, 2026Abhinaba BasuKV-Cache Management KV-Cache Eviction
National Institute of Electronics and Information Technology (NIELIT) · Indian Institute of Information Technology, Allahabad (IIITA)
IceCache: Memory-efficient KV-cache Management for Long-Sequence LLMs
Apr 12, 2026Yuzhen Mao, Qitong Wang, Martin Ester +1KV-Cache Management LLM Inference Acceleration
Simon Fraser University, BC, Canada · Harvard University, Cambridge, USA
LoopGuard: Breaking Self-Reinforcing Attention Loops via Dynamic KV Cache Intervention
Apr 11, 2026Dongjie Xu, Hao Wu, Weijie Shi +7Repetition in Language Generation KV-Cache Management
Soochow University · Tongji University · The Hong Kong University of Science and Technology +2
KV Cache Offloading for Context-Intensive Tasks
Apr 9, 2026Andrey Bocharnikov, Ivan Ermakov, Denis Kuznedelev +2KV-Cache Offloading KV Caching
HSE, Yandex · Yandex · Yandex, NSU
Comparative Characterization of KV Cache Management Strategies for LLM Inference
Apr 6, 2026Oteo Mamo, Olga Kogiou, Hyunjin Yi +1KV-Cache Management LLM Inference Efficiency
Florida State University, Tallahassee, FL, USA
Stochastic KV Routing: Enabling Adaptive Depth-Wise Cache Sharing
Apr 3, 2026Anastasiia Filippova, David Grangier, Marco Cuturi +1KV-Cache Management Memory-Efficient Inference
Apple
QCFuse: Query-Centric Cache Fusion for Efficient RAG Inference
Mar 30, 2026Jianxin Yan, Zeheng Qian, Wangze Ni +6LLM Inference Acceleration KV-Cache Management
Zhejiang University, Hangzhou, China · The University of Sydney, Sydney, Australia · Ant Group, Shanghai, Cina +4
KVSculpt: KV Cache Compression as Distillation
Mar 29, 2026Bo Jiang, Sian JinKV-Cache Management KV-Cache Compression
Temple University
TTKV: Temporal-Tiered KV Cache for Long-Context LLM Inference
Mar 27, 2026Gradwell Dzikanyanga, Weihao Yang, Hao Huang +4KV-Cache Management Long-Context Language Model Inference
Harbin Institute of Technology, Shenzhen · Guangzhou University
GhostServe: A Lightweight Checkpointing System in the Shadow for Fault-Tolerant LLM Serving
Mar 26, 2026Shakya Jayakody, Youpeng Zhao, Chinmay Dhanraj Nehate +1LLM Serving LLM Inference
University of Central Florida, Orlando, FL, USA.
PRISM: Breaking the O(n) Memory Wall in Long-Context LLM Inference via O(1) Photonic Block Selection
Mar 23, 2026Hyoseok Park, Yeonsang ParkLLM Inference Acceleration Long-Context LLM Inference
Department of Physics, Chungnam National University, Daejeon 34134, Republic of Korea
Relax Forcing: Relaxed KV-Memory for Consistent Long Video Generation
Mar 22, 2026Zengqun Zhao, Yanzuo Lu, Ziquan Liu +3Long-Horizon Video Generation Autoregressive Video Generation
Queen Mary University of London · Imperial College London · Huawei R&D UK
Beyond Token Eviction: Mixed-Dimension Budget Allocation for Efficient KV Cache Compression
Mar 21, 2026Ruijie Miao, Zhiming Wang, Wang Li +4KV-Cache Management KV-Cache Compression
Peking University · ByteDance Seed

May 5, 2026J/K move · Enter open · S save

AdapShot: Adaptive Many-Shot In-Context Learning with Semantic-Aware KV Cache Reuse

Jie Ou, Jinyu Guo, Shiyao Guo, Yuang Li+4

School of Information and Software Engineering, University of Electronic Science and Technology of China

Long-Context Language Model Inference KV-Cache Management LLM Inference Acceleration In-Context Learning

Open paper AskPDF arXiv

Many-Shot In-Context Learning (ICL) has emerged as a promising paradigm, leveraging extensive examples to unlock the reasoning potential of Large Language Models (LLMs). However, existing methods typically rely on a predetermined, fixed number of shots. This static approach often fails to adapt to the varying difficulty of different queries, leading to either insufficient context or interference from noise. Furthermore, the prohibitive computational and memory costs of long contexts severely limit Many-Shot's feasibility. To address the above limitations, we propose AdapShot, which dynamically optimizes shot counts and leverages KV cache reuse for efficient inference. Specifically, we design a probe-based evaluation mechanism that utilizes output entropy to determine the optimal number of shots. To bypass the redundant prefilling computation during both the probing and inference phases, we incorporate a semantics-aware KV cache reuse strategy. Within this reuse strategy, to address positional encoding incompatibilities, we introduce a decoupling and re-encoding method that enables the flexible reordering of cached key-value pairs. Extensive experiments demonstrate that AdapShot achieves an average performance gain of around 10% and a 4.64x speedup compared to state-of-the-art DBSA.