KV-Cache Management

Follow this topic Ask about this topic

The field studies how to store, reuse, compress, and move the key-value states that Transformers accumulate during generation, which can dominate memory and data-transfer costs in long-context inference. Methods include token eviction and learned importance scoring, quantization and compact representations, prefix or cross-request sharing, and offloading across GPU, host memory, and storage; serving systems also coordinate these choices with scheduling and batching. Evaluations commonly measure quality on long-context retrieval and reasoning benchmarks such as LongBench and RULER alongside memory use, time to first token, and throughput. The central tension is reducing cache cost without losing information needed later: results depend on the task, workload, hardware, and attention implementation, so robust policies and faithful end-to-end evaluation remain important challenges.

Latest papers

201 papers36 papers in the last four weeks, up 13% on the four weeks before. 0.2% of all new papers.

MKA: Memory-Keyed Attention for Efficient Long-Context Reasoning
Mar 21, 2026Dong Liu, Yanxuan Yu, Ben Lengerich +1Attention Mechanisms Memory-Efficient Attention
University of California, Los Angeles, Los Angeles, California, USA · Columbia University, New York, New York, USA · University of Wisconsin-Madison, Madison, Wisconsin, USA
KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
Mar 20, 2026Yichun Xu, Navjot K. Khaira, Tejinder SinghKV-Cache Management LLM Inference Efficiency
Dell Technologies, Hopkinton, MA 01748, USA · Dell Technologies, Santa Clara, CA 95054, USA
WAND: Windowed Attention and Knowledge Distillation for Efficient Autoregressive Text-to-Speech Models
Mar 17, 2026Hanna Lee, Tan Dat Nguyen, Jaehoon Kang +1TTS Synthesis Self-Attention
Korea Advanced Institute of Science and Technology, Republic of Korea · Sungkyunkwan University, Republic of Korea
Learning to Forget: Sleep-Inspired Memory Consolidation for Resolving Proactive Interference in Large Language Models
Mar 15, 2026Ying XieKV-Cache Management Knowledge Conflicts in Language Models
Kennesaw State University
OxyGen: Unified KV Cache Management for VLA Inference under Multi-Task Parallelism
Mar 15, 2026Xiangyu Li, Huaizhi Tang, Xin Ding +3Efficient VLA Model Inference KV-Cache Management
Institute for AI Industry Research (AIR), Tsinghua University · Department of Electronic Engineering, Tsinghua University · University of Science and Technology of China
Dynamic Sparse Attention: Access Patterns and Architecture
Mar 13, 2026Noam LevyLLM Inference Acceleration Sparse Attention
Intel Corporation.
LycheeCluster: Efficient Long-Context Inference with Structure-Aware Chunking and Hierarchical KV Indexing
Mar 9, 2026Dongfang Li, Zixuan Liu, Gang Lin +2KV-Cache Management Long-Context Language Model Inference
Harbin Institute of Technology · University of Electronic Science and Technology of China
InfoFlow KV: Information-Flow-Aware KV Recomputation for Long Context
Mar 5, 2026Xin Teng, Canyu Zhang, Shaoyi Zheng +3Retrieval-Augmented Generation KV Caching
New York University · New York University Shanghai · Duke University +1
ICaRus: Identical Cache Reuse for Efficient Multi Model Inference
Feb 27, 2026Sunghyeon Woo, Jaeeun Kil, Hoseung Kim +9LLM Inference Acceleration Memory-Efficient Inference
NAVER Cloud
KEEP: A KV-Cache-Centric Memory Management System for Efficient Embodied Planning
Feb 27, 2026Zebin Yang, Tong Xie, Baotong Lu +3Agent Memory Management Memory-Augmented Language Models
Institute for Artificial Intelligence · School of Integrated Circuits · Microsoft Research +2
Going Down Memory Lane: Scaling Tokens for Video Stream Understanding with Dynamic KV-Cache Memory
Feb 20, 2026Vatsal Agarwal, Saksham Suri, Matthew Gwilliam +2Streaming Video Understanding Memory-Augmented VLMs
University of Maryland, College Park · TikTok
ARKV: Adaptive and Resource-Efficient KV Cache Management under Limited Memory Budget for Long-Context Inference in LLMs
Feb 19, 2026Jianlong Lei, Shashikant IlagerKV-Cache Management Long-Context Language Model Inference
Multiscale Networked Systems (MNS) Group, Informatics Institute University of Amsterdam, The Netherlands
Fast KV Compaction via Attention Matching
Feb 18, 2026Adam Zweiger, Xinghong Fu, Han Guo +1KV-Cache Management LLM Context Compression
Massachusetts Institute of Technology.
ThunderAgent: A Simple, Fast and Program-Aware Agentic Inference System
Feb 14, 2026Hao Kang, Ziyang Li, Weili Xu +7LLM Agent Orchestration LLM Inference Scheduling
University of Illinois Urbana-Champaign · 5Together AI · 1Georgia Institute of Technology
Learning to Evict from Key-Value Cache
Feb 10, 2026Luca Moschella, Laura Manduchi, Ozan SenerKV Caching Long-Context Language Model Inference
Apple
Near-Oracle KV Selection via Pre-hoc Sparsity for Long-Context Inference
Feb 9, 2026Yifei Gao, Lei Wang, Rong-Cheng Tu +3Self-Attention KV Caching
University College London, London, UK · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences (CAS), Shenzhen 518055, China · The Chinese University of Hong Kong +1
NOSA: Native and Offloadable Sparse Attention
Oct 15, 2025Yuxiang Huang, Pengjie Wang, Jicheng Han +9KV-Cache Offloading GPU Acceleration
NLP Group, DCST, IAI, BNRIST, Tsinghua University, Beijing, China. · OpenBMB, China. · BUPT, Beijing, China. +1
LLM Serving Optimization with Variable Prefill and Decode Lengths
Aug 8, 2025Meixuan Wang, Yinyu Ye, Zijie ZhouLLM Serving LLM Inference Scheduling
Department of Computer Science and Technology, Tsinghua University · Department of Management Science and Engineering, Stanford University & Department of Industrial Engineering and Decision Analytics, HKUST · Department of Industrial Engineering and Decision Analytics, HKUST
ReFreeKV: Towards Threshold-Free KV Cache Compression
Feb 24, 2025Xuanfan Ni, Liyan Xu, Chenyang Lyu +6KV Caching Memory-Efficient Optimization
Nanjing University of Aeronautics and Astronautics · WeChat AI, Tencent · Independent Researcher +1
Saving GPU Hours in LLM Inference System Development and Online Workloads with Simulation and DBMS-Inspired Cache Replacement Policies
Nov 12, 2024Kyoungmin Kim, Jiacheng Li, Kijae Hong +3LLM Inference Efficiency LLM Inference
EPFL, Switzerland · NUS, Singapore · POSTECH, South Korea
Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving
Jun 24, 2024Ruoyu Qin, Zheming Li, Weiran He +4LLM Serving Disaggregated LLM Serving
Moonshot AI · Tsinghua University

Mar 21, 2026J/K move · Enter open · S save

MKA: Memory-Keyed Attention for Efficient Long-Context Reasoning

Dong Liu, Yanxuan Yu, Ben Lengerich, Ying Nian Wu

University of California, Los Angeles, Los Angeles, California, USA · Columbia University, New York, New York, USA · University of Wisconsin-Madison, Madison, Wisconsin, USA

Attention Mechanisms Memory-Efficient Attention Long-Context Language Modeling KV-Cache Management LLM Inference Acceleration Memory-Augmented Language Models

Open paper AskPDF arXiv

As long-context language modeling becomes increasingly important, the cost of maintaining and attending to large Key/Value (KV) caches grows rapidly, becoming a major bottleneck in both training and inference. While prior works such as Multi-Query Attention (MQA) and Multi-Latent Attention (MLA) reduce memory by sharing or compressing KV features, they often trade off representation quality or incur runtime overhead. We propose Memory-Keyed Attention (MKA), a hierarchical attention mechanism that integrates multi-level KV caches (local, session, and long-term) and learns to route attention across them dynamically. We further introduce Route-Fused MKA (FastMKA), a broadcast-routed variant that fuses memory sources before attention computation for improved efficiency. Experiments on different sequence lengths show that FastMKA achieves a favorable accuracy-efficiency trade-off: comparable perplexity to MLA while achieving up to 5x faster training throughput and 1.8x lower evaluation latency. These results highlight MKA as a practical and extensible framework for efficient long-context attention.