KV-Cache Eviction

Follow this topic Ask about this topic

Key-value cache eviction studies how to shrink the tokens stored during generation so long-context models fit in limited GPU memory and serve requests faster. Common approaches rank tokens using attention, recency, value magnitude, or redundancy; newer methods estimate future or output-level utility, allocate budgets across layers and heads, or combine eviction with quantization and recoverable memory. Methods are commonly evaluated on long-context understanding, retrieval, and reasoning benchmarks such as LongBench, RULER, and Needle-in-a-Haystack, alongside latency and memory measurements. The central tension is that cheap importance signals can miss information needed later, while more accurate prediction or reconstruction may add computation; results also depend on workload, model, hardware, and the rigor of matched-budget evaluation.

Latest papers

99 papers16 papers in the last four weeks, up 7% on the four weeks before. 0.1% of all new papers.

Comparative Characterization of KV Cache Management Strategies for LLM Inference
Apr 6, 2026Oteo Mamo, Olga Kogiou, Hyunjin Yi +1KV-Cache Management LLM Inference Efficiency
Florida State University, Tallahassee, FL, USA
KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
Mar 20, 2026Yichun Xu, Navjot K. Khaira, Tejinder SinghKV-Cache Management LLM Inference Efficiency
Dell Technologies, Hopkinton, MA 01748, USA · Dell Technologies, Santa Clara, CA 95054, USA
Where Matters More Than What: Decoding-aligned KV Cache Compression via Position-aware Pseudo Queries
Mar 12, 2026Zhenxu Tian, Yi Su, Juntao Li +1KV-Cache Compression KV-Cache Eviction
School of Computer Science and Technology, Soochow University, China
LookaheadKV: Fast and Accurate KV Cache Eviction by Glimpsing into the Future without Generation
Mar 11, 2026Jinwoo Ahn, Ingyu Seong, Akhil Kedia +4KV-Cache Eviction Long-Context Language Model Inference
Samsung Research
ARKV: Adaptive and Resource-Efficient KV Cache Management under Limited Memory Budget for Long-Context Inference in LLMs
Feb 19, 2026Jianlong Lei, Shashikant IlagerKV-Cache Management Long-Context Language Model Inference
Multiscale Networked Systems (MNS) Group, Informatics Institute University of Amsterdam, The Netherlands
Learning to Evict from Key-Value Cache
Feb 10, 2026Luca Moschella, Laura Manduchi, Ozan SenerKV Caching Long-Context Language Model Inference
Apple
Fast KVzip: Efficient and Accurate LLM Inference with Gated KV Eviction
Jan 25, 2026Jang-Hyun Kim, Dongyoon Han, Sangdoo YunKV-Cache Eviction KV-Cache Compression
NAVER AI Lab.
RCStat: A Statistical Framework for using Relative Contextualization in Transformers
Jun 24, 2025Debabrata Mahapatra, Shubham Agarwal, Apoorv Saxena +1Attention Mechanisms Transformer Interpretability
Adobe Research, India
Saving GPU Hours in LLM Inference System Development and Online Workloads with Simulation and DBMS-Inspired Cache Replacement Policies
Nov 12, 2024Kyoungmin Kim, Jiacheng Li, Kijae Hong +3LLM Inference Efficiency LLM Inference
EPFL, Switzerland · NUS, Singapore · POSTECH, South Korea

Apr 6, 2026J/K move · Enter open · S save

Comparative Characterization of KV Cache Management Strategies for LLM Inference

Oteo Mamo, Olga Kogiou, Hyunjin Yi, Weikuan Yu

Florida State University, Tallahassee, FL, USA

KV-Cache Management LLM Inference Efficiency LLM Inference KV-Cache Offloading KV-Cache Eviction

Open paper AskPDF arXiv

Efficient inference with Large Language Models (LLMs) increasingly relies on Key-Value (KV) caches to store previously computed key and value vectors at each layer. These caches are essential to minimize redundant computation during autoregressive token generation, lowering computational complexity from quadratic to linear. However, the growth of KV caches has posed significant system-level challenges, particularly as model sizes increase, context lengths grow, and concurrent requests compete for limited memory resources. Even though several recent frameworks for KV cache management have emerged, their comparative trade-offs in memory consumption and inference performance have not been fully understood, especially under varying request sizes and model configurations. In this work, we conduct an empirical study of three state-of-the-art KV cache management frameworks: vLLM, InfiniGen, and H2O. These frameworks employ techniques such as tensor offloading, token eviction heuristics, and speculative scheduling to balance memory usage and performance. We evaluate their performance in terms of a range of metrics such as latency, throughput, and memory usage across a spectrum of key parameters including request rates, model sizes, and sparsity levels. Our results pinpoint the conditions for each framework to perform the best, revealing the most suitable selection and configuration of KV cache strategies under memory and performance constraints.