KV Cache

KV cache, a crucial component in large language model (LLM) inference, aims to accelerate processing by storing previously computed key-value pairs, thereby reducing computational complexity. Current research focuses on optimizing KV cache efficiency through various compression techniques, including quantization, low-rank projection, and selective token eviction, often guided by attention weight analysis and adaptive budget allocation strategies. These advancements are vital for enabling efficient inference of LLMs with expanding context windows, impacting both the scalability of LLM applications and the resource requirements for deploying these powerful models.

31papers

Papers

April 2, 2025

Scaling Test-Time Inference with Policy-Optimized, Dynamic Retrieval-Augmented Generation via KV Caching and Decoding
Sakhinana Sagar Srinivas, Venkataramana Runkana
Bangalore
KV Cache Large Language Model Retrieval Augmented Generation Open Domain Question Answering

March 25, 2025

LogQuant: Log-Distributed 2-Bit Quantization of KV Cache with Superior Accuracy Preservation
Han Chen, Zicong Jiang, Zining Zhang, Bingsheng He, Pingyi Luo, Mian Lu, Yuqiang Chen
National University of Singapore●South China University of Technology●4Paradigm
Low Bit Quantization Large Language Model New Filtering KV Cache Memory Stability

March 24, 2025

March 23, 2025

WindowKV: Task-Adaptive Group-Wise KV Cache Window Selection for Efficient LLM Inference
Youhui Zuo, Sibo Wei, Chen Zhang, Zhuorui Liu, Wenpeng Lu, Dawei Song
Beijing Institute of Technology●Qilu University of Technology (Shandong Academy of Sciences)
KV Cache LLM Inference Long Context

March 20, 2025

SpeCache: Speculative Key-Value Caching for Efficient Generation of LLMs
Shibo Jie, Yehui Tang, Kai Han, Zhi-Hong Deng, Jing Han
Peking University●Huawei Noah’s Ark Lab
DRAM Access Energy KV Cache Compression Large Language Model Efficient Generation Transformer Based Large Language Model Speculative Execution KV Cache

March 17, 2025

Mitigating KV Cache Competition to Enhance User Experience in LLM Inference
Haiying Shen, Tanmoy Sen
University of Virginia
KV Cache Large Language Model User Experience Scientific Inference Minimum Latency

March 16, 2025

CAKE: Cascading and Adaptive KV Cache Eviction with Layer Preferences
Ziran Qin, Yuchen Cao, Mingbao Lin, Wen Hu, Shixuan Fan, Ke Cheng, Weiyao Lin, Jianguo Li
Shanghai Jiao Tong University●Ant Group●Independent Researcher
Key Value Cache Cascade Model Layer Selection Memory Constraint KV Cache Forward Caching

March 14, 2025

X-EcoMLA: Upcycling Pre-Trained Attention into MLA for Efficient and Extreme KV Compression
Guihong Li, Mehdi Rezagholizadeh, Mingyu Yang, Vikram Appia, Emad Barsoum
Inc. (AMD)
Multi Head Attention Feature Wise Attention KV Cache KV Cache Compression High Efficiency Transformer Attention

March 13, 2025

KV-Distill: Nearly Lossless Learnable Context Compression for LLMs
Vivek Chari, Guanghui Qin, Benjamin Van Durme
Johns Hopkins University●Microsoft
Context Compression Medical LLM Lossless Performance Long Context Sequence to Sequence Task KV Cache

March 1, 2025

Streaming Video Question-Answering with In-context Video KV-Cache Retrieval
Shangzhe Di, Zhelun Yu, Guanghao Zhang, Haoyuan Li, Tao Zhong, Hao Cheng, Bolin Li, Wanggui He, Fangxun Shu, Hao Jiang
Shanghai Jiao Tong University●Alibaba Group
Multiple Choice VideoQA KV Cache Large Language Model Video Streaming VideoQA Model App to App Retrieval

February 24, 2025

February 21, 2025

CoKV: Optimizing KV Cache Allocation via Cooperative Game
Qiheng Sun, Hongwei Zhang, Haocheng Xia, Jiayao Zhang, Jinfei Liu, Kui Ren
Model Inference Cooperative Game Large Language Model KV Cache Long Input Scroll Benchmark Model Performance

February 19, 2025

February 18, 2025

A²ATS: Retrieval-Based KV Cache Reduction via Windowed Rotary Position Embedding and Query-Aware Vector Quantization
Junhui He, Junna Xing, Nan Wang, Rui Xu, Shangyu Wu, Peng Zhou, Qiang Liu, Chun Jason Xue, Qingan Li
Wuhan University●Alibaba Cloud Computing●Jinan University●City University of Hong Kong●MBZUAI
KV Cache Large Language Model Vector Quantization Product Specific Position Information KV Cache Compression

February 6, 2025

KVTuner: Sensitivity-Aware Layer-wise Mixed Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inference
Xing Li, Zeyu Xing, Yiming Li, Linping Qu, Hui-Ling Zhen, Wulong Liu, Yiwu Yao, Sinno Jialin Pan, Mingxuan Yuan
KV Cache High Efficiency Cache Quantization Quantization Operator LLM Inference Precision Quantization

KV Cache

Papers

Scaling Test-Time Inference with Policy-Optimized, Dynamic Retrieval-Augmented Generation via KV Caching and Decoding

LogQuant: Log-Distributed 2-Bit Quantization of KV Cache with Superior Accuracy Preservation

xKV: Cross-Layer SVD for KV-Cache Compression

Oaken: Fast and Efficient LLM Serving with Online-Offline Hybrid KV Cache Quantization

WindowKV: Task-Adaptive Group-Wise KV Cache Window Selection for Efficient LLM Inference

SpeCache: Speculative Key-Value Caching for Efficient Generation of LLMs

Mitigating KV Cache Competition to Enhance User Experience in LLM Inference

CAKE: Cascading and Adaptive KV Cache Eviction with Layer Preferences

X-EcoMLA: Upcycling Pre-Trained Attention into MLA for Efficient and Extreme KV Compression

KV-Distill: Nearly Lossless Learnable Context Compression for LLMs

Streaming Video Question-Answering with In-context Video KV-Cache Retrieval

KV-Edit: Training-Free Image Editing for Precise Background Preservation

DBudgetKV: Dynamic Budget in KV Cache Compression for Ensuring Optimal Performance

KVCrush: Key value cache size-reduction using similarity in head-behaviour

CoKV: Optimizing KV Cache Allocation via Cooperative Game

RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression

ETS: Efficient Tree Search for Inference-Time Scaling

FairKV: Balancing Per-Head KV Cache for Fast Multi-GPU Inference

A²ATS: Retrieval-Based KV Cache Reduction via Windowed Rotary Position Embedding and Query-Aware Vector Quantization

KVTuner: Sensitivity-Aware Layer-wise Mixed Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inference

KV Cache

Papers

Scaling Test-Time Inference with Policy-Optimized, Dynamic Retrieval-Augmented Generation via KV Caching and Decoding

LogQuant: Log-Distributed 2-Bit Quantization of KV Cache with Superior Accuracy Preservation

xKV: Cross-Layer SVD for KV-Cache Compression

Oaken: Fast and Efficient LLM Serving with Online-Offline Hybrid KV Cache Quantization

WindowKV: Task-Adaptive Group-Wise KV Cache Window Selection for Efficient LLM Inference

SpeCache: Speculative Key-Value Caching for Efficient Generation of LLMs

Mitigating KV Cache Competition to Enhance User Experience in LLM Inference

CAKE: Cascading and Adaptive KV Cache Eviction with Layer Preferences

X-EcoMLA: Upcycling Pre-Trained Attention into MLA for Efficient and Extreme KV Compression

KV-Distill: Nearly Lossless Learnable Context Compression for LLMs

Streaming Video Question-Answering with In-context Video KV-Cache Retrieval

KV-Edit: Training-Free Image Editing for Precise Background Preservation

DBudgetKV: Dynamic Budget in KV Cache Compression for Ensuring Optimal Performance

KVCrush: Key value cache size-reduction using similarity in head-behaviour

CoKV: Optimizing KV Cache Allocation via Cooperative Game

RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression

ETS: Efficient Tree Search for Inference-Time Scaling

FairKV: Balancing Per-Head KV Cache for Fast Multi-GPU Inference

A2ATS: Retrieval-Based KV Cache Reduction via Windowed Rotary Position Embedding and Query-Aware Vector Quantization

KVTuner: Sensitivity-Aware Layer-wise Mixed Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inference

A²ATS: Retrieval-Based KV Cache Reduction via Windowed Rotary Position Embedding and Query-Aware Vector Quantization