Key Value Cache

Key-value (KV) caching is a crucial technique for accelerating large language model (LLM) inference by storing intermediate computations, but its memory consumption scales linearly with sequence length, hindering efficient deployment. Current research focuses on optimizing KV cache management through various strategies, including low-rank compression, layer-wise allocation and offloading, sliding window attention, and quantization techniques, often combined with novel attention mechanisms or model architectures like MixAttention. These advancements aim to reduce memory footprint and improve inference speed and throughput, significantly impacting the scalability and cost-effectiveness of LLMs in practical applications.

Papers

June 18, 2024

Attention Score is not All You Need for Token Importance Indicator in KV Cache Reduction: Value Also Matters
Zhiyu Guo, Hidetaka Kamigaito, Taro Watanabe
Large Language Model Net Present Value Token Pruning Key Value Cache Attention Score Token Importance

June 17, 2024

A Simple and Effective $L_2$ Norm-Based Strategy for KV Cache Compression
Alessio Devoto, Yu Zhao, Simone Scardapane, Pasquale Minervini
Large Language Model Key Value Cache Decoder Only Transformer Key Value Memory Key Value Cache Compression

June 5, 2024

QJL: 1-Bit Quantized JL Transform for KV Cache Quantization with Zero Overhead
Amir Zandieh, Majid Daliri, Insu Han
Quantization Technique Key Value Cache Cache Quantization Johnson Lindenstrauss Transform

May 23, 2024

MiniCache: KV Cache Compression in Depth Dimension for Large Language Models
Akide Liu, Jing Liu, Zizheng Pan, Yefei He, Gholamreza Haffari, Bohan Zhuang
Large Language Model Sequence Generation Key Value Cache Key Value Cache Compression Lossless Performance

May 21, 2024

Reducing Transformer Key-Value Cache Size with Cross-Layer Attention
William Brandon, Mayank Mishra, Aniruddha Nrusimha, Rameswar Panda, Jonathan Ragan Kelly
Key Value Cache Query Attention Autoregressive Large Language Model Memory Transformer Cross Layer Attention

May 17, 2024

Layer-Condensed KV Cache for Efficient Inference of Large Language Models
Haoyi Wu, Kewei Tu
Large Language Model Language Model Transformer Architecture Large Scale Language Model Efficient Inference Key Value Cache

May 8, 2024

KV-Runahead: Scalable Causal LLM Inference by Parallel Key-Value Cache Generation
Minsik Cho, Mohammad Rastegari, Devang Naik
Large Language Model LLM Inference Key Value Cache KV Cache

May 7, 2024

KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantization
Tianyi Zhang, Jonah Yi, Zhaozhuo Xu, Anshumali Shrivastava
Quantization Operator Large Language Model Inference P Bit Key Value Cache Channel Quality Key Value Cache Compression

April 22, 2024

SnapKV: LLM Knows What You are Looking for Before Generation
Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, Deming Chen
Medical LLM Faithful Generation Input Sequence Key Value Cache Long Input Snap Video

April 18, 2024

TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding
Hanshi Sun, Zhuoming Chen, Xinyu Yang, Yuandong Tian, Beidi Chen
Long Sequence Speculative Decoding Key Value Cache Neural Network Inference Lossless Acceleration

April 15, 2024

LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism
Bingyang Wu, Shengyu Liu, Yinmin Zhong, Peng Sun, Xuanzhe Liu, Xin Jin
Large Language Model Long Context Key Value Cache Level Parallelism Sequence Parallelism

April 7, 2024

SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget
Zihao Wang, Bin Cui, Shaoduo Gan
Large Language Model 2 Dimensional Large Language Model Inference Key Value Cache Key Value Cache Compression Multi Session Budget Optimization Token Sparsification

March 14, 2024

Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inference
Muhammad Adnan, Akhil Arunkumar, Gaurav Jain, Prashant J. Nair, Ilya Soloveychik, Purushotham Kamath
Text Generation Generative Modeling Generative Language Model Token Generation Key Value Cache Token Attention Efficient Generative

March 7, 2024

QAQ: Quality Adaptive Quantization for LLM KV Cache
Shichen Dong, Wen Cheng, Jiayu Qin, Wei Wang
Key Value Cache Adaptive Quantization

February 28, 2024

No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantization
June Yong Yang, Byeongwook Kim, Jeongin Bae, Beomseok Kwon, Gunho Park, Eunho Yang, Se Jung Kwon, Dongsoo Lee
K TOKEN Mixed Precision Quantization Key Value Cache KV Cache Key Value Cache Compression

February 14, 2024

Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference
Harry Dong, Xinyu Yang, Zhenyu Zhang, Zhangyang Wang, Yuejie Chi, Beidi Chen
Large Language Model LLM Inference Key Value Cache Cache Context Temporal Difference Type Recurrence Key Value Cache Compression Memory Bottleneck

February 9, 2024

On the Efficacy of Eviction Policy for Key-Value Constrained Generative Language Model Inference
Siyu Ren, Kenny Q. Zhu
Generative Modeling Generative Language Model Temporal Attention Key Value Cache Caching Strategy Eviction Prediction

February 5, 2024

KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache
Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, Xia Hu
LLM Inference Key Value Cache Kinship Verification Cache Quantization

September 12, 2023

Efficient Memory Management for Large Language Model Serving with PagedAttention
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, Ion Stoica
Large Language Model Key Value Cache Web Page KV Cache Memory Management Virtual Memory

June 9, 2023

S$^{3}$: Increasing GPU Utilization during Generative Inference for Higher Throughput
Yunho Jin, Chun-Feng Wu, David Brooks, Gu-Yeon Wei
Large Language Model Text Generation Large Model Worst Case User Throughput Key Value Cache Sequence Length Learning Generative Model

Key Value Cache

Papers

Attention Score is not All You Need for Token Importance Indicator in KV Cache Reduction: Value Also Matters

A Simple and Effective $L_2$ Norm-Based Strategy for KV Cache Compression

QJL: 1-Bit Quantized JL Transform for KV Cache Quantization with Zero Overhead

MiniCache: KV Cache Compression in Depth Dimension for Large Language Models

Reducing Transformer Key-Value Cache Size with Cross-Layer Attention

Layer-Condensed KV Cache for Efficient Inference of Large Language Models

KV-Runahead: Scalable Causal LLM Inference by Parallel Key-Value Cache Generation

KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantization

SnapKV: LLM Knows What You are Looking for Before Generation

TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding

LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism

SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget

Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inference

QAQ: Quality Adaptive Quantization for LLM KV Cache

No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantization

Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference

On the Efficacy of Eviction Policy for Key-Value Constrained Generative Language Model Inference

KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache

Efficient Memory Management for Large Language Model Serving with PagedAttention

S$^{3}$: Increasing GPU Utilization during Generative Inference for Higher Throughput