KV-Cache Compression

Follow this topic Ask about this topic

Key-value cache compression reduces the memory and bandwidth cost of storing attention states, enabling longer contexts, larger batches, and inference on constrained hardware. Common approaches include token eviction and merging, quantization, low-rank or latent representations, and adaptive per-layer or per-head budgets; some methods also reconstruct or retrieve compressed states on demand. Work is evaluated on long-context retrieval and question answering benchmarks such as LongBench, RULER, and Needle-in-a-Haystack, as well as reasoning, multimodal tasks, and end-to-end serving. The central tension is that aggressive compression can harm accuracy, evidence use, or safety, while a high compression ratio does not guarantee faster inference or lower peak memory, so workload-aware evaluation remains essential.

Latest papers

269 papers49 papers in the last four weeks, up 32% on the four weeks before. 0.3% of all new papers.

Adaptive Mass-Segmented KV Compression for Long-Context Reasoning
May 22, 2026Junzhe Yang, Xiaoyu ShenKV Caching Long-Context Language Model Inference
Shanghai Jiao Tong University · Institute of Digital Twin, Eastern Institute of Technology, Ningbo
WorldKV: Efficient World Memory with World Retrieval and Compression
May 21, 2026Jung Yi, Minjae Kim, Paul Hyunbin Cho +3KV Caching Memory-Augmented Video Generation
1KAIST AI · 2Naver AI Lab
Meta-Soft: Leveraging Composable Meta-Tokens for Context-Preserving KV Cache Compression
May 21, 2026Wei Luo, Yi Huang, Songchen Ma +3KV Caching KV-Cache Eviction
Guangdong Institute of Intelligence Science and Technology · University of Macau · Chinese University of Hong Kong, Shenzhen +1
MuKV: Multi-Grained KV Cache Compression for Long Streaming Video Question-Answering
May 21, 2026Junbin Xiao, Jiajun Chen, Tianxiang Sun +2KV Caching Video QA
University of Science and Technology of China · National University of Singapore
EntmaxKV: Support-Aware Decoding for Entmax Attention
May 20, 2026Gonçalo Duarte, Miguel Couceiro, Marcos V. TrevisoSelf-Attention KV Caching
Instituto Superior Técnico, Universidade de Lisboa. · ELLIS Unit Lisbon. · INESC-ID. +1
OCTOPUS: Optimized KV Cache for Transformers via Octahedral Parametrization Under optimal Squared error quantization
May 20, 2026Mark Boss, Vikram Voleti, Simon Donné +1KV Caching Memory-Efficient Optimization
Stability AI
Head-Aware Key-Value Compression for Efficient Autoregressive Image Generation
May 20, 2026Guotao Liang, Baoquan Zhang, Zhiyuan Wen +1KV Caching Autoregressive Image Generation
Harbin Institute of Technology, Shenzhen · Peng Cheng Laboratory
Latent Cache Flow: Model-to-Model Communication Without Text
May 19, 2026Maximillian Rossi, Prajwal Raghunath, Eugene WuKV Caching KV-Cache Compression
Columbia University
Rotation-Aligned Key Channel Pruning for Efficient Vision-Language Model Inference
May 19, 2026Beomseok Kang, Dongwon Jo, Jiwon Song +2Vision-Language Models Efficient VLM Inference
Seoul National University
Focused Forcing: Content-Aware Per-Frame KV Selection for Efficient Autoregressive Video Diffusion
May 18, 2026Peiliang Cai, Evelyn Zhang, Jiacheng Liu +8Video Diffusion Models Autoregressive Diffusion
1SJTU · 2SDU · 3HUST +4
VeriCache: Turning Lossy KV Cache into Lossless LLM Inference
May 17, 2026Jiayi Yao, Samuel Shen, Kuntai Du +7KV-Cache Offloading KV Caching
University of Chicago · Tensormesh Inc. · Samsung Semiconductor +1
GHOST: Geometry-Hierarchical Online Streaming Token Eviction for Efficient 3D Reconstruction
May 15, 2026Leyang Chen, Junyi Wu, Zhiteng Li +1Streaming 3D Reconstruction KV Caching
Shanghai Jiao Tong University
KVCapsule: Efficient Sequential KV Cache Compression for Vision-Language Models with Asymmetric Redundancy
May 14, 2026Yingbing Huang, Tharun Adithya Srikrishnan, Steven K. Reinhardt +1Vision-Language Models Efficient VLM Inference
University of Illinois Urbana-Champaign · AMD
GQLA: Group-Query Latent Attention for Hardware-Adaptive Decoding
May 14, 2026Fanxu Meng, Zhixin PanGrouped-Query Attention GPU Acceleration
Institute for Artificial Intelligence, Peking University
HeatKV: Head-tuned KV-cache Compression for Visual Autoregressive Modeling
May 14, 2026Jonathan Cederlund, Axel Berg, William Isaksson +3KV Caching Visual Autoregressive Models
Dept. of Automatic Control, Lund University · Arm
CoRDS: Coreset-based Representative and Diverse Selection for Streaming Video Understanding
May 14, 2026Ailar Mahdizadeh, Puria Azadi, Muchen Li +2Streaming Video Understanding KV Caching
University of British Columbia · Vector Institute
Minimal-Intervention KV Retention via Set-Conditioned Diversity
May 14, 2026Libo Sun, Po-wei Harn, Peixiong He +1KV Caching KV-Cache Eviction
Department of Computer Science and Software Engineering Auburn University, Auburn, AL, USA · Department of Information Management National Central University, Taoyuan, Taiwan
Self-Pruned Key-Value Attention: Learning When to Write by Predicting Future Utility
May 13, 2026Gergely Szilvasy, Manuel Faysse, Maria Lomeli +5Self-Attention KV Caching
Meta FAIR · MICS, CentraleSupélec
SPHERICAL KV: Angle-Domain Attention and Rate-Distortion Retention for Efficient Long-Context Inference
May 13, 2026Anay Chauhan, Gurucharan Marthi Krishna Kumar, Arion Das +4KV Caching Long-Context Language Model Inference
1Synopsys, India · 2McGill University, Canada · 3IIIT Ranchi, India +4
KVServe: Service-Aware KV Cache Compression for Communication-Efficient Disaggregated LLM Serving
May 13, 2026Zedong Liu, Xinyang Ma, Dejun Luo +9LLM Serving KV Caching
University of Chinese Academy of Sciences · Institute of Computing Technology, Chinese Academy of Sciences · Shanghai Jiao Tong University
FibQuant: Universal Vector Quantization for Random-Access KV-Cache Compression
May 12, 2026Namyoon Lee, Yongjune KimVector Quantization Low-Bit Quantization
POSTECH
Forcing-KV: Hybrid KV Cache Compression for Efficient Autoregressive Video Diffusion Models
May 10, 2026Yicheng Ji, Zhizhou Zhong, Jun Zhang +7Diffusion Model Caching KV-Cache Compression
ZJU · Video Rebirth · HKUST +1
RDKV: Rate-Distortion Bit Allocation for Joint Eviction and Quantization of the KV Cache
May 8, 2026Junkai Zhang, Hang Guo, Luca Benini +1KV Caching Long-Context Language Model Inference
ETH Zurich · Tsinghua University
Reformulating KV Cache Eviction Problem for Long-Context LLM Inference
May 8, 2026Tho Mai, Joo-Young KimKV Caching Long-Context Language Model Inference
KAIST Daejeon, South Korea
How to Compress KV Cache in RL Post-Training? Shadow Mask Distillation for Memory-Efficient Alignment
May 7, 2026Rui Zhu, Weiheng Bai, Qiushi Wu +3LLM Alignment KV-Cache Compression
Yale University · University of Minnesota Twin Cities · Indiana University Bloomington
Training Transformers for KV Cache Compressibility
May 7, 2026Yoav Gelberg, Yam Eitan, Michael Bronstein +2Long-Context Language Modeling KV-Cache Compression
University of Oxford · 2Technion – Israel Institute of Technology · 3AITHYRA +1
When Quantization Is Free: An int4 KV Cache That Outruns fp16 on Apple Silicon
May 7, 2026Mohamed Amine BergachGPU Kernel Optimization KV Caching
Illumina, San Diego, CA, USA
When Does Value-Aware KV Eviction Help? A Fixed-Contract Diagnostic for Non-Monotone Cache Compression
May 7, 2026Ruijie Zhang, Haozhe Liang, Da Chang +4KV Caching Long-Context Language Model Inference
HeadQ: Model-Visible Distortion and Score-Space Correction for KV-Cache Quantization
May 5, 2026Jorge L. Ruiz WilliamsSelf-Attention KV Caching
Independent Researcher
Benchmarking KV-Cache Optimizations across Task Quality and System Performance for Long-Context Serving
May 3, 2026Nikita Agrawal, Ruben MayerLLM Serving KV Caching
University of Bayreuth

May 22, 2026J/K move · Enter open · S save

Adaptive Mass-Segmented KV Compression for Long-Context Reasoning

Junzhe Yang, Xiaoyu Shen

Shanghai Jiao Tong University · Institute of Digital Twin, Eastern Institute of Technology, Ningbo

KV Caching Long-Context Language Model Inference Memory-Efficient Inference KV-Cache Compression Long-Context Inference KV-Cache Pruning

Open paper AskPDF arXiv

The linear growth of the Key-Value (KV) cache is a critical bottleneck in long-form LLM inference. Existing KV compression methods mitigate this by evicting tokens based on importance scores. However, we show that their reliance on global Top-k selection triggers Region Wipe-out: the severe eviction of contiguous reasoning blocks that derails logical coherence. To address this, we propose Adaptive Mass-Segmented (AMS) KV Compression, a framework that shifts the paradigm from token-level competition to region-aware quota allocation. AMS adaptively partitions the KV cache based on the spatial distribution of attention mass, ensuring structurally vital reasoning segments receive guaranteed memory quotas. To ensure stability during iterative decoding, an EMA-based smoothing mechanism is incorporated to prevent jitter in segment boundaries. Crucially, AMS is a universal plug-and-play layer that is orthogonal to existing scorers. It can be seamlessly integrated into representative methods such as TOVA, Expected Attention, KeyDiff, R-KV and TriAttention. AMS is also system-compatible with modern paged-KV serving frameworks such as vLLM, supporting efficient gather-and-compact KV execution without introducing additional steady-state attention overhead. Extensive experiments across a diverse suite of tasks, including mathematical reasoning (MATH500, AIME, GSM8K), code completion, open-domain QA, and sparse retrieval, demonstrate that AMS consistently mitigates structural fragmentation and boosts model performance.