KV-Cache Compression

Follow this topic Ask about this topic

Momentum

24 papers in the last four weeks, up 71% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 166

All topics

From Rigid to Dynamic: Entropy-Guided Adaptive Inference for Long-Context LLMs
Jun 8, 2026Zhanchao Xu, Haoyang Li, Qingfa Xiao +4Long-Context Language Model Inference LLM Inference Acceleration
Department of Computing, PolyU · DSA, HKUST(GZ) · CSE, HKUST
FlashMemory-DeepSeek-V4: Lightning Index Ultra-Long Context via Lookahead Sparse Attention
Jun 8, 2026Yan Wang, Qifan Zhang, Jiachen Yu +12KV Caching Long-Context Language Model Inference
Independent Researchers · Tencent · The Hong Kong University of Science and Technology (Guangzhou) +1
STAR-KV: Low-Rank KV Cache Compression via Soft Thresholding for Adaptive Rank Control
Jun 7, 2026Priyansh Bhatnagar, Ashkan Moradifirouzabadi, Se-Hyun Yang +3Mixed-Precision Quantization KV Caching
University of California San Diego, San Diego, CA, USA · Dnotitia Inc., Seoul, Republic of Korea · Hanyang University, Seoul, Republic of Korea
HACK++: Towards More Effective Head-Aware Key-Value Compression for Efficient Visual Autoregressive Modeling
Jun 6, 2026Ziran Qin, Yuchen Jiang, Mingbao Lin +4KV Caching Visual Autoregressive Models
Shanghai Jiao Tong University, Shanghai, China · Rakuten, Singapore · Tsinghua University, Beijing, China
IntentKV: Cross-Turn Intent-Aware KV Cache Pruning for Agent Inference
Jun 6, 2026Junjie Li, Jiong Lou, Jie LiKV Caching KV-Cache Management
Shanghai Jiao Tong University
Still: Amortized KV Cache Compaction in a Single Forward Pass
Jun 5, 2026Charles O'Neill, Alex Sandomirsky, Harry Partridge +2KV Caching Long-Context Language Model Inference
Baseten San Francisco · Baseten London
Towards Tight Bounds for Streaming Attention
Jun 5, 2026Justin Y. Chen, Ying Feng, Piotr Indyk +3Efficient Attention KV-Cache Compression
MIT · EPFL
Semantic Cache Distillation: Efficient State Transfer via Reuse and Selective Patching
Jun 5, 2026Qianli Ma, Zhiqing Tang, Hanshuai Cui +2LLM Serving KV Caching
Faculty of Arts and Science, Beijing Normal University, Zhuhai 519087, China · Institute of Artificial Intelligence and Future Networks, Beijing Normal University, Zhuhai 519087, Guangdong, China · School of Artificial Intelligence, Beijing Normal University, Beijing 100875, China +1
Tangram: Unlocking Non-Uniform KV Cache Compression for Efficient Multi-turn LLM Serving
Jun 4, 2026Hyungmin Kim, Minsoo Kim, Hongseok Kim +1LLM Serving KV Caching
RedKnot: Efficient Long-Context LLM Serving with Head-Aware KV Reuse and SegPagedAttention
Jun 4, 2026Yang Liu, ZhaoKai Luo, HuaYi Jin +5LLM Serving KV Caching
†Xiaohongshu Inc., China · ¶Huawei Cloud · §Peking University
YouZhi: Towards High-Concurrency Financial LLMs via Adaptive GQA-to-MLA Transition
Jun 4, 2026PSBC LLM Team, Huawei LLM Team, Ruihan Long +56LLM Compression LLM Inference Acceleration
Cartridges at Scale: Training Modular KV Caches over Large Document Collections
Jun 3, 2026Momchil Hardalov, Gonzalo Iglesias, Adrià de GispertMemory-Augmented Language Models KV Caching
Amazon AGI
Value-Aware Stochastic KV Cache Eviction for Reasoning Models
Jun 2, 2026Ting-Yun Chang, Harvey Yiyun Fu, Deqing Fu +3LLM Inference Efficiency KV Caching
University of Southern California · University of Chicago
STaR-KV: Spatio-Temporal Adaptive Re-weighting for KV Cache Compression in GUI Vision-Language Models
Jun 1, 2026Yuhang Han, Wenzheng Yang, Yujie Chen +4Vision-Language Models KV Caching
1EPIC Lab, SJTU · 2HKUST (GZ) · 3The University of Sydney +3
MomentKV: Closing the Directional Gap in KV Cache Eviction for Long-Context Inference
Jun 1, 2026Yu Li, Binxu Li, Tian LanKV Caching Long-Context Language Model Inference
George Washington University · Princeton University
WaveFilter: Enhancing the Long-Context Capability of Diffusion LLMs via Wavelet-Guided KV Cache Filtering
May 30, 2026Jinnan Yang, Yan Wang, Zhen Bi +5Long-Context Language Model Inference Diffusion Language Model Inference
Nanjing University of Science and Technology · Alibaba Group · Huzhou Normal University +1
Linear Scaling Video VLMs for Long Video Understanding
May 29, 2026Cristobal Eyzaguirre, Jiajun Wu, Juan Carlos NieblesStreaming Video Understanding Efficient VLM Inference
Stanford University
GRKV: Global Regression for Training-Free KV Cache Compression in Long-Context LLMs
May 29, 2026Junjie Peng, You Wu, Haoyi Wu +4KV Caching Long-Context Language Modeling
1Sun Yat-sen University · 2ShanghaiTech University · 3Guangdong Province Key Laboratory of Information Security Technology
Probing the Prompt KV Cache: Where It Becomes Dispensable
May 28, 2026Vinayshekhar Bannihatti Kumar, Manoj Ghuhan Arivazhagan, Disha Makhija +1KV Caching KV-Cache Management
AWS AI Labs
Future Forcing: Future-aware Training-free KV Cache Policy for Autoregressive Video Generation
May 28, 2026Jiayi Luo, Qiyan Liu, Tengyang Wang +8KV Caching Long-Horizon Video Generation
BUAA · ZGCA · PKU +3
Moment-KV: Momentum-Based Decode-Time KV Cache Compression for Long Generation
May 28, 2026Soumyadeep Jana, Sagar Nishad, Sanasam Ranbir SinghKV Caching Long-Context Language Modeling
Indian Institute of Technology Guwahati, India
CIVIC: End-to-End Sequence Compactness for Efficient Vision-Language Models
May 27, 2026Fengze Yang, Bo Yu, Xuewen Luo +2Efficient VLM Inference KV-Cache Compression
Department of Civil & Environmental Engineering University of Utah 201 Presidents’ Cir, Salt Lake City, UT 84112, USA
UNIQUE: Universal Top-k Sparse Attention for Training-free Inference and Sparsity-aware Training
May 26, 2026Keqi Deng, Shaoshi Ling, Ruchao Fan +1Self-Attention Long-Context Language Model Inference
Microsoft, USA
Hurwitz Quaternion Multiplicative Quantization for KV Cache Compression
May 26, 2026Kabir Swain, Sijie Han, Daniel Karl I. Weidele +3LLM Quantization KV Caching
Massachusetts Institute of Technology, Cambridge, MA, USA · University of Toronto, Toronto, Canada · IBM Research, Cambridge, MA, USA.
OmniMem: Perturbation-aware Memory Compression for Streaming Audio-Visual LLMs
May 26, 2026Guangzhi Sun, Yixuan Li, Yudong Yang +1Streaming Video Understanding Multimodal Large Language Models
ByteDance · Department of Engineering, University of Cambridge · Tsinghua University
NestedKV: Nested Memory Routing for Long-Context KV Cache Compression
May 26, 2026Hong Chen, Xiang Liu, Yubo Gao +5KV Caching Long-Context Language Modeling
The Hong Kong University of Science and Technology (Guangzhou) · Jimei University
Polynomial Context-Truncation Sensitivity in Autoregressive Language Models: Sequential Wyner-Ziv Bounds for KV Cache Compression
May 24, 2026Munsik KimKV Caching Rate-Distortion Theory
Independent Researcher.
A Simple Plug-in for Improving Eviction-Based KV Cache Compression
May 22, 2026Yuping Lin, Jiayuan Ding, Yue Xing +3KV Caching Memory-Efficient Optimization
1Michigan State University · 2Hippocratic AI
Adaptive Mass-Segmented KV Compression for Long-Context Reasoning
May 22, 2026Junzhe Yang, Xiaoyu ShenKV Caching Long-Context Language Model Inference
Shanghai Jiao Tong University · Institute of Digital Twin, Eastern Institute of Technology, Ningbo
WorldKV: Efficient World Memory with World Retrieval and Compression
May 21, 2026Jung Yi, Minjae Kim, Paul Hyunbin Cho +3KV Caching Memory-Augmented Video Generation
1KAIST AI · 2Naver AI Lab

Jun 8, 2026, cs.AIJ/K move · Enter open · S save

From Rigid to Dynamic: Entropy-Guided Adaptive Inference for Long-Context LLMs

Zhanchao Xu, Haoyang Li, Qingfa Xiao, Fei Teng+3

Department of Computing, PolyU · DSA, HKUST(GZ) · CSE, HKUST

Long-Context Language Model Inference LLM Inference Acceleration Sparse Attention KV-Cache Compression

Open paper AskPDF arXiv

Existing sparse attention and KV cache compression methods for long-context LLM inference typically apply fixed sparsity patterns or uniform budgets across all attention heads, overlooking the substantial variation in attention behavior among heads and contexts. We observe two distinct entropy patterns among attention heads: Rigid Heads, whose entropy stays near zero across input segments, and Dynamic Heads, whose entropy fluctuates significantly. Crucially, the distribution of these types is context-dependent and cannot be predetermined offline. We therefore propose EntropyInfer, a training-free framework that uses attention entropy to adaptively allocate compute at the granularity of individual heads and segments during prefilling. For decoding, we introduce a latent KV cache compression scheme that leverages generated output tokens, rather than prefill tokens alone, to identify and retain the most critical cache entries. Extensive experiments on Llama, Qwen and openPangu model series show that EntropyInfer consistently outperforms baselines including SnapKV, AdaKV, and CritiPrefill, achieving up to 2.39

\times

end-to-end speedup beyond 100k tokens with minimal quality degradation compared to full attention. The code is released in https://github.com/SHA-4096/EntropyInfer.

Momentum

Related topics

Latest papers 166