KV-Cache Compression

Follow this topic Ask about this topic

Momentum

24 papers in the last four weeks, up 71% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 166

All topics

KARA: Efficient Reasoning LLM Serving via Sliding-Window KV Cache Compression
May 1, 2026Shen Han, Yuyang Wu, Junpu Yu +1KV Caching Memory-Efficient Inference
Carnegie Mellon University
Statistical Inference and Quality Measures of KV Cache Quantisations Inspired by TurboQuant
Apr 27, 2026Paolo D'AlbertoKV Caching Rate-Distortion Optimization
AMD
PolyKV: A Shared Asymmetrically-Compressed KV Cache Pool for Multi-Agent LLM Inference
Apr 27, 2026Ishan Patel, Ishan JoshiLLM Inference Efficiency Multi-Agent LLM Systems
Independent Researcher
Sub-Token Routing for KV Cache Compression
Apr 23, 2026Wei Jiang, Wei WangLLM Inference Efficiency KV Caching
Futurewei Technologies San Jose, CA 95131, USA
Sparse Forcing: Native Trainable Sparse Attention for Real-time Autoregressive Diffusion Video Generation
Apr 23, 2026Boxun Xu, Yuming Du, Zichang Liu +7Autoregressive Diffusion Self-Attention
Meta Superintelligence Labs · University of California, Santa Barbara · Work done at Meta
LKV: End-to-End Learning of Head-wise Budgets and Token Selection for LLM KV Cache Eviction
Apr 22, 2026Enshuai Zhou, Yifan Hao, Chao Wang +7KV Caching Long-Context Language Model Inference
University of Science and Technology of China, Hefei, China · State Key Lab of Processors, Institute of Computing Technology, CAS, Beijing, China · University of Chinese Academy of Sciences, Beijing, China
DASH-KV: Accelerating Long-Context LLM Inference via Asymmetric KV Cache Hashing
Apr 21, 2026Jinyu Guo, Zhihan Zhang, Jiehui Xie +7Long-Context Language Model Inference LLM Inference Acceleration
School of Information and Software Engineering, University of Electronic Science and Technology of China · School of Computer Science and Engineering, University of Electronic Science and Technology of China · Bangladesh University of Engineering and Technology +2
Neural Garbage Collection: Learning to Forget while Learning to Reason
Apr 20, 2026Michael Y. Li, Jubayer Ibn Hamid, Emily B. Fox +1LLM Compression Memory-Efficient Optimization
Stanford University
How Much Cache Does Reasoning Need? Depth-Cache Tradeoffs in KV-Compressed Transformers
Apr 20, 2026Xiao WangTransformer Inference KV Caching
MoE-nD: Per-Layer Mixture-of-Experts Routing for Multi-Axis KV Cache Compression
Apr 20, 2026Libo Sun, Peixiong He, Po-Wei Harn +1LLM Inference KV Caching
Department of Computer Science and Software Engineering, Auburn University, Auburn, AL, USA · Department of Information Management, National Central University, Taoyuan, Taiwan
Reducing Peak Memory Usage for Modern Multimodal Large Language Model Pipelines
Apr 17, 2026Junwan Kim, Hyunkyung BaeEfficient Multimodal Inference Multimodal Large Language Models
New York University
When Less Latent Leads to Better Relay: Information-Preserving Compression for Latent Multi-Agent LLM Collaboration
Apr 14, 2026Yiping Li, Zhiyu An, Wan DuLLM Compression KV Caching
Department of Computer Science and Engineering University of California, Merced Merced, CA 95343
Attention-Weighted Value Projection for KV-Cache Compression
Apr 13, 2026Samuel SalfatiGrouped-Query Attention KV Caching
fraQtl AI Research
KV Cache Offloading for Context-Intensive Tasks
Apr 9, 2026Andrey Bocharnikov, Ivan Ermakov, Denis Kuznedelev +2KV-Cache Offloading KV Caching
HSE, Yandex · Yandex · Yandex, NSU
ResidualKV: Residual-Based KV Cache Compression for Efficient Long-Context Inference
Feb 8, 2026Jitai Hao, Qiang Huang, Yaowei Wang +2Self-Attention KV Caching
Harbin Institute of Technology (Shenzhen), China
ReFreeKV: Towards Threshold-Free KV Cache Compression
Feb 24, 2025Xuanfan Ni, Liyan Xu, Chenyang Lyu +6KV Caching Memory-Efficient Optimization
Nanjing University of Aeronautics and Astronautics · WeChat AI, Tencent · Independent Researcher +1

May 1, 2026, cs.CLJ/K move · Enter open · S save

KARA: Efficient Reasoning LLM Serving via Sliding-Window KV Cache Compression

Shen Han, Yuyang Wu, Junpu Yu, Olexandr Isayev

Carnegie Mellon University

KV Caching Memory-Efficient Inference LLM Inference Acceleration KV-Cache Compression

Open paper AskPDF arXiv

Reasoning language models often generate long chain-of-thought (CoT), which accumulates a massive KV cache during the decoding phase and incurs high decoding latency and limited throughput. To address these issues, KV cache compression has emerged as a promising technique for reducing memory overhead by selectively removing unimportant KV pairs while preserving useful ones for subsequent decoding. Nevertheless, we identify two key limitations in existing KV cache compression methods: 1) their threshold-triggered compression policy may provide limited throughput improvement or even reduce throughput, and may fully eliminate KV pairs from certain blocks of the sequence, potentially worsening information loss. 2) they typically retain either isolated KV pairs or fixed-size chunks with rigid boundaries, failing to preserve important flexible-sized chunks at arbitrary token positions. To overcome these limitations, we propose Kara, a sliding-window KV cache compression method that performs decoding-time compression by operating only on the recently generated context. Kara leverages bidirectional attention to score and select informative KV pairs in the window. To enable flexible preservation of important semantic information, we design a Token2Chunk module to expand a subset of selected KV pairs into chunks. Furthermore, we adapt Kara to PagedAttention and develop KvLLM, an inference framework built upon vLLM, which reduces KV cache memory usage and effectively improves output throughput. Extensive experiments demonstrate consistent performance improvements of proposed Kara and KvLLM.

Momentum

Related topics

Latest papers 166