KV-Cache Quantization

Follow this topic Ask about this topic

KV-cache quantization stores the keys and values used by attention at lower precision, allowing language models to handle longer contexts or more concurrent requests within a fixed memory budget. Approaches include scalar and vector quantization, rotations and other transforms, mixed-precision allocation across tokens or layers, and fused kernels that compute attention directly from compressed data. Methods are assessed on language-model perplexity, long-context retrieval and reasoning benchmarks such as LongBench and RULER, and serving measures including memory, latency, and throughput. The main tension is that aggressive compression can distort attention—especially through key errors—and gains depend on the model, workload, and hardware, so reconstruction quality alone does not predict deployment performance.

Latest papers

100 papers25 papers in the last four weeks, up 108% on the four weeks before. 0.1% of all new papers.

DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression
Sep 17, 2026DeepSeek-AI, :, Anyi Xu +590KV Caching Long-Context Language Model Inference
D-Quant: Driftable Entropy Coding for KV Cache Quantization
Sep 17, 2026Yi Su, Hong Liu, Guanghua Yu +1KV Caching KV-Cache Compression
Tencent
OmniKVQuant: KV Cache Quantization for Omni-LLMs
Sep 10, 2026Suho Yoo, Hyunjong Ok, Jongmin Choi +2Efficient Multimodal Inference KV Caching
KAIST · POSTECH
Interface-Aware KV Cache Quantization for Dense On-Chip NVM in Long-Context LLM Decoding
Sep 4, 2026Jiahao Zheng, Yifan Qin, Xiaobo Sharon Hu +1Hardware-Aware Quantization KV-Cache Quantization
University of Notre Dame, Notre Dame, IN, USA
Quality Recovery for Quantized KV Caches via Low-Rank Attention Adaptation
Sep 2, 2026Seifeldin AbdellatifKV-Cache Quantization Quantization-Aware LoRA
College of Engineering, Al Ain University
HBQ: Hierarchical Scaling Block Quantization with Hardware-Efficiency-Aware Design for Accurate LLM Inference
Aug 31, 2026Chun-Ting Chen, Dongmin Han, Hangyeol Mun +6LLM Quantization LLM Inference Acceleration
Cornell University New York, NY, USA · Intel Corporation Santa Clara, CA, USA · Intel Corporation Hillsboro, OR, USA
Faithfulness Is Not Free: Auditing Offline KV-Cache Quantization in Retrieval-Augmented Generation
Aug 31, 2026Atta Ul Asad, Ahsan Bilal, Muhammad Ali +2KV Caching RAG Evaluation
1Lahore University of Management Sciences (LUMS) · University of Oklahoma · 3Air University +1
SemKV: Semantic Mixed-Precision KV Cache Quantization Guided by the Quality Cliff for Long-Context LLM Inference
Aug 28, 2026Daeha Lee, Do-Hyung Kim, Jae-Hong KimKV-Cache Quantization Mixed-Precision Quantization
Electronics and Telecommunications Research Institute (ETRI), Daejeon, Republic of Korea
HyQuant: Hybrid-Precision Quantization for LLM Attention
Aug 28, 2026Jiatong Ding, Bingxin Xing, Yu Zhang +9Mixed-Precision Quantization LLM Quantization
Shanghai Jiao Tong University · Xi’an Jiaotong University · Xiamen University +1
Minima-KV: Retention-Preserving KV Cache Compression with Mixed-Format Paged Attention
Aug 24, 2026Sergii Kozyrev, Davyd MaiborodaKV-Cache Compression LLM Inference Acceleration
Minima AI, Inc · Minima AI, Inc.
Squeezing the Cache, Preserving the Truth: Monotonic Equipotential Allocation with Geodesia-KV
Aug 19, 2026Vincenzo Dentamaro, Pancrazio Auteri, Giuseppe PirloKV-Cache Compression KV-Cache Management
University of Bari Aldo Moro, Department of Computer Science
KV Cache Compression Through the Lens of Transform Coding
Aug 14, 2026Hannah Laus, Claudio Mayrink Verdun, Hao Wang +2KV-Cache Compression KV-Cache Quantization
Technical University of Darmstadt, the Technical University of Munich and the Munich Center for Machine Learning · Massachusetts Institute of Technology · Red Hat AI Innovation Team and the MIT-IBM Watson AI Lab +1
RotaryQuant: Fitting 120B MoE Models on Consumer Hardware via Fused Compressed-Space Attention
Aug 8, 2026Anthony. Lui, Mohamed. Elsaied, N. P. SavaniExpert Offloading Mixture-of-Experts Language Models
Cognizant AI & Analytics, London, UB8 3PH, UK · Southern Methodist University (SMU), Engineering Management, Lyle School of Engineering, Dallas, TX, USA · Cognizant AI Lab, Responsible AI Office, San Francisco, USA +1
SPECTRA: Pushing the KV Cache Beyond the 2-Bit Cliff via Spectral Transform Coding
Aug 8, 2026Jiamu Zhang, Liang Wu, Kelly Wan +2Mixed-Precision Quantization KV Caching
Nokia · Rice University
QEvict: Recoverable Quantized KV Eviction for Attention-Drift-Robust Long-Context Decoding
Aug 5, 2026Ayushman Garg, Akshita Gupta, Shaswata Bhattacharya +3KV Caching Long-Context Language Model Inference
Mehta Family School of Data Science and Artificial Intelligence, Indian Institute of Technology Roorkee · Yardi School of Artificial Intelligence, Indian Institute of Technology Delhi · Department of Electrical Engineering, Indian Institute of Technology Delhi
Spend Bits Where Queries Look: KV Cache Vector Quantization with Attention-Preserving Transforms
Aug 4, 2026Samuel Fernández-Menduiña, Amir Ziashahabi, Eduardo Pavez +2KV Caching Vector Quantization
Department of Electrical and Computer Engineering, University of Southern California
Output-Aware Rotation for INT2 KV-Cache Quantization
Aug 3, 2026Vincent-Daniel Yun, Woosang Lim, Minsoo Cheong +4LLM Inference KV Caching
University of Southern California · Seoul National University · Inha University
Does Accuracy Equal Evidence? Reasoning Faithfulness under KV Cache Compression
Aug 3, 2026Mengting Ai, Jingrui He, Yue GuoCoT Faithfulness Reasoning Evaluation
University of Illinois Urbana-Champaign
WitCert: Sound Runtime Risk Observability and Gating for KV-Cache Quantization
Jul 30, 2026Fanzhe Wei, Li Liu, Ziyang Wang +1KV Caching Memory-Efficient Inference
Metask Lab
Regularize or Localize: When Training-Time KV-Cache Geometry Pays Under Quantization
Jul 19, 2026Libo Sun, Po-Wei Harn, Zewei Zhang +2KV Caching Attention Quantization
Department of Computer Science and Software Engineering, Auburn University, Auburn, AL 36830, USA · Department of Information Management, National Central University, Taoyuan 320317, Taiwan
A JoLT for the KV cache: Near-Lossless KV Cache Compression via Joint Rank-bit Allocation
Jul 14, 2026Rahul Krishnan, Volker SchulzTucker Decomposition KV Caching
Universität Trier · Fachbereich IV, Mathematik, Universität Trier
Fractal KV-Cache Archives: Lossless Symbolic Storage with In-Place Retrieval for Long-Context LLM Inference
Jul 8, 2026Vladimir GusevResidual VQ KV Caching
Independent Researcher
TriRoute: Unified Learned Routing for Joint Adaptive Attention, Experts, and KV-Cache Allocation
Jul 7, 2026Andrii Balashov, Olena PonomarovaSparse Mixture-of-Experts LLM Routing
Ukrainian State University of Science and Technologies
Lynx: Progressive Speculative Quantization for accelerating KV Transfer in Long-Context Inference
Jul 2, 2026Wenchen Han, Gingfung Matthew Yeung, Marco Barletta +3LLM Inference KV Caching
University College London · Huawei
GSRQ: Gain-Shape Residual Quantization for Sub-1-bit KV Cache
Jul 1, 2026Soosung Kim, Minjae Park, Eui-Young Chung +1LLM Compression Residual VQ
Department of Electrical and Electronic Engineering, Yonsei University, Seoul, Republic of Korea · Department of Systems Semiconductor Engineering, Yonsei University, Seoul, Republic of Korea
RaBitQCache: Rotated Binary Quantization for KVCache in Long Context LLM Inference
Jun 30, 2026Wenhao Li, Jinhao Dong, Hailin Zhang +3KV Caching Long-Context Language Model Inference
School of Information, Renmin University of China, Beijing, China · Peking University, Beijing, China
RoPE-Aware Bit Allocation for KV-Cache Quantization
Jun 23, 2026Fengfeng Liang, Yuechen Zhang, Jiaya JiaKV Caching LLM Inference Acceleration
1Hong Kong University of Science and Technology · 2The Chinese University of Hong Kong · 3MiMo, Xiaomi Corporation
HyperQuant: A Rate-Distortion-Optimal Quantization Pipeline for Large Language and Diffusion Models
Jun 22, 2026Yuval Domb, Hadar Sackstein, Tomer SolbergLLM Quantization Diffusion Model Quantization
UltraQuant: 4-bit KV Caching for Context-Heavy Agents
Jun 18, 2026Inesh Chakrabarti, David Limpus, Aditi Ghai Rana +4LLM Inference Efficiency GPU Kernel Optimization
Advanced Micro Devices · University of California, Los Angeles · Purdue University
Ablation, Statistical Inference, and Validation for KV-Cache Compression
Jun 14, 2026Paolo D'Alberto, Ashish Siarasao, Elliott Delaye +1KV Caching Rotation-Based Quantization
Advanced Micro Devices, Inc.

Sep 17, 2026J/K move · Enter open · S save

DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression

DeepSeek-AI, :, Anyi Xu, B. Li+589

KV Caching Long-Context Language Model Inference LLM Inference Acceleration KV-Cache Compression KV-Cache Quantization

Open paper AskPDF arXiv

The widespread adoption of long-horizon agents has made model workloads increasingly input-heavy. Although prior work has substantially reduced the cost of long-context computation, prefill remains computationally expensive, and large KV caches continue to strain HBM and SSD capacity and data-transfer bandwidth. Together, these compute, storage, and bandwidth demands constitute the primary bottleneck to further lowering deployment costs. To address this challenge, we introduce DeepSeek-V4.1-Flash, a multimodal Mixture-of-Experts (MoE) model with 552B backbone parameters and support for contexts of up to one million tokens. With its Causal Encoder-Decoder (CED) architecture, the model activates 16B parameters per token during decode but only 8B parameters during prefill, substantially improving cost efficiency for agentic workloads. To push the limits of KV cache compression, DeepSeek-V4.1-Flash combines cross-layer KV cache reuse in Compressed Sparse Attention 2 (CSA2) with FP4 KV caching. These designs reduce its global KV cache footprint (always in HBM) to 890 bytes per token, roughly 1/4 of the corresponding footprint of DeepSeek-V4-Flash. Further, through a dedicated deployment optimization known as SWA Bounded Replay, DeepSeek-V4.1-Flash reduces its persistent KV cache footprint (always on SSD or in host memory) to roughly 1/8 of that of DeepSeek-V4-Flash. Despite its much smaller KV cache footprint, the model delivers substantially better performance than the baseline. In addition, we streamline the DeepSeek-V4 architecture and introduce several efficient architectural extensions. We pretrain DeepSeek-V4.1-Flash on a multimodal corpus comprising 45T tokens and conduct comprehensive post-training, yielding strong performance across diverse text-based and multimodal agentic scenarios. Model checkpoints are available at https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash.