KV-Cache Quantization

Ask about this topic

Overview

KV-cache quantization stores the keys and values used by attention at lower precision, allowing language models to handle longer contexts or more concurrent requests within a fixed memory budget. Approaches include scalar and vector quantization, rotations and other transforms, mixed-precision allocation across tokens or layers, and fused kernels that compute attention directly from compressed data. Methods are assessed on language-model perplexity, long-context retrieval and reasoning benchmarks such as LongBench and RULER, and serving measures including memory, latency, and throughput. The main tension is that aggressive compression can distort attention—especially through key errors—and gains depend on the model, workload, and hardware, so reconstruction quality alone does not predict deployment performance.

Latest papers

CardsList

100 papers25 papers in the last four weeks, up 108% on the four weeks before. 0.1% of all new papers.

  1. ReQAT: Achieving Full-Precision Reasoning Accuracy with 4-bit Floating-Point Quantization-Aware Training

    Jun 14, 2026Janghwan Lee, Sihwa Lee, Jinseok Kim +4Activation QuantizationQuantization-Aware Training

    1Hanyang University, 2Rebellions Inc., Republic of Korea.

  2. SpectrumKV: Per-Token Mixed-Precision KV Cache Transfer for Prefill-Decode Disaggregated LLM Serving

    Jun 7, 2026Yang PengjuMixed-Precision QuantizationPrefill-Decode Disaggregation

  3. KVarN: Variance-Normalized KV-Cache Quantization Mitigates Error Accumulation in Reasoning Tasks

    Jun 2, 2026Lorenz K. Muller, Philippe Bich, Chiara Boretti +3LLM QuantizationKV Caching

    Huawei

  4. Alignment Collapse Under KV Cache Quantization: Diagnosis and Mitigation

    Jun 1, 2026Bruce Changlong Xu, Adarsh Kumarappan, Mu ZhouLLM QuantizationKV Caching

    †Stanford University · ‡California Institute of Technology

  5. Hurwitz Quaternion Multiplicative Quantization for KV Cache Compression

    May 26, 2026Kabir Swain, Sijie Han, Daniel Karl I. Weidele +3KV-Cache CompressionLLM Quantization

    Massachusetts Institute of Technology, Cambridge, MA, USA · University of Toronto, Toronto, Canada · IBM Research, Cambridge, MA, USA.

  6. Quantized Keys Steal Attention: Bias Correction for KV-Cache Compression in Video Diffusion

    May 25, 2026Tuna Tuncer, Felix Becker, Thomas PfeilVideo Diffusion ModelsSelf-Attention

    1Technical University of Munich · 2Tensordyne

  7. OCTOPUS: Optimized KV Cache for Transformers via Octahedral Parametrization Under optimal Squared error quantization

    May 20, 2026Mark Boss, Vikram Voleti, Simon Donné +1KV CachingMemory-Efficient Optimization

    Stability AI

  8. Runtime-Certified Bounded-Error Quantized Attention

    May 20, 2026Dean CalverLLM InferenceKV Caching

    Independent Researcher

  9. OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyond

    May 19, 2026Zunhai Su, Rui Yang, Chao Zhang +12KV CachingRotation-Based Quantization

    Tsinghua University · Meituan LongCat Team · The University of Hong Kong +3

  10. OSCAR: Offline Spectral Covariance-Aware Rotation for 2-bit KV Cache Quantization

    May 18, 2026Zhongzhu Zhou, Donglin Zhuang, Jisen Li +4Rotation-Based QuantizationLLM Inference Acceleration

    Together AI · University of Sydney · University of Illinois Urbana-Champaign

  11. TriAxialKV: Toward Extreme Low-Precision KV-Cache Quantization for Agentic Inference Tasks

    May 16, 2026Hanzhang Shen, Haoran Wu, Yiren Zhao +1Mixed-Precision QuantizationLLM Inference Acceleration

    University of Cambridge · Imperial College London

  12. SPHERICAL KV: Angle-Domain Attention and Rate-Distortion Retention for Efficient Long-Context Inference

    May 13, 2026Anay Chauhan, Gurucharan Marthi Krishna Kumar, Arion Das +4KV CachingLong-Context Language Model Inference

    1Synopsys, India · 2McGill University, Canada · 3IIIT Ranchi, India +4

  13. FibQuant: Universal Vector Quantization for Random-Access KV-Cache Compression

    May 12, 2026Namyoon Lee, Yongjune KimVector QuantizationLow-Bit Quantization

    POSTECH

  14. RDKV: Rate-Distortion Bit Allocation for Joint Eviction and Quantization of the KV Cache

    May 8, 2026Junkai Zhang, Hang Guo, Luca Benini +1KV CachingLong-Context Language Model Inference

    ETH Zurich · Tsinghua University

  15. When Quantization Is Free: An int4 KV Cache That Outruns fp16 on Apple Silicon

    May 7, 2026Mohamed Amine BergachGPU Kernel OptimizationKV Caching

    Illumina, San Diego, CA, USA

  16. QKVShare: Quantized KV-Cache Handoff for Multi-Agent On-Device LLMs

    May 5, 2026Pratik Honavar, Tejpratap GVSLMixed-Precision QuantizationMulti-Agent LLM Systems

  17. HeadQ: Model-Visible Distortion and Score-Space Correction for KV-Cache Quantization

    May 5, 2026Jorge L. Ruiz WilliamsSelf-AttentionKV Caching

    Independent Researcher

  18. WindowQuant: Mixed-Precision KV Cache Quantization based on Window-Level Similarity for VLMs Inference Optimization

    May 4, 2026Wei Tao, Xiaoyang Qu, Peiqiang Wang +4Mixed-Precision QuantizationVLM Quantization

    Huazhong University of Science and Technology, Wuhan, Hubei, China · Ping An Technology (Shenzhen) Co., Ltd., Shenzhen, Guangdong, China · Tsinghua University, Shenzhen International Graduate School, Shenzhen, Guangdong, China

  19. Benchmarking KV-Cache Optimizations across Task Quality and System Performance for Long-Context Serving

    May 3, 2026Nikita Agrawal, Ruben MayerLLM ServingKV Caching

    University of Bayreuth

  20. Statistical Inference and Quality Measures of KV Cache Quantisations Inspired by TurboQuant

    Apr 27, 2026Paolo D'AlbertoKV CachingRate-Distortion Optimization

    AMD

  21. PolyKV: A Shared Asymmetrically-Compressed KV Cache Pool for Multi-Agent LLM Inference

    Apr 27, 2026Ishan Patel, Ishan JoshiLLM Inference EfficiencyMulti-Agent LLM Systems

    Independent Researcher

  22. RateQuant: Optimal Mixed-Precision KV Cache Quantization via Rate-Distortion Theory

    Apr 22, 2026Fei Zuo, Zikang Zhou, Hao Cong +2LLM Inference EfficiencyMixed-Precision Quantization

    1BA TechWorks (BMW Group) · 2National University of Singapore · 3Tsinghua University

  23. Revisiting RaBitQ and TurboQuant: A Symmetric Comparison of Methods, Theory, and Experiments

    Apr 21, 2026Jianyang Gao, Yutong Gou, Yuexuan Xu +5Vector QuantizationKV-Cache Quantization

    ETH Zurich · Nanyang Technological University · University of Michigan +2

  24. SAW-INT4: System-Aware 4-Bit KV-Cache Quantization for Real-World LLM Serving

    Apr 21, 2026Jinda Jia, Jisen Li, Zhongzhu Zhou +8LLM ServingKV Caching

    Together AI

  25. AQPIM: Breaking the PIM Capacity Wall for LLMs with In-Memory Activation Quantization

    Apr 20, 2026Kosuke Matsushima, Yasuyuki Okoshi, Masato Motomura +1Compute-in-MemoryLong-Context Language Model Inference

    Institute of Science Tokyo†

  26. MoE-nD: Per-Layer Mixture-of-Experts Routing for Multi-Axis KV Cache Compression

    Apr 20, 2026Libo Sun, Peixiong He, Po-Wei Harn +1LLM InferenceKV Caching

    Department of Computer Science and Software Engineering, Auburn University, Auburn, AL, USA · Department of Information Management, National Central University, Taoyuan, Taiwan

  27. Open-TQ-Metal: Fused Compressed-Domain Attention for Long-Context LLM Inference on Apple Silicon

    Apr 18, 2026Sai VegasenaAI Accelerator InferenceSelf-Attention

    Ensue

  28. Don't Waste Bits! Adaptive KV-Cache Quantization for Lightweight On-Device LLMs

    Apr 6, 2026Sayed Pedram Haeri Boroujeni, Niloufar Mehrabi, Patrick Woods +2KV-Cache QuantizationOn-Device Language Model Inference

    Clemson University, USA

  29. eOptShrinkQ: Near-Lossless KV Cache Compression Through Optimal Spectral Denoising and Quantization

    Apr 6, 2026Pei-Chun SuKV-Cache CompressionKV-Cache Quantization

    Department of Applied and Computational Mathematics, Yale University, New Haven, CT, USA

  30. AXELRAM: Quantize Once, Never Dequantize

    Apr 3, 2026Yasushi NishidaAttention QuantizationKV-Cache Quantization

    Axelidea Inc.