KV-Cache Quantization

Ask about this topic

Overview

KV-cache quantization stores the keys and values used by attention at lower precision, allowing language models to handle longer contexts or more concurrent requests within a fixed memory budget. Approaches include scalar and vector quantization, rotations and other transforms, mixed-precision allocation across tokens or layers, and fused kernels that compute attention directly from compressed data. Methods are assessed on language-model perplexity, long-context retrieval and reasoning benchmarks such as LongBench and RULER, and serving measures including memory, latency, and throughput. The main tension is that aggressive compression can distort attention—especially through key errors—and gains depend on the model, workload, and hardware, so reconstruction quality alone does not predict deployment performance.

Latest papers

CardsList

100 papers25 papers in the last four weeks, up 108% on the four weeks before. 0.1% of all new papers.

  1. IsoQuant: Hardware-Aligned SO(4) Isoclinic Rotations for LLM KV Cache Compression

    Mar 30, 2026Zhongping JiRotation-Based QuantizationKV-Cache Compression

  2. KV Cache Quantization for Self-Forcing Video Generation: A 33-Method Empirical Study

    Mar 29, 2026Suraj Ranganath, Vaishak Menon, Anish PatnaikKV-Cache QuantizationKV-Cache Compression

    University of California, San Diego

  3. TurboAngle: Near-Lossless KV Cache Compression via Uniform Angle Quantization

    Mar 29, 2026Dipkumar PatelKV-Cache CompressionKV-Cache Quantization

    LLMs Research Inc.

  4. VQKV: High-Fidelity and High-Ratio Cache Compression via Vector-Quantization

    Mar 17, 2026Yixuan Wang, Qingyu Shi, Jiayu Zhou +3KV-Cache CompressionKV-Cache Quantization

    LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University · Shanghai Innovation Institute · Shanghai Artificial Intelligence Laboratory +2

  5. Self-Indexing KVCache: Predicting Sparse Attention from Compressed Keys

    Mar 15, 2026Xu Yang, Jiapeng Zhang, Dongyang Zhao +2Binary QuantizationLLM Inference Acceleration

    College of Computer Science and Electronic Engineering, Hunan University, Changsha, China · The Ministry of Education Key Laboratory of “Fusion Computing of Supercomputing and Artificial Intelligence” China · Shenzhen Research Institute, Hunan University, Shenzhen, China

  6. InnerQ: Hardware-Aware Tuning-Free Quantization of KV Cache for Large Language Models

    Feb 26, 2026Sayed Mohammadreza Tayaranian Hosseini, Amir Ardakani, Warren J. GrossHardware-Aware QuantizationKV-Cache Quantization

    Department of Electrical and Computer Engineering McGill University Montreal, QC, Canada

  7. XStreamVGGT: Extremely Memory-Efficient Streaming Vision Geometry Grounded Transformer with KV Cache Compression

    Feb 25, 2026Zunhai Su, Weihao Ye, Hansen Feng +5Streaming 3D ReconstructionKV-Cache Compression

    Shenzhen International Graduate School, Tsinghua University · Institute of Artificial Intelligence, Xiamen University · China Star Optoelectronics Technology +2

  8. ARKV: Adaptive and Resource-Efficient KV Cache Management under Limited Memory Budget for Long-Context Inference in LLMs

    Feb 19, 2026Jianlong Lei, Shashikant IlagerKV-Cache ManagementLong-Context Language Model Inference

    Multiscale Networked Systems (MNS) Group, Informatics Institute University of Amsterdam, The Netherlands

  9. Agent Memory Below the Prompt: Persistent Q4 KV Cache for Multi-Agent LLM Inference on Edge Devices

    Feb 17, 2026Yakov Pyotr ShkolnikovMemory Management for LLM AgentsLLM Agent Memory

  10. NSNQuant: A Double Normalization Approach for Calibration-Free Low-Bit Vector Quantization of KV Cache

    May 23, 2025Donghyun Son, Euntae Choi, Sungjoo YooKV CachingVector Quantization

    Seoul National University