Overview

KV-cache quantization stores the keys and values used by attention at lower precision, allowing language models to handle longer contexts or more concurrent requests within a fixed memory budget. Approaches include scalar and vector quantization, rotations and other transforms, mixed-precision allocation across tokens or layers, and fused kernels that compute attention directly from compressed data. Methods are assessed on language-model perplexity, long-context retrieval and reasoning benchmarks such as LongBench and RULER, and serving measures including memory, latency, and throughput. The main tension is that aggressive compression can distort attention—especially through key errors—and gains depend on the model, workload, and hardware, so reconstruction quality alone does not predict deployment performance.

Latest papers

CardsList

118 papers24 papers in the last four weeks, down 8% on the four weeks before. 0.2% of all new papers.

  1. CALFA-KV: Joint Calibration and Fisher-Guided Anchors for 1-Bit KV Cache Compression

    Nov 7, 2026Xiancai Tian, Leichao Zhang, Shenhui Zhang +5KV-Cache CompressionKV-Cache Quantization

  2. AutoEMatic: Runtime-Adaptive Mixed-Precision LLM Inference via Automatic Exponent-Mantissa Refinement

    Nov 6, 2026Akshat Ramachandran, Reena Elangovan, Rangharajan Venkatesan +4LLM QuantizationMixed-Precision Quantization

  3. BitDecoding-FP4: Native FP4 KV Cache Decoding via Algorithm-System Co-Design

    Nov 6, 2026Dayou Du, Woon Yee Ng, Guangming Sheng +3LLM Inference AccelerationKV-Cache Quantization

  4. Coordinated Mixed-Precision KV Cache Quantization for Efficient LLM-based Multi-agent Systems

    Nov 6, 2026Weizhong Huang, Jinchao Zhang, Xiawu ZhengKV-Cache QuantizationMixed-Precision Quantization

  5. FOSCAR: Accelerating Decode Attention over 2- and 4-Bit KV Caches through Affine Folding

    Nov 6, 2026Donglin Zhuang, Pragaash Ponnusamy, Xiaoxia Wu +2LLM Inference AccelerationGPU Kernel Optimization

  6. Transformer-FSQ: Discrete Latents for Extreme KV-Cache Compression

    Nov 6, 2026Andras Szecsenyi, Thomas L. Lee, Vaisakh Shaj +2KV-Cache CompressionKV-Cache Quantization

  7. StrataFlux: Memory-Hierarchy-Native Low-Bit LLM Inference

    Nov 5, 2026Jake Hyun, Ziyu Zhao, Jae-sun Seo +1LLM InferenceLLM Inference Efficiency

  8. TACT-KV: Tri-Axis Cosine Transform for Compressing Volumetric KV Caches in Medical VLMs

    Nov 5, 2026Chongyu Qu, Ritchie Zhao, Yufan He +12KV-Cache CompressionKV-Cache Quantization

  9. LoopKV: Loop-Aware KV Cache Compression for Serving Looped Transformers

    Nov 3, 2026Tianle Zhou, Yang Zhang, Tianyu Wu +3KV-Cache CompressionLLM Inference Acceleration

  10. FlagRot: Selective Rotation for Low-Bit KV Caches

    Nov 2, 2026Kunming Shao, Xijie Huang, Jierun Chen +7KV-Cache QuantizationRotation-Based Quantization

  11. R2^2-Omni: Reducing Structural and Precision Redundancy in Omni-LLMs for Long-Form Generation

    Nov 2, 2026Bolong Feng, Pengfei Zhou, Zhenglin Wan +2Multimodal Large Language ModelsMultimodal Token Compression

  12. Jet-QSparse: Accelerating Agentic LLM Inference with 4-Bit Sparse Attention

    Oct 31, 2026Haozhan Tang, Hao Kang, Meng Lu +7LLM Inference AccelerationKV-Cache Quantization

  13. KV Cache Quantization for Diffusion Language Models

    Oct 29, 2026Yan Tai, Yunan Ding, Guangzhao Li +5KV-Cache QuantizationDiffusion Language Model Inference

  14. DCKV: Learned Transform Coding of the Prompt KV Cache in Hybrid Language Models

    Oct 27, 2026Egor Maximov, Aleksei Merkelov, Vasiliy Alekseev +5KV-Cache CompressionKV-Cache Quantization

  15. RAQ-MLA: Role-Aware Quantization for Multi-Head Latent Attention Caches

    Oct 24, 2026Tianqi Chen, Xunchao Li, Linchuan Xie +4LLM QuantizationAttention Quantization

  16. RoLaQ: Joint Rotation and Range Calibration for 2-Bit KV Caches in Multi-Head Latent Attention

    Oct 14, 2026Shiyu Ji, Peng Ling, Chen Lin +4KV-Cache QuantizationLow-Bit Quantization

  17. MultiAlloc: Budget-Aware Precision Allocation for KV Cache Quantization

    Oct 13, 2026Yangchengyu Zhou, Zhiping Wu, Runchang Jiang +5KV-Cache QuantizationMemory-Efficient Inference

  18. AlignPred: Error-Controlled KV Cache Compression via Alignment and Predictive Coding

    Oct 13, 2026Xu Yang, Jiapeng Zhang, Zhiwei Zuo +5KV-Cache CompressionMemory Compression

  19. Fast Quantized Johnson−-Lindenstrauss Transforms

    Oct 12, 2026Angelos Korakitis, Vasileios NakosQuantizationRandomized Sketching

  20. E8 Lattice Key Quantization for Low-Bit KV-Cache Compression

    Oct 11, 2026Sasho Nedelkoski, Alexander Acker, Soeren Becker +2KV-Cache CompressionKV-Cache Quantization

  21. TempoKV: Predictive KV Cache Quantization for Autoregressive Video Generation

    Oct 11, 2026Jiahui Hu, Weile Mo, Peiliang Cai +5KV-Cache QuantizationVideo Diffusion Models

  22. RECAP: Calibrating Low-Bit Reasoning Models on the History They Read

    Oct 11, 2026Chi Ruan, Yujie Chen, Zixiang Zhao +2LLM QuantizationPost-Training Quantization

  23. EdgeLLM-Nano: Auditing Polar-Guided Key-Cache Compression and Consistency-Anchored Pruning on a 33.6 MB On-Device Language Model

    Oct 10, 2026S M Mahbub Murshed, Hayden Bright, Rezaul Karim +1KV-Cache CompressionKV-Cache Quantization

  24. KV4-RL: Unlocking Faster Asynchronous Rollouts for RLVR with INT4 KV Cache Quantization

    Oct 10, 2026Sihwan Park, Sung-Yub Kim, Doohyuk Jang +1KV-Cache QuantizationRL Post-Training

  25. Read What Matters: Query-Adaptive Quantization for KV Caches

    Oct 8, 2026Siddharth Bhandari, Lucas Gretta, Krishna Balasubramanian +1KV-Cache QuantizationKV-Cache Compression