KV-Cache Quantization

Follow this topic Ask about this topic

KV-cache quantization stores the keys and values used by attention at lower precision, allowing language models to handle longer contexts or more concurrent requests within a fixed memory budget. Approaches include scalar and vector quantization, rotations and other transforms, mixed-precision allocation across tokens or layers, and fused kernels that compute attention directly from compressed data. Methods are assessed on language-model perplexity, long-context retrieval and reasoning benchmarks such as LongBench and RULER, and serving measures including memory, latency, and throughput. The main tension is that aggressive compression can distort attention—especially through key errors—and gains depend on the model, workload, and hardware, so reconstruction quality alone does not predict deployment performance.

Latest papers

100 papers29 papers in the last four weeks, up 164% on the four weeks before. 0.2% of all new papers.

FuTransformer: FP8 prefill and INT4 decode for Llama-style inference on Hopper
Oct 16, 2026Wei WangEfficient Transformer Inference GPU Kernel Optimization
RoLaQ: Joint Rotation and Range Calibration for 2-Bit KV Caches in Multi-Head Latent Attention
Oct 14, 2026Shiyu Ji, Peng Ling, Chen Lin +4KV-Cache Quantization Low-Bit Quantization
Research Center for Social Computing and Interactive Robotics, Harbin Institute of Technology, China · Tsinghua University, China · Baidu Inc., China
PN-Mix: Pair-Norm Guided Mixed-Precision Key Cache Quantization for Efficient Long-Output LLM Inference
Oct 14, 2026Gil Jae Song, Sungjoo YooKV-Cache Quantization Mixed-Precision Quantization
Seoul National University
MultiAlloc: Budget-Aware Precision Allocation for KV Cache Quantization
Oct 13, 2026Yangchengyu Zhou, Zhiping Wu, Runchang Jiang +5KV-Cache Quantization Memory-Efficient Inference
Nanjing University · The Hong Kong University of Science and Technology · Geely +1
AlignPred: Error-Controlled KV Cache Compression via Alignment and Predictive Coding
Oct 13, 2026Xu Yang, Jiapeng Zhang, Zhiwei Zuo +5KV-Cache Compression Memory Compression
Hunan University · Central China Normal University · Tencent
Fast Quantized Johnson $-$ Lindenstrauss Transforms
Oct 12, 2026Angelos Korakitis, Vasileios NakosQuantization Randomized Sketching
National and Kapodistrian University of Athens
E8 Lattice Key Quantization for Low-Bit KV-Cache Compression
Oct 11, 2026Sasho Nedelkoski, Alexander Acker, Soeren Becker +2KV-Cache Compression KV-Cache Quantization
Technical University of Berlin Berlin, Germany · logsight.ai Berlin, Germany
TempoKV: Predictive KV Cache Quantization for Autoregressive Video Generation
Oct 11, 2026Jiahui Hu, Weile Mo, Peiliang Cai +5KV-Cache Quantization Video Diffusion Models
Shanghai Jiao Tong University · South China University of Technology · Alibaba Group
RECAP: Calibrating Low-Bit Reasoning Models on the History They Read
Oct 11, 2026Chi Ruan, Yujie Chen, Zixiang Zhao +2LLM Quantization Post-Training Quantization
SpinEntropy · ETH Zurich · PolyU
EdgeLLM-Nano: Auditing Polar-Guided Key-Cache Compression and Consistency-Anchored Pruning on a 33.6 MB On-Device Language Model
Oct 10, 2026S M Mahbub Murshed, Hayden Bright, Rezaul Karim +1KV-Cache Compression KV-Cache Quantization
Causal Dynamics Lab · Electrical Engineering and Computer Science Department, University of Missouri-Columbia · Computer Science & Engineering, University of Chittagong
KV4-RL: Unlocking Faster Asynchronous Rollouts for RLVR with INT4 KV Cache Quantization
Oct 10, 2026Sihwan Park, Sung-Yub Kim, Doohyuk Jang +1KV-Cache Quantization RL Post-Training
KAIST · NAVER Cloud · AITRICS
Read What Matters: Query-Adaptive Quantization for KV Caches
Oct 8, 2026Siddharth Bhandari, Lucas Gretta, Krishna Balasubramanian +1KV-Cache Quantization KV-Cache Compression
Amazon · University of California, Berkeley
Bridging KV-Cache Quantization and Linear Attention: From Theory to Pretrained Weight Migration
Oct 8, 2026Kaicheng Xiao, Liran Dong, Haotian Li +1Memory-Augmented Neural Networks KV-Cache Quantization
The Chinese University of Hong Kong
CORAL-KV: Compact Cross-Layer Prediction and Residual Coding for Autoregressive Video KV Caches
Oct 7, 2026Albert Ge, Akash Haridas, Utkarsh Saxena +5KV-Cache Compression KV-Cache Quantization
Advanced Micro Devices Inc. (AMD) · University of Wisconsin-Madison
ResidualQuant: KV Cache Quantization for Looped Transformers with 2-Bit Residuals
Oct 7, 2026Heejun Kim, Junyoung Lee, SangLyul Cho +3KV-Cache Quantization Efficient Transformer Inference
KAIST · Yonsei University · Seoul National University
Dual-QK: Sharp Queries and Flat Keys for Prunable 2-bit KV Caches
Oct 7, 2026Sunjoo Whang, Jungjun Oh, Minsung Kim +5KV-Cache Compression KV-Cache Quantization
KAIST · GIST · Google Research
Few Bits, One Law: Toward W2A4KV2
Oct 6, 2026Kai Yi, Tarek Elgamal, Sruthikesh Surineni +3Neural Network Quantization LLM Quantization
Meta AI
SoloQ: Calibration-Free Quantization for Diffusion Language Models
Oct 5, 2026Donghyun Lee, Arkapravo Ghosh, Varun Manjunath +5Diffusion Model Quantization KV-Cache Quantization
University of Southern California · Yale University · Korea University
Shared Stopping Decisions Change Answers in HQQ Cache Quantization
Oct 5, 2026Seunghui Jwa, Minsu Oh, Chanjun Park +1LLM Inference KV-Cache Quantization
Department of Artificial Intelligence, Jeju National University · School of Software, Soongsil University
Spend Bytes on Breadth: Precision-Count Trade-offs for Decode-Time KV Compression in Long Chain-of-Thought Reasoning
Oct 5, 2026Runguo LiCoT Reasoning KV-Cache Eviction
University of Illinois Urbana-Champaign
Tailoring the Quantization Space for 1-Bit KV Cache Compression
Oct 2, 2026Minsoo Cheong, Donghyun Son, Sungjoo YooBinary Quantization KV-Cache Compression
Stanford University · Seoul National University
WUSH-KV: KV Cache Quantization with Data-Adaptive Transforms
Sep 29, 2026Jiale Chen, Vage Egiazarian, Eldar Kurtić +2KV-Cache Quantization LLM Quantization
Institute of Science and Technology Austria (ISTA) · Red Hat AI · ETH Zürich
KV-Kaizen: Learning Context-Adaptive Cache Compression Choices
Sep 29, 2026Joao Monteiro, Louis Béthune, Anastasiia Filippova +3LLM Inference Acceleration Low-Rank Compression
Apple
Behavioral Capacity Certificates for Quantized Language Models
Sep 29, 2026Arian Eamaz, Mojtaba SoltanalianLLM Quantization KV-Cache Quantization
Department of Electrical and Computer Engineering University of Illinois Chicago
QuantMLA: Function-Aligned Dual-Path Quantization for Low-Bit MLA KV Caching
Sep 29, 2026Zunhai Su, Yuxuan Sun, Jianchao Tan +5LLM Inference Acceleration Attention Quantization
The University of Hong Kong · Meituan LongCat Team · South China University of Technology +1
Does Execution Require Target KV Fidelity? A Mixed-Fidelity KV Runtime for LLM Serving
Sep 27, 2026Jiantong Jiang, Yue Yang, Peiyu Yang +1LLM Serving KV-Cache Management
University of Melbourne · Maincode
The KV Cache Is the New Memory Wall
Sep 25, 2026Tejinder SinghKV Caching Long-Context Language Model Inference
Dell Technologies, Santa Clara, CA 95054, USA
QuantWM: Temporally Consistent 2-Bit KV Cache Quantization for Video World Models
Sep 22, 2026Jiaqi Zhao, Xiaobin Hu, Bo Yin +3KV-Cache Compression KV-Cache Quantization
Harbin Institute of Technology (Shenzhen), Shenzhen, China · National University of Singapore, Singapore
KV-COBRA: KV Cache Compression via Co-Optimized Bit-Rank Allocation
Sep 21, 2026Sihyeon Ha, Jaeho Lee, Yo-Seb JeonLow-Rank Compression KV-Cache Compression
Pohang University of Science and Technology (POSTECH)
HeadGuard: Selective Head Protection for Low-Bit VLM KV-Cache Quantization
Sep 18, 2026Nenad BanficVLM Quantization KV-Cache Quantization
CoreAI, Microsoft

Oct 16, 2026J/K move · Enter open · S save

FuTransformer: FP8 prefill and INT4 decode for Llama-style inference on Hopper

Wei Wang

Efficient Transformer Inference GPU Kernel Optimization LLM Quantization Mixed-Precision Quantization KV-Cache Quantization

Open paper AskPDF arXiv

Transformer prefill reuses weights across tokens and is dominated by matrix multiplication and activation data movement; single-token decode has limited reuse and is dominated by weight and KV-cache reads. We present FUTRANSFORMER, a CUDA implementation of a Llama-style Transformer block for NVIDIA Hopper GPUs. It applies CODA's GEMM-plus-epilogue approach to inference, adding direct KV-cache placement, folding the fixed RMSNorm gain into weights, writing FP8 output directly with delayed scaling, and supporting prefill without padding. Decode uses INT4 projection weights and an INT4 KV-cache copy. Weights and keys follow Tensor Core fragment layouts for direct dequantization into MMA operands without full intermediate tiles. A block-diagonal Hadamard transform reduces cache quantization error from outlier channels. On one H100, prefill is up to 1.48x faster than compiled PyTorch FP8 and reaches 1.29 PFLOP/s. For gate/up projection with SwiGLU at the 8B shape, our BF16 path is 1.13x faster than CODA. Our FP8 path is 2.02-2.18x faster than the evaluated BF16 CODA path. Across 8B and 70B shapes at batches 1-32, INT4 decode is 1.3-2.4x faster than the faster of PyTorch's BF16 and INT4 blocks at each point: 2.1-3.3x faster than compiled, CUDA-graphed BF16 and 1.3-6.3x faster than the same block using PyTorch's INT4 projection kernel, which targets a single query row and loses to BF16 at batch 32. It is also 31-51% faster than our FP8 path. Complete Llama-3.1-8B decode takes 2.8 ms per step at batch one, versus 8.1 ms for the compiled transformers implementation. Decoding every WikiText-2 token through the INT4 weight path raises perplexity from 6.24 to 7.23 with round-to-nearest weights or 6.99 with GPTQ in the same format; the rotated INT4 cache adds 0.20. The kernels match a quantized reference computation within 0.001 perplexity.