Attention Quantization

Follow this topic Ask about this topic

Attention quantization studies how to compute attention and store key-value caches with fewer bits, reducing memory use and speeding up long-context inference and model training. Methods include FP8, FP4, and integer formats; mixed-precision allocation for sensitive heads or token interactions; rotations and calibration; and fused kernels that compute directly on compressed data. Researchers evaluate these approaches using language-model perplexity, long-context retrieval, and vision or video-generation quality, alongside throughput and memory use on specific hardware. A central challenge is that small quantization errors can be amplified by softmax or accumulate during training, so methods must balance quality, speed, hardware constraints, and reliable error detection.

Latest papers

38 papers5 papers in the last four weeks, down 17% on the four weeks before. 0.0% of all new papers.

RAQ-MLA: Role-Aware Quantization for Multi-Head Latent Attention Caches
Oct 24, 2026Tianqi Chen, Xunchao Li, Linchuan Xie +4LLM Quantization Attention Quantization
C2DL, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Tencent
LeapQuant: Efficient Linear Attention with Accurate Recurrent State Quantization
Sep 29, 2026Yi Pan, Haocheng Xi, Kan Zhu +10LLM Quantization LLM Inference Acceleration
UC Berkeley · University of Washington · MIT +2
Delta-Matching: Closing the Final Gap of Native 8-bit Training for LLMs
Sep 29, 2026Haozhan Tang, Hao Kang, Han Cai +2Low-Precision Training LLM Training
Carnegie Mellon University · NVIDIA · NVIDIA & Massachusetts Institute of Technology
QuantMLA: Function-Aligned Dual-Path Quantization for Low-Bit MLA KV Caching
Sep 29, 2026Zunhai Su, Yuxuan Sun, Jianchao Tan +5LLM Inference Acceleration Attention Quantization
The University of Hong Kong · Meituan LongCat Team · South China University of Technology +1
From Attention Sensitivity to Layer Role: Revisiting Mixed-Precision Quantization of Transformers
Sep 28, 2026Nafiseh HosseinpourFardi, Negar Alihadi, Mahmoudreza Babaei +2Mixed-Precision Quantization LLM Quantization
Sharif University of Technology, Tehran, Iran · GISMA University of Applied Sciences, Potsdam, Germany · BRAINS, Brandenburg Research Center for Applied Intelligent Systems, Potsdam, Germany +3
Pretraining Transformers with Quantized Softmax in Attention
Sep 27, 2026Shangzhen Zhu, Muyan Hu, Tomasz KozlowskiLanguage Model Pretraining Quantization-Aware Training
University of Illinois Urbana-Champaign
DraftAttention2: Fast Video Diffusion with Low-Resolution-Guided Mixed-Precision Attention
Sep 26, 2026Rui Ding, Haopeng Li, Weize Ma +10Video Diffusion Models Diffusion Model Inference Acceleration
Nanjing University · HKUST(GZ) · Duke University +3
Global Ranks Survive, Selected Heads Shift: BOS-Sink Topology under 4-bit Weight-Only Quantization
Sep 20, 2026Kuanlin Chen, Chen-Wei Kuo, Cheng-En OuAttention Head Analysis 4-Bit Quantization
Independent Researcher Taoyuan, Taiwan · National Tsing Hua University Hsinchu, Taiwan
Attention Quantization for Tabular Foundation Models
Sep 11, 2026Jonas M. Kübler, Benjamin Jäger, Klemens Flöge +2Efficient Transformer Inference Tabular Foundation Models
Prior Labs
EFQ-Softmax: Exp-Free Quantization for Softmax
Sep 9, 2026Haohui Han, Yuming Wan, Hongni Wang +4Efficient Transformer Inference Transformer Attention
Xi'an Jiaotong University · Huawei Technologies Co., Ltd · Shandong University of Finance and Economics
Hardware-Aware FP4 FlashAttention-4
Sep 3, 2026Robert HuMixed-Precision Quantization GPU Kernel Optimization
GRAPHCORE RESEARCH
HyQuant: Hybrid-Precision Quantization for LLM Attention
Aug 28, 2026Jiatong Ding, Bingxin Xing, Yu Zhang +9Mixed-Precision Quantization LLM Quantization
Shanghai Jiao Tong University · Xi’an Jiaotong University · Xiamen University +1
Jacobian-guided Noise Injection for Quantization Robustness in Large Language Models
Aug 21, 2026Deepanshu Pandey, Arnav Chavan, Nahush Lele +2Quantization Robustness Neural Network Quantization
Amazon
KV Cache Compression Through the Lens of Transform Coding
Aug 14, 2026Hannah Laus, Claudio Mayrink Verdun, Hao Wang +2KV-Cache Compression KV-Cache Quantization
Technical University of Darmstadt, the Technical University of Munich and the Munich Center for Machine Learning · Massachusetts Institute of Technology · Red Hat AI Innovation Team and the MIT-IBM Watson AI Lab +1
RotaryQuant: Fitting 120B MoE Models on Consumer Hardware via Fused Compressed-Space Attention
Aug 8, 2026Anthony. Lui, Mohamed. Elsaied, N. P. SavaniExpert Offloading Mixture-of-Experts Language Models
Cognizant AI & Analytics, London, UB8 3PH, UK · Southern Methodist University (SMU), Engineering Management, Lyle School of Engineering, Dallas, TX, USA · Cognizant AI Lab, Responsible AI Office, San Francisco, USA +1
Output-Aware Rotation for INT2 KV-Cache Quantization
Aug 3, 2026Vincent-Daniel Yun, Woosang Lim, Minsoo Cheong +4LLM Inference KV Caching
University of Southern California · Seoul National University · Inha University
WitCert: Sound Runtime Risk Observability and Gating for KV-Cache Quantization
Jul 30, 2026Fanzhe Wei, Li Liu, Ziyang Wang +1KV Caching Memory-Efficient Inference
Metask Lab
MXAttention: Data-Free Optimal Scaling and Pre-Normalization Quantization for MXFP4 Attention
Jul 27, 2026Jianlin Yu, Jing Lin, Linghui Kong +13Video Diffusion Models Diffusion Model Quantization
Huawei Technologies Co., Ltd.
Regularize or Localize: When Training-Time KV-Cache Geometry Pays Under Quantization
Jul 19, 2026Libo Sun, Po-Wei Harn, Zewei Zhang +2KV Caching Attention Quantization
Department of Computer Science and Software Engineering, Auburn University, Auburn, AL 36830, USA · Department of Information Management, National Central University, Taoyuan 320317, Taiwan
AVQ-Attention: Adaptive Vector-Quantized Attention
Jul 14, 2026Winfried van den dool, Patrick Forré, Amir Habibian +2Transformer Attention Vector Quantization
QUVA Lab, University of Amsterdam, The Netherlands · AMLab, Informatics Institute, University of Amsterdam, The Netherlands · AI4Science Lab, University of Amsterdam, The Netherlands +3
Full-Stack FP4: Stable LLM Pretraining with Quantized Projections, Optimizers, and Attention
Jul 5, 2026Siyu Ding, Mingchuan Ma, Jiabo Tong +3Language Model Pretraining Mixed-Precision Quantization
Institute of Automation, Chinese Academy of Sciences · Sichuan University · Beijing Zhongguancun Academy +3
HiFA4: Training-Free 4-bit FlashAttention on Ascend HIF4 NPUs for LLM Inference
Jul 5, 2026Hui Dong, Yanzhao Li, Jie Gao +5Neural Network Quantization LLM Quantization
Huawei Technologies
RotateAttention: RoPE-Aware Rotation and Range Rectification for INT4 Quantized Attention in Video Generation
Jul 1, 2026Yaofu Liu, Wanli Lan, Jinxi Li +2Mixed-Precision Quantization Rotary Positional Embeddings
The Hong Kong University of Science and Technology, Hong Kong SAR, China · Independent Researcher, China
RoPE-Aware Bit Allocation for KV-Cache Quantization
Jun 23, 2026Fengfeng Liang, Yuechen Zhang, Jiaya JiaKV Caching LLM Inference Acceleration
1Hong Kong University of Science and Technology · 2The Chinese University of Hong Kong · 3MiMo, Xiaomi Corporation
P-Cast Precision in FP8 Attention: Sink-Induced Collapse and the Optimality of S=2^8
Jun 2, 2026Reed LauSoftmax Attention Attention Sinks
Tencent
Quantized Keys Steal Attention: Bias Correction for KV-Cache Compression in Video Diffusion
May 25, 2026Tuna Tuncer, Felix Becker, Thomas PfeilVideo Diffusion Models Self-Attention
1Technical University of Munich · 2Tensordyne
ThriftAttention: Selective Mixed Precision for Long-Context FP4 Attention
May 21, 2026Joe SharrattMixed-Precision Quantization Self-Attention
Runtime-Certified Bounded-Error Quantized Attention
May 20, 2026Dean CalverLLM Inference KV Caching
Independent Researcher
When Quantization Is Free: An int4 KV Cache That Outruns fp16 on Apple Silicon
May 7, 2026Mohamed Amine BergachGPU Kernel Optimization KV Caching
Illumina, San Diego, CA, USA
HeadQ: Model-Visible Distortion and Score-Space Correction for KV-Cache Quantization
May 5, 2026Jorge L. Ruiz WilliamsSelf-Attention KV Caching
Independent Researcher

Oct 24, 2026J/K move · Enter open · S save

RAQ-MLA: Role-Aware Quantization for Multi-Head Latent Attention Caches

Tianqi Chen, Xunchao Li, Linchuan Xie, Guanghua Yu+3

C2DL, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Tencent

LLM Quantization Attention Quantization KV-Cache Quantization Multi-Head Latent Attention Post-Training Quantization

Open paper AskPDF arXiv

Multi-Head Latent Attention (MLA) has been adopted by recent frontier large language models. Instead of per-head keys and values, MLA caches a compressed KV latent that is up-projected into keys and values at decode time, together with a narrow positional key that carries position information into the attention score. Existing KV-cache quantization methods are designed around the keys and values of standard attention and do not account for what MLA actually caches, which costs accuracy at low bit widths. We analyze the MLA cache both in activation distribution and in how quantization error propagates to the attention output, and show that the two cached tensors play different roles: the latent mainly supplies the values, whereas the positional key dominates the attention score. We therefore propose RAQ-MLA (Role-Aware Quantization for Multi-Head Latent Attention Caches), which fits an invertible transform to each cached tensor against the error that tensor delivers to the attention output: attention-weighted value error for the latent and score perturbation for the positional key. Without training, RAQ-MLA quantizes the full MLA cache to about 2.5 bits per element, a 6.5x increase in theoretical KV-cache capacity, while remaining nearly lossless on reasoning and coding tasks across four MLA models.