Attention Quantization

Follow this topic Ask about this topic

Momentum

7 papers in the last four weeks, against 2 the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 27

All topics

LeapQuant: Efficient Linear Attention with Accurate Recurrent State Quantization
Sep 29, 2026Yi Pan, Haocheng Xi, Kan Zhu +10LLM Quantization LLM Inference Acceleration
UC Berkeley · University of Washington · MIT +2
QuantMLA: Function-Aligned Dual-Path Quantization for Low-Bit MLA KV Caching
Sep 29, 2026Zunhai Su, Yuxuan Sun, Jianchao Tan +5LLM Inference Acceleration Attention Quantization
The University of Hong Kong · Meituan LongCat Team · South China University of Technology +1
From Attention Sensitivity to Layer Role: Revisiting Mixed-Precision Quantization of Transformers
Sep 28, 2026Nafiseh HosseinpourFardi, Negar Alihadi, Mahmoudreza Babaei +2Mixed-Precision Quantization LLM Quantization
Sharif University of Technology, Tehran, Iran · GISMA University of Applied Sciences, Potsdam, Germany · BRAINS, Brandenburg Research Center for Applied Intelligent Systems, Potsdam, Germany +3
Pretraining Transformers with Quantized Softmax in Attention
Sep 27, 2026Shangzhen Zhu, Muyan Hu, Tomasz KozlowskiLanguage Model Pretraining Quantization-Aware Training
University of Illinois Urbana-Champaign
Global Ranks Survive, Selected Heads Shift: BOS-Sink Topology under 4-bit Weight-Only Quantization
Sep 20, 2026Kuanlin Chen, Chen-Wei Kuo, Cheng-En OuAttention Head Analysis 4-Bit Quantization
Independent Researcher Taoyuan, Taiwan · National Tsing Hua University Hsinchu, Taiwan
Attention Quantization for Tabular Foundation Models
Sep 14, 2026Jonas M. Kübler, Benjamin Jäger, Klemens Flöge +2Efficient Transformer Inference Tabular Foundation Models
Prior Labs
EFQ-Softmax: Exp-Free Quantization for Softmax
Sep 9, 2026Haohui Han, Yuming Wan, Hongni Wang +4Efficient Transformer Inference Transformer Attention
Xi'an Jiaotong University · Xi’an Jiaotong University · Huawei Technologies Co., Ltd +1
Hardware-Aware FP4 FlashAttention-4
Sep 3, 2026Robert HuMixed-Precision Quantization GPU Kernel Optimization
GRAPHCORE RESEARCH
HyQuant: Hybrid-Precision Quantization for LLM Attention
Aug 28, 2026Jiatong Ding, Bingxin Xing, Yu Zhang +9Mixed-Precision Quantization LLM Quantization
Shanghai Jiao Tong University · Xi’an Jiaotong University · Xiamen University +1
RotaryQuant: Fitting 120B MoE Models on Consumer Hardware via Fused Compressed-Space Attention
Aug 8, 2026Anthony. Lui, Mohamed. Elsaied, N. P. SavaniExpert Offloading Mixture-of-Experts Language Models
Cognizant AI & Analytics, London, UB8 3PH, UK · Southern Methodist University (SMU), Engineering Management, Lyle School of Engineering, Dallas, TX, USA · Cognizant AI Lab, Responsible AI Office, San Francisco, USA +1
Output-Aware Rotation for INT2 KV-Cache Quantization
Aug 3, 2026Vincent-Daniel Yun, Woosang Lim, Minsoo Cheong +4LLM Inference KV Caching
University of Southern California · Seoul National University · Inha University
WitCert: Sound Runtime Risk Observability and Gating for KV-Cache Quantization
Jul 30, 2026Fanzhe Wei, Li Liu, Ziyang Wang +1KV Caching Memory-Efficient Inference
Metask Lab
MXAttention: Data-Free Optimal Scaling and Pre-Normalization Quantization for MXFP4 Attention
Jul 27, 2026Jianlin Yu, Jing Lin, Linghui Kong +13Video Diffusion Models Diffusion Model Quantization
Huawei Technologies Co., Ltd.
Regularize or Localize: When Training-Time KV-Cache Geometry Pays Under Quantization
Jul 19, 2026Libo Sun, Po-Wei Harn, Zewei Zhang +2KV Caching Attention Quantization
Department of Computer Science and Software Engineering, Auburn University, Auburn, AL 36830, USA · Department of Information Management, National Central University, Taoyuan 320317, Taiwan
AVQ-Attention: Adaptive Vector-Quantized Attention
Jul 14, 2026Winfried van den dool, Patrick Forré, Amir Habibian +2Transformer Attention Vector Quantization
QUVA Lab, University of Amsterdam, The Netherlands · AMLab, Informatics Institute, University of Amsterdam, The Netherlands · AI4Science Lab, University of Amsterdam, The Netherlands +3
Full-Stack FP4: Stable LLM Pretraining with Quantized Projections, Optimizers, and Attention
Jul 5, 2026Siyu Ding, Mingchuan Ma, Jiabo Tong +3Language Model Pretraining Mixed-Precision Quantization
Institute of Automation, Chinese Academy of Sciences · Sichuan University · Beijing Zhongguancun Academy +3
HiFA4: Training-Free 4-bit FlashAttention on Ascend HIF4 NPUs for LLM Inference
Jul 5, 2026Hui Dong, Yanzhao Li, Jie Gao +5Softmax Attention LLM Quantization
Huawei Technologies
RotateAttention: RoPE-Aware Rotation and Range Rectification for INT4 Quantized Attention in Video Generation
Jul 1, 2026Yaofu Liu, Wanli Lan, Jinxi Li +2Mixed-Precision Quantization Rotary Positional Embeddings
The Hong Kong University of Science and Technology, Hong Kong SAR, China · Independent Researcher, China
RoPE-Aware Bit Allocation for KV-Cache Quantization
Jun 23, 2026Fengfeng Liang, Yuechen Zhang, Jiaya JiaKV Caching LLM Inference Acceleration
1Hong Kong University of Science and Technology · 2The Chinese University of Hong Kong · 3MiMo, Xiaomi Corporation
P-Cast Precision in FP8 Attention: Sink-Induced Collapse and the Optimality of S=2^8
Jun 2, 2026Reed LauSoftmax Attention Attention Sinks
Tencent
Quantized Keys Steal Attention: Bias Correction for KV-Cache Compression in Video Diffusion
May 25, 2026Tuna Tuncer, Felix Becker, Thomas PfeilVideo Diffusion Models Self-Attention
1Technical University of Munich · 2Tensordyne
ThriftAttention: Selective Mixed Precision for Long-Context FP4 Attention
May 21, 2026Joe SharrattMixed-Precision Quantization Self-Attention
Runtime-Certified Bounded-Error Quantized Attention
May 20, 2026Dean CalverLLM Inference KV Caching
Independent Researcher
When Quantization Is Free: An int4 KV Cache That Outruns fp16 on Apple Silicon
May 7, 2026Mohamed Amine BergachGPU Kernel Optimization KV Caching
Illumina, San Diego, CA, USA
HeadQ: Model-Visible Distortion and Score-Space Correction for KV-Cache Quantization
May 5, 2026Jorge L. Ruiz WilliamsSelf-Attention KV Caching
Independent Researcher
QFlash: Bridging Quantization and Memory Efficiency in Vision Transformer Attention
Apr 28, 2026Sehyeon Oh, Yongin Kwon, Jemin LeeSoftmax Attention GPU Acceleration
University of Science and Technology, Daejeon, Republic of Korea · Electronics and Telecommunications Research Institute, Daejeon, Republic of Korea · Pusan National University, Busan, Republic of Korea +1
Open-TQ-Metal: Fused Compressed-Domain Attention for Long-Context LLM Inference on Apple Silicon
Apr 18, 2026Sai VegasenaAI Accelerator Inference Self-Attention
Ensue

Sep 29, 2026, cs.LGJ/K move · Enter open · S save

LeapQuant: Efficient Linear Attention with Accurate Recurrent State Quantization

Yi Pan, Haocheng Xi, Kan Zhu, Xingyang Li+9

UC Berkeley · University of Washington · MIT+2

LLM Quantization LLM Inference Acceleration Delta-Rule Linear Attention Attention Quantization Neural Network Quantization

Open paper AskPDF arXiv

Recent LLMs increasingly adopt hybrid designs that replace standard attention with linear attention, such as Gated DeltaNet (GDN) and Kimi Delta Attention (KDA). Although they compress the context into a fixed-size recurrent state and substantially reduce the cost of long-context processing, repeatedly reading and updating that state remains a major inference bottleneck. Quantization offers a natural way to reduce this cost, but can significantly degrade model quality, due to the accumulation of rounding errors and the presence of outlier rows and columns in the state. To address these challenges, we propose LeapQuant, a training-free method that achieves near-lossless performance under 8-bit recurrent-state quantization. First, to mitigate error accumulation, we propose per-window quantization, which leaps over a window of tokens and quantizes the state only once at its end. Within a window, outputs are computed from the fixed low-bit state together with high-precision buffered updates. Second, to reduce the error introduced by each quantization, LeapQuant retains the state's largest outliers as a few high-precision Compensator Tokens, which share the update path of real tokens. We then smooth the remaining residual before quantization to further reduce the error. Comprehensive experiments across the Qwen, Kimi, and GLM model families show that LeapQuant substantially reduces memory and compute costs during inference. With accuracy comparable to the FP32 baseline, it achieves average speedups of 2.05--3.70

\times

at the kernel level and 1.47

\times

for end-to-end inference on NVIDIA B200, RTX PRO 6000, and RTX 5090 GPUs.

Momentum

Related topics

Latest papers 27