LLM Quantization

LLM: Large Language Model

LLM quantization studies how to represent model weights, activations, and attention key-value caches with fewer bits while preserving useful behavior and making deployment more efficient. Post-training methods such as GPTQ and AWQ use calibration data to round weights, while quantization-aware training, mixed-precision allocation, rotations, and outlier handling aim to reduce errors that accumulate across layers. Work also co-designs number formats and GPU, CPU, and edge-device kernels, evaluating trade-offs with language-model perplexity, zero-shot and reasoning benchmarks, memory, latency, and energy. Four-bit inference is often practical, but performance at very low bit widths remains model- and task-dependent, and an open challenge is ensuring that gains in aggregate accuracy do not hide losses in reasoning, safety, fairness, or uncertainty.

Latest papers

388 papers84 papers in the last four weeks, up 121% on the four weeks before. 0.5% of all new papers.

Finer is Better (with the Right Scaling)
May 8, 2026Clemens Schaefer, Gil TabakLLM Quantization 4-Bit Quantization
Google LLC, Mountain View, Ca
Litespark Inference For CPUs: Ultra-Fast SIMD Framework for Ternary (1.58-bit) Language Models
May 7, 2026Nii Osae Osae Dade, Tony Morri, Moinul Hossain Rahat +2LLM Quantization Efficient Neural Network Inference
Mindbeam AI
Normalized Architectures are Natively 4-Bit
May 7, 2026Maxim Fishman, Brian Chmiel, Ron Banner +2LLM Quantization Neural Representation Geometry
∧NVIDIA · ◦Department of Electrical and Computer Engineering - Technion, Haifa, Israel
Saliency-Aware Regularized Quantization Calibration for Large Language Models
May 7, 2026Yanlong Zhao, Xiaoyuan Cheng, Huihang Liu +6LLM Quantization Language Model Calibration
University of Science and Technology of China · University College London · 3Shanghai University of Finance and Economics +5
OSAQ: Outlier Self-Absorption for Accurate Low-bit LLM Quantization
May 6, 2026Zhikai Li, Zhen Dong, Xuewen Liu +2LLM Quantization Weight-Only Quantization
Institute of Automation, Chinese Academy of Sciences · University of California, Santa Barbara.
QuIDE: Mastering the Quantized Intelligence Trade-off via Active Optimization
May 5, 2026Xiantao JiangMixed-Precision Quantization LLM Quantization
College of Information Engineering, Shanghai Maritime University Shanghai 201306, China
Statistically-Lossless Quantization of Large Language Models
May 4, 2026Michael Helcig, Eldar Kurtic, Dan AlistarhLLM Quantization LLM Inference Acceleration
ETH Z̈urich · ISTA
Quantization Undoes Alignment: Bias Emergence in Compressed LLMs Across Models and Precision Levels
May 2, 2026Plawan Kumar Rath, Rahul MaliakkalLLM Quantization Social Bias in Language Models
Meta
AGoQ: Activation and Gradient Quantization for Memory-Efficient Distributed Training of LLMs
May 1, 2026Wenxiang Lin, Juntao Huang, Luhan Zhang +5LLM Quantization Communication-Efficient Distributed Training
School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen · Huawei Technologies Ltd.
Silicon Showdown: Performance, Efficiency, and Ecosystem Barriers in Consumer-Grade LLM Inference
May 1, 2026Abdurrahman Javat, Allan KazakovLLM Inference Efficiency LLM Quantization
Bahcesehir University, Istanbul, Türkiye
BWLA: Breaking the Barrier of W1AX Post-Training Quantization for LLMs
May 1, 2026Zhixiong Zhao, Zukang Xu, Dawei YangLLM Quantization LLM Inference Acceleration
1Houmo AI
Technical Report: Activation Residual Hessian Quantization (ARHQ) for Low-Bit LLM Quantization
Apr 30, 2026YiFeng Wang, Zhun Sun, Keisuke SakaguchiLLM Quantization Activation Quantization
Graduate School of Information Sciences Tohoku University
CoQuant: Joint Weight-Activation Subspace Projection for Mixed-Precision LLMs
Apr 29, 2026Zhe Ding, Su Pan, Duowei PanMixed-Precision Quantization LLM Quantization
School of Internet of Things, Nanjing University of Posts and Telecommunications, Nanjing, China · Amazon AGI, Seattle, WA, USA
Density Field State Space Models: 1-Bit Distillation, Efficient Inference, and Knowledge Organization in Mamba-2
Apr 28, 2026Chirag ShindeModel Compression LLM Quantization
Independent Researcher
BitRL: Reinforcement Learning with 1-bit Quantized Language Models for Resource-Constrained Edge Deployment
Apr 27, 2026Md. Ashiq Ul Islam Sajid, Mohammad Sakib Mahmood, Md. Tareq Hasan +3LLM Quantization RL for Language Models
Department of Computer Science BRAC University Dhaka 1212, Bangladesh · Department of Computer Science Missouri State University Springfield, MO, USA · Department of Computer Science & Engineering Prime University Dhaka, Bangladesh +2
Coverage-Based Calibration for Post-Training Quantization via Weighted Set Cover over Outlier Channels
Apr 27, 2026Ibne Farabi Shihab, Sanjeda Akter, Anuj SharmaLLM Quantization Activation Outliers
Department of Computer Science, Iowa State University · Department of Civil, Construction & Environmental Engineering, Iowa State University
QuantClaw: Precision Where It Matters for OpenClaw
Apr 24, 2026Manyi Zhang, Ji-Fu Li, Zhongao Sun +5Mixed-Precision Quantization LLM Quantization
Huawei Technologies · National University of Singapore · University of Science and Technology of China
RateQuant: Optimal Mixed-Precision KV Cache Quantization via Rate-Distortion Theory
Apr 22, 2026Fei Zuo, Zikang Zhou, Hao Cong +2LLM Inference Efficiency Mixed-Precision Quantization
1BA TechWorks (BMW Group) · 2National University of Singapore · 3Tsinghua University
From Signal Degradation to Computation Collapse: Uncovering the Two Failure Modes of LLM Quantization
Apr 21, 2026Chenxi Zhou, Pengfei Cao, Jiang Li +4LLM Quantization Post-Training Quantization
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences · The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences · College of Computer Science, Inner Mongolia University +1
LBLLM: Lightweight Binarization of Large Language Models via Three-Stage Distillation
Apr 21, 2026Siqing Song, Chuang Wang, Yong Lang +2LLM Quantization Activation Quantization
MAIS, Institute of Automation, Chinese Academy of Sciences, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, China · Central Media Technology Institute, Huawei
Benchmarking System Dynamics AI Assistants: Cloud Versus Local LLMs on CLD Extraction and Discussion
Apr 20, 2026Terry LeitchLLM Quantization Dynamical Systems
Ruxton AI
GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling
Apr 20, 2026Alireza Dadgarnia, Soroush Tabesh, Mahdi Nikdan +4LLM Quantization Post-Training Quantization
ISTA · ETH Zürich · ISTA & Red Hat AI +1
Depth Registers Unlock W4A4 on SwiGLU: A Reader/Generator Decomposition
Apr 20, 2026Ziyang LiuLLM Quantization 4-Bit Quantization
Independent Researcher
DuQuant++: Fine-grained Rotation Enhances Microscaling FP4 Quantization
Apr 20, 2026Haokun Lin, Xinle Jia, Haobo Xu +7LLM Quantization 4-Bit Quantization
CASIA · CityU · NJU +3
D-QRELO: Training- and Data-Free Delta Compression for Large Language Models via Quantization and Residual Low-Rank Approximation
Apr 18, 2026Junlin Li, Shuangyong Song, Guodong Du +6LLM Quantization LLM Compression
Harbin Institute of Technology, Shenzhen, China · TeleAI of China Telecom · The Hong Kong Polytechnic University +1
When Flat Minima Fail: Characterizing INT4 Quantization Collapse After FP32 Convergence
Apr 16, 2026Marcus ArmstrongLLM Quantization 4-Bit Quantization
Department of Computer Science University of Houston Houston, TX 77004
Robust Ultra Low-Bit Post-Training Quantization via Stable Diagonal Curvature Estimate
Apr 15, 2026Jaemin Kim, Sungkyun Kim, Junyeol Lee +1LLM Quantization Language Model Calibration
Seoul National University · Hanyang University
A KL Lens on Quantization: Fast, Forward-Only Sensitivity for Mixed-Precision SSM-Transformer Models
Apr 15, 2026Jason Kong, Nilesh Prasad Pandey, Flavio Ponzina +1Quantization Sensitivity Analysis LLM Quantization
OSC: Hardware Efficient W4A4 Quantization via Outlier Separation in Channel Dimension
Apr 14, 2026Zhiyuan Zhang, Yanzhao Li, Zhiqiang Zou +4LLM Quantization Neural Network Quantization
Huawei Technology
ReSpinQuant: Efficient Layer-Wise LLM Quantization via Subspace Residual Rotation Approximation
Apr 13, 2026Suyoung Kim, Sunghyun Wee, Hyeonjin Kim +3LLM Quantization Post-Training Quantization
Seoul National University, Seoul, Republic of Korea · LG Electronics, Seoul, Republic of Korea

May 8, 2026J/K move · Enter open · S save

Finer is Better (with the Right Scaling)

Clemens Schaefer, Gil Tabak

Google LLC, Mountain View, Ca

LLM Quantization 4-Bit Quantization Low-Bit Quantization Neural Network Quantization

Open paper AskPDF arXiv

Microscaling is a critical technique for preserving the quality of Large Language Models (LLMs) quantized to ultra-low precision formats. Intuitively, finer block sizes should yield lower quantization error; however, a paradox recently identified by Fasoli et al. (2026) demonstrates that standard abs-max scaling can actually result in degraded model quality as block sizes shrink. In this work, we investigate the underlying mechanics of this phenomenon. We demonstrate that this degradation is not an inherent limitation of finer granularity, but is primarily driven by how elements in smaller blocks statistically cluster closer to their local block maximum, interacting poorly with the coarse subnormal E4M3 values used as scaling factors. Specifically, we show that i) preventing the scaling factor from underflowing to zero mitigates errors caused by extreme underflow, ii) targeted algorithmic interventions like the 4-over-6 methodology that give more flexibility to the choice of scaling factor resolve the paradox for larger values, and iii) a brute-force search establishes an optimal baseline, confirming that the theoretical Mean Squared Error (MSE) strictly improves with finer block sizes. Ultimately, our findings highlight a critical insight for hardware-software co-design: the block-size paradox is partially an artifact of naive scale selection. While using hierarchical scaling factors or wider formats like UE5M3 interchangeably resolves much of the quality loss, we found the 4-over-6 scale selection heuristic can even further improve quality, especially for very small block sizes. Consequently, maximizing the performance of next-generation ML accelerators will require treating silicon format specifications and software scaling algorithms as tightly coupled design choices.