Low-Bit Quantization

Follow this topic Ask about this topic

Low-bit quantization studies how to represent model weights, activations, attention caches, and training states with fewer bits, reducing memory use and enabling faster or more energy-efficient inference. Post-training methods use calibration data, error reconstruction, Hessian information, rotations, and outlier handling, while quantization-aware training adapts models to the errors they will encounter at deployment. Mixed precision and scalar or vector codebooks allocate limited precision to sensitive components, with specialized kernels and hardware formats needed to turn compression into real speed and energy gains. The central tension is that aggressive compression can damage generation, reasoning, or long-context behavior even when standard accuracy metrics look stable, and improvements in quantization quality do not always translate into faster end-to-end systems.

Latest papers

237 papers46 papers in the last four weeks, up 100% on the four weeks before. 0.3% of all new papers.

MuonQ: Enhancing Low-Bit Muon Quantization via Directional Fidelity Optimization
May 12, 2026Yupeng Su, Ruijie Zhang, Ziyue Liu +24-Bit Quantization Muon Optimizer
University of California, Santa Barbara
ADMM-Q: An Improved Hessian-based Weight Quantizer for Post-Training Quantization of Large Language Models
May 11, 2026Ryan Lucas, Mehdi Makni, Xiang Meng +2LLM Quantization Alternating Direction Method of Multipliers
MIT Operations Research Center · MIT Sloan School of Management · MIT Center for Statistics
Theory-optimal Quantization Based on Flatness
May 11, 2026Xiusheng Huang, Zhe Li, Xuanwu Yin +5LLM Quantization Activation Quantization
The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences · Beijing Academy of Artificial Intelligence +2
PermuQuant: Lowering Per-Group Quantization Error by Reordering Channels for Diffusion Models
May 10, 2026Yongsen Cheng, Kai Liu, Kaiwen Tao +5Diffusion Model Quantization Post-Training Quantization
Shanghai Jiao Tong University · Huawei Noah’s Ark Lab
Fitting Is Not Enough: Smoothness in Extremely Quantized LLMs
May 9, 2026Yuzhuang Xu, Xu Han, Yuxuan Li +2LLM Quantization Post-Training Quantization
1Harbin Institute of Technology, Harbin, China · 2Tsinghua University, Beijing, China
Finer is Better (with the Right Scaling)
May 8, 2026Clemens Schaefer, Gil TabakLLM Quantization 4-Bit Quantization
Google LLC, Mountain View, Ca
Direction-Preserving Number Representations
May 8, 2026Bardia Zadeh, George A. Constantinides4-Bit Quantization Low-Bit Quantization
Department of Electrical and Electronic Engineering Imperial College London London, United Kingdom
LoopQ: Quantization for Looped Language Models
May 8, 2026Rui Fang, Hsi-Wen Chen, Chia-Ming Cheng +1Recurrent Transformers Language Modeling
National Taiwan University
OSAQ: Outlier Self-Absorption for Accurate Low-bit LLM Quantization
May 6, 2026Zhikai Li, Zhen Dong, Xuewen Liu +2LLM Quantization Weight-Only Quantization
Institute of Automation, Chinese Academy of Sciences · University of California, Santa Barbara.
Statistically-Lossless Quantization of Large Language Models
May 4, 2026Michael Helcig, Eldar Kurtic, Dan AlistarhLLM Quantization LLM Inference Acceleration
ETH Z̈urich · ISTA
Colinearity Decay: Training Quantization-Friendly ViTs with Outlier Decay
May 2, 2026Jin Tong, Guang Liang, Peilin Sun +1Vision Transformer Activation Outliers
State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing 210023, China · School of Artificial Intelligence, Nanjing University, Nanjing 210023, China · Zhongguancun Academy, Beijing 100094, China
BWLA: Breaking the Barrier of W1AX Post-Training Quantization for LLMs
May 1, 2026Zhixiong Zhao, Zukang Xu, Dawei YangLLM Quantization LLM Inference Acceleration
1Houmo AI
Technical Report: Activation Residual Hessian Quantization (ARHQ) for Low-Bit LLM Quantization
Apr 30, 2026YiFeng Wang, Zhun Sun, Keisuke SakaguchiLLM Quantization Activation Quantization
Graduate School of Information Sciences Tohoku University
CoQuant: Joint Weight-Activation Subspace Projection for Mixed-Precision LLMs
Apr 29, 2026Zhe Ding, Su Pan, Duowei PanMixed-Precision Quantization LLM Quantization
School of Internet of Things, Nanjing University of Posts and Telecommunications, Nanjing, China · Amazon AGI, Seattle, WA, USA
QFlash: Bridging Quantization and Memory Efficiency in Vision Transformer Attention
Apr 28, 2026Sehyeon Oh, Yongin Kwon, Jemin LeeSoftmax Attention GPU Acceleration
University of Science and Technology, Daejeon, Republic of Korea · Electronics and Telecommunications Research Institute, Daejeon, Republic of Korea · Pusan National University, Busan, Republic of Korea +1
BitRL: Reinforcement Learning with 1-bit Quantized Language Models for Resource-Constrained Edge Deployment
Apr 27, 2026Md. Ashiq Ul Islam Sajid, Mohammad Sakib Mahmood, Md. Tareq Hasan +3LLM Quantization RL for Language Models
Department of Computer Science BRAC University Dhaka 1212, Bangladesh · Department of Computer Science Missouri State University Springfield, MO, USA · Department of Computer Science & Engineering Prime University Dhaka, Bangladesh +2
Efficient INT8 Single-Image Super-Resolution via Deployment-Aware Quantization and Teacher-Guided Training
Apr 22, 2026Pham Phuong Nam Nguyen, Nam Tien Le, Thi Kim Trang Vo +1Efficient Neural Network Inference Quantization-Aware Training
University of Information Technology · 3Vietnam National University, Ho Chi Minh City · 2Ho Chi Minh City University of Technology
From Signal Degradation to Computation Collapse: Uncovering the Two Failure Modes of LLM Quantization
Apr 21, 2026Chenxi Zhou, Pengfei Cao, Jiang Li +4LLM Quantization Post-Training Quantization
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences · The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences · College of Computer Science, Inner Mongolia University +1
Revisiting RaBitQ and TurboQuant: A Symmetric Comparison of Methods, Theory, and Experiments
Apr 21, 2026Jianyang Gao, Yutong Gou, Yuexuan Xu +5Vector Quantization KV-Cache Quantization
ETH Zurich · Nanyang Technological University · University of Michigan +2
LBLLM: Lightweight Binarization of Large Language Models via Three-Stage Distillation
Apr 21, 2026Siqing Song, Chuang Wang, Yong Lang +2LLM Quantization Activation Quantization
MAIS, Institute of Automation, Chinese Academy of Sciences, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, China · Central Media Technology Institute, Huawei
GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling
Apr 20, 2026Alireza Dadgarnia, Soroush Tabesh, Mahdi Nikdan +4LLM Quantization Post-Training Quantization
ISTA · ETH Zürich · ISTA & Red Hat AI +1
A Note on TurboQuant and the Earlier DRIVE/EDEN Line of Work
Apr 20, 2026Ran Ben-Basat, Yaniv Ben-Itzhak, Gal Mendelson +3Residual VQ Vector Quantization
UCL and Broadcom · VMware Research by Broadcom · North Carolina State University +2
Towards Joint Quantization and Token Pruning of Vision-Language Models
Apr 19, 2026Xinqing Li, Xin He, Xindong Zhang +3Vision-Language Models VLM Quantization
VCIP, College of Computer Science, Nankai University · School of Computer Science and Engineering, Tianjin University of Technology · OPPO Research Institute +3
Robust Ultra Low-Bit Post-Training Quantization via Stable Diagonal Curvature Estimate
Apr 15, 2026Jaemin Kim, Sungkyun Kim, Junyeol Lee +1LLM Quantization Language Model Calibration
Seoul National University · Hanyang University
ReSpinQuant: Efficient Layer-Wise LLM Quantization via Subspace Residual Rotation Approximation
Apr 13, 2026Suyoung Kim, Sunghyun Wee, Hyeonjin Kim +3LLM Quantization Post-Training Quantization
Seoul National University, Seoul, Republic of Korea · LG Electronics, Seoul, Republic of Korea
SEPTQ: A Simple and Effective Post-Training Quantization Paradigm for Large Language Models
Apr 11, 2026Han Liu, Haotian Gao, Xiaotong Zhang +5LLM Quantization Post-Training Quantization
Dalian University of Technology, Dalian, China · Peking University, Beijing, China · Shenzhen MSU-BIT University, Shenzhen, China +1
Initialisation Determines the Basin: Efficient Codebook Optimisation for Extreme LLM Quantization
Apr 9, 2026Ian W. Kennedy, Nafise Sadat MoosaviLLM Quantization Neural Network Quantization
Department of Computer Science, University of Sheffield, Sheffield, UK
Bit-by-Bit: Progressive QAT Strategy with Outlier Channel Splitting for Stable Low-Bit LLMs
Apr 9, 2026Binxing Xu, Hao Gu, Lujun Li +8LLM Quantization Quantization-Aware Training
Zhejiang University · Hong Kong University of Science and Technology · City University of Hong Kong
Weight Group-wise Post-Training Quantization for Medical Foundation Model
Apr 9, 2026Yineng Chen, Peng Huang, Aozhong Zhang +9Post-Training Quantization Low-Bit Quantization
University at Albany, SUNY · Southwest Jiaotong University · Purdue University +1
BWTA: Accurate and Efficient Binarized Transformer by Algorithm-Hardware Co-design
Apr 5, 2026Yifu Ding, Xianglong Liu, Shenghao Jin +2Binary Quantization Low-Bit Quantization
School of Computer Science and Engineering, Beihang University, Beijing, China. · Institute of Artificial Intelligence, Beihang University, Beijing, China. · Department of Automation, Tsinghua University, Beijing, China.

May 12, 2026J/K move · Enter open · S save

MuonQ: Enhancing Low-Bit Muon Quantization via Directional Fidelity Optimization

Yupeng Su, Ruijie Zhang, Ziyue Liu, Yequan Zhao+1

University of California, Santa Barbara

4-Bit Quantization Muon Optimizer Low-Bit Quantization Neural Network Quantization

Open paper AskPDF arXiv

The Muon optimizer has emerged as a compelling alternative to Adam for training large language models, achieving remarkable computational savings through gradient orthogonalization. However, Muon's optimizer state is more sensitive to quantization errors: because the orthogonalization discards the magnitudes of singular values and retains only directional information, even small quantization errors in singular vector directions are amplified in the update. In this work, we propose MuonQ, a low-bit Muon training framework built on the principle of directional fidelity optimization. First, we apply a pre-quantization normalization so that each step introduces quantization errors of the same magnitude, preventing the accumulated error from developing a preferred direction. Second, we introduce a structural decomposition that separately quantizes the dominant singular components via power iteration, ensuring that quantization errors perturb only singular value magnitudes rather than rotating singular vector directions. Third, we adopt

μ

-law companding quantization to allocate higher resolution to densely packed momentum values, shifting the quantization objective from outlier preservation to dense-region distinguishability. Together, these techniques enable stable 4-bit quantization of Muon's optimizer states. Pre-training experiments on GPT-style and LLaMA-style models demonstrate that MuonQ at 4-bit precision closely matches full-precision Muon in both training loss and downstream task accuracy, while reducing optimizer state memory by up to 7.3

\times

. Our code is available at https://github.com/YupengSu/MuonQ.