Low-Bit Quantization

Follow this topic Ask about this topic

Low-bit quantization studies how to represent model weights, activations, attention caches, and training states with fewer bits, reducing memory use and enabling faster or more energy-efficient inference. Post-training methods use calibration data, error reconstruction, Hessian information, rotations, and outlier handling, while quantization-aware training adapts models to the errors they will encounter at deployment. Mixed precision and scalar or vector codebooks allocate limited precision to sensitive components, with specialized kernels and hardware formats needed to turn compression into real speed and energy gains. The central tension is that aggressive compression can damage generation, reasoning, or long-context behavior even when standard accuracy metrics look stable, and improvements in quantization quality do not always translate into faster end-to-end systems.

Latest papers

237 papers46 papers in the last four weeks, up 100% on the four weeks before. 0.3% of all new papers.

A Unified Hessian-Guided Framework for Post-Training Quantization
Oct 29, 2026Mads Toftrup, Marco Federici, Boris van Breugel +2Quantization LLM Quantization
Aarhus University · Qualcomm AI Research
Low-Bit GDN Recurrent-State Quantization: Global Gradient-aware Allocation and Pure-Write Sliding Windows
Oct 29, 2026Qinghang Huang, Xuchu Huang, Xunzhao YinLLM Quantization Recurrent-State Quantization
Zhejiang University
CoS-VQ: Coordinated Shared-Scale Vector Quantization for Large Language Models
Oct 27, 2026Zukang Xu, Zhixiong Zhao, Xing Hu +3LLM Quantization Vector Quantization
Houmo AI · Peking University · Southeast University
ThinkBit: Search Codewords, Recalibrate Scales, Select by Reasoning
Oct 26, 2026Rhea Sanjay Sukthanker, Pashmina CameronLLM Quantization Low-Bit Quantization
Microsoft Applied Sciences Group
ResCom: Activation Residual Compensation for Large Language Model Quantization
Oct 25, 2026Ning Yang, Bohan Cui, Zhuowen Liu +1LLM Quantization Quantization Error Compensation
Institute of Automation, Chinese Academy of Sciences · Fudan University · The Chinese University of Hong Kong +1
BiLoCo: Binary Low-Rank Corrections for Post-Training Quantization
Oct 23, 2026David Jin, Beshr IslamBouli, Tarushii Goel +2LLM Quantization Post-Training Quantization
Massachusetts Institute of Technology
MPQ: A Message-Passing View of Post-Training Quantization
Oct 23, 2026Yanshuo Chen, Chenxi Liu, Reza Shirkavand +1Quantization LLM Quantization
Department of Computer Science, University of Maryland, College Park, MD 20742
Decode Late, Multiply Once: Rethinking the Dot Product for Quantized Weights
Oct 22, 2026Jonathan TapsonNeural Network Hardware Acceleration Low-Bit Quantization
BrainChip Inc, Laguna Hills, CA, USA
Low-Bit N-Gram Embeddings for Language Models
Oct 22, 2026Stephen Chung, Wenyu DuLLM Compression Low-Bit Quantization
DualverseAI
Compressing Onboard Language Models for Space Robots under a Behavioral-Fidelity Requirement
Oct 21, 2026Jia Deng, Zizhe Wang, Xin Lu +2LLM Compression LLM Quantization
Astroup Lab, Beijing, China
Extreme Low-State Quantized Transformer for Electricity-Load Forecasting
Oct 20, 2026Muhammad Sabbir Alam, Harindra Sandun Mavikumbure, Victor Cobilean +2Low-Bit Quantization Quantization-Aware Training
Department of Mechanical and Nuclear Engineering, Virginia Commonwealth University, Richmond, VA, 23284, USA · Department of Computer Science, Virginia Commonwealth University, Richmond, VA, 23284, USA · Department of Electrical and Computer Engineering, Virginia Commonwealth University, Richmond, VA, 23284, USA
Beyond 4 bits: Data-Centric Post-training Quantization and Recovery
Oct 20, 2026Rhea Sanjay Sukthanker, Riccardo Grazzi, James Hensman +1LLM Quantization Post-Training Quantization
Applied Sciences Group (ASG), Microsoft · Microsoft Research
E8-QAT: Efficient 2-Bit Quantization-Aware Training with Implicit Lattice Codebooks
Oct 20, 2026Xinyu Lei, Hong Huang, Jiebin Zhang +6LLM Quantization Quantization-Aware Training
Peking University · City University of Hong Kong · Tencent
LIRQuant: An Efficient Locally Isolated Rotation Quantization Method for Large Language Models
Oct 19, 2026Desheng Xu, Hanglin Li, Tianchen Song +2LLM Quantization Post-Training Quantization
Li Auto Inc.
ShiftQAT: Quantization-Aware Training for the Numerical Contract of a Fixed-Function Edge CNN Accelerator
Oct 15, 2026Oguzhan Buyuksolak, Osman Erman Okman, Mehmet Gorkem UlkarHardware-Aware Quantization Neural Network Quantization
Analog Devices, Inc.
RoLaQ: Joint Rotation and Range Calibration for 2-Bit KV Caches in Multi-Head Latent Attention
Oct 14, 2026Shiyu Ji, Peng Ling, Chen Lin +4KV-Cache Quantization Low-Bit Quantization
Research Center for Social Computing and Interactive Robotics, Harbin Institute of Technology, China · Tsinghua University, China · Baidu Inc., China
SpectrumQuant: Spectrum Alignment for LLM Quantization
Oct 14, 2026Dwaipayan Haldar, Indranil Patra, Azhar Yousuf +1LLM Quantization Rotation-Based Quantization
Indian Institute of Science Bengaluru-560012, Karnataka, India
Not All Frames Are Equal:Difference-Aware Quantization for Ultra-Low-Bit ASR
Oct 14, 2026Woori Jeon, Jungmin SoLow-Bit Quantization Automatic Speech Recognition
Sogang University, Seoul, Korea
PN-Mix: Pair-Norm Guided Mixed-Precision Key Cache Quantization for Efficient Long-Output LLM Inference
Oct 14, 2026Gil Jae Song, Sungjoo YooKV-Cache Quantization Mixed-Precision Quantization
Seoul National University
LoopQuant: Accurate and Efficient Post-Training Quantization for Looped Transformers
Oct 12, 2026Zhixiong Zhao, Zukang Xu, Xing Hu +3Post-Training Quantization LLM Quantization
School of Integrated Circuits, Peking University · Houmo AI
Preserving Updates in Low-Bit Recurrent State Quantization
Oct 11, 2026Haotian Li, Liran Dong, Kaicheng Xiao +1Recurrent-State Quantization Low-Bit Quantization
The Chinese University of Hong Kong
E8 Lattice Key Quantization for Low-Bit KV-Cache Compression
Oct 11, 2026Sasho Nedelkoski, Alexander Acker, Soeren Becker +2KV-Cache Compression KV-Cache Quantization
Technical University of Berlin Berlin, Germany · logsight.ai Berlin, Germany
HGA-Quant: Heterogeneity-Aware Quantization for AutoRegressive Visual Generation Models
Oct 11, 2026Xinmeng Liu, Xianglong Yan, Kaicheng Yang +2Post-Training Quantization Neural Network Quantization
Shanghai Jiao Tong University · Xi’an University of Electronic Science and Technology
When Dynamic Scaling Fails: A Scale-State Feedback Governs Recurrent-State Quantization in Linear-Attention and SSM LLMs
Oct 10, 2026Yuhang JiangRecurrent-State Quantization LLM Quantization
Independent Researcher
VLCQ: Post-training quantization for deep neural networks using variable length coding
Oct 10, 2026Reem Abdel-Salam, Ahmed H. Abdel-Gawad, Amr G. WassalPost-Training Quantization Neural Network Quantization
Computer Engineering Department, Faculty of Engineering, Cairo University, Giza, Egypt
FidelityQ: Preserving Predictive Fidelity in Ultra-Low-Bit Language Model Quantization
Oct 10, 2026Yujie Chen, Weiyan Zheng, Yiwu Yao +7Low-Bit Quantization Neural Network Quantization
The Hong Kong Polytechnic University · Huawei Technologies Ltd. · The University of Hong Kong +2
SAGE: Pushing 2-Bit Vector-Quantized LLMs toward Real-World Usability
Oct 10, 2026Xianglong Yan, Zenan Wu, Kewei Zhang +2LLM Quantization Weight-Only Quantization
Shanghai Jiao Tong University
DualQuant: Dual-scale Calibration-free Weight-and-Activation Quantization for LLMs
Oct 9, 2026Mahsa Salmani, Sayeh Sharify, Hesham MostafaLLM Quantization Post-Training Quantization
d-Matrix
ResidualQuant: KV Cache Quantization for Looped Transformers with 2-Bit Residuals
Oct 7, 2026Heejun Kim, Junyoung Lee, SangLyul Cho +3KV-Cache Quantization Efficient Transformer Inference
KAIST · Yonsei University · Seoul National University
Dual-QK: Sharp Queries and Flat Keys for Prunable 2-bit KV Caches
Oct 7, 2026Sunjoo Whang, Jungjun Oh, Minsung Kim +5KV-Cache Compression KV-Cache Quantization
KAIST · GIST · Google Research

Oct 29, 2026J/K move · Enter open · S save

A Unified Hessian-Guided Framework for Post-Training Quantization

Mads Toftrup, Marco Federici, Boris van Breugel, Paul Whatmough+1

Aarhus University · Qualcomm AI Research

Quantization LLM Quantization Post-Training Quantization Low-Bit Quantization Activation Quantization

Open paper AskPDF arXiv

Low-bit quantization enables efficient deployment of large language models by reducing memory footprint and computational cost. Achieving high accuracy at low precision requires selecting quantization ranges, linear transformations, and weight rounding decisions. Existing approaches typically optimize these components independently using local reconstruction objectives, which fail to account for the varying impact of quantization errors on downstream predictions. Direct optimization of task-level metrics would provide a more faithful objective, but is challenging due to the discontinuous nature of quantization operations and the high computational cost of evaluating their downstream effects. In this work, we derive a Hessian-guided approximation of the output change induced by joint weight and activation quantization. Based on a tractable Fisher approximation, we formulate a unified post-training quantization framework that enables the joint optimization of quantization ranges, linear transformations, and weight rounding under a common output-sensitivity objective. The resulting method remains practical for modern large language models and consistently outperforms state-of-the-art transformation and rounding methods across both channel-wise and block-wise quantization settings.