Low-Bit Quantization

Follow this topic Ask about this topic

Low-bit quantization studies how to represent model weights, activations, attention caches, and training states with fewer bits, reducing memory use and enabling faster or more energy-efficient inference. Post-training methods use calibration data, error reconstruction, Hessian information, rotations, and outlier handling, while quantization-aware training adapts models to the errors they will encounter at deployment. Mixed precision and scalar or vector codebooks allocate limited precision to sensitive components, with specialized kernels and hardware formats needed to turn compression into real speed and energy gains. The central tension is that aggressive compression can damage generation, reasoning, or long-context behavior even when standard accuracy metrics look stable, and improvements in quantization quality do not always translate into faster end-to-end systems.

Latest papers

237 papers46 papers in the last four weeks, up 100% on the four weeks before. 0.3% of all new papers.

Improving Quantized Model Performance in Qualitative Analysis with Multi-Pass Prompt Verification
Apr 4, 2026Aisvarya Adeseye, Jouni Isoaho, Adeyemi AdeseyeLLM Quantization Low-Bit Quantization
University of Turku, Turku, Finland · Brilloconnetz Partners avoin yhtiö, Turku, Finland
Zero-Shot Quantization via Weight-Space Arithmetic
Apr 3, 2026Daniele Solombrino, Antonio Andrea Gargiulo, Alessandro Zirilli +3Quantization Robustness Quantization
Sapienza University of Rome · Sapienza University of Rome / Paradigma
IsoQuant: Hardware-Aligned SO(4) Isoclinic Rotations for LLM KV Cache Compression
Mar 30, 2026Zhongping JiRotation-Based Quantization KV-Cache Compression
GlowQ: Group-Shared LOw-Rank Approximation for Quantized LLMs
Mar 26, 2026Selim An, Il hong Suh, Yeseong KimLLM Quantization LLM Inference Acceleration
Department of Artificial Intelligence, DGIST, Korea · COGA robotics, Korea · Department of Electrical Engineering, POSTECH, Korea
Beyond Outliers: A Data-Free Layer-wise Mixed-Precision Quantization Approach Driven by Numerical and Structural Dual-Sensitivity
Mar 18, 2026Hengyuan Zhang, Xinrong Chen, Zunhai Su +10Mixed-Precision Quantization Neural Network Quantization
The University of Hong Kong · Peking University · Tsinghua University +2
KANtize: Exploring Low-bit Quantization of Kolmogorov-Arnold Networks for Efficient Inference
Mar 18, 2026Sohaib Errabii, Olivier Sentieys, Marcello TraiolaLow-Bit Quantization Efficient Neural Network Inference
Univ Rennes, Inria, CNRS, IRISA, Rennes, France
Understanding Quantization of Optimizer States in LLM Pre-training: Dynamics of State Staleness and Effectiveness of State Resets
Mar 17, 2026Kristi Topollai, Anna ChoromanskaLow-Precision Training Low-Bit Quantization
New York University
Quantization Robustness of Monotone Operator Equilibrium Networks
Mar 11, 2026James Li, Philip H. W. Leong, Thomas ChaffeyQuantization Robustness Neural Network Quantization
School of Electrical and Computer Engineering, The University of Sydney, NSW, Australia
SERQ: Saliency-Aware Low-Rank Error Reconstruction for LLM Quantization
Mar 9, 2026Yeonsik Park, Hyeonseong Kim, Seungkyu ChoiLLM Quantization Post-Training Quantization
Kyung Hee University · Yonsei University
Evolution Strategy-Based Calibration for Low-Bit Quantization of Speech Models
Mar 9, 2026Lucas RakotoarivonyLow-Bit Quantization Activation Quantization
Thales, cortAIx Labs, 1 Av. Augustin Fresnel, 91120 Palaiseau, France
Post Training Quantization for Efficient Dataset Condensation
Mar 7, 2026Linh-Tam Tran, Sung-Ho BaeDataset Condensation Post-Training Quantization
Kyung Hee University, Republic of Korea
Sparse-BitNet: 1.58-bit LLMs are Naturally Friendly to Semi-Structured Sparsity
Mar 5, 2026Di Zhang, Xun Wu, Shaohan Huang +9LLM Inference Efficiency LLM Quantization
Microsoft Research · Peking University · South China University of Technology
Bielik-Q2-Sharp: A Comparative Study of Extreme 2-bit Quantization Methods for a Polish 11B Language Model
Mar 4, 2026Jakub PrejznerQuantization LLM Quantization
BitSharp, Independent Researcher, Rzeszów, Poland
Boosting Entropy with Bell Box Quantization
Mar 2, 2026Ningfeng Yang, Tor M. AamodtQuantization Neural Network Quantization
University of British Columbia
The AetherFloat Family: Block-Scale-Free Quad-Radix Floating-Point Architectures for AI Accelerators
Feb 26, 2026Keita MorisakiStochastic Rounding AI Accelerator Inference
Independent Hardware/Software Co-Design Researcher Shinagawa City, Tokyo, Japan
pQuant: Towards Effective Low-Bit Language Models via Decoupled Linear Quantization-Aware Training
Feb 26, 2026Wenzheng Zhang, Bingzheng Liu, Yang Hu +3LLM Quantization Low-Bit Quantization
School of Computer Science, Peking University · College of Future Information Technology, Fudan University · Center for Information Research, Academy of Military Sciences +2
Scaling Laws for Precision in High-Dimensional Linear Regression
Feb 22, 2026Dechen Zhang, Xuan Tang, Yingyu Liang +1High-Dimensional Linear Regression Scaling Laws
Institute of Data Science, The University of Hong Kong · School of Computing & Data Science, The University of Hong Kong · Institute of Data Science and School of Computing & Data Science, The University of Hong Kong +1
Uniform error bounds for quantized dynamical models
Feb 17, 2026Abdelkader Metakalard, Fabien Lauer, Kevin Colin +1System Identification Generalization Bounds
CRAN, SYNALP · ∗Universit´e de Lorraine, CNRS, CRAN, F-54000 Nancy, France · ∗∗Universit´e de Lorraine, CNRS, LORIA, F-54000 Nancy, France +2
1-Bit Wonder: Improving QAT Performance in the Low-Bit Regime through K-Means Quantization
Feb 17, 2026Sohir Maskey, Constantin Eichenberg, Johannes Messner +1Quantization LLM Quantization
Aleph Alpha Research · Graphcore Research
MatGPTQ: Efficient and Accurate Inference over Nested Quantized Models
Feb 3, 2026Maximilian Kleinegger, Elvir Crnčević, Dan AlistarhLLM Quantization LLM Inference Acceleration
1Vienna University of Technology, Vienna, Austria · Institute of Science & Technology Austria (ISTA), Vienna, Austria · 3Red Hat AI, Boston, USA
QuantKAN: A Unified Quantization Framework for Kolmogorov Arnold Networks
Nov 24, 2025Kazi Ahmed Asif Fuad, Lizhong ChenQuantization-Aware Training Kolmogorov-Arnold Networks
Department of EECS · Oregon State University · Corvallis, OR 97331
Activation Quantization of Vision Encoders Needs Prefixing Registers
Oct 6, 2025Seunghyeon Kim, Taesun Yeom, Jinho Kim +3Activation Outliers Activation Quantization
POSTECH · NYU Langone Health2 · Google3
LUQ: Layerwise Ultra-Low Bit Quantization for Multimodal Large Language Models
Sep 28, 2025Shubhang Bhatnagar, Andy Xu, Kar-Han Tan +1Mixed-Precision Quantization LLM Quantization
University of Illinois Urbana-Champaign · Work done as an intern at HP Inc. · University of California, Los Angeles +1
Squeeze10-LLM: Squeezing LLMs' Weights by 10 Times via a Staged Mixed-Precision Quantization Method
Jul 24, 2025Qingcheng Zhu, Yangyang Ren, Linlin Yang +6Mixed-Precision Quantization LLM Quantization
Beihang University · Communication University of China · Beijing Jiaotong University
A Comprehensive Evaluation on Quantization Techniques for Large Language Models
Jul 23, 2025Yutong Liu, Cairong Zhao, Guosheng HuLLM Quantization Post-Training Quantization
School of Computer Science and Technology, Tongji University, Shanghai 201804, China · School of Engineering Math and Technology, University of Bristol, BS8 1QU, UK
FPTQuant: Function-Preserving Transforms for LLM Quantization
Jun 5, 2025Boris van Breugel, Yelysei Bondarenko, Paul Whatmough +1LLM Quantization LLM Inference Acceleration
Qualcomm AI Research, Amsterdam, Netherlands
NSNQuant: A Double Normalization Approach for Calibration-Free Low-Bit Vector Quantization of KV Cache
May 23, 2025Donghyun Son, Euntae Choi, Sungjoo YooKV Caching Vector Quantization
Seoul National University

Apr 4, 2026J/K move · Enter open · S save

Improving Quantized Model Performance in Qualitative Analysis with Multi-Pass Prompt Verification

Aisvarya Adeseye, Jouni Isoaho, Adeyemi Adeseye

University of Turku, Turku, Finland · Brilloconnetz Partners avoin yhtiö, Turku, Finland

LLM Quantization Low-Bit Quantization LLM Hallucination Mitigation LLM Answer Verification

Open paper AskPDF arXiv

Quantized Large Language Models (LLMs) are used more often in qualitative analysis because they run fast and need fewer computing resources. This study examines how different lower bits quantization levels (8-bit, 4-bit, 3-bit, and 2-bit) and quantization types affect the performance of LLaMA-3.1 (8B) on qualitative analysis. The study uses expert and non-expert responses from 82 interview transcripts. Low-bit models often produce higher levels of hallucinations and unstable results, especially when reading non-expert language with unclear terms. To improve performance, we propose a quantization-aware multi-pass prompt verification method. This method guides the model through controlled steps that reduce hallucinations. It removes unreliable content and passes the results to the next transcript after verification, improving accuracy. To validate performance, human coders analyzed transcripts using NVivo and BF16 LLaMA. BF16 LLaMA-3.1 produced high-precision output but had semantic drift and hallucination. These errors were corrected manually. The corrected BF16 output and NVivo human coding were combined to create a gold-standard ground truth (GSGT) for thematic extraction and frequency analysis. The results show that 8-bit models stay closest to the GSGT. The 4-bit models lose accuracy but become stable when the proposed method is applied. The 3-bit and 2-bit models drop in performance because of heavy compression, but they improve with the proposed prompt design and verification. The study also finds that models at the same bit level behave differently depending on quantization type. Overall, the method helps low-resource LLMs become more stable, accurate, and suitable for qualitative research at lower cost.