LLM Quantization

LLM: Large Language Model

Ask about this topic

Overview

LLM quantization studies how to represent model weights, activations, and attention key-value caches with fewer bits while preserving useful behavior and making deployment more efficient. Post-training methods such as GPTQ and AWQ use calibration data to round weights, while quantization-aware training, mixed-precision allocation, rotations, and outlier handling aim to reduce errors that accumulate across layers. Work also co-designs number formats and GPU, CPU, and edge-device kernels, evaluating trade-offs with language-model perplexity, zero-shot and reasoning benchmarks, memory, latency, and energy. Four-bit inference is often practical, but performance at very low bit widths remains model- and task-dependent, and an open challenge is ensuring that gains in aggregate accuracy do not hide losses in reasoning, safety, fairness, or uncertainty.

Latest papers

CardsList

388 papers84 papers in the last four weeks, up 121% on the four weeks before. 0.5% of all new papers.

  1. SERQ: Saliency-Aware Low-Rank Error Reconstruction for LLM Quantization

    Mar 9, 2026Yeonsik Park, Hyeonseong Kim, Seungkyu ChoiLLM QuantizationPost-Training Quantization

    Kyung Hee University · Yonsei University

  2. Diagnosing FP4 inference: a layer-wise and block-wise sensitivity analysis of NVFP4 and MXFP4

    Mar 5, 2026Musa Cim, Burak Topcu, Mahmut Taylan KandemirQuantization Sensitivity AnalysisLLM Quantization

    Department of Computer Science and Engineering The Pennsylvania State University University Park, PA, USA

  3. Sparse-BitNet: 1.58-bit LLMs are Naturally Friendly to Semi-Structured Sparsity

    Mar 5, 2026Di Zhang, Xun Wu, Shaohan Huang +9LLM Inference EfficiencyLLM Quantization

    Microsoft Research · Peking University · South China University of Technology

  4. WaterSIC: Information-Theoretically (Near) Optimal Linear Layer Quantization

    Mar 5, 2026Egor Lifar, Semyon Savkin, Or Ordentlich +1Neural Network QuantizationLLM Quantization

    MIT · Independent Researcher · Hebrew University of Jerusalem

  5. Dissecting Quantization Error: A Concentration-Alignment Perspective

    Mar 4, 2026Marco Federici, Boris van Breugel, Paul Whatmough +1LLM Quantization4-Bit Quantization

    Qualcomm AI Research

  6. Bielik-Q2-Sharp: A Comparative Study of Extreme 2-bit Quantization Methods for a Polish 11B Language Model

    Mar 4, 2026Jakub PrejznerQuantizationLLM Quantization

    BitSharp, Independent Researcher, Rzeszów, Poland

  7. FreeAct: Freeing Activations for LLM Quantization

    Mar 2, 2026Xiaohao Liu, Xiaobo Xia, Manyi Zhang +6LLM QuantizationActivation Quantization

    National University of Singapore · Huawei Technology · Central South University

  8. pQuant: Towards Effective Low-Bit Language Models via Decoupled Linear Quantization-Aware Training

    Feb 26, 2026Wenzheng Zhang, Bingzheng Liu, Yang Hu +3LLM QuantizationLow-Bit Quantization

    School of Computer Science, Peking University · College of Future Information Technology, Fudan University · Center for Information Research, Academy of Military Sciences +2

  9. MoBiQuant: Mixture-of-Bits Quantization for Token-Adaptive Any-Precision LLM

    Feb 21, 2026Dongwei Wang, Jinhee Kim, Seokho Han +8LLM QuantizationNeural Network Quantization

    University of Arizona · Panasonic AI Lab · Duke University +2

  10. SPQ: An Ensemble Technique for Large Language Model Compression

    Feb 20, 2026Jiamin Yao, Eren GultepeLLM CompressionLLM Quantization

    Dept. of Computer Science, Southern Illinois University Edwardsville, Edwardsville, USA

  11. 1-Bit Wonder: Improving QAT Performance in the Low-Bit Regime through K-Means Quantization

    Feb 17, 2026Sohir Maskey, Constantin Eichenberg, Johannes Messner +1QuantizationLLM Quantization

    Aleph Alpha Research · Graphcore Research

  12. QTALE: Quantization-Robust Token-Adaptive Layer Execution for LLMs

    Feb 11, 2026Kanghyun Noh, Jinheon Choi, Yulhwa KimLLM QuantizationLLM Compression

    Department of Semiconductor Convergence Engineering, Sungkyunkwan University · Department of Electrical and Computer Engineering, Sungkyunkwan University · Department of Semiconductor Systems Engineering, Sungkyunkwan University

  13. Investigating Social Bias Changes in Quantized Language Models

    Feb 5, 2026Stanley Z. Hua, Sanae Lotfi, Irene Y. ChenLLM QuantizationSocial Bias in Language Models

    UC Berkeley & UCSF · Meta Superintelligence Labs

  14. MatGPTQ: Efficient and Accurate Inference over Nested Quantized Models

    Feb 3, 2026Maximilian Kleinegger, Elvir Crnčević, Dan AlistarhLLM QuantizationLLM Inference Acceleration

    1Vienna University of Technology, Vienna, Austria · Institute of Science & Technology Austria (ISTA), Vienna, Austria · 3Red Hat AI, Boston, USA

  15. Benford's Law as a Distributional Prior for Post-Training Quantization of Large Language Models

    Jan 29, 2026Arthur Negrão, Pedro Silva, Vander L. S. Freitas +2LLM Quantization4-Bit Quantization

    Postgraduate Program in Computer Science Federal University of Ouro Preto · Computing Department Federal University of Ouro Preto

  16. Preserving Fairness and Safety in Quantized LLMs Through Critical Weight Protection

    Jan 17, 2026Muhammad Alif Al Hakim, Alfan Farizki Wicaksono, Fajri KotoLLM QuantizationMultilingual Language Model Evaluation

    Faculty of Computer Science, Universitas Indonesia · Mohamed bin Zayed University of Artificial Intelligence

  17. Activation Sensitivity as a Unifying Principle for Post-Training Quantization

    Jan 15, 2026Bruce Changlong XuLLM QuantizationPost-Training Quantization

  18. ARCQuant: Boosting NVFP4 Quantization with Augmented Residual Channels for LLMs

    Jan 12, 2026Haoqian Meng, Yilun Luo, Yafei Zhao +3LLM Quantization4-Bit Quantization

    School of Computer Science and Technology, Tianjin University

  19. When Less is More: 8-bit Quantization Improves Continual Learning in Large Language Models

    Dec 22, 2025Michael S. Zhang, Rishi A. Ruia, Arnav Kewalram +3LLM QuantizationContinual Learning for LLMs

    Algoverse

  20. You Had One Job: Per-Task Quantization Using LLMs' Hidden Representations

    Nov 9, 2025Amit LeVi, Raz Lapid, Rom Himelstein +3Weight-Only QuantizationMixed-Precision Quantization

    1Technion—Israel Institute of Technology · 2Intuit · 3Ben-Gurion University of the Negev +1

  21. LUQ: Layerwise Ultra-Low Bit Quantization for Multimodal Large Language Models

    Sep 28, 2025Shubhang Bhatnagar, Andy Xu, Kar-Han Tan +1Mixed-Precision QuantizationLLM Quantization

    University of Illinois Urbana-Champaign · Work done as an intern at HP Inc. · University of California, Los Angeles +1

  22. On-the-Fly Adaptation to Quantization: Configuration-Aware LoRA for Efficient Fine-Tuning of Quantized LLMs

    Sep 22, 2025Rongguang Ye, Ming Tang, Edith C. H. NgaiMixed-Precision QuantizationLLM Quantization

    Southern University of Science and Technology · The University of Hong Kong

  23. Fair-GPTQ: Bias-Aware Quantization for Large Language Models

    Sep 18, 2025Irina Proskurina, Guillaume Metzler, Julien VelcinLLM QuantizationSocial Bias in Language Models

    Université Claude Bernard Lyon 1, Université Lumière Lyon 2, ERIC · École Centrale de Lyon, LIRIS, CNRS UMR 5205

  24. Squeeze10-LLM: Squeezing LLMs' Weights by 10 Times via a Staged Mixed-Precision Quantization Method

    Jul 24, 2025Qingcheng Zhu, Yangyang Ren, Linlin Yang +6Mixed-Precision QuantizationLLM Quantization

    Beihang University · Communication University of China · Beijing Jiaotong University

  25. A Comprehensive Evaluation on Quantization Techniques for Large Language Models

    Jul 23, 2025Yutong Liu, Cairong Zhao, Guosheng HuLLM QuantizationPost-Training Quantization

    School of Computer Science and Technology, Tongji University, Shanghai 201804, China · School of Engineering Math and Technology, University of Bristol, BS8 1QU, UK

  26. FPTQuant: Function-Preserving Transforms for LLM Quantization

    Jun 5, 2025Boris van Breugel, Yelysei Bondarenko, Paul Whatmough +1LLM QuantizationLLM Inference Acceleration

    Qualcomm AI Research, Amsterdam, Netherlands

  27. FlexQuant: Elastic Quantization Framework for Locally Hosted LLM on Edge Devices

    Jan 13, 2025Yuji Chai, Mujin Kwun, David Brooks +1LLM QuantizationOn-Device Language Model Inference

    Harvard John A. Paulson School Of Engineering And Applied Sciences Cambridge, Massachusetts, USA

  28. TernaryLLM: Ternarized Large Language Model

    Jun 11, 2024Tianqi Chen, Zhe Li, Weixiang Xu +6LLM QuantizationTernary Quantization

    Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Advanced Micro Devices, Inc., Beijing, China +1