LLM Quantization

LLM: Large Language Model

Ask about this topic

Overview

LLM quantization studies how to represent model weights, activations, and attention key-value caches with fewer bits while preserving useful behavior and making deployment more efficient. Post-training methods such as GPTQ and AWQ use calibration data to round weights, while quantization-aware training, mixed-precision allocation, rotations, and outlier handling aim to reduce errors that accumulate across layers. Work also co-designs number formats and GPU, CPU, and edge-device kernels, evaluating trade-offs with language-model perplexity, zero-shot and reasoning benchmarks, memory, latency, and energy. Four-bit inference is often practical, but performance at very low bit widths remains model- and task-dependent, and an open challenge is ensuring that gains in aggregate accuracy do not hide losses in reasoning, safety, fairness, or uncertainty.

Latest papers

CardsList

388 papers84 papers in the last four weeks, up 121% on the four weeks before. 0.5% of all new papers.

  1. Spike-driven Large Language Model

    Apr 11, 2026Han Xu, Xuerui Qiu, Baiyu Chen +7Spiking Language ModelsSpiking Neural Networks

    Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China · Brain-Inspired Models Group, Beijing Academy of Artificial Intelligence, Beijing 100862, China +7

  2. SEPTQ: A Simple and Effective Post-Training Quantization Paradigm for Large Language Models

    Apr 11, 2026Han Liu, Haotian Gao, Xiaotong Zhang +5LLM QuantizationPost-Training Quantization

    Dalian University of Technology, Dalian, China · Peking University, Beijing, China · Shenzhen MSU-BIT University, Shenzhen, China +1

  3. EdgeRazor: A Lightweight Framework for Large Language Models via Mixed-Precision Quantization-Aware Distillation

    Apr 10, 2026Shu-Hao Zhang, Le-Tong Huang, Xiang-Sheng Deng +5Quantization-Aware DistillationLLM Quantization

    National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing 210063, China · School of Intelligent Science and Technology, Nanjing University, Suzhou 215163, China · Microsoft AI, Beijing 100080, China +1

  4. Quantisation Reshapes the Metacognitive Geometry of Language Models

    Apr 10, 2026Jon-Paul CacioliLanguage Model Self-AssessmentMetacognition in Language Models

    Independent Researcher, Melbourne, Australia

  5. HiFloat4 Format for Language Model Pre-training on Ascend NPUs

    Apr 9, 2026Mehran Taghian, Yunke Peng, Xing Huang +22Low-Precision TrainingLLM Quantization

    Huawei

  6. Initialisation Determines the Basin: Efficient Codebook Optimisation for Extreme LLM Quantization

    Apr 9, 2026Ian W. Kennedy, Nafise Sadat MoosaviLLM QuantizationNeural Network Quantization

    Department of Computer Science, University of Sheffield, Sheffield, UK

  7. Rethinking Residual Errors in Compensation-based LLM Quantization

    Apr 9, 2026Shuaiting Li, Juncan Deng, Kedong Xu +5Quantization Error CompensationLLM Quantization

    Zhejiang University · vivo Mobile Communication Co., Ltd

  8. Bit-by-Bit: Progressive QAT Strategy with Outlier Channel Splitting for Stable Low-Bit LLMs

    Apr 9, 2026Binxing Xu, Hao Gu, Lujun Li +8LLM QuantizationQuantization-Aware Training

    Zhejiang University · Hong Kong University of Science and Technology · City University of Hong Kong

  9. MUXQ: Mixed-to-Uniform Precision MatriX Quantization via Low-Rank Outlier Decomposition

    Apr 6, 2026Seoungsub Lee, In Seo Kim, Seon Wook KimLLM QuantizationNeural Network Quantization

    School of Electrical Engineering, Korea University, Seoul, Korea

  10. RUQuant: Towards Refining Uniform Quantization for Large Language Models

    Apr 5, 2026Han Liu, Haotian Gao, Changya Li +4LLM QuantizationActivation Quantization

    Dalian University of Technology Dalian, China · Peking University Beijing, China · Macao Polytechnic University Macao, China

  11. Improving Quantized Model Performance in Qualitative Analysis with Multi-Pass Prompt Verification

    Apr 4, 2026Aisvarya Adeseye, Jouni Isoaho, Adeyemi AdeseyeLLM QuantizationLow-Bit Quantization

    University of Turku, Turku, Finland · Brilloconnetz Partners avoin yhtiö, Turku, Finland

  12. Fast NF4 Dequantization Kernels for Large Language Model Inference

    Apr 2, 2026Xiangbo Qi, Chaoyi Jiang, Murali AnnavaramLLM Inference AccelerationGPU Kernel Optimization

    University of Southern California Los Angeles, USA

  13. AdaHOP: Fast and Accurate Low-Precision Training via Outlier-Pattern-Aware Rotation

    Apr 2, 2026Seonggon Kim, Alireza Khodamoradi, Pranathi Vasireddy +2Walsh-Hadamard TransformLow-Precision Training

    Advanced Micro Devices, Inc. · POSTECH

  14. PolarQuant: Optimal Gaussian Weight Quantization via Hadamard Rotation for LLM Compression

    Mar 30, 2026Caio VicentinoLLM QuantizationNeural Network Quantization

  15. Adaptive Block-Scaled Data Types

    Mar 30, 2026Jack Cook, Hyemin S. Lee, Kathryn Le +4Neural Network QuantizationLLM Quantization

    Massachusetts Institute of Technology · NVIDIA

  16. ITQ3_S: High-Fidelity 3-bit LLM Inference via Interleaved Ternary Quantization with Rotation-Domain Smoothing

    Mar 30, 2026Edward J. YoonLLM QuantizationTernary Quantization

  17. GlowQ: Group-Shared LOw-Rank Approximation for Quantized LLMs

    Mar 26, 2026Selim An, Il hong Suh, Yeseong KimLLM QuantizationLLM Inference Acceleration

    Department of Artificial Intelligence, DGIST, Korea · COGA robotics, Korea · Department of Electrical Engineering, POSTECH, Korea

  18. SliderQuant: Accurate Post-Training Quantization for LLMs

    Mar 26, 2026Shigeng Wang, Chao Li, Yangyuxuan Kang +3LLM QuantizationPost-Training Quantization

    Intel Labs China · BUPT

  19. APreQEL: Adaptive Mixed Precision Quantization For Edge LLMs

    Mar 24, 2026Meriem Bouzouad, Yuan-Hao Chang, Jalil BoukhobzaHardware-Aware QuantizationLLM Quantization

    Lab-STICC, CNRS UMR 6285 , ENSTA, Institut Polytechnique de Paris, Brest, France · National Taiwan University, Taipei, Taiwan

  20. FAAR: Format-Aware Adaptive Rounding for NVFP4

    Mar 23, 2026Hanglin Li, Shuchang Tian, Chen Lin +2LLM Quantization4-Bit Quantization

    Li Auto Inc.

  21. DAQ: Delta-Aware Quantization for Post-Training LLM Weight Compression

    Mar 20, 2026Xiaoming Yu, Shize Tang, Guanghua Yu +4QuantizationLLM Quantization

    Yuanbao & Hunyuan AI Infra Team

  22. Weight Clustering Reveals the Importance of Relative Rank in Large Language Models

    Mar 18, 2026Borja Aizpurua, Sukhbinder Singh, Román OrúsLLM CompressionLLM Quantization

    Multiverse Computing, San Sebasti´an, Spain · Department of Basic Sciences, Tecnun - University of Navarra, San Sebasti´an, Spain · Multiverse Computing, Toronto, Ontario, Canada +2

  23. RAMP: Reinforcement Adaptive Mixed Precision Quantization for Efficient On Device LLM Inference

    Mar 18, 2026Arpit Singh Gautam, Saurabh JhaLLM QuantizationNeural Network Quantization

  24. Understanding Quantization of Optimizer States in LLM Pre-training: Dynamics of State Staleness and Effectiveness of State Resets

    Mar 17, 2026Kristi Topollai, Anna ChoromanskaLow-Precision TrainingLow-Bit Quantization

    New York University

  25. BATQuant: Outlier-resilient MXFP4 Quantization via Learnable Block-wise Optimization

    Mar 17, 2026Ji-Fu Li, Manyi Zhang, Xiaobo Xia +4LLM QuantizationNeural Network Quantization

    Huawei Technologies · University of Science and Technology of China

  26. Frequency Matters: Fast Model-Agnostic Data Curation for Pruning and Quantization

    Mar 17, 2026Francesco Pio Monaco, Elia Cunegatti, Flavio Vella +1LLM CompressionLLM Pruning

    University of Trento

  27. Adapting Methods for Domain-Specific Japanese Small LMs: Scale, Architecture, and Quantization

    Mar 12, 2026Takato YasunoLanguage ModelingSmall Language Models

  28. Leech Lattice Vector Quantization for Efficient LLM Compression

    Mar 11, 2026Tycho F. A. van der Ouderaa, Mart van Baalen, Paul Whatmough +1LLM QuantizationLLM Compression

    Qualcomm AI Research

  29. The Curse and Blessing of Mean Bias in FP4-Quantized LLM Training

    Mar 11, 2026Hengjie Cao, Zhendong Huang, Mengyi Chen +15LLM Quantization4-Bit Quantization

    Fudan University · Shanghai Innovation Institute · University of Bath +5

  30. TTQ: Activation-Aware Test-Time Quantization to Accelerate LLM Inference On The Fly

    Mar 11, 2026Toshiaki Koike-Akino, Jing Liu, Ye WangLLM QuantizationNeural Network Quantization

    Mitsubishi Electric Research Laboratories (MERL), 201 Broadway, Cambridge, MA 02139, USA