Post-Training Quantization

Also known as PTQ

Post-training quantization (PTQ) converts a trained model to lower-precision numbers without retraining it, reducing memory use and enabling inference on constrained hardware. Methods range from rounding and calibration-based scaling to Hessian-aware reconstruction such as GPTQ, rotations and outlier handling, codebooks, and mixed-precision allocation across layers, experts, or caches. Calibration usually uses a small set of representative inputs, while evaluation spans language-model perplexity and reasoning, vision accuracy, multimodal tasks, and real-device latency and memory. The central tension is that moderate precision often retains task quality, while very low bit widths can amplify errors and alter behavior; calibration and hardware-aware kernels help, but accuracy, behavioral fidelity, and deployment speed do not always move together.

Latest papers

366 papers72 papers in the last four weeks, up 50% on the four weeks before. 0.4% of all new papers.

LUQ: Layerwise Ultra-Low Bit Quantization for Multimodal Large Language Models
Sep 28, 2025Shubhang Bhatnagar, Andy Xu, Kar-Han Tan +1Mixed-Precision Quantization LLM Quantization
University of Illinois Urbana-Champaign · Work done as an intern at HP Inc. · University of California, Los Angeles +1
SalQ-VLM: Fine-Grained Saliency-Guided Quantization for Vision-Language Models
Aug 5, 2025Yufei Xue, Yushi Huang, Lunjie Zhu +2Vision-Language Models VLM Quantization
Institute of Artificial Intelligence (TeleAI), China Telecom · The Hong Kong University of Science and Technology
Squeeze10-LLM: Squeezing LLMs' Weights by 10 Times via a Staged Mixed-Precision Quantization Method
Jul 24, 2025Qingcheng Zhu, Yangyang Ren, Linlin Yang +6Mixed-Precision Quantization LLM Quantization
Beihang University · Communication University of China · Beijing Jiaotong University
A Comprehensive Evaluation on Quantization Techniques for Large Language Models
Jul 23, 2025Yutong Liu, Cairong Zhao, Guosheng HuLLM Quantization Post-Training Quantization
School of Computer Science and Technology, Tongji University, Shanghai 201804, China · School of Engineering Math and Technology, University of Bristol, BS8 1QU, UK
FedHQ: Hybrid Runtime Quantization for Federated Learning
May 17, 2025Zihao Zheng, Ziyao Wang, Jiawei Yang +7Federated Learning Adaptive Quantization
Peking University · University of Maryland
On Orthogonal Recurrent Neural Networks Quantization
Feb 5, 2024Armand Foucault, Franck Mamalet, François MalgouyresRecurrent Neural Networks Neural Network Quantization
IMT · Institut de Mathématiques de Toulouse, UMR5219. Université de Toulouse, CNRS. UPS IMT, F-31062 Toulouse Cedex 9, France · Comue de Toulouse +1

Sep 28, 2025J/K move · Enter open · S save

LUQ: Layerwise Ultra-Low Bit Quantization for Multimodal Large Language Models

Shubhang Bhatnagar, Andy Xu, Kar-Han Tan, Narendra Ahuja

University of Illinois Urbana-Champaign · Work done as an intern at HP Inc. · University of California, Los Angeles+1

Mixed-Precision Quantization LLM Quantization VLM Quantization Multimodal Large Language Models Multimodal Foundation Models Post-Training Quantization+1

Open paper AskPDF arXiv

Large Language Models (LLMs) with multimodal capabilities have revolutionized vision-language tasks, but their deployment often requires huge memory and computational resources. Post-training quantization (PTQ) has successfully compressed language models to as low as 1-bit precision, its effectiveness for multimodal LLMs (MLLMs) remains unexplored. In this paper, we present the first method for ultra-low-bit (<4-bit) quantization of MLLMs. Our analysis reveals that multimodal tokens and intermediate layer activations produced by them exhibit significantly higher entropy compared to text tokens, indicating greater functional complexity that makes MLLMs less tolerant to ultra-low bit quantization. However, this entropy varies significantly across layers, with some layers producing lower-entropy activation distributions that we empirically show can better tolerate ultra-low bit quantization. Existing PTQ methods optimize weight quantization within each layer but apply the same target precision uniformly, ignoring this variation in complexity across layers. Building on this insight, we propose LUQ: Layerwise Ultra-Low Bit Quantization, which characterizes each transformer layer's functional complexity via its output activation entropy and selectively applies ultra-low bit quantization to layers encoding simpler, more compressible functions. We also show that multimodal calibration (image and text tokens) boosts VQA performance in the ultra-low bit regime. Evaluated on LLaVA-1.5 and Qwen-2.5-VL across 9 VQA benchmarks, LUQ models use 40% and 31% less memory than their 4-bit counterparts while exhibiting less than 10% degradation on MME.

Latest papers

LUQ: Layerwise Ultra-Low Bit Quantization for Multimodal Large Language Models

SalQ-VLM: Fine-Grained Saliency-Guided Quantization for Vision-Language Models

Squeeze10-LLM: Squeezing LLMs' Weights by 10 Times via a Staged Mixed-Precision Quantization Method

A Comprehensive Evaluation on Quantization Techniques for Large Language Models

FedHQ: Hybrid Runtime Quantization for Federated Learning

On Orthogonal Recurrent Neural Networks Quantization