LLM Quantization

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

33 papers in the last four weeks, up 43% on the four weeks before. 0.3% of all new papers.

Jul 13Week of Sep 28

Latest papers 251

All topics

LUQ: Layerwise Ultra-Low Bit Quantization for Multimodal Large Language Models
Sep 28, 2025Shubhang Bhatnagar, Andy Xu, Kar-Han Tan +1Mixed-Precision Quantization LLM Quantization
University of Illinois Urbana-Champaign · Work done as an intern at HP Inc. · University of California, Los Angeles +1
On-the-Fly Adaptation to Quantization: Configuration-Aware LoRA for Efficient Fine-Tuning of Quantized LLMs
Sep 22, 2025Rongguang Ye, Ming Tang, Edith C. H. NgaiMixed-Precision Quantization LLM Quantization
Southern University of Science and Technology · The University of Hong Kong
Fair-GPTQ: Bias-Aware Quantization for Large Language Models
Sep 18, 2025Irina Proskurina, Guillaume Metzler, Julien VelcinLLM Quantization Social Bias in Language Models
Université Claude Bernard Lyon 1, Université Lumière Lyon 2, ERIC · École Centrale de Lyon, LIRIS, CNRS UMR 5205
Squeeze10-LLM: Squeezing LLMs' Weights by 10 Times via a Staged Mixed-Precision Quantization Method
Jul 24, 2025Qingcheng Zhu, Yangyang Ren, Linlin Yang +6Mixed-Precision Quantization LLM Quantization
Beihang University · Communication University of China · Beijing Jiaotong University
FPTQuant: Function-Preserving Transforms for LLM Quantization
Jun 5, 2025Boris van Breugel, Yelysei Bondarenko, Paul Whatmough +1LLM Quantization LLM Inference Acceleration
Qualcomm AI Research, Amsterdam, Netherlands
FlexQuant: Elastic Quantization Framework for Locally Hosted LLM on Edge Devices
Jan 13, 2025Yuji Chai, Mujin Kwun, David Brooks +1LLM Quantization On-Device Language Model Inference
Harvard John A. Paulson School Of Engineering And Applied Sciences Cambridge, Massachusetts, USA
Benford's Law as a Distributional Prior for Post-Training Quantization of Large Language Models
Date pendingArthur Negrão, Pedro Silva, Vander L. S. Freitas +2LLM Quantization 4-Bit Quantization
Postgraduate Program in Computer Science Federal University of Ouro Preto · Computing Department Federal University of Ouro Preto
LC-QAT: Data-Efficient 2-Bit QAT for LLMs via Linear-Constrained Vector Quantization
Date pendingHaoyu Wang, Xingyu Yu, Haiyan Zhao +2LLM Quantization Quantization-Aware Training
Department of Computer Science and Technology, Tsinghua University, Beijing, China · School of Software and Microelectronics, Peking University, Beijing, China · College of Computer Science and Technology, National University of Defense Technology, Hunan, China.
RDQ: Residual Distribution Quantization for Large Language Models
Date pendingPrateek SinghLLM Quantization Post-Training Quantization
REAL-Q: E2E LLM Quantization via Dynamic Gradient Descent
Date pendingQian Zhang, Yaoming Li, Zhewen Tan +9LLM Quantization Post-Training Quantization
Peking University · Northeastern University · ZTE Corporation
Accuracy is Not Enough: A Divergence-Based Approach to Evaluate Fidelity Loss in Quantized LLMs
Date pendingShahzeb Qamar, Lorenz Sparrenberg, Christian Bauckhage +5LLM Quantization LLM Evaluation
Fraunhofer IAIS, Germany · PhenoRob Cluster of Excellence, University of Bonn, Germany · Lamarr Institute for Machine Learning and Artificial Intelligence, Germany +4

Sep 28, 2025, cs.CVJ/K move · Enter open · S save

LUQ: Layerwise Ultra-Low Bit Quantization for Multimodal Large Language Models

Shubhang Bhatnagar, Andy Xu, Kar-Han Tan, Narendra Ahuja

University of Illinois Urbana-Champaign · Work done as an intern at HP Inc. · University of California, Los Angeles+1

Mixed-Precision Quantization LLM Quantization VLM Quantization Multimodal Large Language Models Multimodal Foundation Models Post-Training Quantization+1

Open paper AskPDF arXiv

Large Language Models (LLMs) with multimodal capabilities have revolutionized vision-language tasks, but their deployment often requires huge memory and computational resources. Post-training quantization (PTQ) has successfully compressed language models to as low as 1-bit precision, its effectiveness for multimodal LLMs (MLLMs) remains unexplored. In this paper, we present the first method for ultra-low-bit (<4-bit) quantization of MLLMs. Our analysis reveals that multimodal tokens and intermediate layer activations produced by them exhibit significantly higher entropy compared to text tokens, indicating greater functional complexity that makes MLLMs less tolerant to ultra-low bit quantization. However, this entropy varies significantly across layers, with some layers producing lower-entropy activation distributions that we empirically show can better tolerate ultra-low bit quantization. Existing PTQ methods optimize weight quantization within each layer but apply the same target precision uniformly, ignoring this variation in complexity across layers. Building on this insight, we propose LUQ: Layerwise Ultra-Low Bit Quantization, which characterizes each transformer layer's functional complexity via its output activation entropy and selectively applies ultra-low bit quantization to layers encoding simpler, more compressible functions. We also show that multimodal calibration (image and text tokens) boosts VQA performance in the ultra-low bit regime. Evaluated on LLaVA-1.5 and Qwen-2.5-VL across 9 VQA benchmarks, LUQ models use 40% and 31% less memory than their 4-bit counterparts while exhibiting less than 10% degradation on MME.

Momentum

Related topics

Latest papers 251