LLM Quantization

LLM quantization aims to reduce the substantial memory and computational demands of large language models (LLMs) by representing their weights and activations using lower precision numbers. Current research focuses on developing efficient quantization algorithms, including techniques like post-training quantization (PTQ) with methods such as vector quantization and layer-wise quantization with varying bit-widths, often incorporating adaptive strategies to minimize performance loss. These advancements are crucial for deploying LLMs on resource-constrained devices and improving the efficiency of LLM inference, impacting both the accessibility of large language models and the sustainability of AI infrastructure.

Papers

October 29, 2023

Atom: Low-bit Quantization for Efficient and Accurate LLM Serving
Yilong Zhao, Chien-Yu Lin, Kan Zhu, Zihao Ye, Lequn Chen, Size Zheng, Luis Ceze, Arvind Krishnamurthy, Tianqi Chen, Baris Kasikci
High Efficiency Activation Quantization Low Bit Quantization LLM Quantization Single Atom LLM Accuracy Fine Grained Quantization

October 8, 2023

Revisiting Block-based Quantisation: What is Important for Sub-8-bit LLM Inference?
Cheng Zhang, Jianyi Cheng, Ilia Shumailov, George A. Constantinides, Yiren Zhao
Large Language Model LLM Quantization 8 Bit Quantization Quantization Granularity

August 16, 2023

FineQuant: Unlocking Efficiency with Fine-Grained Weight-Only Quantization for LLMs
Young Jin Kim, Rawn Henry, Raffy Fahim, Hany Hassan Awadalla
Large Language Model Fine Grained Enhancing Efficiency Dense Model LLM Quantization Weight Only Quantization Auto Regressive Decoding

July 25, 2023

QuIP: 2-Bit Quantization of Large Language Models With Guarantees
Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, Christopher De Sa
Large Language Model Quantization Operator Formal Guarantee LLM Quantization Parameter Quantization Incoherence Processing

May 23, 2023

QLoRA: Efficient Finetuning of Quantized LLMs
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, Luke Zettlemoyer
Fine Tuning Small Datasets LLM Quantization Finetuning Method Conversational Search Benchmark Bit Training

February 2, 2023

QCM-SGM+: Improved Quantized Compressed Sensing With Score-Based Generative Models
Xiangming Meng, Yoshiyuki Kabashima
Score Based Generative Compressed Sensing LLM Quantization