Weight Only Quantization

Weight-only quantization aims to reduce the memory footprint and computational cost of large language models (LLMs) and other deep learning models by representing their weights using fewer bits, typically 2-4 bits, without retraining. Current research focuses on techniques like vector quantization, adaptive quantization strategies (e.g., per-channel or per-group), and optimized matrix multiplication kernels to minimize accuracy loss at extremely low bit-widths, often employing lookup tables for efficient dequantization. This research is significant because it enables the deployment of increasingly large models on resource-constrained devices, improving both the efficiency and accessibility of advanced AI applications.

Papers

November 6, 2024

Interactions Across Blocks in Post-Training Quantization of Large Language Models
Khasmamad Shabanovi, Lukas Wiest, Vladimir Golkov, Daniel Cremers, Thomas Pfeil
Language Model Post Training Quantization Multiplier Free Quantization Interaction Strategy Quantization Error Fine Tuning Strategy Weight Only Quantization Honeycomb Block

November 4, 2024

"Give Me BF16 or Give Me Death"? Accuracy-Performance Trade-Offs in LLM Quantization
Eldar Kurtic, Alexandre Marques, Shubhra Pandit, Mark Kurtz, Dan Alistarh
Large Language Model Quantization Operator Activation Quantization Weight Only Quantization Inference Performance Different Quantization Accuracy Efficiency Trade

October 24, 2024

TesseraQ: Ultra Low-Bit LLM Post-Training Quantization with Block Reconstruction
Yuhang Li, Priyadarshini Panda
Large Language Model Natural Language Processing Full State Reconstruction Quantization Operator Post Training Quantization Multi Modal Outer Arithmetic Block Weight Only Quantization

October 16, 2024

October 15, 2024

QSpec: Speculative Decoding with Complementary Quantization Schemes
Juntao Zhao, Wenhao Lu, Sheng Wang, Lingpeng Kong, Chuan Wu
Speculative Decoding Activation Quantization Weight Only Quantization Low Precision Quantization Speculative Exploration

September 25, 2024

VPTQ: Extreme Low-bit Vector Post-Training Quantization for Large Language Models
Yifei Liu, Jicheng Wen, Yang Wang, Shengyu Ye, Li Lyna Zhang, Ting Cao, Cheng Li, Mao Yang
Post Training Quantization Low Bit Quantization Weight Only Quantization

September 10, 2024

Rate-Constrained Quantization for Communication-Efficient Federated Learning
Shayan Mohajer Hamidi, Ali Bereyhi
Quantization Operator Compression Technique Quantization Error Weight Only Quantization Gradient Coding

July 15, 2024

Fast Matrix Multiplications for Lookup Table-Quantized LLMs
Han Guo, William Brandon, Radostin Cholakov, Jonathan Ragan-Kelley, Eric P. Xing, Yoon Kim
Large Language Model Matrix Multiplication Weight Only Quantization Kernel Performance Quantization Scale

May 27, 2024

CLAQ: Pushing the Limits of Low-Bit Post-Training Quantization for LLMs
Haoyu Wang, Bei Liu, Hang Shao, Bo Xiao, Ke Zeng, Guanglu Wan, Yanmin Qian
Large Language Model Continuum Limit Quantization Operator Low Bit LLM Quantization Weight Only Quantization Class Centroid Parameter Quantization

May 24, 2024

BiSup: Bidirectional Quantization Error Suppression for Large Language Models
Minghui Zou, Ronghui Guo, Sai Zhang, Xiaowang Zhang, Zhiyong Feng
Quantization Error Activation Quantization Weight Only Quantization Mixed Quantization

April 30, 2024

Transition Rate Scheduling for Quantization-Aware Training
Junghyup lee, Dohyung Kim, Jeimin Jeon, Bumsub Ham
Learning Rate Quantization Aware Training Weight Freezing Weight Only Quantization Quantization Step Variable Rate

March 26, 2024

Oh! We Freeze: Improving Quantized Knowledge Distillation via Signal Propagation Analysis for Large Language Models
Kartikeya Bhardwaj, Nilesh Prasad Pandey, Sweta Priyadarshi, Kyunggeun Lee, Jun Ma, Harris Teague
Large Language Model Knowledge Distillation Propagation Environment Quantization Level Weight Only Quantization Uniform Quantization Quantization Aware Knowledge Distillation

February 19, 2024

WKVQuant: Quantizing Weight and Key/Value Cache for Large Language Models Gains More
Yuxuan Yue, Zhihang Yuan, Haojie Duanmu, Sifan Zhou, Jianlong Wu, Liqiang Nie
Large Language Model Activation Quantization Weight Only Quantization Different Quantization

February 16, 2024

BitDistiller: Unleashing the Potential of Sub-4-Bit LLMs via Self-Distillation
Dayou Du, Yijia Zhang, Shijie Cao, Jiaqi Guo, Ting Cao, Xiaowen Chu, Ningyi Xu
Knowledge Distillation Full Potential Self Distillation Quantization Aware Training Weight Quantization Weight Only Quantization

December 9, 2023

Agile-Quant: Activation-Guided Quantization for Faster Inference of LLMs on the Edge
Xuan Shen, Peiyan Dong, Lei Lu, Zhenglun Kong, Zhengang Li, Ming Lin, Chao Wu, Yanzhi Wang
Large Language Model Language Model Extreme Edge Fast Inference Activation Quantization Weight Only Quantization

December 6, 2023

SmoothQuant+: Accurate and Efficient 4-bit Post-Training WeightQuantization for LLM
Jiayi Pan, Chengcan Wang, Kaifu Zheng, Yangguang Li, Zhenyu Wang, Bin Feng
Large Language Model Medical LLM Weight Quantization Low Bit Weight Only Quantization Bit Weight Quantization

November 9, 2023

Enhancing Computation Efficiency in Large Language Models through Weight and Activation Quantization
Jangwhan Lee, Minsoo Kim, Seungcheol Baek, Seok Joong Hwang, Wonyong Sung, Jungwook Choi
Post Training Quantization Multiplier Free Quantization Computational Efficiency Activation Quantization Weight Update Weight Only Quantization W4A8 Quantization

October 13, 2023

QUIK: Towards End-to-End 4-Bit Inference on Generative Large Language Models
Saleh Ashkboos, Ilia Markov, Elias Frantar, Tingxuan Zhong, Xincheng Wang, Jie Ren, Torsten Hoefler, Dan Alistarh
Generative Large Language Model Inference Cost Large Generative Model Weight Only Quantization

Weight Only Quantization

Papers

Interactions Across Blocks in Post-Training Quantization of Large Language Models

"Give Me BF16 or Give Me Death"? Accuracy-Performance Trade-Offs in LLM Quantization

TesseraQ: Ultra Low-Bit LLM Post-Training Quantization with Block Reconstruction

Channel-Wise Mixed-Precision Quantization for Large Language Models

DAQ: Density-Aware Post-Training Weight-Only Quantization For LLMs

COMET: Towards Partical W4A4KV4 LLMs Serving

QSpec: Speculative Decoding with Complementary Quantization Schemes

VPTQ: Extreme Low-bit Vector Post-Training Quantization for Large Language Models

Rate-Constrained Quantization for Communication-Efficient Federated Learning

Fast Matrix Multiplications for Lookup Table-Quantized LLMs

CLAQ: Pushing the Limits of Low-Bit Post-Training Quantization for LLMs

BiSup: Bidirectional Quantization Error Suppression for Large Language Models

Transition Rate Scheduling for Quantization-Aware Training

Oh! We Freeze: Improving Quantized Knowledge Distillation via Signal Propagation Analysis for Large Language Models

WKVQuant: Quantizing Weight and Key/Value Cache for Large Language Models Gains More

BitDistiller: Unleashing the Potential of Sub-4-Bit LLMs via Self-Distillation

Agile-Quant: Activation-Guided Quantization for Faster Inference of LLMs on the Edge

SmoothQuant+: Accurate and Efficient 4-bit Post-Training WeightQuantization for LLM

Enhancing Computation Efficiency in Large Language Models through Weight and Activation Quantization

QUIK: Towards End-to-End 4-Bit Inference on Generative Large Language Models