INT4 Quantization

INT4 quantization aims to reduce the memory footprint and computational cost of large language models (LLMs) by representing model weights and activations using only 4 bits, significantly increasing inference speed and reducing hardware requirements. Current research focuses on developing efficient quantization algorithms, such as those based on coordinate descent or mixed-precision strategies, and optimizing their integration with architectures like FlashAttention and various transformer models. This work is crucial for deploying LLMs on resource-constrained devices and improving the efficiency of LLM serving in cloud environments, potentially lowering the cost and energy consumption of AI applications.

Papers

September 25, 2024

INT-FlashAttention: Enabling Flash Attention for INT8 Quantization
Shimao Chen, Zirui Liu, Zhiying Wu, Ce Zheng, Peizhuang Cong, Zihan Jiang, Lei Su, Tong Yang
Self Attention Module Attention Operation FlashAttention 2 INT4 Quantization

June 25, 2024

CDQuant: Greedy Coordinate Descent for Accurate LLM Quantization
Pranav Ajit Nair, Arun Sai Suggala
Large Language Model Quantization Operator Post Training Quantization Large Pre Trained Model Post Training Quantization Level Greedy Coordinate INT4 Quantization

May 7, 2024

QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving
Yujun Lin, Haotian Tang, Shang Yang, Zhekai Zhang, Guangxuan Xiao, Chuang Gan, Song Han
Large Language Model Co Design Inference System INT4 Quantization W4A8 Quantization

December 14, 2023

ZeroQuant(4+2): Redefining LLMs Quantization with a New FP6-Centric Strategy for Diverse Generative Tasks
Xiaoxia Wu, Haojun Xia, Stephen Youn, Zhen Zheng, Shiyang Chen, Arash Bakhtiari, Michael Wyatt, Reza Yazdani Aminabadi, Yuxiong He, Olatunji Ruwase, Leon Song, Zhewei Yao
Large Language Model Generative Task Bit Quantization Floating Point Quantization Fine Grained Quantization INT4 Quantization

November 1, 2023

Efficient LLM Inference on CPUs
Haihao Shen, Hanwen Chang, Bo Dong, Yu Luo, Hengyu Meng
Large Language Model LLM Inference Kernel Optimization INT4 Quantization Optimal Inference

January 27, 2023

Understanding INT4 Quantization for Transformer Models: Latency Speedup, Composability, and Failure Cases
Xiaoxia Wu, Cheng Li, Reza Yazdani Aminabadi, Zhewei Yao, Yuxiong He
Transformer Model Encoder Decoder Model Composable Framework Inference Pipeline Failure Explanation INT4 Quantization

September 19, 2022

SAMP: A Model Inference Toolkit of Post-Training Quantization for Text Processing via Self-Adaptive Mixed-Precision
Rong Tian, Zijing Zhao, Weijie Liu, Haoyan Liu, Weiquan Mao, Zhe Zhao, Kan Zhou
Post Training Quantization Inference Framework Mixed Precision Inference Engine Text Processing INT4 Quantization

September 12, 2022

FP8 Formats for Deep Learning
Paulius Micikevicius, Dusan Stosic, Neil Burgess, Marius Cornea, Pradeep Dubey, Richard Grisenthwaite, Sangwon Ha, Alexander Heinecke, Patrick Judd, John Kamalu, Naveen Mellempudi, Stuart Oberman, Mohammad Shoeybi, Michael Siu, Hao Wu
Deep Learning Post Training Quantization Binary Code FP8 Format INT4 Quantization Modern Neural Network Architecture