W4A8 Quantization

W4A8 quantization aims to improve the efficiency of large language models (LLMs) by representing weights with 4 bits and activations with 8 bits, significantly reducing memory footprint and computational cost. Current research focuses on overcoming accuracy loss inherent in low-precision quantization through techniques like affine transformations, low-rank error reconstruction, and novel quantization algorithms tailored to specific hardware architectures (e.g., GPU-optimized methods). These advancements are crucial for deploying LLMs on resource-constrained devices and reducing the economic cost of LLM serving, impacting both the accessibility and scalability of these powerful models.

Papers

October 12, 2024

FlatQuant: Flatness Matters for LLM Quantization
Yuxuan Sun, Ruikang Liu, Haoli Bai, Han Bao, Kang Zhao, Yuening Li, Jiaxin Hu, Xianzhi Yu, Lu Hou, Chun Yuan, Xin Jiang, Wulong Liu, Jun Yao
Post Training Quantization Affine Transformation LLM Quantization Quantization Step W4A8 Quantization

May 7, 2024

QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving
Yujun Lin, Haotian Tang, Shang Yang, Zhekai Zhang, Guangxuan Xiao, Chuang Gan, Song Han
Large Language Model Co Design Inference System INT4 Quantization W4A8 Quantization

March 19, 2024

AffineQuant: Affine Transformation Quantization for Large Language Models
Yuexiao Ma, Huixia Li, Xiawu Zheng, Feng Ling, Xuefeng Xiao, Rui Wang, Shilei Wen, Fei Chao, Rongrong Ji
Large Scale Language Model Weight Quantization Affine Transformation Bit Quantization W4A8 Quantization

February 4, 2024

LQER: Low-Rank Quantization Error Reconstruction for LLMs
Cheng Zhang, Jianyi Cheng, George A. Constantinides, Yiren Zhao
Large Language Model Low Rank Approximation Quantization Error W4A8 Quantization Low Rank Quantization

November 9, 2023

Enhancing Computation Efficiency in Large Language Models through Weight and Activation Quantization
Jangwhan Lee, Minsoo Kim, Seungcheol Baek, Seok Joong Hwang, Wonyong Sung, Jungwook Choi
Post Training Quantization Multiplier Free Quantization Computational Efficiency Activation Quantization Weight Update Weight Only Quantization W4A8 Quantization

October 26, 2023

ZeroQuant-HERO: Hardware-Enhanced Robust Optimized Post-Training Quantization Framework for W8A8 Transformers
Zhewei Yao, Reza Yazdani Aminabadi, Stephen Youn, Xiaoxia Wu, Elton Zheng, Yuxiong He
Neural Network Post Training Quantization Quantization Technique Dynamic Quantization Transformer XL W4A8 Quantization Intensive Operator

August 30, 2023

FPTQ: Fine-grained Post-Training Quantization for Large Language Models
Qingyuan Li, Yifan Zhang, Liang Li, Peng Yao, Bo Zhang, Xiangxiang Chu, Yerui Sun, Li Du, Yuchen Xie
Large Language Model Fine Grained Quantization Operator Large Scale Language Model Weight Quantization Fine Grained Quantization W4A8 Quantization

July 19, 2023

ZeroQuant-FP: A Leap Forward in LLMs Post-Training W4A8 Quantization Using Floating-Point Formats
Xiaoxia Wu, Zhewei Yao, Yuxiong He
Large Language Model Weight Quantization Leap Forward Uniform Quantization Floating Point Quantization W4A8 Quantization Floating Point Format