Post Training Quantization

Post-training quantization (PTQ) aims to reduce the computational cost and memory footprint of large neural networks, particularly large language models (LLMs) and vision transformers (ViTs), without retraining. Current research focuses on improving PTQ accuracy at extremely low bit-widths (e.g., 2-4 bits) through techniques like vector quantization, adaptive quantization schemes (e.g., per-channel, mixed-precision), and optimization strategies that minimize quantization error by addressing issues such as outliers and activation distribution. This work is significant because efficient quantization is crucial for deploying large models on resource-constrained devices, enabling broader accessibility and reducing the environmental impact of AI.

Papers

April 22, 2024

An Empirical Study of LLaMA3 Quantization: From LLMs to MLLMs
Wei Huang, Xingyu Zheng, Xudong Ma, Haotong Qin, Chengtao Lv, Hong Chen, Jie Luo, Xiaojuan Qi, Xianglong Liu, Michele Magno
Empirical Study Post Training Quantization Low Bit Quantization Bit Quantization LLaMA 3

April 11, 2024

Frame Quantization of Neural Networks
Wojciech Czaja, Sanghoon Na
Neural Network Post Training Quantization Vector Quantization Multiplier Free Quantization

April 8, 2024

Investigating the Impact of Quantization on Adversarial Robustness
Qun Li, Yuan Meng, Chen Tang, Jiacheng Jiang, Zhi Wang
Adversarial Robustness Global Impact Quantization Operator Post Training Quantization Model Robustness Quantization Aware Training Quantization Step

April 4, 2024

Mitigating the Impact of Outlier Channels for Language Model Quantization with Activation Regularization
Aniruddha Nrusimha, Mayank Mishra, Naigang Wang, Dan Alistarh, Rameswar Panda, Yoon Kim
Large Language Model Global Impact Post Training Quantization Quantization Aware Training Activation Quantization Accurate Quantization Task Specific Channel Input Quantization

April 3, 2024

DNN Memory Footprint Reduction via Post-Training Intra-Layer Multi-Precision Quantization
Behnam Ghavami, Amin Kamjoo, Lesley Shannon, Steve Wilton
Deep Neural Network Quantization Operator Edge Device Post Training Quantization Memory Efficient Intermediate Layer Post Training

April 2, 2024

Minimize Quantization Output Error with Bias Compensation
Cheng Gong, Haoshuai Zheng, Mengting Hu, Zheng Lin, Deng-Ping Fan, Yuzhi Zhang, Tao Li
Post Training Quantization Quantization Error Low Precision Quantization

April 1, 2024

Instance-Aware Group Quantization for Vision Transformers
Jaehyeon Moon, Dohyung Kim, Junyong Cheon, Bumsub Ham
Convolutional Neural Network Vision Transformer Post Training Quantization Softmax Attention Extreme Quantization

March 11, 2024

COMQ: A Backpropagation-Free Algorithm for Post-Training Quantization
Aozhong Zhang, Zi Yang, Naigang Wang, Yingyong Qi, Jack Xin, Xin Li, Penghang Yin
Quantization Operator Post Training Quantization Backpropagation Free Bit Quantization Integer Quantization Bit Vision Transformer

March 10, 2024

FrameQuant: Flexible Low-Bit Quantization for Transformers
Harshavardhan Adepu, Zhanpeng Zeng, Li Zhang, Vikas Singh
Transformer Megatron Decepticons Transformer Model Transformer Based Model Quantization Operator Post Training Quantization Bit Quantization

February 28, 2024

Evaluating Quantized Large Language Models
Shiyao Li, Xuefei Ning, Luning Wang, Tengxuan Liu, Xiangsheng Shi, Shengen Yan, Guohao Dai, Huazhong Yang, Yu Wang
Large Language Model Post Training Quantization Quantization Technique

February 25, 2024

Towards Accurate Post-training Quantization for Reparameterized Models
Luoming Zhang, Yefei He, Wen Fei, Zhenyu Lou, Weijia Wu, YangWei Ying, Hong Zhou
Post Training Quantization Reparameterization Model

February 21, 2024

APTQ: Attention-aware Post-Training Mixed-Precision Quantization for Large Language Models
Ziyi Guan, Hantao Huang, Yupeng Su, Hong Huang, Ngai Wong, Hao Yu
Post Training Quantization Mixed Precision Quantization

February 14, 2024

Towards Next-Level Post-Training Quantization of Hyper-Scale Transformers
Junhan Kim, Chungman Lee, Eulrang Cho, Kyungphil Park, Ho-young Kim, Joonyoung Kim, Yongkweon Jeon
Language Model Transformer Model Post Training Quantization Generative AI Model Multiplier Free Quantization Transformer Based Continual Learning

February 8, 2024

RepQuant: Towards Accurate Post-Training Quantization of Large Transformer Models via Scale Reparameterization
Zhikai Li, Xuewen Liu, Jing Zhang, Qingyi Gu
Post Training Quantization Large Transformer Model Accurate Quantization Large Scale Transformer

February 6, 2024

January 9, 2024

EDA-DM: Enhanced Distribution Alignment for Post-Training Quantization of Diffusion Models
Xuewen Liu, Zhikai Li, Junrui Xiao, Qingyi Gu
Diffusion Model Fine Grained Quantization Operator Post Training Quantization Distribution Alignment Denoising Model

December 13, 2023

CBQ: Cross-Block Quantization for Large Language Models
Xin Ding, Xiaoyu Liu, Zhijun Tu, Yun Zhang, Wei Li, Jie Hu, Hanting Chen, Yehui Tang, Zhiwei Xiong, Baoqun Yin, Yunhe Wang
Large Language Model Post Training Quantization Bit Quantization Block Quantization

November 27, 2023

TFMQ-DM: Temporal Feature Maintenance Quantization for Diffusion Models
Yushi Huang, Ruihao Gong, Jing Liu, Tianlong Chen, Xianglong Liu
Diffusion Model Post Training Quantization Quantization Technique Quantization Level Dynamic Quantization

November 21, 2023

Shedding the Bits: Pushing the Boundaries of Quantization with Minifloats on FPGAs
Shivam Aggarwal, Hans Jakob Damsgaard, Alessandro Pappalardo, Giuseppe Franco, Thomas B. Preußer, Michaela Blott, Tulika Mitra
Quantization Operator Field Programmable Gate Array Post Training Quantization Document Boundary Quantization Technique P Bit Floating Point Format