Sharpness Aware Quantization

Sharpness-aware quantization (SAQ) aims to improve the accuracy of quantized deep neural networks (DNNs), a crucial technique for deploying models on resource-constrained hardware. Current research focuses on mitigating the negative impact of quantization on model robustness, particularly addressing issues like sensitivity to hardware faults and instability during training, often employing techniques like Sharpness-Aware Minimization (SAM) to smooth the loss landscape. This work is significant because it tackles the trade-off between model compression and accuracy, leading to more efficient and reliable DNNs across various architectures, including convolutional neural networks, transformers, and even spiking neural networks, with applications in image compression and natural language processing.

Papers

June 15, 2024

Memory Faults in Activation-sparse Quantized Deep Neural Networks: Analysis and Mitigation using Sharpness-aware Training
Akul Malhotra, Sumeet Kumar Gupta
Deep Neural Network General Analysis Activation Sparsity Sparse Activation Memory Access Sharpness Aware Training Sharpness Aware Quantization

April 15, 2024

SQUAT: Stateful Quantization-Aware Training in Recurrent Spiking Neural Networks
Sreyes Venkatesh, Razvan Marinescu, Jason K. Eshraghian
Quantization Aware Training Weight Quantization Quantization Level Uniform Quantization Quad Attention Sharpness Aware Quantization

October 20, 2023

Zero-Shot Sharpness-Aware Quantization for Pre-trained Language Models
Miaoxi Zhu, Qihuang Zhong, Li Shen, Liang Ding, Juhua Liu, Bo Du, Dacheng Tao
Pre Trained Language Model Zero Shot Quantization Sharpness Aware Quantization

September 29, 2023

On Uniform Scalar Quantization for Learned Image Compression
Haotian Zhang, Li Li, Dong Liu
Learned Image Compression Quantization Step Scalar Quantization Sharpness Aware Quantization

November 24, 2021

Sharpness-aware Quantization for Deep Neural Networks
Jing Liu, Jianfei Cai, Bohan Zhuang
Deep Neural Network Sharpness Aware Minimization Multiplier Free Quantization Uniform Quantization Network Quantization Sharpness Aware Quantization

Sharpness Aware Quantization

Papers

Memory Faults in Activation-sparse Quantized Deep Neural Networks: Analysis and Mitigation using Sharpness-aware Training

SQUAT: Stateful Quantization-Aware Training in Recurrent Spiking Neural Networks

Zero-Shot Sharpness-Aware Quantization for Pre-trained Language Models

On Uniform Scalar Quantization for Learned Image Compression

Sharpness-aware Quantization for Deep Neural Networks