Uniform Quantization

Uniform quantization aims to reduce the computational cost and memory footprint of deep neural networks by representing weights and activations using low-bit integer representations. Current research focuses on mitigating the accuracy loss inherent in this process, particularly for complex architectures like Vision Transformers (ViTs) and large language models (LLMs), through techniques such as adaptive quantization schemes, per-layer or per-head bit-width optimization, and the incorporation of floating-point formats. These advancements are crucial for deploying large models on resource-constrained devices and accelerating inference speeds in various applications, from computer vision to natural language processing.

Papers

October 27, 2024

Logarithmically Quantized Distributed Optimization over Dynamic Multi-Agent Networks
Mohammadreza Doostmohammadian, Sérgio Pequito
Optimization Purpose Multi Agent Network Dynamic Optimization Uniform Quantization Discrete Logarithm

October 17, 2024

Progressive Mixed-Precision Decoding for Efficient LLM Inference
Hao Mark Chen, Fuwen Tan, Alexandros Kouris, Royson Lee, Hongxiang Fan, Stylianos I. Venieris
Large Language Model LLM Inference Language Task Uniform Quantization

July 17, 2024

AdaLog: Post-Training Quantization for Vision Transformers with Adaptive Logarithm Quantizer
Zhuguanyu Wu, Jiaxin Chen, Hanwen Zhong, Di Huang, Yunhong Wang
Vision Transformer Quantization Operator Post Training Quantization Computer Vision Community Uniform Quantization

July 3, 2024

ADFQ-ViT: Activation-Distribution-Friendly Post-Training Quantization for Vision Transformers
Yanfeng Jiang, Ning Sun, Xueshuo Xie, Fei Yang, Tao Li
Vision Transformer Quantization Operator Uniform Quantization GPUSQ ViT

May 22, 2024

Clipped Uniform Quantizers for Communication-Efficient Federated Learning
Zavareh Bozorgasl, Hao Chen
Quantization Error Uniform Quantization Adaptive Clipping Stochastic Quantization

May 1, 2024

Gradient-based Automatic Mixed Precision Quantization for Neural Networks On-Chip
Chang Sun, Thea K. Årrestad, Vladimir Loncar, Jennifer Ngadiuba, Maria Spiropulu
Multiplier Free Quantization Quantization Aware Training Mixed Precision Quantization Uniform Quantization Chip Learning Quantization Granularity

April 19, 2024

decoupleQ: Towards 2-bit Post-Training Uniform Quantization via decoupling Parameters into Integer and Floating Points
Yi Guo, Fanliu Kong, Xiaoyang Li, Hui Li, Wei Chen, Xiaogang Tian, Jinping Cai, Yang Zhang, Shouda Liu
Multi Agent Decoupling Coefficient Bit Quantization Floating Point Integer Arithmetic Uniform Quantization

April 15, 2024

SQUAT: Stateful Quantization-Aware Training in Recurrent Spiking Neural Networks
Sreyes Venkatesh, Razvan Marinescu, Jason K. Eshraghian
Quantization Aware Training Weight Quantization Quantization Level Uniform Quantization Quad Attention Sharpness Aware Quantization

April 10, 2024

Differentiable Search for Finding Optimal Quantization Strategy
Lianqiang Li, Chenqian Yan, Yefei Chen
Uniform Quantization Differentiable Search Mixed Quantization Quantization Loss Optimal Quantization Quantization Search

March 26, 2024

Oh! We Freeze: Improving Quantized Knowledge Distillation via Signal Propagation Analysis for Large Language Models
Kartikeya Bhardwaj, Nilesh Prasad Pandey, Sweta Priyadarshi, Kyunggeun Lee, Jun Ma, Harris Teague
Large Language Model Knowledge Distillation Propagation Environment Quantization Level Weight Only Quantization Uniform Quantization Quantization Aware Knowledge Distillation

August 10, 2023

NUPES : Non-Uniform Post-Training Quantization via Power Exponent Search
Edouard Yvinec, Arnaud Dapogny, Kevin Bailly
Quantization Operator Multiplier Free Quantization Neural Network Weight Exponential Mechanism Non Uniform Neural Network Quantization Uniform Quantization Non Uniform Quantization

July 19, 2023

ZeroQuant-FP: A Leap Forward in LLMs Post-Training W4A8 Quantization Using Floating-Point Formats
Xiaoxia Wu, Zhewei Yao, Yuxiong He
Large Language Model Weight Quantization Leap Forward Uniform Quantization Floating Point Quantization W4A8 Quantization Floating Point Format

June 28, 2023

DNA-TEQ: An Adaptive Exponential Quantization of Tensors for DNN Inference
Bahareh Khabbazan, Marc Riera, Antonio González
Quantization Operator Tensor Network Biological Sequence Tensor Type Calculus DNN Inference Uniform Quantization Adaptive Quantization

May 16, 2023

MINT: Multiplier-less INTeger Quantization for Energy Efficient Spiking Neural Networks
Ruokai Yin, Yuhang Li, Abhishek Moitra, Priyadarshini Panda
Neural Network Quantization Uniform Quantization Integer Quantization Membrane Potential

February 22, 2023

Quantized Low-Rank Multivariate Regression with Random Dithering
Junren Chen, Yueqi Wang, Michael K. Ng
Low Rank Constrained Learning Multivariate Regression Uniform Quantization Lasso Estimator

January 20, 2023

Optimized learned entropy coding parameters for practical neural-based image and video compression
Amir Said, Reza Pourreza, Hoang Le
Many Parameter Video Compression Efficient Optimization Entropy Coding Entropy Solution Uniform Quantization Based Image Quantization Challenge Parameter Quantization

July 13, 2022

Learning Representations for CSI Adaptive Quantization and Feedback
Valentina Rizzello, Matteo Nerini, Michael Joham, Bruno Clerckx, Wolfgang Utschick
Representation Learning Human Feedback Post Training Quantization Channel State Information Quantization Technique Uniform Quantization Adaptive Quantization Downlink Adaptive Quantization

May 24, 2022

Approximation speed of quantized vs. unquantized ReLU neural networks and beyond
Antoine Gonon, Nicolas Brisebarre, Rémi Gribonval, Elisa Riccietti
Neural Network ReLU Layer Uniform Quantization

February 18, 2022

LG-LSQ: Learned Gradient Linear Symmetric Quantization
Shih-Ting Lin, Zhaofang Li, Yu-Hsiang Cheng, Hao-Wen Kuo, Chih-Cheng Lu, Kea-Tiong Tang
Quantization Technique Uniform Quantization Gradient Quantization Linear Quantization

January 19, 2022

Q-ViT: Fully Differentiable Quantization for Vision Transformer
Zhexin Li, Tong Yang, Peisong Wang, Jian Cheng
Vision Transformer Uniform Quantization Vision Transformer Quantization Quantization Robust Parameter

Uniform Quantization

Papers

Logarithmically Quantized Distributed Optimization over Dynamic Multi-Agent Networks

Progressive Mixed-Precision Decoding for Efficient LLM Inference

AdaLog: Post-Training Quantization for Vision Transformers with Adaptive Logarithm Quantizer

ADFQ-ViT: Activation-Distribution-Friendly Post-Training Quantization for Vision Transformers

Clipped Uniform Quantizers for Communication-Efficient Federated Learning

Gradient-based Automatic Mixed Precision Quantization for Neural Networks On-Chip

decoupleQ: Towards 2-bit Post-Training Uniform Quantization via decoupling Parameters into Integer and Floating Points

SQUAT: Stateful Quantization-Aware Training in Recurrent Spiking Neural Networks

Differentiable Search for Finding Optimal Quantization Strategy

Oh! We Freeze: Improving Quantized Knowledge Distillation via Signal Propagation Analysis for Large Language Models

NUPES : Non-Uniform Post-Training Quantization via Power Exponent Search

ZeroQuant-FP: A Leap Forward in LLMs Post-Training W4A8 Quantization Using Floating-Point Formats

DNA-TEQ: An Adaptive Exponential Quantization of Tensors for DNN Inference

MINT: Multiplier-less INTeger Quantization for Energy Efficient Spiking Neural Networks

Quantized Low-Rank Multivariate Regression with Random Dithering

Optimized learned entropy coding parameters for practical neural-based image and video compression

Learning Representations for CSI Adaptive Quantization and Feedback

Approximation speed of quantized vs. unquantized ReLU neural networks and beyond

LG-LSQ: Learned Gradient Linear Symmetric Quantization

Q-ViT: Fully Differentiable Quantization for Vision Transformer