Low Bit

Low-bit quantization aims to reduce the computational and memory demands of deep neural networks by representing model parameters and activations using fewer bits, thereby improving efficiency without significant accuracy loss. Current research focuses on developing novel quantization techniques for various architectures, including transformers, convolutional neural networks, and large language models, often employing methods like data-free quantization, layer-wise quantization, and adaptive precision strategies. This area is crucial for deploying large models on resource-constrained devices and accelerating inference, impacting both the efficiency of machine learning research and the practical applications of AI in various domains.

Papers

June 21, 2023

Training Transformers with 4-bit Integers
Haocheng Xi, Changhao Li, Jianfei Chen, Jun Zhu
Neural Network Transformer Training Low Bit Bit Training

May 14, 2023

MBQuant: A Novel Multi-Branch Topology Method for Arbitrary Bit-width Network Quantization
Yunshan Zhong, Yuyao Zhou, Fei Chao, Rongrong Ji
Topological Feature Low Bit Network Quantization Binary Activation Bit Width Quantization

February 24, 2023

DyBit: Dynamic Bit-Precision Numbers for Efficient Quantized Neural Network Inference
Jiajun Zhou, Jiajun Wu, Yizhao Gao, Yuhao Ding, Chaofan Tao, Boyu Li, Fengbin Tu, Kwang-Ting Cheng, Hayden Kwok-Hay So, Ngai Wong
DNN Model Neural Network Weight Low Bit Heterogeneous Quantization

January 30, 2023

The Hidden Power of Pure 16-bit Floating-Point Neural Networks
Juyoung Yun, Byungkon Kang, Zhoulai Fu
Low Bit Low Precision Training

August 20, 2022

DenseShift: Towards Accurate and Efficient Low-Bit Power-of-Two Quantization
Xinlin Li, Bang Liu, Rui Heng Yang, Vanessa Courville, Chao Xing, Vahid Partovi Nia
Neural Network Quantization Operator Low Bit Dense Neural Network Bit Shift Network

August 19, 2022

FP8 Quantization: The Power of the Exponent
Andrey Kuzmin, Mart Van Baalen, Yuwei Ren, Markus Nagel, Jorn Peters, Tijmen Blankevoort
Real Power Post Training Quantization Efficient Inference Exponential Mechanism P Bit Neural Network Inference Low Bit Floating Point Quantization

July 18, 2022

Is Integer Arithmetic Enough for Deep Learning Training?
Alireza Ghaffari, Marzieh S. Tahaei, Mohammadreza Tayaranian, Masoud Asgharian, Vahid Partovi Nia
Deep Learning Model Low Bit Integer Arithmetic Integer Only Training

March 11, 2022

QDrop: Randomly Dropping Quantization for Extremely Low-bit Post-Training Quantization
Xiuying Wei, Ruihao Gong, Yuhang Li, Xianglong Liu, Fengwei Yu
Quantization Operator Multiplier Free Quantization Accuracy Improvement Activation Quantization Low Bit

March 9, 2022

Power-of-Two Quantization for Low Bitwidth and Hardware Compliant Neural Networks
Dominika Przewlocka-Rus, Syed Shakib Sarwar, H. Ekin Sumbul, Yuecheng Li, Barbara De Salvo
Neural Network Quantization Operator Low Bit Non Uniform Quantization Power of Two Quantization

February 23, 2022

Bitwidth Heterogeneous Federated Learning with Progressive Weight Dequantization
Jaehong Yoon, Geon Park, Wonyong Jeong, Sung Ju Hwang
Heterogeneous Federated Learning Low Bit Efficient Dequantization Binary Weight Bit Weight

December 28, 2021

Low Bit

Papers

Training Transformers with 4-bit Integers

MBQuant: A Novel Multi-Branch Topology Method for Arbitrary Bit-width Network Quantization

DyBit: Dynamic Bit-Precision Numbers for Efficient Quantized Neural Network Inference

The Hidden Power of Pure 16-bit Floating-Point Neural Networks

DenseShift: Towards Accurate and Efficient Low-Bit Power-of-Two Quantization

FP8 Quantization: The Power of the Exponent

Is Integer Arithmetic Enough for Deep Learning Training?

QDrop: Randomly Dropping Quantization for Extremely Low-bit Post-Training Quantization

Power-of-Two Quantization for Low Bitwidth and Hardware Compliant Neural Networks

Bitwidth Heterogeneous Federated Learning with Progressive Weight Dequantization

Speedup deep learning models on GPU by taking advantage of efficient unstructured pruning and bit-width reduction

HiKonv: High Throughput Quantized Convolution With Novel Bit-wise Management and Computation