Bit Vision Transformer

Bit vision transformers (BitViTs) aim to reduce the computational cost and memory footprint of vision transformers (ViTs) by representing their weights and activations using fewer bits, thereby enabling efficient deployment on resource-constrained devices. Current research focuses on developing novel quantization techniques, such as those employing learnable scaling factors or softmax-aware binarization, to minimize accuracy loss during this compression. These advancements, applied to architectures like DeiT and Swin, are improving the performance of low-bit ViTs, with some methods even achieving accuracy comparable to or exceeding full-precision models, and others facilitating automated hardware acceleration for real-time applications.

Papers

March 11, 2024

COMQ: A Backpropagation-Free Algorithm for Post-Training Quantization
Aozhong Zhang, Zi Yang, Naigang Wang, Yingyong Qi, Jack Xin, Xin Li, Penghang Yin
Quantization Operator Post Training Quantization Backpropagation Free Bit Quantization Integer Quantization Bit Vision Transformer

February 4, 2023

Oscillation-free Quantization for Low-bit Vision Transformers
Shih-Yang Liu, Zechun Liu, Kwang-Ting Cheng
Quantization Aware Training Quantization Technique Weight Only Quantization Weight Monitoring Scaling Factor Bit Vision Transformer

November 14, 2022

BiViT: Extremely Compressed Binary Vision Transformer
Yefei He, Zhenyu Lou, Luoming Zhang, Jing Liu, Weijia Wu, Hong Zhou, Bohan Zhuang
Vision Transformer Binarization Method Vision Transformer Compression Binary Vision Transformer Bit Vision Transformer Layer Binarization

October 13, 2022

Q-ViT: Accurate and Fully Quantized Low-bit Vision Transformer
Yanjing Li, Sheng Xu, Baochang Zhang, Xianbin Cao, Peng Gao, Guodong Guo
Vision Transformer Multiplier Free Quantization Self Attention Layer Pre Trained Vision Transformer Bit Vision Transformer

January 17, 2022

VAQF: Fully Automatic Software-Hardware Co-Design Framework for Low-Bit Vision Transformer
Mengshu Sun, Haoyu Ma, Guoliang Kang, Yifan Jiang, Tianlong Chen, Xiaolong Ma, Zhangyang Wang, Yanzhi Wang
Vision Transformer Co Design Activation Quantization Quantization Loss Bit Vision Transformer

Bit Vision Transformer

Papers

COMQ: A Backpropagation-Free Algorithm for Post-Training Quantization

Oscillation-free Quantization for Low-bit Vision Transformers

BiViT: Extremely Compressed Binary Vision Transformer

Q-ViT: Accurate and Fully Quantized Low-bit Vision Transformer

VAQF: Fully Automatic Software-Hardware Co-Design Framework for Low-Bit Vision Transformer