Binary Vision Transformer

Binary Vision Transformers (Bi-ViTs) aim to drastically reduce the computational cost and memory footprint of Vision Transformers (ViTs) by binarizing their weights and activations, enabling deployment on resource-constrained devices. Current research focuses on improving the accuracy of these binarized models through techniques like novel binarization methods tailored to the unique characteristics of ViTs (e.g., addressing attention mechanisms and gradient issues), architectural modifications inspired by Convolutional Neural Networks, and knowledge distillation from higher-precision models. The success of Bi-ViTs would significantly broaden the accessibility and applicability of ViTs, impacting various computer vision tasks and expanding the potential of edge computing.

Papers

January 22, 2024

BETA: Binarized Energy-Efficient Transformer Accelerator at the Edge
Yuhao Ji, Chao Fang, Zhongfeng Wang
Extreme Edge Matrix Multiplication FPGA Accelerator Edge Transformer Transformer Accelerator Binary Vision Transformer

December 14, 2023

BiPFT: Binary Pre-trained Foundation Transformer with Low-rank Estimation of Binarization Residual Polynomials
Xingrun Xing, Li Du, Xinyuan Wang, Xianlin Zeng, Yequan Wang, Zheng Zhang, Jiajun Zhang
Foundation Model Binary Neural Network Low Rank Matrix Estimation Binary Vision Transformer Foundation Transformer

June 29, 2023

BinaryViT: Pushing Binary Vision Transformers Towards Convolutional Models
Phuoc-Hoan Charles Le, Xinlin Li
Convolutional Neural Network Vision Transformer Deep Convolutional Neural Network Convolutional Neural Network Model Convolutional Neural Network Architecture ViT Model Binary Vision Transformer

May 24, 2023

BinaryViT: Towards Efficient and Accurate Binary Vision Transformers
Junrui Xiao, Zhikai Li, Lianwei Yang, Qingyi Gu
Neural Network Vision Transformer High Efficiency Binarization Method Binary Vision Transformer

May 21, 2023

Bi-ViT: Pushing the Limit of Vision Transformer Quantization
Yanjing Li, Sheng Xu, Mingbao Lin, Xianbin Cao, Chuanjian Liu, Xiao Sun, Baochang Zhang
Vision Transformer Supervised ImageNet Continuum Limit ViT Lens Large Pre Trained Vision Transformer Quantization Binary Vision Transformer

November 14, 2022

BiViT: Extremely Compressed Binary Vision Transformer
Yefei He, Zhenyu Lou, Luoming Zhang, Jing Liu, Weijia Wu, Hong Zhou, Bohan Zhuang
Vision Transformer Binarization Method Vision Transformer Compression Binary Vision Transformer Bit Vision Transformer Layer Binarization

May 25, 2022

BiT: Robustly Binarized Multi-distilled Transformer
Zechun Liu, Barlas Oguz, Aasish Pappu, Lin Xiao, Scott Yih, Meng Li, Raghuraman Krishnamoorthi, Yashar Mehdad
Transformer Model Pre Trained Transformer Binarization Method P Bit BERT Baseline Binary Vision Transformer