Pre Trained Vision Transformer

Pre-trained Vision Transformers (ViTs) leverage the power of large-scale pre-training to efficiently adapt to diverse downstream computer vision tasks, focusing on parameter and computational efficiency. Current research emphasizes methods like low-rank adaptation, visual prompt tuning, and multiple-exit strategies to minimize the number of trainable parameters and optimize inference speed, particularly for resource-constrained environments like onboard satellite processing and mobile devices. These advancements are significantly impacting various fields, including remote sensing, medical image analysis, and robotics, by enabling high-accuracy visual recognition with reduced computational demands.

Papers

March 1, 2023

Time Series as Images: Vision Transformer for Irregularly Sampled Time Series
Zekun Li, Shiyang Li, Xifeng Yan
Vision Transformer Time Series Time Series Classification Pre Trained Vision Transformer

February 28, 2023

Remote Sensing Scene Classification with Masked Image Modeling (MIM)
Liya Wang, Alex Tien
Vision Transformer Masked Image Modeling Pre Trained Vision Transformer Scene Classification Remote Sensing Scene Classification

February 8, 2023

Adapting Pre-trained Vision Transformers from 2D to 3D through Weight Inflation Improves Medical Image Segmentation
Yuhui Zhang, Shih-Cheng Huang, Zhengping Zhou, Matthew P. Lungren, Serena Yeung
Medical Image Segmentation 2 Dimensional Medical Image Analysis 3D Content Pre Trained Transformer Pre Trained Vision Transformer 3D Medical Image 3D Medical Imaging Weight Dynamic

February 6, 2023

AIM: Adapting Image Models for Efficient Video Action Recognition
Taojiannan Yang, Yi Zhu, Yusheng Xie, Aston Zhang, Chen Chen, Mu Li
Vision Transformer Pre Trained Action Recognition Video Understanding Image Modeling Pre Trained Vision Transformer

February 1, 2023

CertViT: Certified Robustness of Pre-Trained Vision Transformers
Kavya Gupta, Sagar Verma
Vision Transformer Native Robustness Adversarial Robustness Pre Trained Vision Transformer Lipschitz Bound

January 24, 2023

Model soups to increase inference without increasing compute time
Charles Dansereau, Milo Sobral, Maninder Bhogal, Mehdi Zalai
Full Model Scientific Inference Pre Trained Vision Transformer EfficientNet Algorithm Model Soup Computation Time

December 6, 2022

FacT: Factor-Tuning for Lightweight Adaptation on Vision Transformer
Shibo Jie, Zhi-Hong Deng
Vision Transformer Tensor Decomposition Pre Trained Vision Transformer Parameter Efficient Transfer

November 23, 2022

CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learning
James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante-Bonilla, Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, Zsolt Kira
Catastrophic Forgetting Domain Shift Microbial Decomposition Pre Trained Vision Transformer Continual Learning Benchmark Rehearsal Free Continual Learning

November 1, 2022

ViT-DeiT: An Ensemble Model for Breast Cancer Histopathological Images Classification
Amira Alotaibi, Tarik Alafif, Faris Alkhilaiwi, Yasser Alatawi, Hassan Althobaiti, Abdulmajeed Alrefaei, Yousef M Hawsawi, Tin Nguyen
Vision Transformer Histopathological Image Ensemble Model Pre Trained Vision Transformer ViT Lens Breast Cancer Histopathological Image Classification

October 23, 2022

Delving into Masked Autoencoders for Multi-Label Thorax Disease Classification
Junfei Xiao, Yutong Bai, Alan Yuille, Zongwei Zhou
Convolutional Neural Network Vision Transformer Masked Autoencoders Pre Trained Vision Transformer

October 13, 2022

Q-ViT: Accurate and Fully Quantized Low-bit Vision Transformer
Yanjing Li, Sheng Xu, Baochang Zhang, Xianbin Cao, Peng Gao, Guodong Guo
Vision Transformer Multiplier Free Quantization Self Attention Layer Pre Trained Vision Transformer Bit Vision Transformer

October 6, 2022

SynBench: Task-Agnostic Benchmarking of Pretrained Representations using Synthetic Data
Ching-Yun Ko, Pin-Yu Chen, Jeet Mohapatra, Payel Das, Luca Daniel
Synthetic Data Pre Trained Model Evaluation Benchmark Pre Trained Vision Transformer Pre Trained Representation Task Agnostic Representation Pretrained Model

July 27, 2022

Boosting Point-BERT by Multi-choice Tokens
Kexue Fu, Mingzhi Yuan, Manning Wang
Masked Language Pre Trained Vision Transformer Token Attention

July 10, 2022

Facilitated machine learning for image-based fruit quality assessment
Manuel Knott, Fernando Perez-Cruz, Thijs Defraeye
Convolutional Neural Network Pre Trained Vision Transformer Fruit Ripeness Apple Detection

June 22, 2022

Parallel Pre-trained Transformers (PPT) for Synthetic Data-based Instance Segmentation
Ming Li, Jie Wu, Jinhang Cai, Jie Qin, Yuxi Ren, Xuefeng Xiao, Min Zheng, Rui Wang, Xin Pan
Transformer Megatron Decepticons Pre Trained Transformer Synthetic to Real Pre Trained Vision Transformer

May 30, 2022

Zero-Shot and Few-Shot Learning for Lung Cancer Multi-Label Classification using Vision Transformer
Fu-Ming Guo, Yingfang Fan
LeArning Abstract Vision Transformer Zero Shot Classification Code Pre Trained Vision Transformer Lung Cancer

May 26, 2022

AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition
Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, Ping Luo
Vision Transformer Visual Recognition Pre Trained Vision Transformer Video Recognition Scalable Vision

May 20, 2022

Self-supervised 3D anatomy segmentation using self-distilled masked image transformer (SMIT)
Jue Jiang, Neelam Tyagi, Kathryn Tringale, Christopher Crane, Harini Veeraraghavan
Vision Transformer Pre Trained Vision Transformer Organ Segmentation Anatomical Segmentation Abdominal Organ Segmentation Image Transformer

April 8, 2022

Underwater Image Enhancement Using Pre-trained Transformer
Abderrahmene Boudiaf, Yuhang Guo, Adarsh Ghimire, Naoufel Werghi, Giulia De Masi, Sajid Javed, Jorge Dias
Underwater Image Pre Trained Vision Transformer Underwater Image Enhancement Transformer Based Approach Image Transformer

March 10, 2022

Self Pre-training with Masked Autoencoders for Medical Image Classification and Segmentation
Lei Zhou, Huidong Liu, Joseph Bae, Junjun He, Dimitris Samaras, Prateek Prasanna
Segmentation Based Approach Medical Image Classification Masked Autoencoders Pre Trained Vision Transformer Masked AutoEncoder Self Information Large Scale Medical Medical Imaging Task