Pre Trained Vision Transformer

Pre-trained Vision Transformers (ViTs) leverage the power of large-scale pre-training to efficiently adapt to diverse downstream computer vision tasks, focusing on parameter and computational efficiency. Current research emphasizes methods like low-rank adaptation, visual prompt tuning, and multiple-exit strategies to minimize the number of trainable parameters and optimize inference speed, particularly for resource-constrained environments like onboard satellite processing and mobile devices. These advancements are significantly impacting various fields, including remote sensing, medical image analysis, and robotics, by enabling high-accuracy visual recognition with reduced computational demands.

Papers

October 27, 2023

October 18, 2023

Exploring Fairness in Pre-trained Visual Transformer based Natural and GAN Generated Image Detection Systems and Understanding the Impact of Image Compression in Fairness
Manjary P. Gangan, Anoop Kadan, Lajish V L
Procedural Fairness Image Compression Model Bias Pre Trained Vision Transformer Fake Image Bias Metric Visual Naturalness GAN Image

October 9, 2023

Hierarchical Side-Tuning for Vision Transformers
Weifeng Lin, Ziheng Wu, Wentao Yang, Mingxin Huang, Jun Huang, Lianwen Jin
Vision Transformer Fine Tuning Vision Task Pre Trained Vision Transformer Recognition Task Parameter Efficient Transfer Learning Side Tuning

September 17, 2023

MVP: Meta Visual Prompt Tuning for Few-Shot Remote Sensing Image Scene Classification
Junjie Zhu, Yiying Li, Chunping Qiu, Ke Yang, Naiyang Guan, Xiaodong Yi
Vision Transformer Visual Prompt Pre Trained Vision Transformer Shot Image Classification Visual Prompt Tuning Remote Sensing Scene Classification

September 15, 2023

September 7, 2023

DropPos: Pre-Training Vision Transformers by Reconstructing Dropped Positions
Haochen Wang, Junsong Fan, Yuxi Wang, Kaiyou Song, Tong Wang, Zhaoxiang Zhang
Vision Transformer Pre Trained Vision Transformer Attention Based Model Positional Embeddings Self Supervised Pretext Task

September 6, 2023

Combining pre-trained Vision Transformers and CIDER for Out Of Domain Detection
Grégor Jouet, Clément Duhart, Francis Rousseaux, Julio Laborde, Cyril de Runz
Vision Transformer Pre Trained Transformer Pre Trained Vision Transformer Domain Discriminator Domain Detection

August 20, 2023

Large Transformers are Better EEG Learners
Bingxin Wang, Xiaowen Fu, Yuan Lan, Luchan Zhang, Wei Zheng, Yang Xiang
Pre Trained Pre Trained Vision Transformer Large Transformer Model Large Scale Transformer EEG Task

July 27, 2023

Pre-training Vision Transformers with Very Limited Synthesized Images
Ryo Nakamura, Hirokatsu Kataoka, Sora Takashima, Edgar Josafat Martinez Noriega, Rio Yokota, Nakamasa Inoue
Vision Transformer Synthetic Data Synthetic Image Pre Trained Vision Transformer ImageNet 1k Synthesized Image

July 9, 2023

Cross-modal Orthogonal High-rank Augmentation for RGB-Event Transformer-trackers
Zhiyu Zhu, Junhui Hou, Dapeng Oliver Wu
Cross Modal Pre Trained Vision Transformer Hierarchical Mask Cross Modal Fusion Network RGBD Tracker

June 12, 2023

Learning to Mask and Permute Visual Tokens for Vision Transformer Pre-Training
Lorenzo Baraldi, Roberto Amoroso, Marcella Cornia, Lorenzo Baraldi, Andrea Pilzer, Rita Cucchiara
LeArning Abstract Self Supervised Pre Training Masked Image Modeling Pre Trained Vision Transformer Visual Token Mask Frozen DETR Mask Estimation Module

May 31, 2023

Self-supervised Vision Transformers for 3D Pose Estimation of Novel Objects
Stefan Thalhammer, Jean-Baptiste Weibel, Markus Vincze, Jose Garcia-Rodriguez
Vision Transformer Pre Trained Vision Transformer Self Supervised Vision Transformer Deep Template Pose Estimator Novel Object Template Matching

April 4, 2023

Strong Baselines for Parameter Efficient Few-Shot Fine-tuning
Samyadeep Basu, Daniela Massiceti, Shell Xu Hu, Soheil Feizi
Fine Tuning Shot Classification Strong Baseline Parameter Efficient Pre Trained Vision Transformer Efficient Fine Tuning Shot Adaptation Pre Trained ViT

March 28, 2023

ASIC: Aligning Sparse in-the-wild Image Collections
Kamal Gupta, Varun Jampani, Carlos Esteves, Abhinav Shrivastava, Ameesh Makadia, Noah Snavely, Abhishek Kar
Pre Trained Vision Transformer Dense Correspondence Image Collection Sparse Keypoints

March 24, 2023

Video Pre-trained Transformer: A Multimodal Mixture of Pre-trained Experts
Kastan Day, Daniel Christl, Rohan Salvi, Pranav Sriram
Pre Trained State of the Art Pre Trained Vision Transformer Encoder Side Encoder Model Joint Encoder

March 13, 2023

Pretrained ViTs Yield Versatile Representations For Medical Images
Christos Matsoukas, Johan Fredin Haslum, Magnus Söderberg, Kevin Smith
Convolutional Neural Network Vision Transformer Medical Image Medical Image Classification Pre Trained Vision Transformer

March 8, 2023

Centroid-centered Modeling for Efficient Vision Transformer Pre-training
Xin Yan, Zuchao Li, Lefei Zhang, Bo Du, Dacheng Tao
Vision Transformer Self Supervised Masked Image Modeling Pre Trained Vision Transformer Discrete Token Centroid Encoder

March 2, 2023

Visual Atoms: Pre-training Vision Transformers with Sinusoidal Waves
Sora Takashima, Ryo Hayamizu, Nakamasa Inoue, Hirokatsu Kataoka, Rio Yokota
Vision Transformer Synthetic Data Synthetic Image Pre Trained Vision Transformer ImageNet 1k Neural Assembly