Pre Trained Vision Transformer

Pre-trained Vision Transformers (ViTs) leverage the power of large-scale pre-training to efficiently adapt to diverse downstream computer vision tasks, focusing on parameter and computational efficiency. Current research emphasizes methods like low-rank adaptation, visual prompt tuning, and multiple-exit strategies to minimize the number of trainable parameters and optimize inference speed, particularly for resource-constrained environments like onboard satellite processing and mobile devices. These advancements are significantly impacting various fields, including remote sensing, medical image analysis, and robotics, by enabling high-accuracy visual recognition with reduced computational demands.

Papers

November 11, 2024

ScaleKD: Strong Vision Transformers Could Be Excellent Teachers
Jiawei Fan, Chao Li, Xiaolong Liu, Anbang Yao
Convolutional Neural Network Vision Transformer Good Teacher Pre Trained Vision Transformer

November 8, 2024

Image inpainting enhancement by replacing the original mask with a self-attended region from the input image
Kourosh Kiani, Razieh Rastgoo, Alireza Chaji, Sergio Escalera
Vision Transformer Feature Enhancement Pre Trained Vision Transformer Image Patch Input Image Pre Processing Image Inpainting

October 30, 2024

Efficient Adaptation of Pre-trained Vision Transformer via Householder Transformation
Wei Dong, Yuan Sun, Yiting Yang, Xing Zhang, Zhijun Lin, Qingsen Yan, Haokui Zhang, Peng Wang, Yang Yang, Hengtao Shen
Vision Transformer Low Rank Adaptation Pre Trained Vision Transformer Character Transformation Efficient Adaptation Projection Matrix Residential Household Covariance Matrix Adaptation

October 27, 2024

PViT: Prior-augmented Vision Transformer for Out-of-distribution Detection
Tianhao Zhang, Zhixiang Chen, Lyudmila S. Mihaylova
Vision Transformer Data Detection Distribution Data Pre Trained Vision Transformer Large Vision Model ImageNet Benchmark Distribution Image

September 23, 2024

Designing Pre-training Datasets from Unlabeled Data for EEG Classification with Transformers
Tim Bary, Benoit Macq
Data Set Transformer Megatron Decepticons Product Design Unlabeled Data Pre Trained Vision Transformer Self Supervised Training EEG Classification Seizure Prediction Temple University Hospital

September 21, 2024

Multiple-Exit Tuning: Towards Inference-Efficient Adaptation for Vision Transformer
Zheng Liu, Jinchao Zhu, Nannan Li, Gao Huang
Vision Transformer Pre Trained Vision Transformer Inference Time Parameter Efficient Transfer

September 5, 2024

Onboard Satellite Image Classification for Earth Observation: A Comparative Study of ViT Models
Thanh-Dung Le, Vu Nguyen Ha, Ti Ti Nguyen, Geoffrey Eappen, Prabhu Thiruvasagam, Luis M. Garces-Socarras, Hong-fu Chou, Jorge L. Gonzalez-Rios, Juan Carlos Merlano-Duncan, Symeon Chatzinotas
Convolutional Neural Network Comparative Study Satellite Image Pre Trained Vision Transformer Local Inference EfficientViT SAM

August 13, 2024

Dynamic and Compressive Adaptation of Transformers From Images to Videos
Guozhen Zhang, Jingyu Liu, Shengming Cao, Xiaotong Zhao, Kevin Zhao, Kai Ma, Limin Wang
Transformer Megatron Decepticons Gameplay Video Pre Trained Vision Transformer Frame Interpolation Compressive Learning Naive Adaptation Image to Video Adaptation

August 3, 2024

Downstream Transfer Attack: Adversarial Attacks on Downstream Models with Pre-trained Vision Transformers
Weijie Zheng, Xingjun Ma, Hanxun Huang, Zuxuan Wu, Yu-Gang Jiang
Adversarial Attack Adversarial Example Pre Trained Vision Transformer Adversarial Vulnerability Pre Trained ViT Downstream Model Transfer Based Attack

July 16, 2024

Siamese Transformer Networks for Few-shot Image Classification
Weihao Jiang, Shuoxi Zhang, Kun He
Pre Trained Vision Transformer Shot Image Classification Siamese Transformer

July 9, 2024

Parameter-Efficient and Memory-Efficient Tuning for Vision Transformer: A Disentangled Approach
Taolin Zhang, Jiawang Bai, Zhihe Lu, Dongze Lian, Genping Wang, Xinchao Wang, Shu-Tao Xia
Vision Transformer Pre Trained Model Task Specific Pre Trained Vision Transformer Recognition Task Memory Efficient Fine Tuning Disentanglement Method

July 5, 2024

PDiscoFormer: Relaxing Part Discovery Constraints with Vision Transformers
Ananthu Aniraj, Cassio F. Dantas, Dino Ienco, Diego Marcos
Vision Transformer Pre Trained Vision Transformer Pre Trained ViT Part Discovery

June 24, 2024

Priorformer: A UGC-VQA Method with content and distortion priors
Yajing Pei, Shiyu Huang, Yiting Lu, Xin Li, Zhibo Chen
Pre Trained Vision Transformer Video Quality Assessment Implicit Feature Blind Video Quality Assessment

June 16, 2024

ExPLoRA: Parameter-Efficient Extended Pre-Training to Adapt Vision Transformers under Domain Shifts
Samar Khanna, Medhanie Irgau, David B. Lobell, Stefano Ermon
Vision Transformer Transfer Learning Foundation Model Domain Shift Pre Training Pre Trained Vision Transformer

June 15, 2024

Self Pre-training with Topology- and Spatiality-aware Masked Autoencoders for 3D Medical Image Segmentation
Pengfei Gu, Yejia Zhang, Huimin Li, Chaoli Wang, Danny Z. Chen
State of the Art Masked Autoencoders Topological Feature Pre Trained Vision Transformer 3D Medical Image Segmentation Masked AutoEncoder Self Information ViT Encoder

June 5, 2024

Learning Visual Prompts for Guiding the Attention of Vision Transformers
Razieh Rezaei, Masoud Jalili Sabet, Jindong Gu, Daniel Rueckert, Philip Torr, Ashkan Khakzar
Vision Transformer Pre Trained Human Attention Visual Prompt Pre Trained Vision Transformer Visual Prompting

May 31, 2024

A Comparative Study of CNN, ResNet, and Vision Transformers for Multi-Classification of Chest Diseases
Ananya Jain, Aviral Bhardwaj, Kaushik Murali, Isha Surani
Convolutional Neural Network Vision Transformer Supervised ImageNet Transformer Architecture CNN Network Pre Trained Vision Transformer ResNet Based Multi Category

May 28, 2024

Near-Infrared and Low-Rank Adaptation of Vision Transformers in Remote Sensing
Irem Ulku, O. Ozgur Tanriover, Erdem Akagündüz
Vision Transformer Remote Sensing Adaptation Concern Pre Trained Vision Transformer Near Infrared

May 24, 2024

Recasting Generic Pretrained Vision Transformers As Object-Centric Scene Encoders For Manipulation Policies
Jianing Qian, Anastasios Panagopoulos, Dinesh Jayaraman
Pre Trained Pre Trained Vision Transformer Manipulation Policy Robot Representation Object Centric Scene

May 23, 2024

Sparse-Tuning: Adapting Vision Transformers with Efficient Fine-tuning and Inference
Ting Liu, Xuyang Liu, Siteng Huang, Liangtao Shi, Zunnan Xu, Yi Xin, Quanjun Yin, Xiaohong Liu
Vision Transformer Fine Tuning Scientific Inference Parameter Efficient Fine Tuning Pre Trained Vision Transformer Efficient Fine Tuning

Pre Trained Vision Transformer

Papers

ScaleKD: Strong Vision Transformers Could Be Excellent Teachers

Image inpainting enhancement by replacing the original mask with a self-attended region from the input image

Efficient Adaptation of Pre-trained Vision Transformer via Householder Transformation

PViT: Prior-augmented Vision Transformer for Out-of-distribution Detection

Designing Pre-training Datasets from Unlabeled Data for EEG Classification with Transformers

Multiple-Exit Tuning: Towards Inference-Efficient Adaptation for Vision Transformer

Onboard Satellite Image Classification for Earth Observation: A Comparative Study of ViT Models

Dynamic and Compressive Adaptation of Transformers From Images to Videos

Downstream Transfer Attack: Adversarial Attacks on Downstream Models with Pre-trained Vision Transformers

Siamese Transformer Networks for Few-shot Image Classification

Parameter-Efficient and Memory-Efficient Tuning for Vision Transformer: A Disentangled Approach

PDiscoFormer: Relaxing Part Discovery Constraints with Vision Transformers

Priorformer: A UGC-VQA Method with content and distortion priors

ExPLoRA: Parameter-Efficient Extended Pre-Training to Adapt Vision Transformers under Domain Shifts

Self Pre-training with Topology- and Spatiality-aware Masked Autoencoders for 3D Medical Image Segmentation

Learning Visual Prompts for Guiding the Attention of Vision Transformers

A Comparative Study of CNN, ResNet, and Vision Transformers for Multi-Classification of Chest Diseases

Near-Infrared and Low-Rank Adaptation of Vision Transformers in Remote Sensing

Recasting Generic Pretrained Vision Transformers As Object-Centric Scene Encoders For Manipulation Policies

Sparse-Tuning: Adapting Vision Transformers with Efficient Fine-tuning and Inference