Pure Transformer

Pure Transformer models are emerging as powerful alternatives to convolutional neural networks (CNNs) in various computer vision and time-series tasks, aiming to leverage the attention mechanism's ability to capture long-range dependencies and global context. Current research focuses on adapting Transformer architectures, such as Swin Transformers and Vision Transformers (ViTs), for specific applications, often incorporating modifications to improve efficiency and address limitations in handling local details or multi-scale information. These advancements demonstrate the potential of pure Transformers to achieve state-of-the-art performance in diverse fields, including image processing, video analysis, and time series forecasting, while also offering opportunities for improved efficiency and reduced computational costs compared to traditional methods.

Papers

October 7, 2024

PredFormer: Transformers Are Effective Spatial-Temporal Predictive Learners
Yujin Tang, Lu Qi, Fei Xie, Xiangtai Li, Chao Ma, Ming-Hsuan Yang
Vision Transformer Transformer Megatron Decepticons Temporal Attention Pure Transformer Spatiotemporal Predictive Learning Spatio Temporal Predictive Learning

June 19, 2024

SwinStyleformer is a favorable choice for image inversion
Jiawei Mao, Guangyi Zhao, Xuesong Yin, Yuanqi Chang
Self Attention Swin Transformer Image Inversion Pure Transformer

May 23, 2024

PuTR: A Pure Transformer for Decoupled and Online Multi-Object Tracking
Chongwei Liu, Haojie Li, Zhihui Wang, Rui Xu
Multi Object Tracking Structured Output Trajectory Data Tracking by Detection Multi Agent Decoupling Coefficient Pure Transformer Long Term Tracking Online Multi Object Tracking

March 24, 2024

A Transformer approach for Electricity Price Forecasting
Oscar Llorente, Jose Portela
Transformer Model Recurrent Network Transformer Based Approach Pure Transformer Electricity Price Forecasting

February 4, 2024

A Graph is Worth $K$ Words: Euclideanizing Graph using Pure Transformer
Zhangyang Gao, Daize Dong, Cheng Tan, Jun Xia, Bozhen Hu, Stan Z. Li
Graph Drawing Graph Representation Learning Word List Non Euclidean Pure Transformer Word Usage Graph Non Euclidean Graph

October 22, 2023

ConViViT -- A Deep Neural Network Combining Convolutions and Factorized Self-Attention for Human Activity Recognition
Rachid Reda Dokkar, Faten Chaieb, Hassen Drira, Arezki Aberkane
Self Attention Transformer Architecture Activity Recognition Video Representation Spatial Token Pure Transformer Combining Deep Learning

September 30, 2023

Pubic Symphysis-Fetal Head Segmentation Using Pure Transformer with Bi-level Routing Attention
Pengzhou Cai, Lu Jiang, Yanxin Li, Libin Lan
Ball Rolling Pure Transformer Transrectal Ultrasound Routing Attention

March 23, 2023

POTTER: Pooling Attention Transformer for Efficient Human Mesh Recovery
Ce Zheng, Xianpeng Liu, Guo-Jun Qi, Chen Chen
Attention Module Human Mesh Recovery Attention Based Pooling Pure Transformer Attention Transformer

March 14, 2023

FPTN: Fast Pure Transformer Network for Traffic Flow Forecasting
Junhao Zhang, Junjie Tang, Juncheng Jin, Zehui Qu
Traffic Data Traffic Flow Forecasting Spatiotemporal Correlation Pure Transformer

December 28, 2022

OVO: One-shot Vision Transformer Search with Online distillation
Zimian Wei, Hengyue Pan, Xin Niu, Dongsheng Li
Supervised ImageNet Domain Sample Pure Transformer Online Distillation

November 9, 2022

Pure Transformer with Integrated Experts for Scene Text Recognition
Yew Lee Tan, Adams Wai-kin Kong, Jung-Jae Kim
Transformer Architecture Scene Text Recognition Transformer Decoder Hybrid CNN Transformer Scene Text Image Pure Transformer

October 18, 2022

Transfer-learning for video classification: Video Swin Transformer on multiple domains
Daniel Oliveira, David Martins de Matos
Transfer Learning Swin Transformer Multi Domain Video Classification Computer Vision Community Pure Transformer

August 1, 2022

TransDeepLab: Convolution-Free Transformer-based DeepLab v3+ for Medical Image Segmentation
Reza Azad, Moein Heidari, Moein Shariatnia, Ehsan Khodapanah Aghdam, Sanaz Karimijafarbigloo, Ehsan Adeli, Dorit Merhof
Convolutional Neural Network Vision Transformer Medical Image Segmentation Deep Model Swin Transformer Pure Transformer Convolution Free

July 6, 2022

Pure Transformers are Powerful Graph Learners
Jinwoo Kim, Tien Dat Nguyen, Seonwoo Min, Sungjun Cho, Moontae Lee, Honglak Lee, Seunghoon Hong
Graph Transformer Graph Learning Message Passing Graph Neural Network Token Embeddings Pure Transformer Inductive Graph Invariant Graph Network

June 2, 2022

EfficientFormer: Vision Transformers at MobileNet Speed
Yanyu Li, Geng Yuan, Yang Wen, Ju Hu, Georgios Evangelidis, Sergey Tulyakov, Yanzhi Wang, Jian Ren
Convolutional Neural Network Vision Transformer L MobileNet Faster Model Pure Transformer

April 29, 2022

SideRT: A Real-time Pure Transformer Architecture for Single Image Depth Estimation
Chang Shu, Ziming Chen, Lei Chen, Kuan Ma, Minghui Wang, Haibing Ren
Transformer Based Network Single Image Depth Estimation Pure Transformer Cross Scale Attention

April 25, 2022

SwinFuse: A Residual Swin Transformer Fusion Network for Infrared and Visible Images
Zhishe Wang, Yanlin Chen, Wenyu Shao, Hui Li, Lei Zhang
Convolutional Neural Network Richer RGB Infrared Feature Convolutional Operation Pure Transformer Visible Image

March 31, 2022

TransGeo: Transformer Is All You Need for Cross-view Image Geo-localization
Sijie Zhu, Mubarak Shah, Chen Chen
Convolutional Neural Network Transformer Based Cross View Pure Transformer

December 9, 2021

PE-former: Pose Estimation Transformer
Paschalis Panteleris, Antonis Argyros
Encoder Decoder Vision Transformer Architecture Convolutional Backbone Pure Transformer X Former CNN Backbone Pose Induced Video Transformer