Transformer Backbone

Transformer backbones are increasingly used as the foundation for various deep learning models, aiming to improve efficiency and performance across diverse tasks. Current research focuses on enhancing their robustness, compositionality, and parameter efficiency through modifications like selective attention mechanisms, shared backbones for multi-modal data, and novel fine-tuning strategies such as generative parameter-efficient fine-tuning. These advancements are impacting numerous fields, including computer vision (image classification, segmentation, and object detection), natural language processing (various reasoning and text understanding tasks), and remote sensing, by enabling more accurate and efficient model training and deployment.

Papers

October 17, 2024

Analyzing Deep Transformer Models for Time Series Forecasting via Manifold Learning
Ilya Kaufman, Omri Azencot
Transformer Model Time Series Forecasting Manifold Learning Deep Learning Based Transformer Based Deep Latent Manifold Transformer Backbone

August 2, 2024

UNER: A Unified Prediction Head for Named Entity Recognition in Visually-rich Documents
Yi Tu, Chong Zhang, Ya Guo, Huan Chen, Jinyang Tang, Huijia Zhu, Qi Zhang
Entity Recognition Visually Rich Document Transformer Backbone Entity Relation Extraction Prediction Head Entity Recognition Performance

April 24, 2024

SPARO: Selective Attention for Robust and Compositional Transformer Encodings for Vision
Ankit Vani, Bac Nguyen, Samuel Lavoie, Ranjay Krishna, Aaron Courville
Representation Learning Supervised ImageNet Vision Paper Robust Version Structured Representation Compositional Learning Transformer Backbone

January 15, 2024

One for All: Toward Unified Foundation Models for Earth Vision
Zhitong Xiong, Yi Wang, Fahong Zhang, Xiao Xiang Zhu
Foundation Model Multi Modal Remote Sensing Visual Encoder Transformer Backbone

December 1, 2023

Generative Parameter-Efficient Fine-Tuning
Chinmay Savadikar, Xi Song, Tianfu Wu
Vision Transformer Fine Tuning Parameter Efficient Fine Tuning Built in Interpretability Transformer Backbone

August 31, 2023

Document Layout Analysis on BaDLAD Dataset: A Comprehensive MViTv2 Based Approach
Ashrafur Rahman Khan, Asif Azad
Pre Trained Transformer Architecture MAESTRO Dataset Document Layout Analysis Mask R CNN Transformer Backbone Mask Specific Loss

July 18, 2023

U-shaped Transformer: Retain High Frequency Context in Time Series Analysis
Qingkui Chen, Yiqin Zhang
Time Series Time Series Analysis Long Range Context U Shaped Transformer Backbone Simple MLP

June 5, 2023

Transformer-Based UNet with Multi-Headed Cross-Attention Skip Connections to Eliminate Artifacts in Scanned Documents
David Kreuzer, Michael Munz
Convolutional Neural Network Optical Character Recognition Ring Artifact Transformer Backbone Scanned Document Transformer Based UNet

May 19, 2023

Enhancing Transformer Backbone for Egocentric Video Action Segmentation
Sakib Reza, Balaji Sundareshan, Mohsen Moghaddam, Octavia Camps
Hierarchical Representation Action Segmentation Transformer Backbone

February 9, 2023

In-Context Learning with Many Demonstration Examples
Mukai Li, Shansan Gong, Jiangtao Feng, Yiheng Xu, Jun Zhang, Zhiyong Wu, Lingpeng Kong
Context Learning Pre Trained Large Language Model Efficient Transformer Transformer Backbone Long Range Language Demonstration Example

December 27, 2022

Exploring Transformer Backbones for Image Diffusion Models
Princy Chahal
End to End Transformer Model Image Synthesis Image Diffusion Model Transformer Backbone Domain Diffusion

September 20, 2022

Dynamic Graph Message Passing Networks for Visual Recognition
Li Zhang, Mohan Chen, Anurag Arnab, Xiangyang Xue, Philip H. S. Torr
Visual Recognition Self Attention Module Message Passing Graph Neural Network Fully Connected Transformer Backbone

August 15, 2022

Conv-Adapter: Exploring Parameter Efficient Transfer Learning for ConvNets
Hao Chen, Ran Tao, Han Zhang, Yidong Wang, Xiang Li, Wei Ye, Jindong Wang, Guosheng Hu, Marios Savvides
Transformer Architecture High Performing 1D ConvNet Parameter Efficient Transfer Learning Transformer Backbone

July 29, 2022

ScaleFormer: Revisiting the Transformer-based Backbones from a Scale-wise Perspective for Medical Image Segmentation
Huimin Huang, Shiao Xie1, Lanfen Lin, Yutaro Iwamoto, Xianhua Han, Yen-Wei Chen, Ruofeng Tong
Vision Transformer Medical Image Segmentation Global Context Transformer Backbone Scale Dependency

June 15, 2022

Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbone
Zi-Yi Dou, Aishwarya Kamath, Zhe Gan, Pengchuan Zhang, Jianfeng Wang, Linjie Li, Zicheng Liu, Ce Liu, Yann LeCun, Nanyun Peng, Jianfeng Gao, Lijuan Wang
Fine Grained Vision Language Hybrid Fusion Multimodal Fusion Numerous Cutting Edge Backbone Transformer Backbone

February 2, 2022

Exploring Transformer Backbones for Heterogeneous Treatment Effect Estimation
Yi-Fan Zhang, Hanlin Zhang, Zachary C. Lipton, Li Erran Li, Eric P. Xing
Treatment Effect Propensity Score Transformer Backbone Heterogeneous Treatment Effect Estimation Continuous Treatment Effect

December 28, 2021

Pale Transformer: A General Vision Transformer Backbone with Pale-Shaped Attention
Sitong Wu, Tianyi Wu, Haoru Tan, Guodong Guo
Vision Transformer Attention Layer Transformer Backbone Vision Transformer Backbone

December 23, 2021

SeMask: Semantically Masked Transformers for Semantic Segmentation
Jitesh Jain, Anukriti Singh, Nikita Orlov, Zilong Huang, Jiachen Li, Steven Walton, Humphrey Shi
Semantic Segmentation Vision Transformer Network Map Prior Transformer Backbone Masked Transformer