Deep Transformer

Deep Transformers are complex neural networks aiming to improve upon the limitations of previous architectures by leveraging self-attention mechanisms for processing sequential data like text and images. Current research focuses on enhancing efficiency and stability through techniques like adaptive token processing, modified attention blocks, and optimized residual connections, often within architectures such as Vision Transformers (ViTs) and variations of the standard Transformer block. These advancements are significant because they enable the application of deep Transformers to resource-constrained environments and improve performance on various tasks, including image classification, natural language processing, and even circuit design.

Papers

May 4, 2023

BranchNorm: Robustly Scaling Extremely Deep Transformers
Yijin Liu, Xianfeng Zeng, Fandong Meng, Jie Zhou
Robust Version Deep Model Deep Transformer Strong Scaling Stable Training

April 28, 2023

ResiDual: Transformer with Dual Residual Connections
Shufang Xie, Huishuai Zhang, Junliang Guo, Xu Tan, Jiang Bian, Hany Hassan Awadalla, Arul Menezes, Tao Qin, Rui Yan
Transformer Based Transformer Network Layer Normalization Deep Transformer Residual Connection Residual Stream Residual Transformer

April 4, 2023

Effective Theory of Transformers at Initialization
Emily Dinan, Sho Yaida, Susan Zhang
Transformer Megatron Decepticons New Initialization Deep Transformer Backward Propagation Residual Neural Network Effective Theory

February 20, 2023

Deep Transformers without Shortcuts: Modifying Self-attention for Faithful Signal Propagation
Bobby He, James Martens, Guodong Zhang, Aleksandar Botev, Andrew Brock, Samuel L Smith, Yee Whye Teh
Self Attention Propagation Environment Deep Transformer Normalization Layer Shortcut Feature Vanilla Transformer Deep Kernel

January 30, 2023

Alternating Updates for Efficient Transformers
Cenk Baykal, Dylan Cutler, Nishanth Dikkala, Nikhil Ghosh, Rina Panigrahy, Xin Wang
Inference Latency Efficient Transformer Supervised Baseline Deep Transformer Information System Update Dynamic Update to Data Ratio

January 17, 2023

The Recent Advances in Automatic Term Extraction: A survey
Hanh Thi Hong Tran, Matej Martinc, Jaya Caporusso, Antoine Doucet, Senja Pollak
Natural Language Processing Timely Survey Machine Translation Recent Advance Deep Transformer Domain Corpus Term Extraction

January 8, 2023

DeepMatcher: A Deep Transformer-based Network for Robust and Accurate Local Feature Matching
Tao Xie, Kun Dai, Ke Wang, Ruifeng Li, Lijun Zhao
Robust Version Transformer Based Network Deep Transformer Local Feature Matching Low Level Feature Dense Matching

December 24, 2022

Optimizing Deep Transformers for Chinese-Thai Low-Resource Translation
Wenjie Hao, Hongfei Xu, Lingling Mu, Hongying Zan
Low Resource Translation Quality Layer Transformer Deep Transformer Transformer Based Deep

October 26, 2022

Automatic Diagnosis of Myocarditis Disease in Cardiac MRI Modality using Deep Transformers and Explainable Artificial Intelligence
Mahboobeh Jafari, Afshin Shoeibi, Navid Ghassemi, Jonathan Heras, Sai Ho Ling, Amin Beheshti, Yu-Dong Zhang, Shui-Hua Wang, Roohallah Alizadehsani, Juan M. Gorriz, U. Rajendra Acharya, Hamid Alinejad Rokny
Explainable Artificial Intelligence Cardiac Magnetic Resonance Deep Transformer Automatic Diagnosis Coronary Artery Disease

September 16, 2022

Transformer-based Detection of Multiword Expressions in Flower and Plant Names
Damith Premasiri, Amal Haddad Haddad, Tharindu Ranasinghe, Ruslan Mitkov
Long Short Term Memory Deep Transformer Many Natural Language Processing Plant Identification Multiword Expression Transformer Based Detection

June 30, 2022

Rethinking Surgical Captioning: End-to-End Window-Based MLP Transformer Using Patches
Mengya Xu, Mobarakol Islam, Hongliang Ren
End to End Captioning Model Caption Generation Deep Transformer Class Relevant Patch

June 23, 2022

ICOS Protein Expression Segmentation: Can Transformer Networks Give Better Results?
Vivek Kumar Singh, Paul O Reilly, Jacqueline James, Manuel Salto Tellez, Perry Maxwell
Transformer Network Performance Improvement Cell Segmentation Deep Transformer Immune Checkpoint

June 21, 2022

Neural Transformers for Intraductal Papillary Mucosal Neoplasms (IPMN) Classification in MRI images
Federica Proietto Salanitri, Giovanni Bellitto, Simone Palazzo, Ismail Irmakci, Michael B. Wallace, Candice W. Bolan, Megan Engels, Sanne Hoogenboom, Marco Aldinucci, Ulas Bagci, Daniela Giordano, Concetto Spampinato
Classification Code Magnetic Resonance Image Deep Transformer Intraductal Papillary Mucosal Neoplasm

June 9, 2022

STNDT: Modeling Neural Population Activity with a Spatiotemporal Transformer
Trung Le, Eli Shlizerman
Neural Activity Deep Transformer Neural Dynamic Spatiotemporal Transformer Neural Population

June 1, 2022

B2T Connection: Serving Stability and Performance in Deep Transformers
Sho Takase, Shun Kiyono, Sosuke Kobayashi, Jun Suzuki
System Performance Back Propagation Core Stability Layer Normalization Deep Transformer Shallow Transformer

May 22, 2022

Dynamic Query Selection for Fast Visual Perceiver
Corentin Dancette, Matthieu Cord
Cross Attention Perception Model Deep Transformer Latent Token Efficient Perception

May 21, 2022

A Study on Transformer Configuration and Training Objective
Fuzhao Xue, Jianghai Chen, Aixin Sun, Xiaozhe Ren, Zangwei Zheng, Xiaoxin He, Yongming Chen, Xin Jiang, Yang You
Supervised Autoencoder Study Feature Deep Transformer Autoencoder Model Transformer Layer Deeper Model Transformer Performance Training Objective

April 14, 2022

3D Shuffle-Mixer: An Efficient Context-Aware Vision Learner of Transformer-MLP Paradigm for Dense Prediction in Medical Volume
Jianye Pang, Cheng Jiang, Yihao Chen, Jianbo Chang, Ming Feng, Renzhi Wang, Jianhua Yao
Vision Transformer Dense Prediction Deep Transformer Local Transformer Visual in Context Learning MRI Volume CNN Backbone

March 9, 2022

Anti-Oversmoothing in Deep Vision Transformers via the Fourier Domain Analysis: From Theory to Practice
Peihao Wang, Wenqing Zheng, Tianlong Chen, Zhangyang Wang
Convolutional Neural Network Vision Transformer Theoretical Understanding Deep Transformer Squashing Mitigation Approach Self Attention Block Self Attention Matrix

March 1, 2022

DeepNet: Scaling Transformers to 1,000 Layers
Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, Furu Wei
Transformer Megatron Decepticons Transformer Based Deep Network Multi Layer Deep Transformer