Transformer Model

Transformer models are a class of neural networks built upon an attention mechanism, enabling them to process sequential data like text and time series with remarkable effectiveness. Current research focuses on improving training stability (e.g., mitigating loss spikes), enhancing expressiveness through novel attention mechanisms and embedding techniques, and optimizing performance for various applications by exploring different architectures (e.g., hybrid Transformer-Mamba models) and parallelization strategies. This work is significant due to the widespread adoption of transformers in diverse fields, from natural language processing and computer vision to scientific computing and engineering, driving advancements in both theoretical understanding and practical applications.

Papers

April 14, 2023

Masked Pre-Training of Transformers for Histology Image Analysis
Shuai Jiang, Liesbeth Hondelink, Arief A. Suriawinata, Saeed Hassanpour
Transformer Megatron Decepticons Transformer Model Digital Pathology Histopathological Image Image Analysis Cancer Subtype Masked Pre Training

April 4, 2023

ConvFormer: Parameter Reduction in Transformer Models for 3D Human Pose Estimation by Leveraging Dynamic Multi-Headed Convolutional Attention
Alec Diaz-Arias, Dmitriy Shin
Transformer Model 3D Human Pose Estimation Multi Head Self Attention Fully Convolutional Parameter Reduction

April 1, 2023

Vision Transformers with Mixed-Resolution Tokenization
Tomer Ronen, Omer Levy, Avram Golbert
Vision Transformer Transformer Model Image Tokenizer Accurate Saliency

March 27, 2023

TransCODE: Co-design of Transformers and Accelerators for Efficient Training and Inference
Shikhar Tuli, Niraj K. Jha
Transformer Megatron Decepticons Scientific Inference Transformer Architecture Transformer Model Efficient Training Transformer Inference Transformer Accelerator Accelerator Tuning Video Transcoding

March 24, 2023

EdgeTran: Co-designing Transformers for Efficient Inference on Mobile Edge Platforms
Shikhar Tuli, Niraj K. Jha
Transformer Architecture Transformer Model Efficient Inference Efficient Transformer Large Transformer Model Transformer Performance Edge Platform

March 17, 2023

March 15, 2023

Transformer Models for Type Inference in the Simply Typed Lambda Calculus: A Case Study in Deep Learning for Code
Brando Miranda, Avi Shinnar, Vasily Pestun, Barry Trager
Deep Learning Transformer Model Real World Code Deep Learning Architecture Lambda Calculus Type Inference

March 14, 2023

March 13, 2023

Transformer Models for Acute Brain Dysfunction Prediction
Brandon Silva, Miguel Contreras, Tezcan Ozrazgat Baslanti, Yuanfang Ren, Guan Ziyuan, Kia Khezeli, Azra Bihorac, Parisa Rashidi
Transformer Model Intensive Care Unit Acute Brain Dysfunction

March 8, 2023

X-Pruner: eXplainable Pruning for Vision Transformers
Lu Yu, Wei Xiang
Vision Transformer Transformer Model Based Pruning LD Pruner

March 6, 2023

Angel-PTM: A Scalable and Economical Large-scale Pre-training System in Tencent
Xiaonan Nie, Yi Liu, Fangcheng Fu, Jinbao Xue, Dian Jiao, Xupeng Miao, Yangyu Tao, Bin Cui
Pre Trained Model Transformer Model System Description Deep Learning System Large Scale Pre Trained Model Large Scale Model

March 5, 2023

Comparative study of Transformer and LSTM Network with attention mechanism on Image Captioning
Pranav Dandwate, Chaitanya Shahane, Vandana Jagtap, Shridevi C. Karande
Transformer Based Comparative Study Attention Mechanism Transformer Model Image Captioning LSTM Network COCO Dataset InceptionV3 Model

March 4, 2023

Calibrating Transformers via Sparse Gaussian Processes
Wenlong Chen, Yingzhen Li
Transformer Model Transformer Training Sparse Gaussian Process Distribution Robustness

February 28, 2023

February 18, 2023

Transformadores: Fundamentos teoricos y Aplicaciones
Jordi de la Torre
Transformer Megatron Decepticons Transformer Based Self Attention Attention Mechanism Transformer Model Neural Network Architecture

February 17, 2023

ViTA: A Vision Transformer Inference Accelerator for Edge Applications
Shashank Nag, Gourav Datta, Souvik Kundu, Nitin Chandrachoodan, Peter A. Beerel
Transformer Model Swin Transformer Hardware Accelerator Transformer Inference Edge Application MLP Layer

February 16, 2023

Efficiency 360: Efficient Vision Transformers
Badri N. Patro, Vijay Srinivas Agneeswaran
Vision Transformer Computer Vision Transformer Megatron Decepticons High Efficiency Image Classification Transformer Model