Transformer Model

Transformer models are a class of neural networks built upon an attention mechanism, enabling them to process sequential data like text and time series with remarkable effectiveness. Current research focuses on improving training stability (e.g., mitigating loss spikes), enhancing expressiveness through novel attention mechanisms and embedding techniques, and optimizing performance for various applications by exploring different architectures (e.g., hybrid Transformer-Mamba models) and parallelization strategies. This work is significant due to the widespread adoption of transformers in diverse fields, from natural language processing and computer vision to scientific computing and engineering, driving advancements in both theoretical understanding and practical applications.

Papers

July 29, 2022

Effectiveness of Transformer Models on IoT Security Detection in StackOverflow Discussions
Nibir Chandra Mandal, G. M. Shahariar, Md. Tanvir Rouf Shawon
Transformer Model Internet of Thing Stack Overflow IoT Intrusion Detection

July 21, 2022

Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?
Yi Tay, Mostafa Dehghani, Samira Abnar, Hyung Won Chung, William Fedus, Jinfeng Rao, Sharan Narang, Vinh Q. Tran, Dani Yogatama, Donald Metzler
Transformer Model Inductive Bias Scaling Law Model Architecture Diverse Model

July 13, 2022

N-Grammer: Augmenting Transformers with latent n-grams
Aurko Roy, Rohan Anil, Guangda Lai, Benjamin Lee, Jeffrey Zhao, Shuyuan Zhang, Shibo Wang, Ye Zhang, Shen Wu, Rigel Swavely, Tao, Yu, Phuong Dao, Christopher Fifty, Zhifeng Chen, Yonghui Wu
Language Model Natural Language Processing Transformer Megatron Decepticons Transformer Model Transformer Based Large Language Model N Gram

July 12, 2022

eX-ViT: A Novel eXplainable Vision Transformer for Weakly Supervised Semantic Segmentation
Lu Yu, Wei Xiang, Juan Fang, Yi-Ping Phoebe Chen, Lianhua Chi
Vision Transformer Transformer Model Weakly Supervised Semantic Segmentation Novel Vision Transformer ViT LoS V2X Interpretable Transformer

July 10, 2022

Horizontal and Vertical Attention in Transformers
Litao Yu, Jian Zhang
Transformer Megatron Decepticons Transformer Model Dot Product Attention Vertical Pole Vertical Attention Differential Attention

July 4, 2022

Understanding Performance of Long-Document Ranking Models through Comprehensive Evaluation and Leaderboarding
Leonid Boytsov, David Akinpelu, Tianyi Lin, Fangwei Gao, Yutian Zhao, Jeffrey Huang, Nipun Katyal, Eric Nyberg
System Performance Transformer Model Long Context Comprehensive Evaluation Long Document Leaderboard Extraction

June 30, 2022

DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale
Reza Yazdani Aminabadi, Samyam Rajbhandari, Minjia Zhang, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Jeff Rasley, Shaden Smith, Olatunji Ruwase, Yuxiong He
Transformer Model Transformer Based Model Efficient Inference Transformer Inference Neural Network Inference Global Scale GPU Inference

June 29, 2022

What Can Secondary Predictions Tell Us? An Exploration on Question-Answering with SQuAD-v2.0
Michael Kamfonas, Gabriel Alon
Natural Language Processing Question Answering Transformer Model Environment Exploration Pop Quiz High Confidence Prediction Intermediate Prediction

June 5, 2022

Federated Adversarial Training with Transformers
Ahmed Aldahdooh, Wassim Hamidouche, Olivier Déforges
Vision Transformer Adversarial Example Transformer Megatron Decepticons Adversarial Training Transformer Model Federated Adversarial

June 2, 2022

June 1, 2022

May 27, 2022

Transformers from an Optimization Perspective
Yongyi Yang, Zengfeng Huang, David Wipf
Deep Learning Model Transformer Megatron Decepticons Transformer Based Transformer Model Energy Minimization

May 25, 2022

May 23, 2022

Simple Recurrence Improves Masked Language Models
Tao Lei, Ran Tian, Jasmijn Bastings, Ankur P. Parikh
Fine Tuning Transformer Based Transformer Architecture Transformer Model Masked Language Recurrence Relation Recurrent Module

May 20, 2022

Pre-training Transformer Models with Sentence-Level Objectives for Answer Sentence Selection
Luca Di Liello, Siddhant Garg, Luca Soldaini, Alessandro Moschitti
Transformer Model Sentence Level Pre Trained Transformer Sentence Representation Paragraph Speech Level Semantics Answer Sentence Selection

Transformer Model

Papers

Effectiveness of Transformer Models on IoT Security Detection in StackOverflow Discussions

Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?

N-Grammer: Augmenting Transformers with latent n-grams

eX-ViT: A Novel eXplainable Vision Transformer for Weakly Supervised Semantic Segmentation

Horizontal and Vertical Attention in Transformers

Understanding Performance of Long-Document Ranking Models through Comprehensive Evaluation and Leaderboarding

DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale

What Can Secondary Predictions Tell Us? An Exploration on Question-Answering with SQuAD-v2.0

Federated Adversarial Training with Transformers

Transforming medical imaging with Transformers? A comparative review of key properties, current progresses, and future perspectives

Modeling Image Composition for Complex Scene Generation

BayesFormer: Transformer with Uncertainty Estimation

Romantic-Computing

A comparative study between vision transformers and CNNs in digital pathology

THE-X: Privacy-Preserving Transformer Inference with Homomorphic Encryption

Transformers from an Optimization Perspective

BiT: Robustly Binarized Multi-distilled Transformer

Eliciting and Understanding Cross-Task Skills with Task-Level Mixture-of-Experts

Simple Recurrence Improves Masked Language Models

Pre-training Transformer Models with Sentence-Level Objectives for Answer Sentence Selection