Transformer Model

Transformer models are a class of neural networks built upon an attention mechanism, enabling them to process sequential data like text and time series with remarkable effectiveness. Current research focuses on improving training stability (e.g., mitigating loss spikes), enhancing expressiveness through novel attention mechanisms and embedding techniques, and optimizing performance for various applications by exploring different architectures (e.g., hybrid Transformer-Mamba models) and parallelization strategies. This work is significant due to the widespread adoption of transformers in diverse fields, from natural language processing and computer vision to scientific computing and engineering, driving advancements in both theoretical understanding and practical applications.

Papers

February 12, 2023

Transformer models: an introduction and catalog
Xavier Amatriain, Ananth Sankar, Jie Bing, Praveen Kumar Bodigutla, Timothy J. Hazen, Michaeel Kazi
Self Supervised Learning Transformer Based Transformer Model Gentle Introduction Instruct Model

February 3, 2023

Coinductive guide to inductive transformer heads
Adam Nemecek
Transformer Model Head Transformer Coherence Aware Hopf Algebra

February 1, 2023

Attention Link: An Efficient Attention-Based Low Resource Machine Translation Architecture
Zeping Min
Machine Translation Transformer Model Parallel Corpus Attention Head Attention Based Architecture Low Resource Neural Machine Translation NMT Model

January 27, 2023

Understanding INT4 Quantization for Transformer Models: Latency Speedup, Composability, and Failure Cases
Xiaoxia Wu, Cheng Li, Reza Yazdani Aminabadi, Zhewei Yao, Yuxiong He
Transformer Model Encoder Decoder Model Composable Framework Inference Pipeline Failure Explanation INT4 Quantization

January 25, 2023

Transfer Learning in Deep Learning Models for Building Load Forecasting: Case of Limited Data
Menna Nawar, Moustafa Shomer, Samy Faddel, Huangjie Gong
Deep Learning Model Transfer Learning Long Short Term Memory Transformer Model Case Relevance Limited Data Load Forecasting

January 17, 2023

Tracing and Manipulating Intermediate Values in Neural Math Problem Solvers
Yuta Matsumoto, Benjamin Heinzerling, Masashi Yoshikawa, Kentaro Inui
Language Model Inherent Interpretability Scientific Inference Transformer Model Neural Solver Complex Valued Source Tracing

January 5, 2023

Towards Long-Term Time-Series Forecasting: Feature, Pattern, and Distribution
Yan Li, Xinjiang Lu, Haoyi Xiong, Jian Tang, Jiantao Su, Bo Jin, Dejing Dou
Self Attention Transformer Model Time Series Forecasting Complex Pattern Long Term Time Series Forecasting Content Based Feature Product Distribution

December 27, 2022

December 19, 2022

(Psycho-)Linguistic Features Meet Transformer Models for Improved Explainable and Controllable Text Simplification
Yu Qiao, Xiaofei Li, Daniel Wiechmann, Elma Kerz
Pre Trained Language Model Transformer Model Complexity Matter Text Simplification Seq2seq Model

December 15, 2022

Rethinking Vision Transformers for MobileNet Size and Speed
Yanyu Li, Ju Hu, Yang Wen, Georgios Evangelidis, Kamyar Salahi, Yanzhi Wang, Sergey Tulyakov, Jian Ren
Vision Transformer Transformer Model Speed Effect L MobileNet Lightweight CNN

December 5, 2022

December 3, 2022

Recognition and Prediction of Surgical Gestures and Trajectories Using Transformer Models in Robot-Assisted Surgery
Chang Shi, Yi Zheng, Ann Majewicz Fey
Human Prediction Transformer Model Handwritten Trajectory Robot Assisted Surgery End Effector Surgical Gesture Surgical Activity Recognition

December 2, 2022

Tackling Low-Resourced Sign Language Translation: UPC at WMT-SLT 22
Laia Tarrés, Gerard I. Gàllego, Xavier Giró-i-Nieto, Jordi Torres
Machine Translation Transformer Model Sign Language Translation Task Sign Language Translation Camouflage Attack

November 25, 2022

Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelism
Xupeng Miao, Yujie Wang, Youhe Jiang, Chunan Shi, Xiaonan Nie, Hailin Zhang, Bin Cui
Neural Network Transformer Model Multi GPU Expert Parallelism 3D Parallelism

November 18, 2022

Vision Transformers in Medical Imaging: A Review
Emerald U. Henry, Onyeka Emebob, Conrad Asotie Omonhinmin
Convolutional Neural Network Vision Transformer Transformer Based Narrative Review Transformer Model Medical Imaging Attention Based Encoder Decoder

November 9, 2022

Efficiently Scaling Transformer Inference
Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Anselm Levskaya, Jonathan Heek, Kefan Xiao, Shivani Agrawal, Jeff Dean
Transformer Model Transformer Based Model Inference Efficiency Transformer Inference Large Deep Learning Model Efficient Generative

November 5, 2022

Privacy-Preserving Models for Legal Natural Language Processing
Ying Yin, Ivan Habernal
Full Model Differential Privacy Transformer Model Transformer Language Model Large Transformer Model Legal Natural Language Processing Adversarial Privacy

November 4, 2022

BERT for Long Documents: A Case Study of Automated ICD Coding
Arash Afkanpour, Shabir Adeel, Hansenclever Bassani, Arkady Epshteyn, Hongbo Fan, Isaac Jones, Mahan Malihi, Adrian Nauth, Raj Sinha, Sanjana Woonna, Shiva Zamani, Elli Kanal, Mikhail Fomitchev, Donny Cheung
Case Study Transformer Model Ticket BERT Long Document Long Text Medical Coding ICD Code NLP Problem