Transformer Model

Transformer models are a class of neural networks built upon an attention mechanism, enabling them to process sequential data like text and time series with remarkable effectiveness. Current research focuses on improving training stability (e.g., mitigating loss spikes), enhancing expressiveness through novel attention mechanisms and embedding techniques, and optimizing performance for various applications by exploring different architectures (e.g., hybrid Transformer-Mamba models) and parallelization strategies. This work is significant due to the widespread adoption of transformers in diverse fields, from natural language processing and computer vision to scientific computing and engineering, driving advancements in both theoretical understanding and practical applications.

Papers

August 6, 2023

Average-Hard Attention Transformers are Constant-Depth Uniform Threshold Circuits
Lena Strobl
Transformer Model Precision Transformer Threshold Circuit

August 1, 2023

July 28, 2023

BARTPhoBEiT: Pre-trained Sequence-to-Sequence and Image Transformers Models for Vietnamese Visual Question Answering
Khiem Vinh Tran, Kiet Van Nguyen, Ngan Luu Thuy Nguyen
Transformer Model Visual Question Answering Sequence to Sequence Image Transformer Vietnamese Language Model Vietnamese Image Captioning

July 27, 2023

Improving Natural Language Inference in Arabic using Transformer Models and Linguistically Informed Pre-Training
Mohammad Majd Saad Al Deen, Maren Pielka, Jörn Hees, Bouthaina Soulef Abdou, Rafet Sifa
Natural Language Processing Entity Recognition Transformer Model Natural Language Inference Arabic Speaker Arabic Dataset Contradiction Detection

July 26, 2023

Are Transformers with One Layer Self-Attention Using Low-Rank Weight Matrices Universal Approximators?
Tokio Kajitsuka, Issei Sato
Transformer Megatron Decepticons Transformer Model Softmax Function Self Attention Layer Data Memorization Universal Approximators Head Transformer

July 22, 2023

Identifying Misinformation on YouTube through Transcript Contextual Analysis with Transformer Models
Christos Christodoulou, Nikos Salamanos, Pantelitsa Leonidou, Michail Papadakis, Michael Sirivianos
Transfer Learning Transformer Model Fake News Misinformation Claim Speech to Text Misinformation Video

July 20, 2023

July 19, 2023

Improving Domain Generalization for Sound Classification with Sparse Frequency-Regularized Transformer
Honglin Mu, Wentian Xia, Wanxiang Che
Transformer Based Domain Generalization Transformer Model State of the Art Many Sparse Classification Model Audio Classification Long Term Recurrent Convolutional Network Regularized Transformer

July 7, 2023

ITA: An Energy-Efficient Attention and Softmax Accelerator for Quantized Transformers
Gamze İslamoğlu, Moritz Scherer, Gianna Paulin, Tim Fischer, Victor J. B. Jung, Angelo Garofalo, Luca Benini
Transformer Model Softmax Function Transformer Network Hardware Acceleration Accelerator Architecture Quantized Transformer Transformer Accelerator

July 5, 2023

Improving Automatic Parallel Training via Balanced Memory Workload Optimization
Yujie Wang, Youhe Jiang, Xupeng Miao, Fangcheng Fu, Shenhan Zhu, Xiaonan Nie, Yaofeng Tu, Bin Cui
Transformer Model Parallel Training GPU Memory Expert Parallelism Memory Optimization Large Scale Deep Learning

June 24, 2023

L3Cube-MahaSent-MD: A Multi-domain Marathi Sentiment Analysis Dataset and Transformer Models
Aabha Pingle, Aditya Vyawahare, Isha Joshi, Rahul Tangsali, Raviraj Joshi
Sentiment Analysis Transformer Model Sentiment Polarity Multi Domain L3Cube MahaSocialNER Domain Sentiment

June 23, 2023

Swin-Free: Achieving Better Cross-Window Attention and Efficiency with Size-varying Window
Jinkyu Koo, John Yang, Le An, Gwenaelle Cunha Sergio, Su Inn Park
Vision Transformer High Efficiency Human Attention Transformer Model Swin Transformer

June 22, 2023

Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing
Yelysei Bondarenko, Markus Nagel, Tijmen Blankevoort
Language Model Vision Transformer Transformer Model Attention Head Quantized Transformer

June 20, 2023

June 19, 2023

ExpPoint-MAE: Better interpretability and performance for self-supervised point cloud transformers
Ioannis Romanelis, Vlassis Fotis, Konstantinos Moustakas, Adrian Munteanu
Transformer Based System Performance Transformer Model Learned Representation Better Interpretability Point Cloud Transformer Learned Feature

June 18, 2023

Gender Bias in Transformer Models: A comprehensive survey
Praneeth Nemani, Yericherla Deepak Joel, Palla Vijay, Farhana Ferdousi Liza
Language Model Transformer Model Absolute Stance Bias Comprehensive Survey Gender Bias Linguistic Perspective

June 13, 2023

Reviving Shift Equivariance in Vision Transformers
Peijian Ding, Davit Soselia, Thomas Armstrong, Jiahao Su, Furong Huang
Vision Transformer Transformer Model Shift Equivariance

Transformer Model

Papers

Average-Hard Attention Transformers are Constant-Depth Uniform Threshold Circuits

ViT2EEG: Leveraging Hybrid Pretrained Vision Transformers for EEG Data

LGViT: Dynamic Early Exiting for Accelerating Vision Transformer

BARTPhoBEiT: Pre-trained Sequence-to-Sequence and Image Transformers Models for Vietnamese Visual Question Answering

Improving Natural Language Inference in Arabic using Transformer Models and Linguistically Informed Pre-Training

Are Transformers with One Layer Self-Attention Using Low-Rank Weight Matrices Universal Approximators?

Identifying Misinformation on YouTube through Transcript Contextual Analysis with Transformer Models

PASTA: Pretrained Action-State Transformer Agents

Learned Thresholds Token Merging and Pruning for Vision Transformers

Improving Domain Generalization for Sound Classification with Sparse Frequency-Regularized Transformer

ITA: An Energy-Efficient Attention and Softmax Accelerator for Quantized Transformers

Improving Automatic Parallel Training via Balanced Memory Workload Optimization

L3Cube-MahaSent-MD: A Multi-domain Marathi Sentiment Analysis Dataset and Transformer Models

Swin-Free: Achieving Better Cross-Window Attention and Efficiency with Size-varying Window

Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing

Exploring the Performance and Efficiency of Transformer Models for NLP on Mobile Devices

LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximation

ExpPoint-MAE: Better interpretability and performance for self-supervised point cloud transformers

Gender Bias in Transformer Models: A comprehensive survey

Reviving Shift Equivariance in Vision Transformers