Transformer Model

Transformer models are a class of neural networks built upon an attention mechanism, enabling them to process sequential data like text and time series with remarkable effectiveness. Current research focuses on improving training stability (e.g., mitigating loss spikes), enhancing expressiveness through novel attention mechanisms and embedding techniques, and optimizing performance for various applications by exploring different architectures (e.g., hybrid Transformer-Mamba models) and parallelization strategies. This work is significant due to the widespread adoption of transformers in diverse fields, from natural language processing and computer vision to scientific computing and engineering, driving advancements in both theoretical understanding and practical applications.

Papers

March 25, 2022

A Comparative Evaluation Of Transformer Models For De-Identification Of Clinical Text Data
Christopher Meaney, Wali Hakimpour, Sumeet Kalia, Rahim Moineddin
Transformer Model Comparative Evaluation Clinical Data Sensitive Patient De Identification

March 24, 2022

March 23, 2022

Mokey: Enabling Narrow Fixed-Point Inference for Out-of-the-Box Floating-Point Transformer Models
Ali Hadi Zadeh, Mostafa Mahmoud, Ameer Abdelhadi, Andreas Moshovos
Transformer Model Shelf Model Precision Transformer

February 23, 2022

FastRPB: a Scalable Relative Positional Encoding for Long Sequence Tasks
Maksim Zubkov, Daniil Gavrilov
Transformer Model Linear Transformer Relative Positional Encoding Sequence Task Shift Invariant Kernel

February 22, 2022

Social Computational Design Method for Generating Product Shapes with GAN and Transformer Models
Maolin Yang, Pingyu Jiang
GAN Model Transformer Model 3D Shape AI Model Artificial Intelligence Technology Shape Generation

February 17, 2022

Transformer for Graphs: An Overview from Architecture Perspective
Erxue Min, Runfa Chen, Yatao Bian, Tingyang Xu, Kangfei Zhao, Wenbing Huang, Peilin Zhao, Junzhou Huang, Sophia Ananiadou, Yu Rong
Transformer Based Graph Drawing Transformer Model Graph Transformer Graph Structured Data Architecture Perspective Graph Level Task

February 16, 2022

The NLP Task Effectiveness of Long-Range Transformers
Guanghui Qin, Yukun Feng, Benjamin Van Durme
Natural Language Processing Transformer Model Long Span Long Range Attention Transformer Variant Long Range Transformer

February 15, 2022

XAI for Transformers: Better Explanations through Conservative Propagation
Ameen Ali, Thomas Schnake, Oliver Eberle, Grégoire Montavon, Klaus-Robert Müller, Lior Wolf
Transformer Megatron Decepticons Transformer Model xAI Community Interpretability Method Gradient Information Better Explainability Explanation Performance Hierarchical Propagation

February 8, 2022

Logical Reasoning for Task Oriented Dialogue Systems
Sajjad Beygi, Maryam Fazel-Zarandi, Alessandra Cervone, Prakash Krishnan, Siddhartha Reddy Jonnalagadda
Transformer Model Dialogue System Reasoning Capability Task Oriented Dialogue System Logical Reasoning

February 7, 2022

Universal Spam Detection using Transfer Learning of BERT Model
Vijay Srinivas Tida, Sonya Hsu
Transfer Learning Transformer Model Bidirectional Encoder Representation From Transformer Pretrained Model Spam Detection

January 24, 2022

Transformers in Medical Imaging: A Survey
Fahad Shamshad, Salman Khan, Syed Waqas Zamir, Muhammad Haris Khan, Munawar Hayat, Fahad Shahbaz Khan, Huazhu Fu
Convolutional Neural Network Timely Survey Computer Vision Transformer Megatron Decepticons Medical Image Segmentation Transformer Model Medical Imaging Transformer Based Medical

January 16, 2022

Video Transformers: A Survey
Javier Selva, Anders S. Johansen, Sergio Escalera, Kamal Nasrollahi, Thomas B. Moeslund, Albert Clapés
Timely Survey Transformer Megatron Decepticons Transformer Model 3d CNN Video Transformer Video Customization

December 29, 2021

Fake or Genuine? Contextualised Text Representation for Fake Review Detection
Rami Mohawesh, Shuxiang Xu, Matthew Springer, Muna Al-Hawawreh, Sumbal Maqsood
Transformer Model Fake Speech Fake Review Novel Ensemble User Profiling Fake Review Detection

December 27, 2021

Transformer Uncertainty Estimation with Hierarchical Stochastic Attention
Jiahuan Pei, Cheng Wang, György Szarvas
Transformer Model Uncertainty Estimation Transformer Based Structure Transformer Prediction

December 20, 2021

Lite Vision Transformer with Enhanced Self-Attention
Chenglin Yang, Yilin Wang, Jianming Zhang, He Zhang, Zijun Wei, Zhe Lin, Alan Yuille
Vision Transformer Self Attention Transformer Model Lite Transformer

December 16, 2021

Learning Bounded Context-Free-Grammar via LSTM and the Transformer:Difference and Explanations
Hui Shi, Sicun Gao, Yuandong Tian, Xinyun Chen, Jishen Zhao
Transformer Based Long Short Term Memory Line by Line Explanation Transformer Model Qualitative Difference Probabilistic Context Free Grammar

December 15, 2021

Evaluating Pretrained Transformer Models for Entity Linking in Task-Oriented Dialog
Sai Muralidhar Jayanthi, Varsha Embar, Karthik Raghunathan
Transformer Model Task Oriented Entity Linking Natural Language Task Unsupervised Entity

December 10, 2021

Human Guided Exploitation of Interpretable Attention Patterns in Summarization and Topic Segmentation
Raymond Li, Wen Xiao, Linzi Xing, Lanjun Wang, Gabriel Murray, Giuseppe Carenini
Attention Mechanism Transformer Model Structured Summary Multi Head Self Attention Attention Pattern Topic Segmentation

December 6, 2021

Scaling Up Influence Functions
Andrea Schioppa, Polina Zablotskaia, David Vilar, Artem Sokolov
Transformer Model Influence Function Newton Iteration