Transformer Model

Transformer models are a class of neural networks built upon an attention mechanism, enabling them to process sequential data like text and time series with remarkable effectiveness. Current research focuses on improving training stability (e.g., mitigating loss spikes), enhancing expressiveness through novel attention mechanisms and embedding techniques, and optimizing performance for various applications by exploring different architectures (e.g., hybrid Transformer-Mamba models) and parallelization strategies. This work is significant due to the widespread adoption of transformers in diverse fields, from natural language processing and computer vision to scientific computing and engineering, driving advancements in both theoretical understanding and practical applications.

Papers

December 4, 2023

FaultFormer: Pretraining Transformers for Adaptable Bearing Fault Classification
Anthony Zhou, Amir Barati Farimani
Deep Learning Transformer Megatron Decepticons Self Supervised Transformer Model Adaptive Importance Machine Condition Monitoring Fault Classification

December 3, 2023

December 2, 2023

A Comprehensive Study of Vision Transformers in Image Classification Tasks
Mahmoud Khalil, Ahmad Khalil, Alioune Ngom
Vision Transformer Image Classification Transformer Model Comprehensive Study Image Classification Task Fine Grained Visual Image Classification Datasets

December 1, 2023

Mitigating Over-smoothing in Transformers via Regularized Nonlocal Functionals
Tam Nguyen, Tan M. Nguyen, Richard G. Baraniuk
Transformer Megatron Decepticons Transformer Model Deep Transformer Transformer Based Deep Baseline Transformer Smoothing Problem

November 29, 2023

RACE-IT: A Reconfigurable Analog CAM-Crossbar Engine for In-Memory Transformer Acceleration
Lei Zhao, Luca Buonanno, Ron M. Roth, Sergey Serebryakov, Archit Gajjar, John Moon, Jim Ignowski, Giacomo Pedretti
Transformer Model Memory Computing Crossbar Array Content Addressable Memory Memory Based Transformer Analog Accelerator

November 21, 2023

InteRACT: Transformer Models for Human Intent Prediction Conditioned on Robot Actions
Kushal Kedia, Atiksh Bhardwaj, Prithwish Dan, Sanjiban Choudhury
Transformer Model Human Robot Robot Action Digital Action

November 20, 2023

A novel transformer-based approach for soil temperature prediction
Muhammet Mucahit Enes Yurtsever, Ayhan Kucukmanisa, Zeynep Hilal Kilimci
Transformer Model Transformer Based Model Vanilla Transformer Soil Temperature Novel Transformer Based Approach Soil Respiration

November 19, 2023

Inspecting Explainability of Transformer Models with Additional Statistical Information
Hoang C. Nguyen, Haeil Lee, Junmo Kim
Transformer Based High Explainability Transformer Model Swin Transformer Attention Layer Based Method

November 15, 2023

Converting Transformers to Polynomial Form for Secure Inference Over Homomorphic Encryption
Itamar Zimerman, Moran Baruch, Nir Drucker, Gilad Ezov, Omri Soceanu, Lior Wolf
Transformer Megatron Decepticons Transformer Architecture Transformer Model Homomorphic Encryption Polynomial System Secure Inference

November 7, 2023

Interpreting Shared Circuits for Ordered Sequence Prediction in a Large Language Model
Michael Lan, Fazl Barez
Large Language Model Language Model Transformer Model Interpretable Representation Sequence Prediction Circuit Analysis Interpretability Scale

November 6, 2023

Machine Learning-Based Tea Leaf Disease Detection: A Comprehensive Review
Faruk Ahmed, Md. Taimur Ahad, Yousuf Rayhan Emon
Convolutional Neural Network Image Classification New Machine Transformer Model Comprehensive Review Leaf Disease

November 1, 2023

Pretraining Data Mixtures Enable Narrow Model Selection Capabilities in Transformer Models
Steve Yadlowsky, Lyric Doshi, Nilesh Tripuraneni
Large Language Model Transformer Model Sequence Model Pre Training Model Explicit Feature Map

October 29, 2023

Analyzing Vision Transformers for Image Classification in Class Embedding Space
Martina G. Vilas, Timothy Schaumlöffel, Gemma Roig
Vision Transformer Image Classification Transformer Model Deep Space Internal Representation True Class Class Representation Categorical Representation

October 26, 2023

BERT-PIN: A BERT-based Framework for Recovering Missing Data Segments in Time-series Load Profiles
Yi Hu, Kai Ye, Hyeonjin Kim, Ning Lu
Transformer Model Bidirectional Encoder Representation From Transformer Missing Value Bidirectional Encoder Representation

October 24, 2023

Transformer Model

Papers

FaultFormer: Pretraining Transformers for Adaptable Bearing Fault Classification

The mechanistic basis of data dependence and abrupt learning in an in-context classification task

MABViT -- Modified Attention Block Enhances Vision Transformers

A Comprehensive Study of Vision Transformers in Image Classification Tasks

Mitigating Over-smoothing in Transformers via Regularized Nonlocal Functionals

RACE-IT: A Reconfigurable Analog CAM-Crossbar Engine for In-Memory Transformer Acceleration

InteRACT: Transformer Models for Human Intent Prediction Conditioned on Robot Actions

A novel transformer-based approach for soil temperature prediction

Inspecting Explainability of Transformer Models with Additional Statistical Information

Converting Transformers to Polynomial Form for Secure Inference Over Homomorphic Encryption

Interpreting Shared Circuits for Ordered Sequence Prediction in a Large Language Model

Machine Learning-Based Tea Leaf Disease Detection: A Comprehensive Review

Pretraining Data Mixtures Enable Narrow Model Selection Capabilities in Transformer Models

Analyzing Vision Transformers for Image Classification in Class Embedding Space

BERT-PIN: A BERT-based Framework for Recovering Missing Data Segments in Time-series Load Profiles

Transferring a molecular foundation model for polymer property predictions

Divide et Impera: Multi-Transformer Architectures for Complex NLP-Tasks

Understanding Code Semantics: An Evaluation of Transformer Models in Summarization

Mixture of Tokens: Continuous MoE through Cross-Example Aggregation

Unveiling Multilinguality in Transformer Models: Exploring Language Specificity in Feed-Forward Networks