Transformer Architecture

Transformer architectures are a dominant deep learning paradigm, primarily known for their self-attention mechanism enabling efficient processing of sequential data like text and time series. Current research focuses on addressing the quadratic time complexity of self-attention through alternative architectures (e.g., state space models like Mamba) and optimized algorithms (e.g., local attention, quantized attention), as well as exploring the application of transformers to diverse domains including computer vision, robotics, and blockchain technology. These efforts aim to improve the efficiency, scalability, and interpretability of transformers, leading to broader applicability and enhanced performance across numerous fields.

Papers

April 21, 2023

Transformer-based models and hardware acceleration analysis in autonomous driving: A survey
Juan Zhong, Zheng Liu, Xi Chen
Convolutional Neural Network Timely Survey Autonomous Driving Full Model Transformer Architecture Transformer Based Model Autonomous Driving Application Hardware Acceleration

April 20, 2023

April 17, 2023

Attention Mixtures for Time-Aware Sequential Recommendation
Viet-Anh Tran, Guillaume Salha-Galvan, Bruno Sguerra, Romain Hennequin
Transformer Architecture Sequential Recommendation Temporal Attention Sequential Recommender System Temporal Context Mixture of Attention

April 11, 2023

March 29, 2023

Adaptive Spot-Guided Transformer for Consistent Local Feature Matching
Jiahuan Yu, Jiahao Chang, Jianfeng He, Tianzhu Zhang, Feng Wu
Transformer Architecture Coarse to Fine Local Feature Matching Boundary Transformer

March 27, 2023

TransCODE: Co-design of Transformers and Accelerators for Efficient Training and Inference
Shikhar Tuli, Niraj K. Jha
Transformer Megatron Decepticons Scientific Inference Transformer Architecture Transformer Model Efficient Training Transformer Inference Accelerator Tuning Transformer Accelerator Video Transcoding

March 24, 2023

EdgeTran: Co-designing Transformers for Efficient Inference on Mobile Edge Platforms
Shikhar Tuli, Niraj K. Jha
Transformer Architecture Transformer Model Efficient Inference Efficient Transformer Large Transformer Model Transformer Performance Edge Platform

March 21, 2023

Machine Learning for Brain Disorders: Transformers and Visual Transformers
Robin Courant, Maika Edberg, Nicolas Dufour, Vicky Kalogeiton
Machine Learning Vision Transformer Transformer Megatron Decepticons Transformer Architecture Transformer Encoder Brain Disorder

March 16, 2023

A transformer-based approach to video frame-level prediction in Affective Behaviour Analysis In-the-wild
Dang-Khanh Nguyen, Ngoc-Huynh Ho, Sudarshan Pant, Hyung-Jeong Yang
Transformer Architecture Transformer Based Model Source Video Affective Computing Transformer Based Approach Behavior Analysis in the Wild

March 11, 2023

Recursive Generalization Transformer for Image Super-Resolution
Zheng Chen, Yulun Zhang, Jinjin Gu, Linghe Kong, Xiaokang Yang
Self Attention Transformer Architecture Image Super Resolution Local Self Attention General Transformer

February 28, 2023

GNOT: A General Neural Operator Transformer for Operator Learning
Zhongkai Hao, Zhengyi Wang, Hang Su, Chengyang Ying, Yinpeng Dong, Songming Liu, Ze Cheng, Jian Song, Jun Zhu
Transformer Architecture Attention Layer Operator Learning Effective Transformer General Neural Operator Transformer

February 27, 2023

February 13, 2023

February 8, 2023

Mitigating Bias in Visual Transformers via Targeted Alignment
Sruthi Sudhakar, Viraj Prabhu, Arvindkumar Krishnakumar, Judy Hoffman
Vision Transformer Computer Vision Transformer Architecture Mitigating Bias Fairness Implication

January 23, 2023

AttMEMO : Accelerating Transformers with Memoization on Big Memory Systems
Yuan Feng, Hyeran Jeon, Filip Blagojevic, Cyril Guyot, Qing Li, Dong Li
Training Data Transformer Megatron Decepticons Transformer Architecture Transformer Inference Attention Computation Data Memorization Memory System

Transformer Architecture

Papers

Transformer-based models and hardware acceleration analysis in autonomous driving: A survey

An Introduction to Transformers

Dynamic Graph Representation Learning via Edge Temporal States Modeling and Structure-reinforced Transformer

Attention Mixtures for Time-Aware Sequential Recommendation

Multi-granulariy Time-based Transformer for Knowledge Tracing

Data-Efficient Image Quality Assessment with Attention-Panel Decoder

Adaptive Spot-Guided Transformer for Consistent Local Feature Matching

TransCODE: Co-design of Transformers and Accelerators for Efficient Training and Inference

EdgeTran: Co-designing Transformers for Efficient Inference on Mobile Edge Platforms

Machine Learning for Brain Disorders: Transformers and Visual Transformers

A transformer-based approach to video frame-level prediction in Affective Behaviour Analysis In-the-wild

Recursive Generalization Transformer for Image Super-Resolution

GNOT: A General Neural Operator Transformer for Operator Learning

Full Stack Optimization of Transformer Inference: a Survey

A low latency attention module for streaming self-supervised speech representation learning

A Study on ReLU and Softmax in Transformer

Encoding Sentence Position in Context-Aware Neural Machine Translation with Concatenation

One Transformer for All Time Series: Representing and Training with Time-Dependent Heterogeneous Tabular Data

Mitigating Bias in Visual Transformers via Targeted Alignment

AttMEMO : Accelerating Transformers with Memoization on Big Memory Systems