Simple Transformer

Simple Transformers aim to streamline the architecture of standard transformers, reducing complexity and computational cost without sacrificing performance. Current research focuses on developing efficient variations for diverse applications, including robotics, anomaly detection, and time series forecasting, often incorporating modifications to attention mechanisms and employing single-scale feature processing. These efforts are significant because they improve the scalability and applicability of transformer models across various domains, enabling faster training and deployment in resource-constrained environments. The resulting simplified architectures demonstrate comparable or superior performance to more complex counterparts, paving the way for broader adoption of transformer-based solutions.

Papers

June 11, 2024

BAKU: An Efficient Transformer for Multi-Task Policy Learning
Siddhant Haldar, Zhuoran Peng, Lerrel Pinto
Efficient Transformer Offline Imitation Learning Real World Manipulation Task Task Specific Policy Multi Task Policy Simple Transformer

May 3, 2024

CVTGAD: Simplified Transformer with Cross-View Attention for Unsupervised Graph-level Anomaly Detection
Jindong Li, Qianli Xing, Qi Wang, Yi Chang
Simple Transformer Cross View Attention Inter Graph

November 3, 2023

Simplifying Transformer Blocks
Bobby He, Thomas Hofmann
Deep Transformer Transformer Block Simple Transformer Standard Transformer

October 11, 2023

Linear Latent World Models in Simple Transformers: A Case Study on Othello-GPT
Dean S. Hazineh, Zechen Zhang, Jeffery Chiu
Case Study Model Complexity Decision Theory MIMICRY Change Optimisation Dynamic Simple Transformer Latent World Model

October 9, 2023

SimPLR: A Simple and Plain Transformer for Scaling-Efficient Object Detection and Segmentation
Duy-Kien Nguyen, Martin R. Oswald, Cees G. M. Snoek
Object Detection Segmentation Based Approach Transformer Based Detector Scale Aware Attention Simple Transformer Multi Scale Vision Transformer

December 6, 2022

A K-variate Time Series Is Worth K Words: Evolution of the Vanilla Transformer Architecture for Long-term Multivariate Time Series Forecasting
Zanwei Zhou, Ruizhe Zhong, Chen Yang, Yan Wang, Xiaokang Yang, Wei Shen
Transformer Based Multivariate Time Series Specie Evolution Multivariate Time Series Forecasting Vanilla Transformer Worth Multiple Word Simple Transformer

June 5, 2022

Performance Comparison of Simple Transformer and Res-CNN-BiLSTM for Cyberbullying Classification
Raunak Joshi, Abhishek Gupta
Text Classification Performance Comparison Traditional Deep Learning Cyberbullying Detection CNN BiLSTM Simple Transformer

April 28, 2022

Depth Estimation with Simplified Transformer
John Yang, Le An, Anurag Dixit, Jinkyu Koo, Su Inn Park
Transformer Based Depth Estimation Monocular Depth Estimation Dense Prediction Dense Prediction Task Simple Transformer

December 28, 2021

Synchronized Audio-Visual Frames with Fractional Positional Encoding for Transformers in Video-to-Text Translation
Philipp Harzig, Moritz Einfalt, Rainer Lienhart
Transformer Megatron Decepticons Transformer Architecture Text to Video Positional Encoding Vanilla Transformer Video Description Simple Transformer

November 23, 2021

SimpleTRON: Simple Transformer with O(N) Complexity
Uladzislau Yorsh, Alexander Kovalenko, Vojtěch Vančura, Daniel Vašata, Pavel Kordík, Tomáš Mikolov
Transformer Model Transformer Based Model Complexity Matter Attention Layer Simple Transformer Quadratic Attention

Simple Transformer

Papers

BAKU: An Efficient Transformer for Multi-Task Policy Learning

CVTGAD: Simplified Transformer with Cross-View Attention for Unsupervised Graph-level Anomaly Detection

Simplifying Transformer Blocks

Linear Latent World Models in Simple Transformers: A Case Study on Othello-GPT

SimPLR: A Simple and Plain Transformer for Scaling-Efficient Object Detection and Segmentation

A K-variate Time Series Is Worth K Words: Evolution of the Vanilla Transformer Architecture for Long-term Multivariate Time Series Forecasting

Performance Comparison of Simple Transformer and Res-CNN-BiLSTM for Cyberbullying Classification

Depth Estimation with Simplified Transformer

Synchronized Audio-Visual Frames with Fractional Positional Encoding for Transformers in Video-to-Text Translation

SimpleTRON: Simple Transformer with O(N) Complexity