Transformer Architecture

Transformer architectures are a dominant deep learning paradigm, primarily known for their self-attention mechanism enabling efficient processing of sequential data like text and time series. Current research focuses on addressing the quadratic time complexity of self-attention through alternative architectures (e.g., state space models like Mamba) and optimized algorithms (e.g., local attention, quantized attention), as well as exploring the application of transformers to diverse domains including computer vision, robotics, and blockchain technology. These efforts aim to improve the efficiency, scalability, and interpretability of transformers, leading to broader applicability and enhanced performance across numerous fields.

Papers

October 31, 2023

A Systematic Review for Transformer-based Long-term Series Forecasting
Liyilei Su, Xumin Zuo, Rui Li, Xin Wang, Heng Zhao, Bingding Huang
Transformer Based Transformer Architecture Time Series Forecasting Systematic Review Long Term Time Series Forecasting

October 30, 2023

Partial Tensorized Transformers for Natural Language Processing
Subhadra Vadlamannati, Ryan Solgi
Natural Language Processing Transformer Architecture Tensor Decomposition Tensor Based Tensor Train Decomposition Tensorial Template

October 25, 2023

October 24, 2023

October 22, 2023

ConViViT -- A Deep Neural Network Combining Convolutions and Factorized Self-Attention for Human Activity Recognition
Rachid Reda Dokkar, Faten Chaieb, Hassen Drira, Arezki Aberkane
Self Attention Transformer Architecture Activity Recognition Video Representation Spatial Token Pure Transformer Combining Deep Learning

October 19, 2023

Sequence Length Independent Norm-Based Generalization Bounds for Transformers
Jacob Trauger, Ambuj Tewari
Strong Generalization Transformer Megatron Decepticons Transformer Architecture General Bound Transformer Training Norm Based Generalization Bound

October 17, 2023

October 16, 2023

How Do Transformers Learn In-Context Beyond Simple Functions? A Case Study on Learning with Representations
Tianyu Guo, Wei Hu, Song Mei, Huan Wang, Caiming Xiong, Silvio Savarese, Yu Bai
Large Language Model LeArning Abstract Transformer Megatron Decepticons Context Learning Transformer Architecture Meaningful Representation Simple Function Different Context Representation Function

October 15, 2023

When can transformers reason with abstract symbols?
Enric Boix-Adsera, Omid Saremi, Emmanuel Abbe, Samy Bengio, Etai Littwin, Joshua Susskind
Transformer Megatron Decepticons Transformer Architecture Transformer Model Relational Reasoning Symbolic Concept

October 10, 2023

iTransformer: Inverted Transformers Are Effective for Time Series Forecasting
Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, Mingsheng Long
Transformer Megatron Decepticons Transformer Based Transformer Architecture Time Series Forecasting Covariate Representation Linear Forecasting

October 9, 2023

October 7, 2023

Uncovering hidden geometry in Transformers via disentangling position and context
Jiajun Song, Yiqiao Zhong
Transformer Megatron Decepticons Jina Embeddings Transformer Architecture Context Information Model Interpretability Product Specific Position Information Interpretable Feature Information Decomposition

October 6, 2023

Functional Interpolation for Relative Positions Improves Long Context Transformers
Shanda Li, Chong You, Guru Guruganesh, Joshua Ainslie, Santiago Ontanon, Manzil Zaheer, Sumit Sanghai, Yiming Yang, Sanjiv Kumar, Srinadh Bhojanapalli
Transformer Architecture Gradient Approximation Relative Positional Encoding Context Transformer Text Benchmark General Transformer Position Encoding Relative Position

Transformer Architecture

Papers

A Systematic Review for Transformer-based Long-term Series Forecasting

Partial Tensorized Transformers for Natural Language Processing

MCUFormer: Deploying Vision Transformers on Microcontrollers with Limited Memory

Divide et Impera: Multi-Transformer Architectures for Complex NLP-Tasks

AGaLiTe: Approximate Gated Linear Transformers for Online Reinforcement Learning

TRAMS: Training-free Memory Selection for Long-range Language Modeling

ConViViT -- A Deep Neural Network Combining Convolutions and Factorized Self-Attention for Human Activity Recognition

Sequence Length Independent Norm-Based Generalization Bounds for Transformers

Transformers for scientific data: a pedagogical review for astronomers

Deep learning based on Transformer architecture for power system short-term voltage stability assessment with class imbalance

Superiority of Softmax: Unveiling the Performance Edge Over Linear Attention

The Efficacy of Transformer-based Adversarial Attacks in Security Domains

BitNet: Scaling 1-bit Transformers for Large Language Models

How Do Transformers Learn In-Context Beyond Simple Functions? A Case Study on Learning with Representations

When can transformers reason with abstract symbols?

iTransformer: Inverted Transformers Are Effective for Time Series Forecasting

Transformers and Large Language Models for Chemistry and Drug Discovery

A Meta-Learning Perspective on Transformers for Causal Language Modeling

Uncovering hidden geometry in Transformers via disentangling position and context

Functional Interpolation for Relative Positions Improves Long Context Transformers