Transformer Architecture

Transformer architectures are a dominant deep learning paradigm, primarily known for their self-attention mechanism enabling efficient processing of sequential data like text and time series. Current research focuses on addressing the quadratic time complexity of self-attention through alternative architectures (e.g., state space models like Mamba) and optimized algorithms (e.g., local attention, quantized attention), as well as exploring the application of transformers to diverse domains including computer vision, robotics, and blockchain technology. These efforts aim to improve the efficiency, scalability, and interpretability of transformers, leading to broader applicability and enhanced performance across numerous fields.

Papers

December 4, 2023

December 1, 2023

Mamba: Linear-Time Sequence Modeling with Selective State Spaces
Albert Gu, Tri Dao
Transformer Architecture Mamba in Mamba Recurrent Network Sequence Model Selective State Space

November 28, 2023

On the Long Range Abilities of Transformers
Itamar Zimerman, Lior Wolf
Transformer Megatron Decepticons Transformer Architecture Long Range Global Convolution RNN Layer

November 22, 2023

Input Compression with Positional Consistency for Efficient Training and Inference of Transformer Neural Networks
Amrit Nagarajan, Anand Raghunathan
Scientific Inference Transformer Architecture Efficient Training Temporal Resolution Positional Information Input Compression

November 21, 2023

November 15, 2023

Converting Transformers to Polynomial Form for Secure Inference Over Homomorphic Encryption
Itamar Zimerman, Moran Baruch, Nir Drucker, Gilad Ezov, Omri Soceanu, Lior Wolf
Transformer Megatron Decepticons Transformer Architecture Transformer Model Homomorphic Encryption Polynomial System Secure Inference

November 11, 2023

VT-Former: An Exploratory Study on Vehicle Trajectory Prediction for Highway Surveillance through Graph Isomorphism and Transformer
Armin Danesh Pazho, Ghazal Alinezhad Noghre, Vinit Katariya, Hamed Tabkhi
Autonomous Driving Transformer Based Transformer Architecture Vehicle Trajectory Exploratory Study Vehicle Trajectory Prediction Graph Isomorphism Traffic Camera Transformer Based Trajectory Prediction

November 10, 2023

The Shape of Learning: Anisotropy and Intrinsic Dimensions in Transformer-Based Models
Anton Razzhigaev, Matvey Mikhalchuk, Elizaveta Goncharova, Ivan Oseledets, Denis Dimitrov, Andrey Kuznetsov
LeArning Abstract Representation Learning Jina Embeddings Transformer Architecture Transformer Based Model Object Shape Encoder Side Transformer Decoder Intrinsic Dimension

November 9, 2023

Legal-HNet: Mixing Legal Long-Context Tokens with Hartley Transform
Daniele Giofré, Sneha Ghantasala
Language Model Transformer Architecture Abstractive Text Summarization Attention Based Encoder Context Token

November 7, 2023

Character-Level Bangla Text-to-IPA Transcription Using Transformer Architecture with Sequence Alignment
Jakir Hasan, Shrestha Datta, Ameya Debnath
Transformer Architecture International Phonetic Alphabet Text to Speech System Sequence Alignment

November 1, 2023

Rethinking Decision Transformer via Hierarchical Reinforcement Learning
Yi Ma, Chenjun Xiao, Hebin Liang, Jianye Hao
Reinforcement Learning Transformer Architecture Hierarchical Reinforcement Learning Sequential Decision Decision Transformer

October 31, 2023

Global Transformer Architecture for Indoor Room Temperature Forecasting
Alfredo V Clemente, Alessandro Nocente, Massimiliano Ruocco
Transformer Architecture Local Transformer Multi Room Temperature Prediction Temperature Forecasting