Transformer Model

Transformer models are a class of neural networks built upon an attention mechanism, enabling them to process sequential data like text and time series with remarkable effectiveness. Current research focuses on improving training stability (e.g., mitigating loss spikes), enhancing expressiveness through novel attention mechanisms and embedding techniques, and optimizing performance for various applications by exploring different architectures (e.g., hybrid Transformer-Mamba models) and parallelization strategies. This work is significant due to the widespread adoption of transformers in diverse fields, from natural language processing and computer vision to scientific computing and engineering, driving advancements in both theoretical understanding and practical applications.

Papers

November 16, 2024

One-Layer Transformer Provably Learns One-Nearest Neighbor In Context
Zihao Li, Yuan Cao, Cheng Gao, Yihan He, Han Liu, Jason M. Klusowski, Jianqing Fan, Mengdi Wang
Context Learning Transformer Model Context Information Layer Transformer Softmax Attention

November 14, 2024

Re-Parameterization of Lightweight Transformer for On-Device Speech Emotion Recognition
Zixing Zhang, Zhongren Dong, Weixiang Xu, Jing Han
Emotion Recognition Transformer Model Lightweight Transformer

November 12, 2024

Unraveling the Gradient Descent Dynamics of Transformers
Bingqing Song, Boran Han, Shuai Zhang, Jie Ding, Mingyi Hong
Transformer Megatron Decepticons Transformer Architecture Transformer Model Layer Transformer Softmax Attention Gradient Descent Dynamic

November 11, 2024

1-800-SHARED-TASKS @ NLU of Devanagari Script Languages: Detection of Language, Hate Speech, and Targets using LLMs
Jebish Purbey, Siddartha Pullakhandam, Kanwal Mehreen, Muhammad Arham, Drishti Sharma, Ashay Srivastava, Ram Mohan Rao Kadiyala
Large Language Model Natural Language Processing Data Detection Medical LLM Transformer Model Hate Speech NLU Model Indic Script Language Detection

November 5, 2024

Enhancing Transformer Training Efficiency with Dynamic Dropout
Hanrui Yan, Dan Shao
Transformer Model Regularization Model Transformer Training Dropout Layer

November 4, 2024

Shrinking the Giant : Quasi-Weightless Transformers for Low Energy Inference
Shashank Nag, Alan T. L. Bacellar, Zachary Susskind, Anshul Jha, Logan Liberty, Aishwarya Sivakumar, Eugene B. John, Krishnan Kailas, Priscila M. V. Lima, Neeraja J. Yadwadkar, Felipe M. G. Franca, Lizy K. John
Scientific Inference Transformer Model Transformer Based Model Low Resource Energy Efficient Inference Lightweight Transformer Mini Giant

October 31, 2024

LSEAttention is All You Need for Time Series Forecasting
Dizhen Liang
Transformer Model Time Series Forecasting Multivariate Time Series Multivariate Time Series Forecasting Transformer Based Architecture

October 30, 2024

ProTransformer: Robustify Transformers via Plug-and-Play Paradigm
Zhichao Hou, Weizhi Gao, Yuchen Shen, Feiyi Wang, Xiaorui Liu
Transformer Model Plug and Play Transformer Based Architecture Robust Transformer Robust Attention

October 26, 2024

Deep Optimizer States: Towards Scalable Training of Transformer Models Using Interleaved Offloading
Avinash Maurya, Jie Ye, M. Mustafa Rafique, Franck Cappello, Bogdan Nicolae
Large Language Model Transformer Model Task Offloading GPU Memory Scalable Training Hybrid CPU GPU

October 25, 2024

Multi-Agent Reinforcement Learning with Selective State-Space Models
Jemma Daniel, Ruan de Kock, Louay Ben Nessir, Sasha Abramowitz, Omayma Mahjoub, Wiem Khlifi, Claude Formanek, Arnu Pretorius
Multi Agent Reinforcement Learning Transformer Model Mamba Based Multi Agent Environment Selective State Space Transformer Agent

October 24, 2024

October 23, 2024

October 22, 2024

Audio-to-Score Conversion Model Based on Whisper methodology
Hongyao Zhang, Bohang Sun
Transformer Model Audio Processing Automated Conversion Chord Formation

October 20, 2024

October 18, 2024

SignAttention: On the Interpretability of Transformer Models for Sign Language Translation
Pedro Alejandro Dal Bianco, Oscar Agustín Stanchi, Facundo Manuel Quiroga, Franco Ronchetti, Enzo Ferrante
Inherent Interpretability Transformer Model Cross Attention Sign Language Translation Human Translation Interpretability Research Cultural Sign

Transformer Model

Papers

One-Layer Transformer Provably Learns One-Nearest Neighbor In Context

Re-Parameterization of Lightweight Transformer for On-Device Speech Emotion Recognition

Unraveling the Gradient Descent Dynamics of Transformers

1-800-SHARED-TASKS @ NLU of Devanagari Script Languages: Detection of Language, Hate Speech, and Targets using LLMs

Enhancing Transformer Training Efficiency with Dynamic Dropout

Shrinking the Giant : Quasi-Weightless Transformers for Low Energy Inference

LSEAttention is All You Need for Time Series Forecasting

ProTransformer: Robustify Transformers via Plug-and-Play Paradigm

Deep Optimizer States: Towards Scalable Training of Transformer Models Using Interleaved Offloading

Multi-Agent Reinforcement Learning with Selective State-Space Models

On Explaining with Attention Matrices

A Comprehensive Survey of Time Series Forecasting: Architectural Diversity and Open Challenges

The Nature of Mathematical Modeling and Probabilistic Optimization Engineering in Generative AI

Beyond position: how rotary embeddings shape representations and memory in autoregressive transfomers

Locating Information in Large Language Models via Random Matrix Theory

PETAH: Parameter Efficient Task Adaptation for Hybrid Transformers in a resource-limited Context

Audio-to-Score Conversion Model Based on Whisper methodology

SEA: State-Exchange Attention for High-Fidelity Physics-Based Transformers

Comparative Analysis of LSTM, GRU, and Transformer Models for Stock Price Prediction

SignAttention: On the Interpretability of Transformer Models for Sign Language Translation