Transformer Model

Transformer models are a class of neural networks built upon an attention mechanism, enabling them to process sequential data like text and time series with remarkable effectiveness. Current research focuses on improving training stability (e.g., mitigating loss spikes), enhancing expressiveness through novel attention mechanisms and embedding techniques, and optimizing performance for various applications by exploring different architectures (e.g., hybrid Transformer-Mamba models) and parallelization strategies. This work is significant due to the widespread adoption of transformers in diverse fields, from natural language processing and computer vision to scientific computing and engineering, driving advancements in both theoretical understanding and practical applications.

Papers

July 17, 2024

Hybrid Dynamic Pruning: A Pathway to Efficient Transformer Inference
Ghadeer Jaradat, Mohammed Tolba, Ghada Alsuhli, Hani Saleh, Mahmoud Al-Qutayri, Thanos Stouraitis, Baker Mohammad
Deep Learning Transformer Model Attention Matrix Biological Pathway Transformer Inference Attention Computation

July 9, 2024

Enhancing Social Media Personalization: Dynamic User Profile Embeddings and Multimodal Contextual Analysis Using Transformer Models
Pranav Vachharajani
Transformer Model Context Aware Personalization Performance User Engagement User Profiling Multimodal Contextual

July 6, 2024

Integer-only Quantized Transformers for Embedded FPGA-based Time-series Forecasting in AIoT
Tianheng Ling, Chao Qian, Gregor Schiele
Transformer Model Time Series Forecasting Hardware Accelerator Embedded System Device Inference Integer Quantization Quantized Transformer

July 4, 2024

Mixture of A Million Experts
Xu Owen He
Transformer Model Mixture Component Sparse Retrieval Sparse Mixture of Expert Expert Retrieval

June 27, 2024

Time Matters: Scaling Laws for Any Budget
Itay Inbar, Luke Sernau
Training Data Transformer Model Time Matter Large Model Related Hyperparameters Local Authority Budget

June 25, 2024

Structured Unrestricted-Rank Matrices for Parameter Efficient Fine-tuning
Arijit Sehanobish, Avinava Dubey, Krzysztof Choromanski, Somnath Basu Roy Chowdhury, Deepali Jain, Vikas Sindhwani, Snigdha Chaturvedi
Transformer Model Parameter Efficient Fine Tuning Efficient Fine Tuning

June 24, 2024

June 20, 2024

June 13, 2024

Fredformer: Frequency Debiased Transformer for Time Series Forecasting
Xihao Piao, Zheng Chen, Taichi Murayama, Yasuko Matsubara, Yasushi Sakurai
Transformer Based Transformer Model Time Series Forecasting High Frequency Transformer Based Framework Frequency Feature Frequency Bias

June 12, 2024

June 11, 2024

Transformer Model

Papers

Hybrid Dynamic Pruning: A Pathway to Efficient Transformer Inference

Exploring Quantization for Efficient Pre-Training of Transformer Language Models

Co-Designing Binarized Transformer and Hardware Accelerator for Efficient End-to-End Edge Deployment

Counting in Small Transformers: The Delicate Interplay between Attention and Feed-Forward Layers

Enhancing Social Media Personalization: Dynamic User Profile Embeddings and Multimodal Contextual Analysis Using Transformer Models

Integer-only Quantized Transformers for Embedded FPGA-based Time-series Forecasting in AIoT

Mixture of A Million Experts

Time Matters: Scaling Laws for Any Budget

Structured Unrestricted-Rank Matrices for Parameter Efficient Fine-tuning

Finding Transformer Circuits with Edge Pruning

Feature Fusion for Human Activity Recognition using Parameter-Optimized Multi-Stage Graph Convolutional Network and Transformer Models

Insights into LLM Long-Context Failures: When Transformers Know but Don't Tell

On Layer-wise Representation Similarity: Application for Multi-Exit Models with a Single Classifier

Fredformer: Frequency Debiased Transformer for Time Series Forecasting

Analyzing Multi-Head Attention on Trojan BERT Models

An Empirical Study of Mamba-based Language Models

Transformer Models in Education: Summarizing Science Textbooks with AraBART, MT5, AraT5, and mBART

ReduceFormer: Attention with Tensor Reduction by Summation

Towards Generalized Hydrological Forecasting using Transformer Models for 120-Hour Streamflow Prediction

Dynamical Mean-Field Theory of Self-Attention Neural Networks