Transformer Model

Transformer models are a class of neural networks built upon an attention mechanism, enabling them to process sequential data like text and time series with remarkable effectiveness. Current research focuses on improving training stability (e.g., mitigating loss spikes), enhancing expressiveness through novel attention mechanisms and embedding techniques, and optimizing performance for various applications by exploring different architectures (e.g., hybrid Transformer-Mamba models) and parallelization strategies. This work is significant due to the widespread adoption of transformers in diverse fields, from natural language processing and computer vision to scientific computing and engineering, driving advancements in both theoretical understanding and practical applications.

Papers

July 26, 2024

QT-TDM: Planning with Transformer Dynamics Model and Autoregressive Q-Learning
Mostafa Kotb, Cornelius Weber, Muhammad Burhan Hafez, Stefan Wermter
Reinforcement Learning Task Planning Transformer Architecture Transformer Model Q Function Q Transformer

July 23, 2024

SINDER: Repairing the Singular Defects of DINOv2
Haoqi Wang, Tong Zhang, Mathieu Salzmann
Transformer Model Unsupervised Segmentation Pre Trained Network Token Classification Industrial Defect DINOv2 Model Smoothness Regularization

July 22, 2024

Dissecting Multiplication in Transformers: Insights into LLMs
Luyu Qiu, Jianing Li, Chi Su, Chen Jason Zhang, Lei Chen
Transformer Megatron Decepticons Transformer Based Medical LLM Transformer Model DCU Insight AQ Vanilla Transformer Transformer Performance Multiplication Task

July 17, 2024

July 9, 2024

Enhancing Social Media Personalization: Dynamic User Profile Embeddings and Multimodal Contextual Analysis Using Transformer Models
Pranav Vachharajani
Transformer Model Context Aware Personalization Performance User Engagement User Profiling Multimodal Contextual

July 6, 2024

Integer-only Quantized Transformers for Embedded FPGA-based Time-series Forecasting in AIoT
Tianheng Ling, Chao Qian, Gregor Schiele
Transformer Model Time Series Forecasting Hardware Accelerator Embedded System Device Inference Integer Quantization Quantized Transformer

July 4, 2024

Mixture of A Million Experts
Xu Owen He
Transformer Model Mixture Component Sparse Retrieval Sparse Mixture of Expert Expert Retrieval

June 27, 2024

Time Matters: Scaling Laws for Any Budget
Itay Inbar, Luke Sernau
Training Data Transformer Model Time Matter Large Model Related Hyperparameters Local Authority Budget

June 25, 2024

Structured Unrestricted-Rank Matrices for Parameter Efficient Fine-tuning
Arijit Sehanobish, Avinava Dubey, Krzysztof Choromanski, Somnath Basu Roy Chowdhury, Deepali Jain, Vikas Sindhwani, Snigdha Chaturvedi
Transformer Model Parameter Efficient Fine Tuning Efficient Fine Tuning

June 24, 2024

June 20, 2024

June 13, 2024

Fredformer: Frequency Debiased Transformer for Time Series Forecasting
Xihao Piao, Zheng Chen, Taichi Murayama, Yasuko Matsubara, Yasushi Sakurai
Transformer Based Transformer Model Time Series Forecasting High Frequency Transformer Based Framework Frequency Feature Frequency Bias

June 12, 2024

Transformer Model

Papers

QT-TDM: Planning with Transformer Dynamics Model and Autoregressive Q-Learning

SINDER: Repairing the Singular Defects of DINOv2

Dissecting Multiplication in Transformers: Insights into LLMs

ARTEMIS: A Mixed Analog-Stochastic In-DRAM Accelerator for Transformer Neural Networks

Hybrid Dynamic Pruning: A Pathway to Efficient Transformer Inference

Exploring Quantization for Efficient Pre-Training of Transformer Language Models

Co-Designing Binarized Transformer and Hardware Accelerator for Efficient End-to-End Edge Deployment

Counting in Small Transformers: The Delicate Interplay between Attention and Feed-Forward Layers

Enhancing Social Media Personalization: Dynamic User Profile Embeddings and Multimodal Contextual Analysis Using Transformer Models

Integer-only Quantized Transformers for Embedded FPGA-based Time-series Forecasting in AIoT

Mixture of A Million Experts

Time Matters: Scaling Laws for Any Budget

Structured Unrestricted-Rank Matrices for Parameter Efficient Fine-tuning

Finding Transformer Circuits with Edge Pruning

Feature Fusion for Human Activity Recognition using Parameter-Optimized Multi-Stage Graph Convolutional Network and Transformer Models

Insights into LLM Long-Context Failures: When Transformers Know but Don't Tell

On Layer-wise Representation Similarity: Application for Multi-Exit Models with a Single Classifier

Fredformer: Frequency Debiased Transformer for Time Series Forecasting

Analyzing Multi-Head Attention on Trojan BERT Models

An Empirical Study of Mamba-based Language Models