Transformer Prediction

Transformer prediction models are being extensively studied to understand their internal mechanisms and improve their performance and efficiency across diverse applications. Current research focuses on analyzing the relationship between transformer predictions and simpler statistical rules, developing specialized architectures for specific tasks like vibration prediction, and addressing challenges like oversmoothing and computational cost through techniques such as sparse attention and memory-efficient explanation methods. These advancements are crucial for enhancing the reliability and interpretability of transformer models, enabling their wider adoption in various fields including natural language processing, time-series analysis, and beyond.

Papers

June 30, 2024

Understanding Transformers via N-gram Statistics
Timothy Nguyen
Large Language Model Transformer Megatron Decepticons N Gram Transformer Prediction

February 16, 2024

Can Transformers Predict Vibrations?
Fusataka Kuniyoshi, Yoshihide Sawada
Transformer Megatron Decepticons Thin Shell Low Cost Ambient Vibration Transformer Prediction

January 9, 2024

Setting the Record Straight on Transformer Oversmoothing
Gbètondji J-S Dovonon, Michael M. Bronstein, Matt J. Kusner
Strong Generalization Deeper Model Squashing Mitigation Approach Fact Extraction Music Recording Transformer Based Structure Transformer Prediction

August 29, 2023

Learning the greatest common divisor: explaining transformer predictions
François Charton
Training Distribution Transformer Prediction

January 19, 2023

AtMan: Understanding Transformer Predictions Through Memory Efficient Attention Manipulation
Björn Deiseroth, Mayukh Deb, Samuel Weinbach, Manuel Brack, Patrick Schramowski, Kristian Kersting
Transformer Megatron Decepticons Transformer Based Generative Relevance Map Memory Efficient Attention Image Text Benchmark Transformer Prediction

December 27, 2021

Transformer Uncertainty Estimation with Hierarchical Stochastic Attention
Jiahuan Pei, Cheng Wang, György Szarvas
Transformer Model Uncertainty Estimation Transformer Based Structure Transformer Prediction

November 24, 2021

Sparse is Enough in Scaling Transformers
Sebastian Jaszczur, Aakanksha Chowdhery, Afroz Mohiuddin, Łukasz Kaiser, Wojciech Gajewski, Henryk Michalewski, Jonni Kanerva
Many Sparse Large Transformer Model Scaling Transformer Transformer Prediction