Transformer Megatron Decepticons

Transformer models are being extensively investigated for various sequence processing tasks, moving beyond natural language processing to encompass time series forecasting, image recognition, and scientific computing applications like solving partial differential equations. Current research focuses on improving efficiency (e.g., through mixed-precision quantization and optimized architectures), enhancing generalization capabilities (particularly to longer sequences), and understanding the underlying mechanisms of in-context learning. These advancements have significant implications for diverse fields, improving the accuracy and efficiency of numerous applications while simultaneously deepening our theoretical understanding of these powerful models.

Papers

June 1, 2023

May 31, 2023

May 30, 2023

May 29, 2023

May 28, 2023

Adaptive Sparsity Level during Training for Efficient Time Series Forecasting with Transformers
Zahra Atashgahi, Mykola Pechenizkiy, Raymond Veldhuis, Decebal Constantin Mocanu
Training Data Transformer Megatron Decepticons Sparse Neural Network Sparse Training Efficient Time Series Forecasting Forecasting Analysis Adaptive Sparsity Sparsity Tradeoff

May 27, 2023

May 26, 2023

May 25, 2023

Landmark Attention: Random-Access Infinite Context Length for Transformers
Amirkeivan Mohtashami, Martin Jaggi
Transformer Megatron Decepticons Retrieval Augmentation Long Term Memory Context Length Context Retrieval Landmark Attention

May 24, 2023

Editing Common Sense in Transformers
Anshita Gupta, Debanjan Mondal, Akshay Krishna Sheshadri, Wenlong Zhao, Xiang Lorraine Li, Sarah Wiegreffe, Niket Tandon
Transformer Megatron Decepticons Commonsense Reasoning Commonsense Knowledge Model Editing Non Rigid Editing Commonsense Question

Transformer Megatron Decepticons

Papers

Transformers learn to implement preconditioned gradient descent for in-context learning

Training-free Neural Architecture Search for RNNs and Transformers

Bytes Are All You Need: Transformers Operating Directly On File Bytes

A Universal Latent Fingerprint Enhancer Using Transformers

Toward Understanding Why Adam Converges Faster Than SGD for Transformers

Humans in 4D: Reconstructing and Tracking Humans with Transformers

LAIT: Efficient Multi-Segment Encoding in Transformers with Layer-Adjustable Interaction

The Impact of Positional Encoding on Length Generalization in Transformers

Are Large Kernels Better Teachers than Transformers for ConvNets?

Approximation and Estimation Ability of Transformers for Sequence-to-Sequence Functions with Infinite Dimensional Input

Faith and Fate: Limits of Transformers on Compositionality

Representation Learning on Hyper-Relational and Numeric Knowledge Graphs with Transformers

Adaptive Sparsity Level during Training for Efficient Time Series Forecasting with Transformers

Graph Inductive Biases in Transformers without Message Passing

Diagnosing Transformers: Illuminating Feature Spaces for Clinical Decision-Making

How Powerful are Decoder-Only Transformer Neural Models?

Randomized Positional Encodings Boost Length Generalization of Transformers

Improving Position Encoding of Transformers for Multivariate Time Series Classification

Landmark Attention: Random-Access Infinite Context Length for Transformers

Editing Common Sense in Transformers