Transformer Training

Transformer training focuses on optimizing the performance and efficiency of transformer-based models, primarily through advancements in algorithms and hardware. Current research emphasizes improving training stability, addressing the quadratic complexity of self-attention (e.g., through linear attention mechanisms), and developing efficient training strategies for long sequences and large datasets. These efforts are crucial for expanding the applicability of transformers to increasingly complex tasks and resource-constrained environments, impacting fields ranging from natural language processing and computer vision to scientific modeling and forecasting.

Papers

October 19, 2023

Sequence Length Independent Norm-Based Generalization Bounds for Transformers
Jacob Trauger, Ambuj Tewari
Strong Generalization Transformer Megatron Decepticons Transformer Architecture General Bound Transformer Training Norm Based Generalization Bound

October 2, 2023

Linear attention is (maybe) all you need (to understand transformer optimization)
Kwangjun Ahn, Xiang Cheng, Minhak Song, Chulhee Yun, Ali Jadbabaie, Suvrit Sra
Transformer Based Transformer Model Linear Attention Linear Transformer Transformer Training Shallow Transformer

September 29, 2023

Benchmarking and In-depth Performance Study of Large Language Models on Habana Gaudi Processors
Chengming Zhang, Baixi Sun, Xiaodong Yu, Zhen Xie, Weijian Zheng, Kamil Iskra, Pete Beckman, Dingwen Tao
Transformer Model Transformer Based Model Benchmark Platform Performance Analysis Transformer Training Tensor Core

August 20, 2023

TransFace: Calibrating Transformer Training for Face Recognition from a Data-Centric Perspective
Jun Dan, Yang Liu, Haoyu Xie, Jiankang Deng, Haoran Xie, Xuansong Xie, Baigui Sun
Vision Transformer Face Recognition Data Centric Transformer Training

June 21, 2023

Training Transformers with 4-bit Integers
Haocheng Xi, Changhao Li, Jianfei Chen, Jun Zhu
Neural Network Transformer Training Low Bit Bit Training

June 19, 2023

Transformer Training Strategies for Forecasting Multiple Load Time Series
Matthias Hertel, Maximilian Beichter, Benedikt Heidrich, Oliver Neumann, Benjamin Schäfer, Ralf Mikut, Veit Hagenmeyer
Load Forecasting Smart Grid Transformer Training Transfer Learning Strategy

June 16, 2023

Trained Transformers Learn Linear Models In-Context
Ruiqi Zhang, Spencer Frei, Peter L. Bartlett
Gradient Flow Attention Layer Linear Model Transformer Training Attention Based Neural Network Flat Lattice Transformer

June 3, 2023

Lightweight Structure-aware Transformer Network for VHR Remote Sensing Image Change Detection
Tao Lei, Yetong Xu, Hailong Ning, Zhiyong Lv, Chongdan Min, Yaochu Jin, Asoke K. Nandi
Convolutional Neural Network Change Detection Transformer Training Visual Transformer Structure Aware Transformer

May 26, 2023

Generalizing Adam to Manifolds for Efficiently Training Transformers
Benedikt Brantner
Class Manifold Low Dimensional Manifold Adam Optimizer Transformer Training Grassmann Manifold Adam Algorithm

March 11, 2023

Stabilizing Transformer Training by Preventing Attention Entropy Collapse
Shuangfei Zhai, Tatiana Likhomanenko, Etai Littwin, Dan Busbridge, Jason Ramapuram, Yizhe Zhang, Jiatao Gu, Josh Susskind
Vision Transformer Attention Layer Transformer Training Stable Training Attention Entropy

March 4, 2023

Calibrating Transformers via Sparse Gaussian Processes
Wenlong Chen, Yingzhen Li
Transformer Model Transformer Training Sparse Gaussian Process Distribution Robustness

March 2, 2023

Learning to Grow Pretrained Models for Efficient Transformer Training
Peihao Wang, Rameswar Panda, Lucas Torroba Hennigen, Philip Greengard, Leonid Karlinsky, Rogerio Feris, David Daniel Cox, Zhangyang Wang, Yoon Kim
LeArning Abstract Vision Transformer Transformer Training Scaling Transformer Growth Operator

February 2, 2023

A Survey on Efficient Training of Transformers
Bohan Zhuang, Jing Liu, Zizheng Pan, Haoyu He, Yuetian Weng, Chunhua Shen
Timely Survey Transformer Megatron Decepticons Computation Method Efficient Training Transformer Training Training Strategy

January 6, 2023

Does compressing activations help model parallel training?
Song Bian, Dacheng Li, Hongyi Wang, Eric P. Xing, Shivaram Venkataraman
Parallel Training Model Parallelism Transformer Training Large Scale Transformer Model Activation Compression

October 19, 2022

Tempo: Accelerating Transformer-Based Model Training through Memory Footprint Reduction
Muralidhar Andoorveedu, Zhanda Zhu, Bojian Zheng, Gennady Pekhimenko
Deep Learning Model Transformer Based Model Efficient Training Transformer Training Training Throughput Tempo Estimation Memory Reduction

October 18, 2022

ViTCoD: Vision Transformer Acceleration via Dedicated Algorithm and Accelerator Co-Design
Haoran You, Zhanyi Sun, Huihong Shi, Zhongzhi Yu, Yang Zhao, Yongan Zhang, Chaojian Li, Baopu Li, Yingyan Lin
Vision Transformer Practical Algorithm Sparse Attention Attention Computation Transformer Training SParse ATtention Accelerator Design

July 7, 2022

Training Transformers Together
Alexander Borzunov, Max Ryabinin, Tim Dettmers, Quentin Lhoest, Lucile Saulnier, Michael Diskin, Yacine Jernite, Thomas Wolf
Training Data Text to Image Transformer Training Collaborative Training

March 23, 2022

AbductionRules: Training Transformers to Explain Unexpected Inputs
Nathan Young, Qiming Bao, Joshua Bensemann, Michael Witbrock
Scientific Inference Abductive Reasoning Transformer Training Ambiguous Input Contrastive Meta Abduction Method Textual Knowledge Abductive Learning

December 14, 2021

Co-training Transformer with Videos and Images Improves Action Recognition
Bowen Zhang, Jiahui Yu, Christopher Fifty, Wei Han, Andrew M. Dai, Ruoming Pang, Fei Sha
Vision Transformer Action Recognition Gameplay Video Video Representation Video Transformer Transformer Training