Training Instability

Training instability in large-scale machine learning models, particularly deep neural networks and transformers, is a significant challenge hindering reliable model development and deployment. Current research focuses on identifying and mitigating sources of instability, such as numerical precision limitations in algorithms like Adam and Flash Attention, the interplay between optimizers and normalization layers (e.g., Batch Normalization), and the impact of data heterogeneity in federated learning. Addressing these issues is crucial for improving the robustness and efficiency of training, leading to more reliable and accurate models across various applications.

Papers

November 8, 2024

On Training of Kolmogorov-Arnold Networks
Shairoz Sohail
Training Data Multi Layer Kolmogorov Arnold Network Stable Training Training Instability

June 9, 2024

OD-DETR: Online Distillation for Stabilizing Training of Detection Transformer
Shengjian Wu, Li Sun, Qingli Li
Training Data Detection Transformer CNN Based Detector Online Distillation Training Instability

May 5, 2024

Is Flash Attention Stable?
Alicia Golden, Samuel Hsia, Fei Sun, Bilge Acun, Basil Hosmer, Yejin Lee, Zachary DeVito, Jeff Johnson, Gu-Yeon Wei, David Brooks, Carole-Jean Wu
Subject Level Deviation Training Instability

September 25, 2023

Small-scale proxies for large-scale Transformer training instabilities
Mitchell Wortsman, Peter J. Liu, Lechao Xiao, Katie Everett, Alex Alemi, Ben Adlam, John D. Co-Reyes, Izzeddin Gur, Abhishek Kumar, Roman Novak, Jeffrey Pennington, Jascha Sohl-dickstein, Kelvin Xu, Jaehoon Lee, Justin Gilmer, Simon Kornblith
Transformer Based Model Proxy Dataset Scaling Behavior Stable Training Training Instability

July 30, 2023

Stable Adam Optimization for 16-bit Neural Networks Training
Juyoung Yun
Neural Network Adam Optimizer New Hyperparameter Training Instability Numerical Stability

April 23, 2023

The Disharmony between BN and ReLU Causes Gradient Explosion, but is Offset by the Correlation between Activations
Inyoung Paik, Jaesik Choi
Learning Rate Total Correlation Targeted Activation Penalty ReLU Layer ReLU Activation Training Instability

March 16, 2023

Enhancing Text Generation with Cooperative Training
Tong Wu, Hao Wang, Zhongshen Zeng, Wei Wang, Hai-Tao Zheng, Jiaxing Zhang
Language Model Generative Model Pre Trained Language Model Text Generation Discriminative Model Collaborative Training Training Instability

February 28, 2023

$(\alpha_D,\alpha_G)$-GANs: Addressing GAN Training Instabilities via Dual Objectives
Monica Welfert, Kyle Otstot, Gowtham R. Kurri, Lalitha Sankar
GAN Model GAN Training Training Instability

February 25, 2023

Better Generative Replay for Continual Federated Learning
Daiqing Qi, Handong Zhao, Sheng Li
Continual LEArning Model Merging Generative Replay Federated Continual Learning Training Instability

February 24, 2023

On the Training Instability of Shuffling SGD with Batch Normalization
David X. Wu, Chulhee Yun, Suvrit Sra
Gradient Descent Batch Normalization Inverse Divergence Training Loss Training Instability Shuffling Gradient

February 17, 2023

Improving Training Stability for Multitask Ranking Models in Recommender Systems
Jiaxi Tang, Yoel Drori, Daryl Chang, Maheswaran Sathiamoorthy, Justin Gilmer, Li Wei, Xinyang Yi, Lichan Hong, Ed H. Chi
Recommender System Recommendation Model Multi Task Model Stable Training Training Instability

February 16, 2023

Distributed Learning in Heterogeneous Environment: federated learning with adaptive aggregation and computation reduction
Jingxin Li, Toktam Mahmoodi, Hak-Keung Lam
LeArning Abstract Heterogeneous Data Global Mixing Heterogeneous Environment Training Instability Complexity Reduction Adaptive Aggregation Asynchronous Update

December 20, 2022

OBMO: One Bounding Box Multiple Objects for Monocular 3D Object Detection
Chenxi Huang, Tong He, Haidong Ren, Wenxiao Wang, Binbin Lin, Deng Cai
Bounding Box Monocular 3D Object Detection Depth Network Object Box Monocular 3D Detector Training Instability Depth Ambiguity

July 3, 2022

Stabilizing Off-Policy Deep Reinforcement Learning from Pixels
Edoardo Cetin, Philip J. Ball, Steve Roberts, Oya Celiktutan
Policy Reinforcement Learning Tetromino Pixel Convolutional Encoder DeepMind Control Suite Catastrophic Overfitting Policy Deep Reinforcement Learning Training Instability

June 10, 2022

The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenon
Vimal Thilak, Etai Littwin, Shuangfei Zhai, Omid Saremi, Roni Paiss, Joshua Susskind
Strong Generalization Empirical Study Grokking Phenomenon Adaptive Optimizers Adaptive Gradient Training Instability

May 30, 2022

CalFAT: Calibrated Federated Adversarial Training with Label Skewness
Chen Chen, Yuchen Liu, Xingjun Ma, Lingjuan Lyu
Adversarial Attack Adversarial Training Adversarial Robustness Label Distribution Training Instability Federated Adversarial