Stable Training

Stable training in machine learning focuses on mitigating the instability issues that hinder the reliable and efficient training of various models, impacting performance and resource consumption. Current research investigates techniques to improve stability across diverse architectures, including Transformers, normalizing flows, and generative adversarial networks (GANs), often employing methods like regularization, orthogonal constraints, and novel loss functions or parameter scaling. These efforts are crucial for advancing the reliability and scalability of machine learning models in various applications, from natural language processing and computer vision to reinforcement learning and recommender systems. Improved stability translates to more robust models, faster training times, and reduced resource requirements.

Papers

November 8, 2024

On Training of Kolmogorov-Arnold Networks
Shairoz Sohail
Training Data Multi Layer Kolmogorov Arnold Network Stable Training Training Instability

October 11, 2024

MAD-TD: Model-Augmented Data stabilizes High Update Ratio RL
Claas A Voelcker, Marcel Hussing, Eric Eaton, Amir-massoud Farahmand, Igor Gilitschenski
Reinforcement Learning Deep Reinforcement Learning Policy Reinforcement Learning Temporal Difference Learning DeepMind Control Suite Stable Training Augmented Dataset Mad Icp

September 12, 2024

Stable Language Model Pre-training by Reducing Embedding Variability
Woojin Chung, Jiwoo Hong, Na Min An, James Thorne, Se-Young Yun
Language Model Layer Normalization Non Pathological Variability Stable Training

July 11, 2024

Towards stable training of parallel continual learning
Li Yuepan, Fan Lyu, Yuyang Li, Wei Feng, Guangcan Liu, Fanhua Shang
Continual LEArning Continual Learning Stable Training Hierarchical Propagation

February 26, 2024

Stable Training of Normalizing Flows for High-dimensional Variational Inference
Daniel Andrade
High Dimensional Variational Inference Normalizing Flow Stable Training

November 23, 2023

Periodically Exchange Teacher-Student for Source-Free Object Detection
Qipeng Liu, Luojun Lin, Zhifeng Shen, Zhifeng Yang
Pseudo Label Self Training Stable Training Source Detection Source Free Object Detection

November 1, 2023

Flooding Regularization for Stable Training of Generative Adversarial Networks
Iu Yahiro, Takashi Ishida, Naoto Yokoya
Generative Adversarial Network Adversarial Loss GAN Training Stable Training

October 20, 2023

ScaleLong: Towards More Stable Training of Diffusion Model via Scaling Network Long Skip Connection
Zhongzhan Huang, Pan Zhou, Shuicheng Yan, Liang Lin
Diffusion Model Stable Training Higher Order Loss

October 5, 2023

Stable Training of Probabilistic Models Using the Leave-One-Out Maximum Log-Likelihood Objective
Kutay Bölat, Simon H. Tindemans, Peter Palensky
Probabilistic Model Expectation Maximization Power System Kernel Density Stable Training

September 25, 2023

Small-scale proxies for large-scale Transformer training instabilities
Mitchell Wortsman, Peter J. Liu, Lechao Xiao, Katie Everett, Alex Alemi, Ben Adlam, John D. Co-Reyes, Izzeddin Gur, Abhishek Kumar, Roman Novak, Jeffrey Pennington, Jascha Sohl-dickstein, Kelvin Xu, Jaehoon Lee, Justin Gilmer, Simon Kornblith
Transformer Based Model Proxy Dataset Stable Training Scaling Behavior Training Instability

July 3, 2023

Achieving Stable Training of Reinforcement Learning Agents in Bimodal Environments through Batch Learning
E. Hurwitz, N. Peace, G. Cevora
Reinforcement Learning Reinforcement Learning Agent Stochastic Environment Reinforcement Learning Problem Batch Learning Stable Training Tabular Q Learning Multimodal Environment

May 4, 2023

BranchNorm: Robustly Scaling Extremely Deep Transformers
Yijin Liu, Xianfeng Zeng, Fandong Meng, Jie Zhou
Robust Version Deep Model Deep Transformer Strong Scaling Stable Training

March 11, 2023

Stabilizing Transformer Training by Preventing Attention Entropy Collapse
Shuangfei Zhai, Tatiana Likhomanenko, Etai Littwin, Dan Busbridge, Jason Ramapuram, Yizhe Zhang, Jiatao Gu, Josh Susskind
Vision Transformer Attention Layer Transformer Training Stable Training Attention Entropy

February 17, 2023

Improving Training Stability for Multitask Ranking Models in Recommender Systems
Jiaxi Tang, Yoel Drori, Daryl Chang, Maheswaran Sathiamoorthy, Justin Gilmer, Li Wei, Xinyang Yi, Lichan Hong, Ed H. Chi
Recommender System Recommendation Model Multi Task Model Stable Training Training Instability

February 3, 2023

On a continuous time model of gradient descent dynamics and instability in deep learning
Mihaela Rosca, Yan Wu, Chongli Qin, Benoit Dherin
Deep Learning Gradient Descent Core Stability Gradient Based Optimization Continuous Time Model Stable Training Gradient Descent Dynamic

February 1, 2023

Stable Target Field for Reduced Variance Score Estimation in Diffusion Models
Yilun Xu, Shangyuan Tong, Tommi Jaakkola
Diffusion Model Noise Estimation Stable Training Dynamic Target

October 7, 2022

Understanding Edge-of-Stability Training Dynamics with a Minimalist Example
Xingyu Zhu, Zixuan Wang, Xiang Wang, Mo Zhou, Rong Ge
Gradient Descent Stable Training Stability Threshold Stability Score

April 10, 2022

Reducing Model Jitter: Stable Re-training of Semantic Parsers in Production Environments
Christopher Hidey, Fei Liu, Rahul Goel
Semantic Parser Stable Training Model Jitter Model Agreement Conversational Semantic