Early Stage Convergence

Early stage convergence in machine learning focuses on understanding and improving the initial phases of training algorithms, aiming to accelerate convergence speed and enhance generalization performance. Current research investigates this through the lens of various optimization algorithms (e.g., Adam, SGD, FedProx), model architectures (e.g., transformers, diffusion models), and specific problem domains (e.g., federated learning, collaborative filtering). These studies leverage techniques from dynamical systems theory and optimal transport to establish convergence guarantees and bounds, ultimately contributing to more efficient and robust machine learning systems across diverse applications.

Papers

May 16, 2022

On the Convergence of the Shapley Value in Parametric Bayesian Learning Games
Lucas Agussurja, Xinyi Xu, Bryan Kian Hsiang Low
Early Stage Convergence Bayesian Inference Shapley Value Fisher Information Cooperative Game Theory Bayesian Game

May 13, 2022

Convergence of Deep Neural Networks with General Activation Functions and Pooling
Wentao Huang, Yuesheng Xu, Haizhang Zhang
Deep Learning Deep Neural Network Deep Convolutional Neural Network Early Stage Convergence Activation Function Rectified Linear Unit Leaky ReLU

May 12, 2022

$\alpha$-GAN: Convergence and Estimation Guarantees
Gowtham R. Kurri, Monica Welfert, Tyler Sypherd, Lalitha Sankar
Early Stage Convergence GAN Loss GAN Framework

May 11, 2022

May 3, 2022

April 28, 2022

On the Convergence of Momentum-Based Algorithms for Federated Bilevel Optimization Problems
Hongchang Gao
Machine Learning Early Stage Convergence Bilevel Optimization Convergence Rate Momentum Based Federated Bilevel Optimization

April 26, 2022

Convergence of neural networks to Gaussian mixture distribution
Yasuhiko Asao, Ryotaro Sakamoto, Shiro Takagi
Neural Network Early Stage Convergence Gaussian Mixture Fully Connected Layer Output Deep Random Neural Network

April 22, 2022

Convergence of the Riemannian Langevin Algorithm
Khashayar Gatmiry, Santosh S. Vempala
Early Stage Convergence Langevin Dynamic Hessian Matrix Smooth Density

April 14, 2022

Convergence and Implicit Regularization Properties of Gradient Descent for Deep Residual Networks
Rama Cont, Alain Rossier, RenYuan Xu
Loss Function Gradient Descent Early Stage Convergence Residual Network Linear Convergence Regularization Property

April 13, 2022

Edge-enabled Metaverse: The Convergence of Metaverse and Mobile Edge Computing
Sahraoui Dhelim, Tahar Kechadi, Liming Chen, Nyothiri Aung, Huansheng Ning, Luigi Atzori
Early Stage Convergence Ubiquitous Semantic Metaverse Mobile Edge Computing Metaverse Application Edge Computing Paradigm

March 30, 2022

Convergence of gradient descent for deep neural networks
Sourav Chatterjee
Neural Network Deep Neural Network Gradient Descent Early Stage Convergence Transformer Feed Forward Layer Random Initialization

March 29, 2022

Convergence of First-Order Methods for Constrained Nonconvex Optimization with Dependent Data
Ahmet Alacaoglu, Hanbaek Lyu
Early Stage Convergence Stochastic Gradient Gradient Method Nonconvex Optimization Dependent Data First Order Method Smooth Nonconvex Stochastic Proximal Gradient

March 16, 2022

On the Convergence of Certified Robust Training with Interval Bound Propagation
Yihan Wang, Zhouxing Shi, Quanquan Gu, Cho-Jui Hsieh
Gradient Descent Adversarial Perturbation Early Stage Convergence Robust Training Interval Bound Propagation

February 24, 2022

On the influence of stochastic roundoff errors and their bias on the convergence of the gradient descent method with low-precision floating-point computation
Lu Xia, Stefano Massei, Michiel E. Hochstenbach, Barry Koren
Gradient Descent Absolute Stance Bias Early Stage Convergence Stochastic Rounding

February 22, 2022

February 14, 2022

On the Convergence of SARSA with Linear Function Approximation
Shangtong Zhang, Remi Tachet, Romain Laroche
Reinforcement Learning Early Stage Convergence Linear Function Approximation Fast Convergence Lipschitz Constant

February 13, 2022

On the Convergence of Clustered Federated Learning
Jie Ma, Guodong Long, Tianyi Zhou, Jing Jiang, Chengqi Zhang
Early Stage Convergence Personalized Model Model Personalization Clustered Federated Learning Personalized Fl

Early Stage Convergence

Papers

On the Convergence of the Shapley Value in Parametric Bayesian Learning Games

Convergence of Deep Neural Networks with General Activation Functions and Pooling

$\alpha$-GAN: Convergence and Estimation Guarantees

Deep Architecture Connectivity Matters for Its Convergence: A Fine-Grained Analysis

An Efficient Summation Algorithm for the Accuracy, Convergence and Reproducibility of Parallel Numerical Methods

On the Convergence of Fictitious Play: A Decomposition Approach

Convergence of Stochastic Approximation via Martingale and Converse Lyapunov Methods

On the Convergence of Momentum-Based Algorithms for Federated Bilevel Optimization Problems

Convergence of neural networks to Gaussian mixture distribution

Convergence of the Riemannian Langevin Algorithm

Convergence and Implicit Regularization Properties of Gradient Descent for Deep Residual Networks

Edge-enabled Metaverse: The Convergence of Metaverse and Mobile Edge Computing

Convergence of gradient descent for deep neural networks

Convergence of First-Order Methods for Constrained Nonconvex Optimization with Dependent Data

On the Convergence of Certified Robust Training with Interval Bound Propagation

On the influence of stochastic roundoff errors and their bias on the convergence of the gradient descent method with low-precision floating-point computation

On the Rate of Convergence of Payoff-based Algorithms to Nash Equilibrium in Strongly Monotone Games

Convergence of online $k$-means

On the Convergence of SARSA with Linear Function Approximation

On the Convergence of Clustered Federated Learning