Early Stage Convergence

Early stage convergence in machine learning focuses on understanding and improving the initial phases of training algorithms, aiming to accelerate convergence speed and enhance generalization performance. Current research investigates this through the lens of various optimization algorithms (e.g., Adam, SGD, FedProx), model architectures (e.g., transformers, diffusion models), and specific problem domains (e.g., federated learning, collaborative filtering). These studies leverage techniques from dynamical systems theory and optimal transport to establish convergence guarantees and bounds, ultimately contributing to more efficient and robust machine learning systems across diverse applications.

Papers

September 7, 2022

September 6, 2022

Rates of Convergence for Regression with the Graph Poly-Laplacian
Nicolás García Trillos, Ryan Murray, Matthew Thorpe
Early Stage Convergence Novel Regression Graph Laplacian B Spline Universal Rate Laplacian Regularization

September 3, 2022

Suppressing Noise from Built Environment Datasets to Reduce Communication Rounds for Convergence of Federated Learning
Rahul Mishra, Hari Prabhat Gupta, Tanima Dutta, Sajal K. Das
Data Driven Early Stage Convergence Privacy Preserving Federated Learning Approach Building Dataset Noise Suppression Intelligent Sensing Communication Round

August 23, 2022

Naive Penalized Spline Estimators of Derivatives Achieve Optimal Rates of Convergence
Bright Antwi Boasiako, John Staudenmayer
Early Stage Convergence Optimal Rate Derivative Process

August 16, 2022

A Review of the Convergence of 5G/6G Architecture and Deep Learning
Olusola T. Odeyomi, Olubiyi O. Akintade, Temitayo O. Olowu, Gergely Zaruba
Deep Learning Artificial Intelligence Early Stage Convergence Deep Learning Technology

August 10, 2022

Convergence of denoising diffusion models under the manifold hypothesis
Valentin De Bortoli
Diffusion Model Generative Model Early Stage Convergence Audio Synthesis Manifold Hypothesis

July 28, 2022

Regret Minimization and Convergence to Equilibria in General-sum Markov Games
Liad Erez, Tal Lancewicki, Uri Sherman, Tomer Koren, Yishay Mansour
Early Stage Convergence Regret Minimization Sublinear Regret Markov Game Efficient Equilibrium General Sum Markov Game

July 21, 2022

Metropolis Monte Carlo sampling: convergence, localization transition and optimality
Alexei D. Chepelianskii, Satya N. Majumdar, Hendrik Schawe, Emmanuel Trizac
Large Scale Early Stage Convergence Random Walk Near Optimality Monte Carlo Sampling

July 3, 2022

On Convergence of Gradient Descent Ascent: A Tight Local Analysis
Haochuan Li, Farzan Farnia, Subhro Das, Ali Jadbabaie
Generative Adversarial Network Early Stage Convergence Convergence Rate Gradient Descent Ascent GAN Algorithm Stochastic Gradient Descent Ascent Tight Analysis

July 2, 2022

Informed Learning by Wide Neural Networks: Convergence, Generalization and Sampling Complexity
Jianyi Yang, Shaolei Ren
Deep Learning Machine Learning Strong Generalization Domain Knowledge Sample Complexity Early Stage Convergence Wide Neural Network Informed Learning

June 30, 2022

On the Convergence of Distributed Stochastic Bilevel Optimization Algorithms over a Network
Hongchang Gao, Bin Gu, My T. Thai
Network Programming Early Stage Convergence Bilevel Optimization Gradient Estimator Stochastic Bilevel Optimization Strongly Convex

June 22, 2022

June 20, 2022

A Note on the Convergence of Mirrored Stein Variational Gradient Descent under $(L_0,L_1)-$Smoothness Condition
Lukang Sun, Peter Richtárik
Early Stage Convergence Stein Variational Gradient Descent Descent Property

June 15, 2022

Convergence and Price of Anarchy Guarantees of the Softmax Policy Gradient in Markov Potential Games
Dingyang Chen, Qi Zhang, Thinh T. Doan
Policy Gradient Early Stage Convergence Markov Game Curious Price Natural Policy Gradient Stability Guarantee Softmax Policy Markov Potential Game

June 13, 2022

June 12, 2022

Convergence and Stability of Coupled Belief--Strategy Learning Dynamics in Continuous Games
Manxi Wu, Saurabh Amin, Asuman Ozdaglar
Early Stage Convergence Core Stability Learning Dynamic Belief State Strategic Agent Continuous Game Strategy Training

June 10, 2022

On Convergence of FedProx: Local Dissimilarity Invariant Bounds, Non-smoothness and Beyond
Xiao-Tong Yuan, Ping Li
Early Stage Convergence Convergence Guarantee Proximal Point FedProx Algorithm

Early Stage Convergence

Papers

On the Convergence of the ELBO to Entropy Sums

On the Convergence of Monte Carlo UCB for Random-Length Episodic MDPs

Rates of Convergence for Regression with the Graph Poly-Laplacian

Suppressing Noise from Built Environment Datasets to Reduce Communication Rounds for Convergence of Federated Learning

Naive Penalized Spline Estimators of Derivatives Achieve Optimal Rates of Convergence

A Review of the Convergence of 5G/6G Architecture and Deep Learning

Convergence of denoising diffusion models under the manifold hypothesis

Regret Minimization and Convergence to Equilibria in General-sum Markov Games

Metropolis Monte Carlo sampling: convergence, localization transition and optimality

On Convergence of Gradient Descent Ascent: A Tight Local Analysis

Informed Learning by Wide Neural Networks: Convergence, Generalization and Sampling Complexity

On the Convergence of Distributed Stochastic Bilevel Optimization Algorithms over a Network

Bi-stochastically normalized graph Laplacian: convergence to manifold Laplacian and robustness to outlier noise

A view of mini-batch SGD via generating functions: conditions of convergence, phase transitions, benefit from negative momenta

A Note on the Convergence of Mirrored Stein Variational Gradient Descent under $(L_0,L_1)-$Smoothness Condition

Convergence and Price of Anarchy Guarantees of the Softmax Policy Gradient in Markov Potential Games

Convergence for score-based generative modeling with polynomial complexity

On the Convergence to a Global Solution of Shuffling-Type Gradient Algorithms

Convergence and Stability of Coupled Belief--Strategy Learning Dynamics in Continuous Games

On Convergence of FedProx: Local Dissimilarity Invariant Bounds, Non-smoothness and Beyond