Convergence Proof

Convergence proofs in machine learning aim to rigorously establish that optimization algorithms used to train models, such as stochastic gradient descent (SGD), reliably reach optimal solutions. Current research focuses on proving convergence for various architectures, including deep neural networks (specifically two-layer nets) and generative models like OT-Flow, often under specific conditions like regularization or bounded activations. These proofs are crucial for understanding the behavior of learning algorithms and improving their reliability and efficiency, impacting both theoretical understanding and practical applications of machine learning. Furthermore, research extends to distributed learning settings, addressing challenges like Byzantine attacks and distributional shifts.

Papers

July 10, 2024

Probabilistic learning rate scheduler with provable convergence
Dahlia Devapriya, Thulasi Tholeti, Janani Suresh, Sheetal Kalyani
Learning Rate Provable Convergence Variable Rate Convergence Proof

March 24, 2024

Convergence analysis of OT-Flow for sample generation
Yang Jing, Lei Li
Generative Model Convergence Analysis Sample Generation Convergence Proof Optimal Transport Flow

September 17, 2023

Global Convergence of SGD For Logistic Loss on Two Layer Neural Nets
Pulkit Gopalani, Samyak Jha, Anirbit Mukherjee
Stochastic Gradient Descent Convergence Rate Global Convergence Logistics System Convergence Proof

October 29, 2022

Robust Distributed Learning Against Both Distributional Shifts and Byzantine Attacks
Guanqiang Zhou, Ping Xu, Yue Wang, Zhi Tian
LeArning Abstract Distribution Shift Robust Version Robust Optimization Byzantine Attack Convergence Proof

October 20, 2022

Global Convergence of SGD On Two Layer Neural Nets
Pulkit Gopalani, Anirbit Mukherjee
Loss Function Stochastic Gradient Descent Global Convergence Convergence Proof Frobenius Norm Minimization

July 9, 2022

Multi-Model Federated Learning with Provable Guarantees
Neelkamal Bhuyan, Sharayu Moharir, Gauri Joshi
Federated Learning FedAvg Algorithm Provable Guarantee Convergence Proof

May 3, 2022

Convergence of Stochastic Approximation via Martingale and Converse Lyapunov Methods
M. Vidyasagar
Early Stage Convergence Lyapunov Function Stochastic Approximation Martingale Difference Global Asymptotic Stability Convergence Proof

December 13, 2021

Convergence proof for stochastic gradient descent in the training of deep neural networks with ReLU activation for constant target functions
Martin Hutzenthaler, Arnulf Jentzen, Katharina Pohl, Adrian Riekert, Luca Scarpa
Deep Neural Network Gradient Descent Stochastic Gradient Descent Stochastic Way Convergence Proof