Global Convergence

Global convergence in optimization and machine learning focuses on establishing theoretical guarantees that algorithms will reach a globally optimal solution, rather than getting stuck in suboptimal local minima. Current research intensely investigates this for various algorithms, including gradient descent variants, quasi-Newton methods, actor-critic methods, and Markov Chain Monte Carlo techniques, often applied to models like neural networks and Gaussian mixture models. These studies are crucial for ensuring the reliability and efficiency of machine learning applications and provide deeper insights into the behavior of optimization algorithms in complex, high-dimensional spaces. The resulting theoretical frameworks and improved algorithms have significant implications for the robustness and scalability of machine learning across diverse fields.

Papers

October 31, 2024

Global Convergence in Training Large-Scale Transformers
Cheng Gao, Yuan Cao, Zihao Li, Yihan He, Mengdi Wang, Han Liu, Jason Matthew Klusowski, Jianqing Fan
Training Data Gradient Flow Global Convergence Transformer Training Large Scale Transformer Decay Regularization

October 21, 2024

October 15, 2024

Convergence to the Truth
Hanti Lin
Scientific Inference Early Stage Convergence Global Convergence Great Truth Epistemic Space Explanation Model

October 13, 2024

Phase retrieval: Global convergence of gradient descent with optimal sample complexity
Théodore Fougereux, Cédric Josz, Xiaopeng Li
Gradient Descent Sample Complexity Global Convergence Phase Retrieval Compressed Measurement Phase Retrieval Problem

October 11, 2024

Improved Sample Complexity for Global Convergence of Actor-Critic Algorithms
Navdeep Kumar, Priyank Agrawal, Giorgia Ramponi, Kfir Yehuda Levy, Shie Mannor
Practical Algorithm Sample Complexity Convergence Guarantee Global Convergence Local Convergence Gradient Dominance Property

October 7, 2024

Aiding Global Convergence in Federated Learning via Local Perturbation and Mutual Similarity Information
Emanuel Buttaci, Giuseppe Carlo Calafiore
Global Convergence Client Selection Gradient Based Optimization Additive Perturbation FedAvg Algorithm

October 3, 2024

Online Learning Guided Quasi-Newton Methods with Global Non-Asymptotic Convergence
Ruichen Jiang, Aryan Mokhtari
Online Learning Global Convergence Monotone Function Quasi Newton Method Superlinear Convergence Non Asymptotic Convergence

June 29, 2024

Toward Global Convergence of Gradient EM for Over-Parameterized Gaussian Mixture Models
Weihang Xu, Maryam Fazel, Simon S. Du
Gaussian Mixture Model Expectation Maximization Gaussian Mixture Global Convergence

June 7, 2024

Gradient Descent on Logistic Regression with Non-Separable Data and Large Step Sizes
Si Yi Meng, Antonio Orvieto, Daniel Yiming Cao, Christopher De Sa
Gradient Descent Logistic Regression Global Convergence Step Size Local Convergence

May 23, 2024

Global Behavior of Learning Dynamics in Zero-Sum Games with Memory Asymmetry
Yuma Fujimoto, Kaito Ariu, Kenshi Abe
Nash Equilibrium Learning Dynamic Zero Sum Game Global Convergence Information Asymmetry Global Interaction

May 19, 2024

Global Convergence of Decentralized Retraction-Free Optimization on the Stiefel Manifold
Youbang Sun, Shixiang Chen, Alfredo Garcia, Shahin Shahrampour
Global Convergence Decentralized Optimization Gradient Update Orthogonality Constraint Decentralized Nonconvex Stiefel Manifold Retraction Based Method

May 3, 2024

Closing the Gap: Achieving Global Convergence (Last Iterate) of Actor-Critic under Markovian Sampling with Neural Network Parametrization
Mudit Gaur, Amrit Singh Bedi, Di Wang, Vaneet Aggarwal
Neural Network Global Convergence State Action Space Sample Complexity Bound Critic Model Markovian Sampling Local Optimality Gradient Dominance Property

April 16, 2024

EMC$^2$: Efficient MCMC Negative Sampling for Contrastive Learning with Global Convergence
Chung-Yiu Yau, Hoi-To Wai, Parameswaran Raman, Soumajyoti Sarkar, Mingyi Hong
Contrastive Learning Contrastive Loss Negative Sampling Global Convergence McMc Sampling

April 15, 2024

Safeguarding adaptive methods: global convergence of Barzilai-Borwein and other stepsize choices
Hongjia Ou, Andreas Themelis
Natural Gradient Global Convergence Proximal Gradient Adaptive Method Anderson Acceleration Convex Minimization

April 3, 2024

Analyzing Economic Convergence Across the Americas: A Survival Analysis Approach to GDP per Capita Trajectories
Diego Vallarino
Handwritten Trajectory Global Convergence Temporal Dynamic Complex Dynamic General Data Protection Regulation

March 11, 2024

On the Global Convergence of Policy Gradient in Average Reward Markov Decision Processes
Navdeep Kumar, Yashaswini Murthy, Itai Shufaro, Kfir Y. Levy, R. Srikant, Shie Mannor
Policy Gradient Global Convergence Tabular Markov Decision Process Average Reward Markov Decision Process Hybrid Reward Average Reward Reinforcement Learning

March 8, 2024

Evidence, Definitions and Algorithms regarding the Existence of Cohesive-Convergence Groups in Neural Network Optimization
Thien An L. Nguyen
Neural Network Practical Algorithm Evidence Piece Global Convergence Group Cohesion

February 9, 2024

Convergence of Gradient Descent with Small Initialization for Unregularized Matrix Completion
Jianhao Ma, Salar Fattahi
Gradient Descent Early Stage Convergence Matrix Completion Global Convergence Explicit Regularization Vanilla Gradient Descent Small Initialization Positive Semidefinite

January 2, 2024

Global Convergence of Natural Policy Gradient with Hessian-aided Momentum Variance Reduction
Jie Feng, Ke Wei, Jinchi Chen
Global Convergence Natural Policy Gradient Policy Parametrization Gradient Dominance Property

Global Convergence

Papers

Global Convergence in Training Large-Scale Transformers

Simplicity Bias via Global Convergence of Sharpness Minimization

On The Global Convergence Of Online RLHF With Neural Parametrization

Convergence to the Truth

Phase retrieval: Global convergence of gradient descent with optimal sample complexity

Improved Sample Complexity for Global Convergence of Actor-Critic Algorithms

Aiding Global Convergence in Federated Learning via Local Perturbation and Mutual Similarity Information

Online Learning Guided Quasi-Newton Methods with Global Non-Asymptotic Convergence

Toward Global Convergence of Gradient EM for Over-Parameterized Gaussian Mixture Models

Gradient Descent on Logistic Regression with Non-Separable Data and Large Step Sizes

Global Behavior of Learning Dynamics in Zero-Sum Games with Memory Asymmetry

Global Convergence of Decentralized Retraction-Free Optimization on the Stiefel Manifold

Closing the Gap: Achieving Global Convergence (Last Iterate) of Actor-Critic under Markovian Sampling with Neural Network Parametrization

EMC$^2$: Efficient MCMC Negative Sampling for Contrastive Learning with Global Convergence

Safeguarding adaptive methods: global convergence of Barzilai-Borwein and other stepsize choices

Analyzing Economic Convergence Across the Americas: A Survival Analysis Approach to GDP per Capita Trajectories

On the Global Convergence of Policy Gradient in Average Reward Markov Decision Processes

Evidence, Definitions and Algorithms regarding the Existence of Cohesive-Convergence Groups in Neural Network Optimization

Convergence of Gradient Descent with Small Initialization for Unregularized Matrix Completion

Global Convergence of Natural Policy Gradient with Hessian-aided Momentum Variance Reduction