Global Convergence Rate

Global convergence rate research focuses on determining how quickly optimization algorithms reach a global optimum, a crucial aspect for the efficiency and scalability of machine learning models. Current investigations explore this rate across various architectures, including deep equilibrium models and over-parameterized neural networks with ReLU activations, and algorithms like Sharpness-Aware Minimization and Frank-Wolfe, often analyzing the impact of factors such as perturbation size, activation functions, and data dimensionality. Understanding these rates is vital for improving the performance and theoretical understanding of numerous machine learning applications, particularly in high-dimensional settings and distributed environments like federated learning. The field is actively refining convergence bounds and exploring techniques to accelerate convergence, particularly in challenging non-convex optimization landscapes.

Papers

June 16, 2023

Practical Sharpness-Aware Minimization Cannot Converge All the Way to Optima
Dongkuk Si, Chulhee Yun
Sharpness Aware Minimization New Way Global Minimum Gradient Normalization Global Convergence Rate

February 11, 2023

Global Convergence Rate of Deep Equilibrium Models with General Activations
Lan V. Truong
Gradient Descent Activation Function Targeted Activation Penalty Deep Equilibrium Model Gram Matrix Quadratic Loss Global Convergence Rate

June 5, 2022

Demystifying the Global Convergence Puzzle of Learning Over-parameterized ReLU Nets in Very High Dimensions
Peng He
LeArning Abstract High Dimension Gradient Norm Global Convergence Rectified Linear Unit Random Neural Network Global Convergence Rate

June 2, 2022

Resource Allocation for Compression-aided Federated Learning with High Distortion Rate
Xuan-Tung Nguyen, Minh-Duong Nguyen, Quoc-Viet Pham, Vinh-Quang Do, Won-Joo Hwang
Federated Learning Model Compression Resource Allocation Distortion Aware Communication Efficiency Federated Learning Convergence Global Convergence Rate

May 24, 2022

Accelerating Frank-Wolfe via Averaging Step Directions
Zhaoyue Chen, Yifan Sun
Human Instruction Iteration Complexity Frank Wolfe Global Convergence Rate Sparse Manifold Transform

May 11, 2022

A globally convergent fast iterative shrinkage-thresholding algorithm with a new momentum factor for single and multi-objective convex optimization
Hiroki Tanabe, Ellen H. Fukuda, Nobuo Yamashita
Multi Objective Optimization Global Convergence Proximal Gradient Iterative Algorithm Convex Composite Optimization Shrinkage Thresholding Algorithm Global Convergence Rate

February 2, 2022

On the Global Convergence Rates of Decentralized Softmax Gradient Play in Markov Potential Games
Runyu Zhang, Jincheng Mei, Bo Dai, Dale Schuurmans, Na Li
Policy OpTimization Single Agent Reinforcement Learning Softmax Policy Markov Potential Game Global Convergence Rate Decentralized Policy Gradient

Global Convergence Rate

Papers

Practical Sharpness-Aware Minimization Cannot Converge All the Way to Optima

Global Convergence Rate of Deep Equilibrium Models with General Activations

Demystifying the Global Convergence Puzzle of Learning Over-parameterized ReLU Nets in Very High Dimensions

Resource Allocation for Compression-aided Federated Learning with High Distortion Rate

Accelerating Frank-Wolfe via Averaging Step Directions

A globally convergent fast iterative shrinkage-thresholding algorithm with a new momentum factor for single and multi-objective convex optimization

On the Global Convergence Rates of Decentralized Softmax Gradient Play in Markov Potential Games