Stochastic Gradient Descent

Stochastic Gradient Descent (SGD) is an iterative optimization algorithm used to find the minimum of a function, particularly useful in machine learning for training large models where computing the exact gradient is computationally prohibitive. Current research focuses on improving SGD's efficiency and convergence properties, exploring variations like Adam, incorporating techniques such as momentum, adaptive learning rates, and line search methods, and analyzing its behavior in high-dimensional and non-convex settings. These advancements are crucial for training complex models like deep neural networks and improving the performance of various machine learning applications, impacting fields ranging from natural language processing to healthcare.

Papers

September 11, 2024

Convergence of continuous-time stochastic gradient descent with applications to linear deep neural networks
Gabor Lugosi, Eulalia Nualart
Deep Neural Network Financial Application Gradient Descent Stochastic Gradient Descent Early Stage Convergence Linear Neural Network Sufficient Condition

September 7, 2024

September 5, 2024

Reducing Bias in Deep Learning Optimization: The RSGDM Approach
Honglin Qin, Hongye Zheng, Bingxing Wang, Zhizhong Wu, Bingyao Liu, Yuanfang Yang
Neural Network Stochastic Gradient Descent Learning Rate SGDA Algorithm Fast AltGDA Type Algorithm

August 30, 2024

Towards Hyper-parameter-free Federated Learning
Geetika, Drishya Uniyal, Bapi Chatterjee
Stochastic Gradient Descent Convergence Rate Step Size Auto Scaling Adaptive Step Size

August 29, 2024

A Minibatch-SGD-Based Learning Meta-Policy for Inventory Systems with Myopic Optimal Policy
Jiameng Lyu, Jinxing Xie, Shilin Yuan, Yuan Zhou
Stochastic Gradient Descent Meta Reinforcement Inventory Control

August 15, 2024

Stochastic Semi-Gradient Descent for Learning Mean Field Games with Population-Aware Function Approximation
Chenyu Zhang, Xu Chen, Xuan Di
Stochastic Gradient Descent Mean Field Mean Field Game Finite Time Convergence Multi Population

August 13, 2024

Information Geometry and Beta Link for Optimizing Sparse Variational Student-t Processes
Jian Xu, Delu Zeng, John Paisley
Stochastic Gradient Descent Tiltable Link Information Geometry Sparse Variational Gaussian Process Variational Parameter

August 10, 2024

Incremental Gauss-Newton Descent for Machine Learning
Mikalai Korbit, Mario Zanon
Machine Learning Stochastic Gradient Descent Gauss Newton

August 5, 2024

Optimizing Cox Models with Stochastic Gradient Descent: Theoretical Foundations and Practical Guidances
Lang Zeng, Weijing Tang, Zhao Ren, Ying Ding
Stochastic Gradient Descent Practical Guide Theoretical Foundation Cox Model

August 3, 2024

Can LLMs predict the convergence of Stochastic Gradient Descent?
Oussama Zekri, Abdelhakim Benechehab, Ievgen Redko
Large Language Model Medical LLM Stochastic Gradient Descent Early Stage Convergence Markov Chain Non Convex Optimization Large Deep Learning Model

August 2, 2024

Data Debugging is NP-hard for Classifiers Trained with SGD
Zizheng Guo, Pengyu Chen, Yanzhang Fu, Dongjing Miao
Training Data Machine Learning Model Stochastic Gradient Descent Simple Classifier Code Debugging Improved Algorithm

August 1, 2024

Convergence Analysis of Natural Gradient Descent for Over-parameterized Physics-Informed Neural Networks
Xianliang Xu, Ting Du, Wang Kong, Ye Li, Zhongyi Huang
Neural Network Gradient Descent Physic Informed Neural Network Stochastic Gradient Descent Convergence Rate Convergence Analysis Natural Gradient Descent Benign Over Parameterization

July 30, 2024

July 29, 2024

July 17, 2024

July 15, 2024

Enhancing Stochastic Optimization for Statistical Efficiency Using ROOT-SGD with Diminishing Stepsize
Tong Zhang, Chris Junchi Li
Optimization Purpose Stochastic Gradient Descent Stochastic Optimization Optimization Algorithm Sample Efficiency Step Size Optimal Convergence Rate

Stochastic Gradient Descent

Papers

Convergence of continuous-time stochastic gradient descent with applications to linear deep neural networks

NGD converges to less degenerate solutions than SGD

Enhancing Deep Learning with Optimized Gradient Descent: Bridging Numerical Methods and Neural Network Training

Reducing Bias in Deep Learning Optimization: The RSGDM Approach

Towards Hyper-parameter-free Federated Learning

A Minibatch-SGD-Based Learning Meta-Policy for Inventory Systems with Myopic Optimal Policy

Stochastic Semi-Gradient Descent for Learning Mean Field Games with Population-Aware Function Approximation

Information Geometry and Beta Link for Optimizing Sparse Variational Student-t Processes

Incremental Gauss-Newton Descent for Machine Learning

Optimizing Cox Models with Stochastic Gradient Descent: Theoretical Foundations and Practical Guidances

Can LLMs predict the convergence of Stochastic Gradient Descent?

Data Debugging is NP-hard for Classifiers Trained with SGD

Convergence Analysis of Natural Gradient Descent for Over-parameterized Physics-Informed Neural Networks

An Effective Dynamic Gradient Calibration Method for Continual Learning

No learning rates needed: Introducing SALSA -- Stable Armijo Line Search Adaptation

Convergence rates for the Adam optimizer

Characterizing Dynamical Stability of Stochastic Gradient Descent in Overparameterized Learning

Comparing Federated Stochastic Gradient Descent and Federated Averaging for Predicting Hospital Length of Stay

A Methodology Establishing Linear Convergence of Adaptive Gradient Methods under PL Inequality

Enhancing Stochastic Optimization for Statistical Efficiency Using ROOT-SGD with Diminishing Stepsize