Stochastic Gradient Descent

Stochastic Gradient Descent (SGD) is an iterative optimization algorithm used to find the minimum of a function, particularly useful in machine learning for training large models where computing the exact gradient is computationally prohibitive. Current research focuses on improving SGD's efficiency and convergence properties, exploring variations like Adam, incorporating techniques such as momentum, adaptive learning rates, and line search methods, and analyzing its behavior in high-dimensional and non-convex settings. These advancements are crucial for training complex models like deep neural networks and improving the performance of various machine learning applications, impacting fields ranging from natural language processing to healthcare.

Papers

June 18, 2022

Provable Generalization of Overparameterized Meta-learning Trained with SGD
Yu Huang, Yingbin Liang, Longbo Huang
Stochastic Gradient Descent Meta Learning Generalization Bound Model Agnostic Meta Learning Provable Generalization

June 15, 2022

June 13, 2022

June 12, 2022

June 10, 2022

Stochastic Zeroth order Descent with Structured Directions
Marco Rando, Cesare Molinari, Silvia Villa, Lorenzo Rosasco
Stochastic Gradient Descent Stochastic Gradient Zeroth Order Interpretable Direction

June 8, 2022

June 7, 2022

June 5, 2022

Sharper Rates and Flexible Framework for Nonconvex SGD with Client and Data Sampling
Alexander Tyurin, Lukang Sun, Konstantin Burlachenko, Peter Richtárik
Stochastic Gradient Descent Fast Rate Flexible Framework Data Sampling Client Collaboration Stochastic First Order Method Nonconvex Function Smooth Nonconvex

June 1, 2022

Computing the Variance of Shuffling Stochastic Gradient Algorithms via Power Spectral Density Analysis
Carles Domingo-Enrich
Stochastic Gradient Descent Stochastic Gradient Stochastic Approximation Digital Computing Variance Information Gradient Noise Nesterov Momentum Spectral Estimator Finite Sum Minimization

May 31, 2022

Asynchronous Hierarchical Federated Learning
Xing Wang, Yijun Wang
Federated Learning Stochastic Gradient Descent Hierarchical Learning Asynchronous Learning

May 30, 2022

Stochastic Gradient Descent

Papers

Provable Generalization of Overparameterized Meta-learning Trained with SGD

Asynchronous SGD Beats Minibatch SGD Under Arbitrary Delays

Implicit Regularization or Implicit Conditioning? Exact Risk Trajectories of SGD in High Dimensions

Markov Chain Score Ascent: A Unifying Framework of Variational Inference with Markovian Gradients

On the Convergence to a Global Solution of Shuffling-Type Gradient Algorithms

Stochastic Gradient Descent without Full Data Shuffle

Characterizing the Implicit Bias of Regularized SGD in Rank Minimization

Stochastic Zeroth order Descent with Structured Directions

High-dimensional limit theorems for SGD: Effective dynamics and critical scaling

Boosting the Confidence of Generalization for $L_2$-Stable Randomized Learning Algorithms

Few-Shot Learning by Dimensionality Reduction in Gradient Space

Generalization Error Bounds for Deep Neural Networks Trained by SGD

Sharper Rates and Flexible Framework for Nonconvex SGD with Client and Data Sampling

Algorithmic Stability of Heavy-Tailed Stochastic Gradient Descent on Least Squares

Stochastic gradient descent introduces an effective landscape-dependent regularization favoring flat solutions

Trajectory of Mini-Batch Momentum: Batch Size Saturation and Convergence in High Dimensions

Computing the Variance of Shuffling Stochastic Gradient Algorithms via Power Spectral Density Analysis

Asynchronous Hierarchical Federated Learning

Special Properties of Gradient Descent with Large Learning Rates

Agnostic Physics-Driven Deep Learning