Stochastic Gradient Descent

Stochastic Gradient Descent (SGD) is an iterative optimization algorithm used to find the minimum of a function, particularly useful in machine learning for training large models where computing the exact gradient is computationally prohibitive. Current research focuses on improving SGD's efficiency and convergence properties, exploring variations like Adam, incorporating techniques such as momentum, adaptive learning rates, and line search methods, and analyzing its behavior in high-dimensional and non-convex settings. These advancements are crucial for training complex models like deep neural networks and improving the performance of various machine learning applications, impacting fields ranging from natural language processing to healthcare.

Papers

July 19, 2023

Stochastic Subgradient Methods with Guaranteed Global Stability in Nonsmooth Nonconvex Optimization
Nachuan Xiao, Xiaoyin Hu, Kim-Chuan Toh
Stochastic Gradient Descent Stability Guarantee Nonsmooth Nonconvex Optimization Stochastic Subgradient Nonsmooth Nonconvex

July 13, 2023

July 11, 2023

Implicit regularisation in stochastic gradient descent: from single-objective to two-player games
Mihaela Rosca, Marc Peter Deisenroth
Neural Network Stochastic Gradient Descent Two Player Regularization Property Implicit Regularization Effect Gradient Based Optimisation

June 30, 2023

Resetting the Optimizer in Deep RL: An Empirical Study
Kavosh Asadi, Rasool Fakoor, Shoham Sabach
Deep Reinforcement Learning Stochastic Gradient Descent Empirical Study Deep RL Superior Optimizer Optimization Landscape Optimal Value Function

June 28, 2023

Ordering for Non-Replacement SGD
Yuetong Xu, Baharan Mirzasoleiman
Stochastic Gradient Descent Convergence Rate Shuffle Model Variable Ordering Incremental Gradient

June 22, 2023

Don't be so Monotone: Relaxing Stochastic Line Search in Over-Parameterized Models
Leonardo Galli, Holger Rauhut, Mark Schmidt
Stochastic Gradient Descent Parameterized Model Line Search Non Negative Monotone Stochastic Line Search Stochastic Polyak

June 21, 2023

Empirical Risk Minimization with Shuffled SGD: A Primal-Dual Perspective and Improved Bounds
Xufeng Cai, Cheuk Yin Lin, Jelena Diakonikolas
Gradient Descent Stochastic Gradient Descent Empirical Risk Minimization Primal Dual Faster Convergence Improved Bound Finite Sum Shuffling Gradient

June 20, 2023

June 19, 2023

Adaptive Federated Learning with Auto-Tuned Clients
Junhyung Lyle Kim, Mohammad Taha Toghani, César A. Uribe, Anastasios Kyrillidis
Stochastic Gradient Descent Machine Learning Framework Step Size Adaptive Federated Learning

June 16, 2023

Gradient is All You Need?
Konstantin Riedl, Timo Klock, Carina Geldhauser, Massimo Fornasier
Stochastic Gradient Descent Natural Gradient Derivative Free Optimization Intrinsic Feature Relaxation Parameter Consensus Based Optimization

June 15, 2023

June 14, 2023

Beyond Implicit Bias: The Insignificance of SGD Noise in Online Learning
Nikhil Vyas, Depen Morwani, Rosie Zhao, Gal Kaplun, Sham Kakade, Boaz Barak
Deep Learning Stochastic Gradient Descent Online Learning Implicit Bias Offline Learning Gradient Step Rating Meaningfulness Multi Epoch Training

June 13, 2023

Implicit Compressibility of Overparametrized Neural Networks Trained with Heavy-Tailed SGD
Yijun Wan, Melih Barsbey, Abdellatif Zaidi, Umut Simsekli
Neural Network Stochastic Gradient Descent Neural Network Compression Implicit Neural Heavy Tailed Noise

June 12, 2023

Fast Diffusion Model
Zike Wu, Pan Zhou, Kenji Kawaguchi, Hanwang Zhang
Diffusion Model Stochastic Gradient Descent Momentum Method

June 8, 2023

Correlated Noise in Epoch-Based Stochastic Gradient Descent: Implications for Weight Variances
Marcel Kühn, Bernd Rosenow
Stochastic Gradient Descent Future Implication Weight Update Correlated Noise Epoch Wise Double Descent

June 7, 2023

Stochastic Gradient Descent

Papers

Stochastic Subgradient Methods with Guaranteed Global Stability in Nonsmooth Nonconvex Optimization

Weighted Averaged Stochastic Gradient Descent: Asymptotic Normality and Optimality

Efficient SGD Neural Network Training via Sublinear Activated Neuron Identification

Implicit regularisation in stochastic gradient descent: from single-objective to two-player games

Resetting the Optimizer in Deep RL: An Empirical Study

Ordering for Non-Replacement SGD

Don't be so Monotone: Relaxing Stochastic Line Search in Over-Parameterized Models

Empirical Risk Minimization with Shuffled SGD: A Primal-Dual Perspective and Improved Bounds

The Implicit Bias of Batch Normalization in Linear Models and Two-layer Linear Convolutional Neural Networks

Sampling from Gaussian Process Posteriors using Stochastic Gradient Descent

Adaptive Federated Learning with Auto-Tuned Clients

Gradient is All You Need?

When and Why Momentum Accelerates SGD:An Empirical Study

Evaluation and Optimization of Gradient Compression for Distributed Deep Learning

Beyond Implicit Bias: The Insignificance of SGD Noise in Online Learning

Implicit Compressibility of Overparametrized Neural Networks Trained with Heavy-Tailed SGD

Fast Diffusion Model

Correlated Noise in Epoch-Based Stochastic Gradient Descent: Implications for Weight Variances

Catapults in SGD: spikes in the training loss and their impact on generalization through feature learning

Stochastic Collapse: How Gradient Noise Attracts SGD Dynamics Towards Simpler Subnetworks