Stochastic Gradient Descent

Stochastic Gradient Descent (SGD) is an iterative optimization algorithm used to find the minimum of a function, particularly useful in machine learning for training large models where computing the exact gradient is computationally prohibitive. Current research focuses on improving SGD's efficiency and convergence properties, exploring variations like Adam, incorporating techniques such as momentum, adaptive learning rates, and line search methods, and analyzing its behavior in high-dimensional and non-convex settings. These advancements are crucial for training complex models like deep neural networks and improving the performance of various machine learning applications, impacting fields ranging from natural language processing to healthcare.

Papers

June 4, 2024

June 3, 2024

June 1, 2024

Stochastic Restarting to Overcome Overfitting in Neural Networks with Noisy Labels
Youngkyoung Bae, Yeongwoo Song, Hawoong Jeong
Neural Network Deep Neural Network Stochastic Gradient Descent Noisy Label Generalization Performance Stochastic Way Model Overfitting

May 30, 2024

Sharpness-Aware Minimization Enhances Feature Quality via Balanced Learning
Jacob Mitchell Springer, Vaishnavh Nagarajan, Aditi Raghunathan
Strong Generalization Stochastic Gradient Descent Sharpness Aware Minimization Flat Minimum Simplicity Bias

May 28, 2024

May 27, 2024

May 25, 2024

Does SGD really happen in tiny subspaces?
Minhak Song, Kwangjun Ahn, Chulhee Yun
Deep Neural Network Stochastic Gradient Descent Hessian Matrix Large Learning Rate Active Subspace

May 24, 2024

May 23, 2024

Exact Gauss-Newton Optimization for Training Deep Neural Networks
Mikalai Korbit, Adeyemi D. Adeoye, Alberto Bemporad, Mario Zanon
Stochastic Gradient Descent Hessian Matrix Second Order Training Deep Gauss Newton

May 22, 2024

May 21, 2024

Uncertainty quantification by block bootstrap for differentially private stochastic gradient descent
Holger Dette, Carina Graw
Differential Privacy Stochastic Gradient Descent Uncertainty Quantification Local Differential Privacy Private Stochastic Gradient Descent Private Convex Optimization

Stochastic Gradient Descent

Papers

Online Learning and Information Exponents: On The Importance of Batch size, and Time/Complexity Tradeoffs

Understanding Stochastic Natural Gradient Variational Inference

Neural network learns low-dimensional polynomials with SGD near the information-theoretic limit

Demystifying SGD with Doubly Stochastic Gradients

Stochastic Restarting to Overcome Overfitting in Neural Networks with Noisy Labels

Sharpness-Aware Minimization Enhances Feature Quality via Balanced Learning

SGD method for entropy error function with smoothing l0 regularization for neural networks

A Hessian-Aware Stochastic Differential Equation for Modelling SGD

Improving Linear System Solvers for Hyperparameter Optimisation in Iterative Gaussian Processes

Bias in Motion: Theoretical Insights into the Dynamics of Bias in SGD Training

Adaptive debiased SGD in high-dimensional GLMs with streaming data

Understanding Forgetting in Continual Learning with Linear Regression

Convergence of SGD with momentum in the nonconvex case: A time window-based analysis

Does SGD really happen in tiny subspaces?

Derivatives of Stochastic Gradient Descent

Repetita Iuvant: Data Repetition Allows SGD to Learn High-Dimensional Multi-Index Functions

Exact Gauss-Newton Optimization for Training Deep Neural Networks

Banded Square Root Matrix Factorization for Differentially Private Model Training

Score-based Generative Models with Adaptive Momentum

Uncertainty quantification by block bootstrap for differentially private stochastic gradient descent