Stochastic Gradient Descent

Stochastic Gradient Descent (SGD) is an iterative optimization algorithm used to find the minimum of a function, particularly useful in machine learning for training large models where computing the exact gradient is computationally prohibitive. Current research focuses on improving SGD's efficiency and convergence properties, exploring variations like Adam, incorporating techniques such as momentum, adaptive learning rates, and line search methods, and analyzing its behavior in high-dimensional and non-convex settings. These advancements are crucial for training complex models like deep neural networks and improving the performance of various machine learning applications, impacting fields ranging from natural language processing to healthcare.

Papers

September 4, 2023

Corgi^2: A Hybrid Offline-Online Approach To Storage-Aware Data Shuffling For SGD
Etay Livne, Gal Kaplun, Eran Malach, Shai Shalev-Schwatz
Stochastic Gradient Descent Online Algorithm Shuffle Model Random Access

September 3, 2023

Modified Step Size for Enhanced Stochastic Gradient Descent: Convergence and Experiments
M. Soheil Shamaee, S. Fathi Hafshejani
Gradient Descent Stochastic Gradient Descent Early Stage Convergence Optical Experiment Step Size

August 30, 2023

A numerical approach for the fractional Laplacian via deep neural networks
Nicolás Valenzuela
Deep Neural Network Stochastic Gradient Descent Numerical Method Fractional Derivative Boundary Condition

August 29, 2023

ABS-SGD: A Delayed Synchronous Stochastic Gradient Descent Algorithm with Adaptive Batch Size for Heterogeneous GPU Clusters
Xin Zhou, Ling Chen, Houming Wu
Stochastic Gradient Descent Stale Gradient Adaptive Batch Size Heterogeneous GPU Gradient Delay

August 25, 2023

Network Embedding Using Sparse Approximations of Random Walks
Paula Mercurio, Di Liu
Stochastic Gradient Descent Network Programming Node Embeddings Random Walk Low Dimensional Representation Sparse Approximation Wavelet Diffusion

August 18, 2023

Towards Understanding the Generalizability of Delayed Stochastic Gradient Descent
Xiaoge Deng, Li Shen, Shengwei Li, Tao Sun, Dongsheng Li, Dacheng Tao
Stochastic Gradient Descent Stronger Generalizability Generalization Error Bound

August 17, 2023

Hitting the High-Dimensional Notes: An ODE for SGD learning dynamics on GLMs and multi-index models
Elizabeth Collins-Woodfin, Courtney Paquette, Elliot Paquette, Inbar Seroussi
High Dimensional Stochastic Gradient Descent Covariance Conditioning Multi Index Model Note Level

August 13, 2023

August 6, 2023

August 5, 2023

K-band: Self-supervised MRI Reconstruction via Stochastic Gradient Descent over K-space Subsets
Frederic Wang, Han Qi, Alfredo De Goyeneche, Reinhard Heckel, Michael Lustig, Efrat Shimron
Self Supervised Stochastic Gradient Descent mmWave Signal K Space Multi Parametric MRI

August 3, 2023

Online covariance estimation for stochastic gradient descent under Markovian sampling
Abhishek Roy, Krishnakumar Balasubramanian
Gradient Descent Stochastic Gradient Descent Stochastic Way Covariance Matrix Markovian Sampling Covariance Regularization

July 28, 2023

On Single Index Models beyond Gaussian Data
Joan Bruna, Loucas Pillaud-Vivien, Aaron Zweig
High Dimensional Stochastic Gradient Descent Gaussian Data High Dimensional Regime Single Index Model Link Function

July 27, 2023

Linear Convergence of Black-Box Variational Inference: Should We Stick the Landing?
Kyurae Kim, Yian Ma, Jacob R. Gardner
Stochastic Gradient Descent Variational Inference Linear Convergence Safe Landing Variational Family Entropy Estimation Black Box Variational Inference

July 26, 2023

Function Value Learning: Adaptive Learning Rates Based on the Polyak Stepsize and Function Splitting in ERM
Guillaume Garrigos, Robert M. Gower, Fabian Schaipp
Gradient Descent Stochastic Gradient Descent Learning Rate Empirical Risk Minimization Value Function Step Size Splitting Algorithm Polyak Step Size

July 25, 2023

July 21, 2023

Convergence of SGD for Training Neural Networks with Sliced Wasserstein Losses
Eloi Tanguy
Neural Network Stochastic Gradient Descent Early Stage Convergence Gradient Flow Wasserstein Distance Sliced Wasserstein Gradient Descent Trajectory

July 19, 2023

Properties of Discrete Sliced Wasserstein Losses
Eloi Tanguy, Rémi Flamary, Julie Delon
Stochastic Gradient Descent Wasserstein Distance Many Property Sliced Wasserstein Almost Sure Convergence

Stochastic Gradient Descent

Papers

Corgi^2: A Hybrid Offline-Online Approach To Storage-Aware Data Shuffling For SGD

Modified Step Size for Enhanced Stochastic Gradient Descent: Convergence and Experiments

A numerical approach for the fractional Laplacian via deep neural networks

ABS-SGD: A Delayed Synchronous Stochastic Gradient Descent Algorithm with Adaptive Batch Size for Heterogeneous GPU Clusters

Network Embedding Using Sparse Approximations of Random Walks

Towards Understanding the Generalizability of Delayed Stochastic Gradient Descent

Hitting the High-Dimensional Notes: An ODE for SGD learning dynamics on GLMs and multi-index models

Understanding the robustness difference between stochastic gradient descent and adaptive gradient methods

Law of Balance and Stationary Distribution of Stochastic Gradient Descent

The Effect of SGD Batch Size on Autoencoder Learning: Sparsity, Sharpness, and Feature Learning

Learning-Rate-Free Learning: Dissecting D-Adaptation and Probabilistic Line Search

K-band: Self-supervised MRI Reconstruction via Stochastic Gradient Descent over K-space Subsets

Online covariance estimation for stochastic gradient descent under Markovian sampling

On Single Index Models beyond Gaussian Data

Linear Convergence of Black-Box Variational Inference: Should We Stick the Landing?

Function Value Learning: Adaptive Learning Rates Based on the Polyak Stepsize and Function Splitting in ERM

Relationship between Batch Size and Number of Steps Needed for Nonconvex Optimization of Stochastic Gradient Descent using Armijo Line Search

How to Scale Your EMA

Convergence of SGD for Training Neural Networks with Sliced Wasserstein Losses

Properties of Discrete Sliced Wasserstein Losses