Stochastic Gradient Descent

Stochastic Gradient Descent (SGD) is an iterative optimization algorithm used to find the minimum of a function, particularly useful in machine learning for training large models where computing the exact gradient is computationally prohibitive. Current research focuses on improving SGD's efficiency and convergence properties, exploring variations like Adam, incorporating techniques such as momentum, adaptive learning rates, and line search methods, and analyzing its behavior in high-dimensional and non-convex settings. These advancements are crucial for training complex models like deep neural networks and improving the performance of various machine learning applications, impacting fields ranging from natural language processing to healthcare.

Papers

December 4, 2023

December 1, 2023

From Mutual Information to Expected Dynamics: New Generalization Bounds for Heavy-Tailed SGD
Benjamin Dupuis, Paul Viallard
Strong Generalization Stochastic Gradient Descent Mutual Information Generalization Bound Learning Dynamic Heavy Tailed Generalization Theory

November 28, 2023

Adaptive Step Sizes for Preconditioned Stochastic Gradient Descent
Frederik Köhne, Leonie Kreis, Anton Schiela, Roland Herzog
Stochastic Gradient Descent Second Order Stochastic Optimization Step Size Hyperparameter Free Preconditioned Gradient Descent

November 27, 2023

MAST: Model-Agnostic Sparsified Training
Yury Demidovich, Grigory Malinovsky, Egor Shulgin, Peter Richtárik
Stochastic Gradient Descent Sparse Training Random Sketching

November 24, 2023

Efficient Gradient Estimation via Adaptive Sampling and Importance Sampling
Corentin Salaün, Xingchang Huang, Iliyan Georgiev, Niloy J. Mitra, Gurprit Singh
Stochastic Gradient Descent Importance Sampling Adaptive Sampling Gradient Estimation Importance Distribution

November 23, 2023

November 15, 2023

Using Stochastic Gradient Descent to Smooth Nonconvex Functions: Analysis of Implicit Graduated Optimization
Naoki Sato, Hideaki Iiduka
General Analysis Optimization Purpose Stochastic Gradient Descent Mini Batch Gradient Descent Optimization Framework Nonconvex Function Smooth Nonconvex Noise Scheduling

November 14, 2023

Sparsity-Preserving Differentially Private Training of Large Embedding Models
Badih Ghazi, Yangsibo Huang, Pritish Kamath, Ravi Kumar, Pasin Manurangsi, Amer Sinha, Chiyuan Zhang
Full Model Differential Privacy Stochastic Gradient Descent User Privacy Private Model Training Gradient Sparsification

November 12, 2023

Inference and Interference: The Role of Clipping, Pruning and Loss Landscapes in Differentially Private Stochastic Gradient Descent
Lauren Watson, Eric Gan, Mohan Dantam, Baharan Mirzasoleiman, Rik Sarkar
Neural Network Gradient Descent Stochastic Gradient Descent Scientific Inference Edge Pruning Loss Landscape Private Stochastic Private Stochastic Gradient Descent High Dimensional Clipping

November 6, 2023

Signal Processing Meets SGD: From Momentum to Filter
Zhipeng Yao, Guiyuan Fu, Ying Li, Yu Zhang, Dazhou Li, Rui Yu
Stochastic Gradient Descent Learning Rate Signal Processing Residual Momentum Slow Convergence

November 1, 2023

Implicit biases in multitask and continual learning from a backward error analysis perspective
Benoit Dherin
Continual LEArning Stochastic Gradient Descent Multi Task Implicit Bias Implicit Regularization Task Gradient Flatness Aware

October 31, 2023

October 29, 2023

Escaping Saddle Points in Heterogeneous Federated Learning via Distributed SGD with Communication Compression
Sijin Chen, Zhize Li, Yuejie Chi
Stochastic Gradient Descent Heterogeneous Federated Learning Saddle Point Communication Compression

October 28, 2023

High-probability Convergence Bounds for Nonlinear Stochastic Gradient Descent Under Heavy-tailed Noise
Aleksandar Armacki, Pranay Sharma, Gauri Joshi, Dragana Bajovic, Dusan Jakovetic, Soummya Kar
Stochastic Gradient Descent Heavy Tailed Noise Stochastic Nonlinear System High Probability Convergence Convex Cost

October 27, 2023

Approximate Heavy Tails in Offline (Multi-Pass) Stochastic Gradient Descent
Krunoslav Lehman Pavasovic, Alain Durmus, Umut Simsekli
Stochastic Gradient Descent Action Free Offline Heavy Tailed Heavy Tail Offline Training

October 26, 2023

Benign Oscillation of Stochastic Gradient Descent with Large Learning Rates
Miao Lu, Beining Wu, Xiaodong Yang, Difan Zou
Stochastic Gradient Descent Generalization Property Large Learning Rate Non Converging Artificial Oscillation

October 24, 2023

Decentralized Learning over Wireless Networks with Broadcast-Based Subgraph Sampling
Daniel Pérez Herrera, Zheng Chen, Erik G. Larsson
Gradient Descent Stochastic Gradient Descent Decentralized Learning Wireless Network Communication Constraint Subgraph Sampling Decentralized Stochastic

Stochastic Gradient Descent

Papers

Unlocking optimal batch size schedules using continuous-time control and perturbation theory

AGD: an Auto-switchable Optimizer using Stepwise Gradient Difference for Preconditioning Matrix

From Mutual Information to Expected Dynamics: New Generalization Bounds for Heavy-Tailed SGD

Adaptive Step Sizes for Preconditioned Stochastic Gradient Descent

MAST: Model-Agnostic Sparsified Training

Efficient Gradient Estimation via Adaptive Sampling and Importance Sampling

Weight fluctuations in (deep) linear neural networks and a derivation of the inverse-variance flatness relation

DPSUR: Accelerating Differentially Private Stochastic Gradient Descent Using Selective Update and Release

Using Stochastic Gradient Descent to Smooth Nonconvex Functions: Analysis of Implicit Graduated Optimization

Sparsity-Preserving Differentially Private Training of Large Embedding Models

Inference and Interference: The Role of Clipping, Pruning and Loss Landscapes in Differentially Private Stochastic Gradient Descent

Signal Processing Meets SGD: From Momentum to Filter

Implicit biases in multitask and continual learning from a backward error analysis perspective

Bandit-Driven Batch Selection for Robust Learning under Label Noise

Stochastic Gradient Descent for Gaussian Processes Done Right

Escaping Saddle Points in Heterogeneous Federated Learning via Distributed SGD with Communication Compression

High-probability Convergence Bounds for Nonlinear Stochastic Gradient Descent Under Heavy-tailed Noise

Approximate Heavy Tails in Offline (Multi-Pass) Stochastic Gradient Descent

Benign Oscillation of Stochastic Gradient Descent with Large Learning Rates

Decentralized Learning over Wireless Networks with Broadcast-Based Subgraph Sampling