Stochastic Gradient Descent

Stochastic Gradient Descent (SGD) is an iterative optimization algorithm used to find the minimum of a function, particularly useful in machine learning for training large models where computing the exact gradient is computationally prohibitive. Current research focuses on improving SGD's efficiency and convergence properties, exploring variations like Adam, incorporating techniques such as momentum, adaptive learning rates, and line search methods, and analyzing its behavior in high-dimensional and non-convex settings. These advancements are crucial for training complex models like deep neural networks and improving the performance of various machine learning applications, impacting fields ranging from natural language processing to healthcare.

Papers

October 7, 2024

Nonasymptotic Analysis of Stochastic Gradient Descent with the Richardson-Romberg Extrapolation
Marina Sheshukova, Denis Belomestny, Alain Durmus, Eric Moulines, Alexey Naumov, Sergey Samsonov
Stochastic Gradient Descent Cartesian Product Extrapolation Non Asymptotic Analysis Smooth Convex Optimization

October 5, 2024

SGD with memory: fundamental properties and stochastic acceleration
Dmitry Yarotsky, Maksim Velikanov
Stochastic Gradient Descent Memory Trace Mini Batch Exponential Mechanism Memory Based Accelerated Gradient Fundamental Property Stable Algorithm

October 3, 2024

October 1, 2024

Convergent Privacy Loss of Noisy-SGD without Convexity and Smoothness
Eli Chien, Pan Li
Differential Privacy Stochastic Gradient Descent Convex Loss Privacy Analysis Lower Level Convexity Simplification

September 26, 2024

Does Worst-Performing Agent Lead the Pack? Analyzing Agent Dynamics in Unified Distributed SGD
Jie Hu, Yi-Ting Ma, Do Young Eun
Stochastic Gradient Descent Agent Smith Local SGD Agent Dynamic Decentralized SGD Centralized Stochastic Gradient Descent

September 25, 2024

Differential Privacy Regularization: Protecting Training Data Through Loss Function Regularization
Francisco Aguilera-Martínez, Fernando Berzal
Training Data Differential Privacy Stochastic Gradient Descent Regularization Loss Standard Gradient Descent Novel Regularization

September 15, 2024

The Optimality of (Accelerated) SGD for High-Dimensional Quadratic Optimization
Haihan Zhang, Yuanshi Liu, Qianwen Chen, Cong Fang
High Dimensional Stochastic Gradient Descent Convergence Rate Near Optimality Low Rate ACCELERATION Convex Quadratic

September 14, 2024

A Dynamic Weighting Strategy to Mitigate Worker Node Failure in Distributed Deep Learning
Yuesheng Xu, Arielle Carr
Deep Learning Stochastic Gradient Descent Convergence Rate Distributed Learning Straggler Mitigation Critical Node Weighting Strategy Dynamic Weighting

September 13, 2024

Increasing Both Batch Size and Learning Rate Accelerates Stochastic Gradient Descent
Hikaru Umeda, Hideaki Iiduka
Gradient Descent Stochastic Gradient Descent Gradient Norm Mini Batch Accelerated Gradient Descent Batch Size

September 11, 2024

Asymptotics of Stochastic Gradient Descent with Dropout Regularization in Linear Models
Jiaqi Li, Johannes Schmidt-Hieber, Wei Biao Wu
Stochastic Gradient Descent Covariance Matrix Linear Model Asymptotic Behavior Online Inference Dropout Regularization

Stochastic Gradient Descent

Papers

Nonasymptotic Analysis of Stochastic Gradient Descent with the Richardson-Romberg Extrapolation

The Optimization Landscape of SGD Across the Feature Learning Strength

Upsample or Upweight? Balanced Training on Heavily Imbalanced Datasets

A Comprehensive Framework for Analyzing the Convergence of Adam: Bridging the Gap with SGD

SGD with memory: fundamental properties and stochastic acceleration

Estimating Generalization Performance Along the Trajectory of Proximal SGD in Robust Regression

Learning K-U-Net with constant complexity: An Application to time series forecasting

Towards Better Generalization: Weight Decay Induces Low-rank Bias for Neural Networks

Universality in Transfer Learning for Linear Models

Review Non-convex Optimization Method for Machine Learning

Truncated Kernel Stochastic Gradient Descent on Spheres

On the SAGA algorithm with decreasing step

Stochastic Gradient Descent with Adaptive Data

Convergent Privacy Loss of Noisy-SGD without Convexity and Smoothness

Does Worst-Performing Agent Lead the Pack? Analyzing Agent Dynamics in Unified Distributed SGD

Differential Privacy Regularization: Protecting Training Data Through Loss Function Regularization

The Optimality of (Accelerated) SGD for High-Dimensional Quadratic Optimization

A Dynamic Weighting Strategy to Mitigate Worker Node Failure in Distributed Deep Learning

Increasing Both Batch Size and Learning Rate Accelerates Stochastic Gradient Descent

Asymptotics of Stochastic Gradient Descent with Dropout Regularization in Linear Models