Stochastic Gradient Descent

Stochastic Gradient Descent (SGD) is an iterative optimization algorithm used to find the minimum of a function, particularly useful in machine learning for training large models where computing the exact gradient is computationally prohibitive. Current research focuses on improving SGD's efficiency and convergence properties, exploring variations like Adam, incorporating techniques such as momentum, adaptive learning rates, and line search methods, and analyzing its behavior in high-dimensional and non-convex settings. These advancements are crucial for training complex models like deep neural networks and improving the performance of various machine learning applications, impacting fields ranging from natural language processing to healthcare.

Papers

February 14, 2023

Stochastic Modified Flows, Mean-Field Limits and Dynamics of Stochastic Gradient Descent
Benjamin Gess, Sebastian Kassing, Vitalii Konarovskyi
Stochastic Gradient Descent Learning Rate Brownian Motion Limit Cycle Mean Field Limit Stochastic Gradient Flow

February 12, 2023

From high-dimensional & mean-field dynamics to dimensionless ODEs: A unifying approach to SGD in two-layers networks
Luca Arnaboldi, Ludovic Stephan, Florent Krzakala, Bruno Loureiro
High Dimensional Stochastic Gradient Descent Mean Field Unifying Theory High Dimensional Regime Two Layer Network Limit Cycle

February 10, 2023

February 8, 2023

February 6, 2023

Easy Learning from Label Proportions
Robert Istvan Busa-Fekete, Heejin Choi, Travis Dick, Claudio Gentile, Andres Munoz medina
Loss Function Stochastic Gradient Descent Debiasing Method Label Proportion Aggregated Label

February 2, 2023

Implicit regularization in Heavy-ball momentum accelerated stochastic gradient descent
Avrajit Ghosh, He Lyu, Xitong Zhang, Rongrong Wang
Gradient Descent Stochastic Gradient Descent Implicit Regularization Heavy Ball

January 31, 2023

Dissecting the Effects of SGD Noise in Distinct Regimes of Deep Learning
Antonio Sclocchi, Mario Geiger, Matthieu Wyart
Deep Learning Deep Neural Network Gradient Descent Stochastic Gradient Descent Mixed Effect Chaotic Regime

January 27, 2023

Algorithmic Stability of Heavy-Tailed SGD with General Loss Functions
Anant Raj, Lingjiong Zhu, Mert Gürbüzbalaban, Umut Şimşekli
Stochastic Gradient Descent Generalization Bound Algorithmic Stability Novel Loss

January 21, 2023

January 19, 2023

An SDE for Modeling SAM: Theory and Insights
Enea Monzio Compagnoni, Luca Biggio, Antonio Orvieto, Frank Norbert Proske, Hans Kersting, Aurelien Lucchi
Stochastic Gradient Descent Real Time Theoretical Understanding DCU Insight AQ Hessian Matrix Sharpness Aware Minimization Human Model Diffusion SDE Continuous Time Model

January 16, 2023

A Robust Classification Framework for Byzantine-Resilient Stochastic Gradient Descent
Shashank Reddy Chirra, Kalyan Varma Nadimpalli, Shrisha Rao
Stochastic Gradient Descent Byzantine Robust Robust Classification Byzantine Seal Byzantine Fault Tolerance Byzantine Fault

January 14, 2023

CEDAS: A Compressed Decentralized Stochastic Gradient Method with Improved Convergence
Kun Huang, Shi Pu
Stochastic Gradient Descent Stochastic Gradient Convergence Rate Convex Objective Decentralized Stochastic Gradient Descent Centralized Stochastic Gradient Descent

January 9, 2023

Fast and Correct Gradient-Based Optimisation for Probabilistic Programming via Smoothing
Basim Khajwal, C. -H. Luke Ong, Dominik Wagner
Stochastic Gradient Descent Variational Inference Posterior Inference Optimization Process Probabilistic Programming Smoothing Factor

January 5, 2023

Training trajectories, mini-batch losses and the curious role of the learning rate
Mark Sandler, Andrey Zhmoginov, Max Vladymyrov, Nolan Miller
Loss Function Gradient Descent Stochastic Gradient Descent Integral Role Learning Rate Mini Batch Stochastic Weight Averaging Training Trajectory Gradient Descent Trajectory

January 4, 2023

On the Convergence of Stochastic Gradient Descent in Low-precision Number Formats
Matteo Cacciola, Antonio Frangioni, Masoud Asgharian, Alireza Ghaffari, Vahid Partovi Nia
Stochastic Gradient Descent Early Stage Convergence Low Precision

January 2, 2023

GPU accelerated matrix factorization of large scale data using block based approach
Prasad Bhavana, Vineet Padmanabhan
Stochastic Gradient Descent Constructive Approach Matrix Factorization Single GPU Prime Factorization Multi Modal Outer Arithmetic Block Large Scale Data

Stochastic Gradient Descent

Papers

Stochastic Modified Flows, Mean-Field Limits and Dynamics of Stochastic Gradient Descent

From high-dimensional & mean-field dynamics to dimensionless ODEs: A unifying approach to SGD in two-layers networks

Cyclic and Randomized Stepsizes Invoke Heavier Tails in SGD than Constant Stepsize

Achieving acceleration despite very noisy gradients

On the Convergence of Stochastic Gradient Descent for Linear Inverse Problems in Banach Spaces

Unsupervised Learning of Initialization in Deep Neural Networks via Maximum Mean Discrepancy

DoG is SGD's Best Friend: A Parameter-Free Dynamic Step Size Schedule

Easy Learning from Label Proportions

Implicit regularization in Heavy-ball momentum accelerated stochastic gradient descent

Dissecting the Effects of SGD Noise in Distinct Regimes of Deep Learning

Algorithmic Stability of Heavy-Tailed SGD with General Loss Functions

Genetically Modified Wolf Optimization with Stochastic Gradient Descent for Optimising Deep Neural Networks

ScaDLES: Scalable Deep Learning over Streaming data at the Edge

An SDE for Modeling SAM: Theory and Insights

A Robust Classification Framework for Byzantine-Resilient Stochastic Gradient Descent

CEDAS: A Compressed Decentralized Stochastic Gradient Method with Improved Convergence

Fast and Correct Gradient-Based Optimisation for Probabilistic Programming via Smoothing

Training trajectories, mini-batch losses and the curious role of the learning rate

On the Convergence of Stochastic Gradient Descent in Low-precision Number Formats

GPU accelerated matrix factorization of large scale data using block based approach