Stochastic Gradient Descent

Stochastic Gradient Descent (SGD) is an iterative optimization algorithm used to find the minimum of a function, particularly useful in machine learning for training large models where computing the exact gradient is computationally prohibitive. Current research focuses on improving SGD's efficiency and convergence properties, exploring variations like Adam, incorporating techniques such as momentum, adaptive learning rates, and line search methods, and analyzing its behavior in high-dimensional and non-convex settings. These advancements are crucial for training complex models like deep neural networks and improving the performance of various machine learning applications, impacting fields ranging from natural language processing to healthcare.

Papers

August 1, 2022

Dynamic Batch Adaptation
Cristian Simionescu, George Stoica, Robert Herscovici
Stochastic Gradient Descent Learning Rate Model Convergence Adaptive Batch Size Adaptive Optimization Method

July 31, 2022

Formal guarantees for heuristic optimization algorithms used in machine learning
Xiaoyu Li
Machine Learning Stochastic Gradient Descent Stochastic Gradient Stochastic Convex Optimization Formal Guarantee Large Scale Optimization Optimization Heuristic

July 28, 2022

One-Pass Learning via Bridging Orthogonal Gradient Descent and Recursive Least-Squares
Youngjae Min, Kwangjun Ahn, Navid Azizan
Deep Learning LeArning Abstract Stochastic Gradient Descent Least Square Orthogonal Gradient Orthogonal Parameterization

July 27, 2022

Learning with Combinatorial Optimization Layers: a Probabilistic Approach
Guillaume Dalle, Léo Baty, Louis Bouvier, Axel Parmentier
LeArning Abstract Stochastic Gradient Descent Probabilistic Approach

July 20, 2022

Adaptive Step-Size Methods for Compressed SGD
Adarsh M. Subramaniam, Akshayaa Magesh, Venugopal V. Veeravalli
Stochastic Gradient Descent Adaptive Step Size Compressed Stochastic Gradient Descent

July 19, 2022

Moment Centralization based Gradient Descent Optimizers for Convolutional Neural Networks
Sumanth Sadu, Shiv Ram Dubey, SR Sreeja
Convolutional Neural Network Stochastic Gradient Descent Momentum Based Centralized Stochastic Gradient Descent Gradient Descent Optimizers Moment Localization

July 16, 2022

Adaptive Sketches for Robust Regression with Importance Sampling
Sepideh Mahabadi, David P. Woodruff, Samson Zhou
Stochastic Gradient Descent Importance Sampling Robust Regression Standard Gradient Descent Sketch and Lift

July 13, 2022

Towards understanding how momentum improves generalization in deep learning
Samy Jelassi, Yuanzhi Li
Deep Learning Strong Generalization Gradient Descent Stochastic Gradient Descent Residual Momentum Novel Generalization

July 12, 2022

Conservative SPDEs as fluctuating mean field limits of stochastic gradient descent
Benjamin Gess, Rishabh S. Gvalani, Vitalii Konarovskyi
Stochastic Gradient Descent Mean Field Stochastic Partial Differential Equation Mean Field Limit

July 11, 2022

July 9, 2022

Improved Binary Forward Exploration: Learning Rate Scheduling Method for Stochastic Optimization
Xin Cao
Gradient Descent Stochastic Gradient Descent Learning Rate Gradient Based Stochastic Optimization Binary Forward Exploration

July 6, 2022

The alignment property of SGD noise and how it helps select flat minima: A stability analysis
Lei Wu, Mingze Wang, Weijie Su
Stochastic Gradient Descent Implicit Regularization Stability Analysis Flat Minimum Better Alignment Square Loss

July 3, 2022

Distributed Online System Identification for LTI Systems Using Reverse Experience Replay
Ting-Jui Chang, Shahin Shahrampour
Stochastic Gradient Descent System Identification Experience Replay Linear Time Invariant System

July 1, 2022

Analysis of Kinetic Models for Label Switching and Stochastic Gradient Descent
Martin Burger, Alex Rossi
General Analysis Stochastic Gradient Descent Gradient Flow Label Shift Particle System Energy Landscape Kinetic Model

June 25, 2022

Topology-aware Generalization of Decentralized SGD
Tongtian Zhu, Fengxiang He, Lan Zhang, Zhengyang Niu, Mingli Song, Dacheng Tao
Stochastic Gradient Descent Vanilla Gradient Descent Decentralized SGD Decentralized Stochastic Gradient Descent Topological Generalization

June 21, 2022

Stochastic Gradient Descent

Papers

Dynamic Batch Adaptation

Formal guarantees for heuristic optimization algorithms used in machine learning

One-Pass Learning via Bridging Orthogonal Gradient Descent and Recursive Least-Squares

Learning with Combinatorial Optimization Layers: a Probabilistic Approach

Adaptive Step-Size Methods for Compressed SGD

Moment Centralization based Gradient Descent Optimizers for Convolutional Neural Networks

Adaptive Sketches for Robust Regression with Importance Sampling

Towards understanding how momentum improves generalization in deep learning

Conservative SPDEs as fluctuating mean field limits of stochastic gradient descent

UrduFake@FIRE2021: Shared Track on Fake News Identification in Urdu

Stochastic Gradient Descent and Anomaly of Variance-flatness Relation in Artificial Neural Networks

On uniform-in-time diffusion approximation for stochastic gradient descent

Improved Binary Forward Exploration: Learning Rate Scheduling Method for Stochastic Optimization

The alignment property of SGD noise and how it helps select flat minima: A stability analysis

Distributed Online System Identification for LTI Systems Using Reverse Experience Replay

Analysis of Kinetic Models for Label Switching and Stochastic Gradient Descent

Topology-aware Generalization of Decentralized SGD

On the Maximum Hessian Eigenvalue and Generalization

sqSGD: Locally Private and Communication Efficient Federated Learning

A General Theory for Federated Optimization with Asynchronous and Heterogeneous Clients Updates