Stochastic Gradient Descent

Stochastic Gradient Descent (SGD) is an iterative optimization algorithm used to find the minimum of a function, particularly useful in machine learning for training large models where computing the exact gradient is computationally prohibitive. Current research focuses on improving SGD's efficiency and convergence properties, exploring variations like Adam, incorporating techniques such as momentum, adaptive learning rates, and line search methods, and analyzing its behavior in high-dimensional and non-convex settings. These advancements are crucial for training complex models like deep neural networks and improving the performance of various machine learning applications, impacting fields ranging from natural language processing to healthcare.

Papers

February 23, 2024

February 19, 2024

February 18, 2024

OptEx: Expediting First-Order Optimization with Approximately Parallelized Iterations
Yao Shu, Jiongfeng Fang, Ying Tiffany He, Fei Richard Yu
Optimization Purpose Stochastic Gradient Descent Gradient Estimation First Order Optimization Iteration Head Gradient Prediction

February 14, 2024

Mean-Field Analysis for Learning Subspace-Sparse Polynomials with Gaussian Input
Ziang Chen, Rong Ge
Stochastic Gradient Descent Mean Field Sparse Polynomial Gaussian Input

February 13, 2024

Implicit Bias in Noisy-SGD: With Applications to Differentially Private Training
Tom Sander, Maxime Sylvestre, Alain Durmus
Financial Application Stochastic Gradient Descent Implicit Bias Differential Privacy SGD Private Model Training

February 11, 2024

Loss Symmetry and Noise Equilibrium of Stochastic Gradient Descent
Liu Ziyin, Mingze Wang, Hongchao Li, Lei Wu
Stochastic Gradient Descent Equilibrium State Gradient Noise Logarithmic Barrier Symmetric Loss

February 10, 2024

Should I try multiple optimizers when fine-tuning pre-trained Transformers for NLP tasks? Should I tune their hyperparameters?
Nefeli Gkouti, Prodromos Malakasiotis, Stavros Toumpis, Ion Androutsopoulos
Stochastic Gradient Descent NLP Task Neural Network Architecture Pre Trained Transformer Related Hyperparameters Better Optimizers Adaptive Optimizers

February 7, 2024

Stochastic Gradient Descent

Papers

Iteration and Stochastic First-order Oracle Complexities of Stochastic Gradient Descent using Constant and Decaying Learning Rates

Dynamic Memory Based Adaptive Optimization

On the Convergence of Gradient Descent for Large Learning Rates

SGD with Clipping is Secretly Estimating the Median Gradient

Training Artificial Neural Networks by Coordinate Search Algorithm

Stochastic Hessian Fittings with Lie Groups

Communication-Efficient Distributed Learning with Local Immediate Error Compensation

Diagonalisation SGD: Fast & Convergent SGD for Non-Differentiable Models via Reparameterisation and Smoothing

OptEx: Expediting First-Order Optimization with Approximately Parallelized Iterations

Mean-Field Analysis for Learning Subspace-Sparse Polynomials with Gaussian Input

Implicit Bias in Noisy-SGD: With Applications to Differentially Private Training

Loss Symmetry and Noise Equilibrium of Stochastic Gradient Descent

Should I try multiple optimizers when fine-tuning pre-trained Transformers for NLP tasks? Should I tune their hyperparameters?

RQP-SGD: Differential Private Machine Learning through Noisy SGD and Randomized Quantization

Convergence of a L2 regularized Policy Gradient Algorithm for the Multi Armed Bandit

How Uniform Random Weights Induce Non-uniform Bias: Typical Interpolating Neural Networks Generalize with Narrow Teachers

AdaBatchGrad: Combining Adaptive Batch Size and Adaptive Step Size

Non-convergence to global minimizers for Adam and stochastic gradient descent optimization and constructions of local minimizers in the training of artificial neural networks

Shadowheart SGD: Distributed Asynchronous SGD with Optimal Time Complexity Under Arbitrary Computation and Communication Heterogeneity

Learning Operators with Stochastic Gradient Descent in General Hilbert Spaces