Stochastic Gradient Descent
Also known as SGD
Overview
Stochastic Gradient Descent (SGD) updates model parameters using gradients from individual examples or mini-batches, making optimization practical when full-dataset gradients are too costly. Research studies convergence and generalization, and develops variants such as momentum, adaptive learning rates, clipping, variance reduction, and decentralized or federated updates. Analyses span convex and nonconvex objectives, neural-network dynamics, heavy-tailed or dependent noise, and statistical inference from training trajectories. A central tension is that gradient noise can improve exploration and favor flatter solutions, yet can also destabilize training or slow convergence, so its effects depend on batch size, learning-rate schedule, model geometry, and data conditions.
Latest papers
252 papers28 papers in the last four weeks, down 33% on the four weeks before. 0.2% of all new papers.
Taming the Tails of Stochastic Gradient Descent in the Interpolation Regime
Nearly Tight Almost Sure Convergence Rates for the Last iterate of SGD
An inexact stochastic variance-reduced gradient method for constrained convex finite-sum optimization
LACCO: Local Accumulation with Computation--Communication Overlap
When and How SignSGD Outperforms SGD: A Functional Scaling Law Perspective
Improved last-iterate bounds for SGD with and without replacement
Settling the Burn-in Gap: Sharp Risk Bounds for Tail-Averaged SGD in Linear Regression
Learning Rate Decay Can Exponentially Accelerate SGD for Global Optimization of Nonconvex Functions
Orthogonal Updates for the Win: Towards Accelerated Adaptive Minimax Optimization
Full Fine-Tuning is not the Problem: Why Adam Fails and SGD Succeeds in Few-Shot CLIP Adaptation
The Multiscale Single-Index Model: An MSE Analysis of Online SGD Dynamics
On the Few-Shot Adaptation of Vision-Language Models: A Few Gradient Signs Are All You Need
Compressed Per-Player Local Methods for Multiplayer Federated Learning: Exact Convergence via Increment Sharing
Last-Iterate Convergence of Stochastic Gradient Methods for Unbounded Problems
Stepsize modulation for cross entropy from local quadratic optimization
ZEMA: confidence-calibrated signed descent with only one state
Lower Bounds on Inference Complexity for Polyak-Ruppert Averaged Stochastic Gradient Descent
Stochastic learning flow: moment equations for stochastic gradient descent at finite step size
Exact Theory and Optimal Stepsizes for Stochastic Gradient Descent in Smooth Convex Minimization
Variance Reduction for Decentralized Stochastic Optimization with Multiple Local Updates
Where, When, and How Many Bits? Scaling Laws for SGD with Stochastic Rounding
Implicit Bias of (Stochastic) Gradient Descent on Deep Linear Diagonal Networks
Rennala NSGD: Asynchronous Stochastic Optimization Beyond Euclidean Geometry
B2P-Corr: Batch-to-Population Gradient Estimators for Non-Decomposable Correlation Losses
batch-to-population'' (B2P) gap, we propose B2P-Corr, a novel framework that transforms correlation objectives into decomposable pseudo-losses driven by lightweight global moment sketches''. We introduce two efficient variants: B2P-EMA, which utilizes exponential moving averages and stop-gradient operators to ensure asymptotic gradient tracking with cost ( is batch size); and B2P-U, which uses sampled cross-batch U-statistic estimators and a reservoir-based stabilization with an implementation. Theoretically, we quantify the bias of naive mini-batch optimization, prove a moving-target tracking result for B2P-EMA under a fast-sketch / slow-parameter regime, and provide an idealized control-variate analysis for B2P-U. Experiments on two synthetic and three real-world datasets demonstrate B2P-Corr achieves stable optimization and higher correlation than naive mini-batch optimization with negligible overhead.