Stochastic Gradient Descent
Also known as SGD
Overview
Stochastic Gradient Descent (SGD) updates model parameters using gradients from individual examples or mini-batches, making optimization practical when full-dataset gradients are too costly. Research studies convergence and generalization, and develops variants such as momentum, adaptive learning rates, clipping, variance reduction, and decentralized or federated updates. Analyses span convex and nonconvex objectives, neural-network dynamics, heavy-tailed or dependent noise, and statistical inference from training trajectories. A central tension is that gradient noise can improve exploration and favor flatter solutions, yet can also destabilize training or slow convergence, so its effects depend on batch size, learning-rate schedule, model geometry, and data conditions.
Latest papers
252 papers28 papers in the last four weeks, down 33% on the four weeks before. 0.2% of all new papers.
Uniform-in-time concentration in two-layer neural networks via transportation inequalities
Generalization Bounds of Stochastic Gradient Descent in Homogeneous Neural Networks
Neural network optimization strategies and the topography of the loss landscape
On the Convergence of Stochastic Gradient Descent with Perturbed Forward-Backward Passes
Stability and Generalization of Push-Sum Based Decentralized Optimization over Directed Graphs
Exponential Convergence of (Stochastic) Gradient Descent for Separable Logistic Regression
Interactive Learning of Single-Index Models via Stochastic Gradient Descent
burn-in'' phase before entering the learning'' phase in this setting. Moreover, with an appropriately chosen learning rate schedule, a single SGD procedure simultaneously achieves near-optimal (or best-known) sample complexity and regret guarantees across both phases, for a broad class of link functions. Our results demonstrate that SGD remains highly competitive for learning single-index models under adaptive data.