Stochastic Gradient Descent

Also known as SGD

Overview

Stochastic Gradient Descent (SGD) updates model parameters using gradients from individual examples or mini-batches, making optimization practical when full-dataset gradients are too costly. Research studies convergence and generalization, and develops variants such as momentum, adaptive learning rates, clipping, variance reduction, and decentralized or federated updates. Analyses span convex and nonconvex objectives, neural-network dynamics, heavy-tailed or dependent noise, and statistical inference from training trajectories. A central tension is that gradient noise can improve exploration and favor flatter solutions, yet can also destabilize training or slow convergence, so its effects depend on batch size, learning-rate schedule, model geometry, and data conditions.

Latest papers

CardsList

252 papers28 papers in the last four weeks, down 33% on the four weeks before. 0.2% of all new papers.

  1. Convergence of Stochastic Gradient Methods for Wide Two-Layer Physics-Informed Neural Networks for the Poisson Equation

    Aug 29, 2025Bangti Jin, Longjun WuNeural PDE SolversPDE Solving

  2. Better Convergence Guarantees for Sign-Based Momentum Methods

    Jul 16, 2025Wei Jiang, Dingzhi Yu, Sifan Yang +3Sign-Based OptimizationMomentum Methods

  3. On the Effectiveness of the z-Transform Method in Quadratic Optimization

    Jul 4, 2025Francis BachOptimization Convergence AnalysisGradient Descent Dynamics

  4. Scaling Textual Gradients via Sampling-Based Momentum

    May 31, 2025Zixin Ding, Junyuan Hong, Zhan Shi +6Large Language Model-Guided OptimizationPrompt Optimization

  5. Private Rate-Constrained Optimization with Applications to Fair Learning

    May 28, 2025Mohammad Yaghini, Tudor Cebere, Michael Menart +2Primal-Dual OptimizationGroup Fairness

  6. Bias-Optimal Bounds for Stochastic Gradient Descent

    May 23, 2025Daniel Cortild, Lucas Ketels, Juan Peypouquet +1Stochastic Gradient DescentStochastic Optimization Convergence

  7. Online simultaneous inference for quantiles via smoothed stochastic gradient descent

    May 19, 2025Likai Chen, Georg Keilbar, Wei Biao WuQuantile RegressionStochastic Gradient Descent

  8. Projected gradient methods for nonconvex and stochastic smooth optimization: new complexities and auto-conditioned stepsizes

    Dec 18, 2024Guanghui Lan, Tianjiao Li, Yangyang XuGradient ProjectionNonconvex Optimization

  9. Learning rate adaptive stochastic gradient descent optimization methods: numerical simulations for deep learning methods for partial differential equations and convergence analyses

    Jun 20, 2024Steffen Dereich, Arnulf Jentzen, Adrian RiekertAdaptive Learning RateStochastic Optimization Convergence

  10. The Limits and Potentials of Local SGD for Distributed Heterogeneous Learning with Intermittent Communication

    May 19, 2024Kumar Kshitij Patel, Margalit Glasgow, Ali Zindari +5Communication-Efficient Distributed TrainingDistributed Optimization