Stochastic Gradient Descent

Also known as SGD

Overview

Stochastic Gradient Descent (SGD) updates model parameters using gradients from individual examples or mini-batches, making optimization practical when full-dataset gradients are too costly. Research studies convergence and generalization, and develops variants such as momentum, adaptive learning rates, clipping, variance reduction, and decentralized or federated updates. Analyses span convex and nonconvex objectives, neural-network dynamics, heavy-tailed or dependent noise, and statistical inference from training trajectories. A central tension is that gradient noise can improve exploration and favor flatter solutions, yet can also destabilize training or slow convergence, so its effects depend on batch size, learning-rate schedule, model geometry, and data conditions.

Latest papers

CardsList

252 papers28 papers in the last four weeks, down 33% on the four weeks before. 0.2% of all new papers.

  1. Broken scale symmetries in undercomplete linear autoencoders

    Oct 2, 2026Farhad Pashakhanloo, Jacob A. Zavatone-VethNeural Network Training DynamicsSymmetry Breaking

  2. Learning the identity: a case study of how SGD selects among functional decompositions

    Sep 30, 2026Andy Arditi, Weian Xie, David Bau +1Deep Linear NetworksImplicit Bias

  3. From Spectra to Joint Schedules in LLM Pre-training: 3+3(+2) Scaling-Law Regimes

    Sep 30, 2026Yichen Wang, Fanghui Liu, Yudong ChenLanguage Model PretrainingDeep Learning Optimization

  4. Robustifying Asynchronous SGD via Soft Throttling

    Sep 30, 2026Kaoru Otsuka, Maxime Meyer, Yuki Takezawa +2Distributed OptimizationFault-Tolerant Distributed Training

  5. AutoLoCo: Communication Efficient Distributed LLM Training via Adaptive Synchronization

    Sep 29, 2026Pengyu He, Yan Zhang, Ruien Li +1Communication-Efficient Distributed TrainingDistributed Training

  6. Stochastic Optimization Under Power-Law Spectra: Tight Bounds and Shuffling Analysis

    Sep 28, 2026Christopher De Sa, Thomas Dybdahl Ahle, Yaroslav Bulatov +1Random ReshufflingStochastic Optimization Convergence

  7. Clipped or Unclipped? Finite-Sample Trade-offs for Averaged SGD under Heavy-Tailed Noise

    Sep 26, 2026Alexandra Suvorikova, Egor Gladin, Darina Dvinskikh +5Polyak-Ruppert AveragingStochastic Optimization

  8. Muon Under Gradient Noise and the Limits of Orthogonalization Near Optima

    Sep 26, 2026Xiaohui XieMuon OptimizerGradient Orthogonalization

  9. High-Probability Guarantees for SGD under ββ-Heavy-Tailed Gradient Noise

    Sep 26, 2026Qijun Tong, Masahiro Ikeda, Ryota KawasumiStochastic Gradient DescentStochastic Optimization Convergence

  10. Precise Convergence Speed of Clipped SGD

    Sep 24, 2026David A. R. RobinGradient ClippingNonconvex Stochastic Optimization

  11. Resource-Adaptive Stochastic Gradient Descent for Online Linear Programming without Re-solving

    Sep 23, 2026Jiameng LyuOnline Convex OptimizationOnline LP

  12. Optimization Geometry of Equivalent Brownian RKHS Representations

    Sep 18, 2026Mahdi Mohammadigohari, Gustau Camps-VallsReproducing Kernel Hilbert SpacesStochastic Gradient Descent

  13. Same Flow, Different Paths: Variance Reduction in Flow Matching

    Sep 15, 2026Alexander TyurinFlow MatchingStochastic Optimization Convergence

  14. Quantile-based Loss Filtering for Outlier-Robust Stochastic Gradient Descent

    Sep 11, 2026Jamie Haddock, Anna Ma, Elizaveta RebrovaStochastic OptimizationStochastic Optimization Convergence

  15. Stochastic Gradient Descent over P2

    Sep 11, 2026Maria Oprea, Qin Li, Yunan YangStochastic OptimizationStochastic Gradient Descent

  16. Convergence of Stochastic Gradient Methods under Heavy-Tailed Noise and Hölder Smoothness

    Sep 11, 2026Misbah Uz Zaman, Anirbit MukherjeeGradient ClippingNonconvex Stochastic Optimization

  17. High-Probability Convergence of SGD via Batched Updates

    Sep 11, 2026Feng Zhu, Robert W. Heath, Aritra MitraCommunication-Efficient Distributed TrainingStochastic Optimization Convergence

  18. Almost Sure Convergence Analysis of Stochastic Gradient Methods with Clipping and Additive Noise

    Sep 10, 2026Amartya Mukherjee, Jun LiuStochastic OptimizationGradient Clipping

  19. The Exact Time-Uniform Rate Frontier for Stochastic Gradient Descent on Smooth Convex Objectives

    Sep 8, 2026Ruijie Li, Kang Chen, Tianyu WangLearning Rate SchedulingLast-Iterate Convergence

  20. Speed limits for information acquisition in stochastic learning dynamics

    Sep 8, 2026Shuta Kobayashi, Andreas DechantNeural Network Training DynamicsInformation-Theoretic Lower Bounds

  21. SGD in Multiclass Logistic Regression: Sequential Learning and Scaling Laws

    Sep 7, 2026Konstantinos Christopher Tsiolis, Denny Wu, Christos Thrampoulidis +1Logistic RegressionMulticlass Classification

  22. The Role of Gradient Modification in Heavy-Tailed Nonconvex Stochastic Min-Max Optimization

    Sep 5, 2026Tianxi Zhu, Yi Xu, Xiangyang JiMin-Max OptimizationStochastic Optimization Convergence

  23. Percolation Dynamics in Optimization: Variance Cascades and Nested Symmetry

    Sep 2, 2026Sai Niranjan Ramachandran, Suvrit SraPhase Transitions in Neural NetworksStochastic Gradient Descent

  24. Beyond Optimal Rates in Stochastic Optimization: Trajectory-Adaptive Stopping Rules

    Aug 26, 2026Liviu Aolaritei, Lucas Lévy, Francis Bach +1Stochastic OptimizationStochastic Gradient Descent