Stochastic Gradient Descent

Also known as SGD

Overview

Stochastic Gradient Descent (SGD) updates model parameters using gradients from individual examples or mini-batches, making optimization practical when full-dataset gradients are too costly. Research studies convergence and generalization, and develops variants such as momentum, adaptive learning rates, clipping, variance reduction, and decentralized or federated updates. Analyses span convex and nonconvex objectives, neural-network dynamics, heavy-tailed or dependent noise, and statistical inference from training trajectories. A central tension is that gradient noise can improve exploration and favor flatter solutions, yet can also destabilize training or slow convergence, so its effects depend on batch size, learning-rate schedule, model geometry, and data conditions.

Latest papers

CardsList

252 papers28 papers in the last four weeks, down 33% on the four weeks before. 0.2% of all new papers.

  1. When Momentum Helps Polyak--Ruppert Averaging

    Oct 20, 2026Alexandra Suvorikova, Darina Dvinskikh, Yuriy Dorn +1Stochastic OptimizationStochastic Optimization Convergence

  2. Near-Optimal Last-Iterate Convergence of SGD in the Smooth Interpolation Regime

    Oct 18, 2026Matan Schliserman, Tomer KorenSmooth Convex OptimizationLast-Iterate Convergence

  3. Understanding Stability of Outer Nesterov Momentum on Stochastic Pseudo-Gradients

    Oct 15, 2026Nishant Sharma, Laura Gomezjurado Gonzalez, Ganesh Talluri +2Neural Network StabilityNeural Network Training Dynamics

  4. Pairwise AUC Optimization Needs Corrective Power: A Unified View

    Oct 15, 2026Jia Chen, Zhiyong Yang, Shilong Bao +2U-StatisticsLearning to Rank

  5. Personalized Projected Ascent for Federated Stackelberg Optimization

    Oct 15, 2026M Yashwanth, Sharannya Ghosh, Saikiran Bulusu +1Federated LearningStochastic Gradient Descent

  6. Stochastic Gradient Descent: A Nonlinear Time Series Perspective

    Oct 14, 2026Jiaqi Li, Zhipeng Lou, Stefan Richter +1Richardson ExtrapolationStochastic Gradient Descent

  7. Memory-Efficient Subspace Optimization with Residual Correction: From Momentum SGD to Muon

    Oct 14, 2026Sifan Yang, Hongyi Tao, Dingzhi Yu +5Subspace OptimizationMemory-Efficient Training

  8. Large-Step Last-Iterate Convergence of Random-Reshuffling Stochastic Gradient Descent

    Oct 14, 2026Zijun Chen, Zihan ZhangStochastic OptimizationLast-Iterate Convergence

  9. Generalization Analysis of Biased Stochastic Gradient Methods for Minimax Problems

    Oct 14, 2026Shuang Zeng, Yunwen Lei, Yiming YingMin-Max OptimizationGeneralization Bounds

  10. How Inner Optimizers Shape Replica Scaling

    Oct 13, 2026Eugene Belilovsky, Stefan Horoi, Edouard Oyallon +1LLM TrainingDistributed Training

  11. Last-Iterate Convergence of Clipped SGD with Linear Decay

    Oct 13, 2026Wei Jiang, Haomin Bai, Zechao LiLast-Iterate ConvergenceStochastic Gradient Descent

  12. From Parallel to Sequential: Small-Batch Noise Restores Simplicity Bias of Muon

    Oct 12, 2026Tom Jacobs, Rebekka BurkholzMuon OptimizerNonconvex Optimization

  13. Feature Normalization under Finite Training Budgets

    Oct 12, 2026Johan LarssonStatistical Learning TheoryStochastic Gradient Descent

  14. Asymptotic Risk of Learning-Rate Schedules for Least-Squares Regression with Discretely Distributed Features

    Oct 12, 2026Noé Colomban, Francis Bach, Fabian SchaippLearning Rate SchedulingStochastic Gradient Descent

  15. A Nonlinear Time-Series Theory for Constant-step SGD with Momentum

    Oct 12, 2026Younghun Lee, Jake Zummo, Wei Biao WuSGD with MomentumStochastic Gradient Descent

  16. Probabilistic Multi-Source Data Selection via Learnable Categorical Priors: Variance Reduction and Guaranteed Derandomization

    Oct 10, 2026Shizhan Cai, Qixin Zhang, Siyuan Ma +3Data SelectionBilevel Optimization

  17. Formalizing Forgetting: Theory of Forgetting for Online Continual Learning

    Oct 10, 2026Renjie Cao, Yifei Cheng, Li ShenOnline Continual LearningCatastrophic Forgetting

  18. Rare Gate Disagreements Can Limit Plasticity: When Gradient Flow Mispredicts Finite-Batch SGD

    Oct 8, 2026Ruoyu Zhao, Mingxuan Zhang, Jianbo Dai +3Neural Network Training DynamicsStochastic Gradient Descent

  19. Decentralized SGD under Heavy-Tailed Noise: Optimal Convergence Rates and the Role of Gradient Clipping

    Oct 7, 2026Aleksandar Armacki, Haoyuan Cai, Ali H. SayedDecentralized OptimizationStochastic Gradient Descent

  20. A Solvable Model of Adaptive Learning Rate Rescaling: Acceleration, Stability & Scaling

    Oct 5, 2026Itay Lavie, Clarissa Lauditi, Cengiz PehlevanEdge of StabilityNeural Scaling Laws

  21. Finite-Sample Distribution Theory and Efficient Large-Scale Inference for Online Quantile Regression

    Oct 5, 2026Ziyang Wei, Jiaqi Li, Lan Wang +1Quantile RegressionEfficient Inference