Stochastic Gradient Descent

Also known as SGD

Follow this topic Ask about this topic

Momentum

21 papers in the last four weeks, up 600% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 132

All topics

Rare Gate Disagreements Can Limit Plasticity: When Gradient Flow Mispredicts Finite-Batch SGD
Oct 8, 2026Ruoyu Zhao, Mingxuan Zhang, Jianbo Dai +3Neural Network Training Dynamics Stochastic Gradient Descent
City University of Hong Kong · Microsoft · Copula Lab +1
Decentralized SGD under Heavy-Tailed Noise: Optimal Convergence Rates and the Role of Gradient Clipping
Oct 7, 2026Aleksandar Armacki, Haoyuan Cai, Ali H. SayedDecentralized Optimization Stochastic Gradient Descent
Ecole Polytechnique Fédérale de Lausanne, Lausanne, Switzerland
Stochastic Gradient Descent Ascent is Suboptimal for Nonconvex-PL Min-Max Games
Oct 6, 2026Junsoo HaOptimization Convergence Analysis Min-Max Optimization
A Solvable Model of Adaptive Learning Rate Rescaling: Acceleration, Stability & Scaling
Oct 5, 2026Itay Lavie, Clarissa Lauditi, Cengiz PehlevanEdge of Stability Neural Scaling Laws
John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, MA · Center for Brain Science, Harvard University, Cambridge, MA · Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University, Cambridge, MA
Finite-Sample Distribution Theory and Efficient Large-Scale Inference for Online Quantile Regression
Oct 5, 2026Ziyang Wei, Jiaqi Li, Lan Wang +1Quantile Regression Efficient Inference
Department of Statistics, University of Chicago · Department of Statistics, Rice University · Department of Management Science, University of Miami
Gaussian Limits for SGD Without Stationary Moments
Oct 4, 2026Xiaoli Li, Wei Biao WuStochastic Optimization Convergence Stochastic Gradient Descent
University of Chicago
Moment-Accurate Gaussian Mixtures for Constant-Step Stochastic Approximation
Oct 4, 2026Xiaoli Li, Wei Biao WuGaussian Mixture Models Stochastic Approximation
University of Chicago
Learning the identity: a case study of how SGD selects among functional decompositions
Sep 30, 2026Andy Arditi, Weian Xie, David Bau +1Deep Linear Networks Implicit Bias
Northeastern University · Massachusetts Institute of Technology
Exact information accounting for SGD methods
Sep 30, 2026Akshay BalsubramaniNeural Network Generalization Stochastic Optimization Convergence
From Spectra to Joint Schedules in LLM Pre-training: 3+3(+2) Scaling-Law Regimes
Sep 30, 2026Yichen Wang, Fanghui Liu, Yudong ChenLanguage Model Pretraining Deep Learning Optimization
Department of Computer Sciences, University of Wisconsin–Madison, USA. · School of Mathematical Sciences, Institute of Natural Sciences and MOE-LSC, Shanghai Jiao Tong University, China.
Robustifying Asynchronous SGD via Soft Throttling
Sep 30, 2026Kaoru Otsuka, Maxime Meyer, Yuki Takezawa +2Distributed Optimization Fault-Tolerant Distributed Training
Okinawa Institute of Science and Technology · National University of Singapore · Toyota Motor Corporation +1
Sharp Stationary Gaussian Approximation for Constant-Stepsize SGD
Sep 30, 2026Junghoon SeoStochastic Approximation Stochastic Gradient Descent
PIT IN Corp., South Korea
AutoLoCo: Communication Efficient Distributed LLM Training via Adaptive Synchronization
Sep 29, 2026Pengyu He, Yan Zhang, Ruien Li +1Communication-Efficient Distributed Training Distributed Training
Department of Electrical Engineering, California Institute of Technology · Department of Computer Science and Technology, Tsinghua University · Department of Computer Sciences, University of Wisconsin - Madison
Ordinary Nonconvex SGD under Distance-Dependent Moments: Finite-Horizon Stationarity and Nagaev Bounds
Sep 24, 2026Wei Biao WuNonconvex Stochastic Optimization Stochastic Optimization Convergence
Precise Convergence Speed of Clipped SGD
Sep 24, 2026David A. R. RobinGradient Clipping Nonconvex Stochastic Optimization
LAMSADE, Université Paris-Dauphine, PSL Research University
Resource-Adaptive Stochastic Gradient Descent for Online Linear Programming without Re-solving
Sep 23, 2026Jiameng LyuOnline Convex Optimization Online Resource Allocation
Department of Management Science, School of Management, Fudan University, Shanghai 200433, China
Continuous Delayed-Memory Stochastic Gradient Descent and Continuous-Time Reinforcement Learning from History of Astrophysical Time Series Studies
Sep 17, 2026Juncheng YiContinuous-Time RL Policy Gradient Methods
Same Flow, Different Paths: Variance Reduction in Flow Matching
Sep 15, 2026Alexander TyurinFlow Matching Stochastic Optimization Convergence
AXXX, Moscow, Russia · Applied AI Institute, Moscow, Russia
High-Probability Convergence of SGD via Batched Updates
Sep 14, 2026Feng Zhu, Robert W. Heath Jr., Aritra MitraCommunication-Efficient Distributed Training Stochastic Optimization Convergence
Convergence of Stochastic Gradient Methods under Heavy-Tailed Noise and H"{o}lder Smoothness
Sep 14, 2026Misbah Uz Zaman, Anirbit MukherjeeGradient Clipping Nonconvex Stochastic Optimization
Department of Mathematics and Statistics Indian Institute of Science Education and Research Kolkata · Department of Computer Science The University of Manchester
Quantile-based Loss Filtering for Outlier-Robust Stochastic Gradient Descent
Sep 14, 2026Jamie Haddock, Anna Ma, Elizaveta RebrovaStochastic Optimization Stochastic Optimization Convergence
Department of Mathematics, Harvey Mudd College, Claremont, CA 91711-3116, USA. · Department of Mathematics, University of California, Irvine, CA 92617, USA. · Mathematical Institute, University of Oxford, Oxford, OX1 2JD, UK.
Stochastic Gradient Descent over P2
Sep 11, 2026Maria Oprea, Qin Li, Yunan YangStochastic Optimization Stochastic Gradient Descent
Center for Applied Mathematics, Cornell University · Department of Mathematics, University of Wisconsin–Madison, USA · Department of Mathematics, Cornell University, USA
Almost Sure Convergence Analysis of Stochastic Gradient Methods with Clipping and Additive Noise
Sep 10, 2026Amartya Mukherjee, Jun LiuStochastic Optimization Gradient Clipping
Department of Applied Mathematics, University of Waterloo, Waterloo, Ontario, Canada N2L 3G1
The Exact Time-Uniform Rate Frontier for Stochastic Gradient Descent on Smooth Convex Objectives
Sep 8, 2026Ruijie Li, Kang Chen, Tianyu WangLearning Rate Scheduling Last-Iterate Convergence
School of Data Science, Fudan University · Shanghai Center for Mathematical Sciences, Fudan University
Speed Limit for Information Acquisition in Stochastic Learning Dynamics
Sep 8, 2026Shuta Kobayashi, Andreas DechantNeural Network Training Dynamics Information-Theoretic Lower Bounds
Department of Physics, Kyoto University, Kyoto 606-8502, Japan
SGD in Multiclass Logistic Regression: Sequential Learning and Scaling Laws
Sep 7, 2026Konstantinos Christopher Tsiolis, Denny Wu, Christos Thrampoulidis +1Logistic Regression Multiclass Classification
University of Toronto, Vector Institute · New York University, Flatiron Institute · University of British Columbia +1
Percolation Dynamics in Optimization: Variance Cascades and Nested Symmetry
Sep 2, 2026Sai Niranjan Ramachandran, Suvrit SraStochastic Gradient Descent Adaptive Gradient Methods
School of Computation, Information and Technology, Technical University of Munich, Germany · Munich Center for Machine Learning (MCML)
Adaptive Bregman Proximal Stochastic Gradient with a Stabilized Barzilai--Borwein Step Size
Aug 12, 2026Chenhan Jin, Shengze Xu, Binghui Xie +4Nonconvex Stochastic Optimization Adaptive Gradient Methods
1The Chinese University of Hong Kong · University of Utah · 3Beijing Normal-Hong Kong Baptist University +1
Generalized Convexity and Smoothness via Conjugate Duality: Optimization Theory for Deep Neural Networks
Aug 10, 2026Binchuan QiGradient Descent Neural Network Optimization
College of Electronics and Information Engineering Tongji University Shanghai, 200092, China
Stochastic gradient descent with discontinuity across a manifold
Aug 7, 2026Vivek S. BorkarStochastic Approximation Gradient Descent Dynamics
Department of Electrical Engineering, Indian Institute of Technology Bombay, Powai, Mumbai 400076, INDIA.

Oct 8, 2026, cs.LGJ/K move · Enter open · S save

Rare Gate Disagreements Can Limit Plasticity: When Gradient Flow Mispredicts Finite-Batch SGD

Ruoyu Zhao, Mingxuan Zhang, Jianbo Dai, Jiaqi Wu+2

City University of Hong Kong · Microsoft · Copula Lab+1

Neural Network Training Dynamics Stochastic Gradient Descent Gradient Descent Dynamics ReLU Neural Networks

Open paper AskPDF arXiv

Population gradient flow is a common tool for reasoning about how neural networks adapt, including after pretraining. We show that it can mispredict finite-batch stochastic gradient descent (SGD) qualitatively, and we trace the discrepancy to a specific mechanism. In a two-unit ReLU regression, a source task drives the two neurons toward positive proportionality and a target task rewards separating them. After source training for time

T

, gradient flow recovers on the target in time linear in

T

. Online SGD with batch size

b

and step size

η

in both phases instead fails with high probability throughout a horizon of order

e^{c/η}

once

T \gtrsim \log(b/η)

, uniformly on an explicit set of initializations with Gaussian probability above one percent. For each fixed

T

, small-step SGD still recovers, so the failure requires the joint limit of small steps and long pretraining. At the target clone, the population instability is carried entirely by inputs on which the two ReLU gates disagree. For units at angle

δ

these inputs form a wedge of probability

δ/π

, and weight decay shrinks the angle exponentially during pretraining. On every other input both units receive the same random linear update, which contracts their separation in conditional expectation. Bounding the cumulative probability of sampling the wedge along the exact online recursion, without a diffusion approximation, shows that recovery with fixed probability from an identical source-gradient-flow checkpoint, within

e^{c/η}

updates, requires

Nb \gtrsim e^{λT}

target samples and batch size

b \gtrsim ηe^{λT}

, where

N

counts updates and

λ

is the weight decay. In simulations, recovery is approximately a function of the disagreement budget

bδ/η

and saturates in the horizon.

Momentum

Related topics

Latest papers 132