Gradient Descent Dynamics

Ask about this topic

Overview

This field studies how parameters and predictions change during training, and how learning rates, initialization, data structure, and optimizer choice affect convergence and generalization. Researchers use gradient-flow and ODE approximations, discrete dynamical-systems tools such as Lyapunov and bifurcation analysis, and stochastic or spectral methods to study settings from linear regression and matrix factorization to neural networks. A major theme is the edge of stability, where large steps can produce oscillations that alter sharpness, feature learning, and implicit bias; related work examines early stopping, grokking, and the effects of minibatch noise and momentum. An open tension is that continuous-time and simplified-model theories often give clear guarantees, while finite-step training in modern networks can show cycles, transient amplification, and optimizer-specific behavior that those approximations miss.

Latest papers

CardsList

157 papers23 papers in the last four weeks, up 109% on the four weeks before. 0.1% of all new papers.

  1. Conservation Law Breaking at the Edge of Stability: A Spectral Theory of Non-Convex Neural Network Optimization

    Apr 8, 2026Daniel Nobrega MedeirosNeural Network OptimizationGradient Descent Dynamics

    University of Colorado Boulder

  2. Stochastic Gradient Descent in the Saddle-to-Saddle Regime of Deep Linear Networks

    Apr 7, 2026Guillaume Corlouer, Avi Semler, Alexander Strang +1Gradient Descent DynamicsStochastic Gradient Descent

    Moirai · University of Oxford · University of California, Berkeley +1

  3. Grokking as Dimensional Phase Transition in Neural Networks

    Apr 6, 2026Ping WangPhase Transitions in Neural NetworksGrokking

    Institute of High Energy Physics, Chinese Academy of Science, 100049 Beijing, China

  4. Product-Stability: Provable Convergence for Gradient Descent on the Edge of Stability

    Apr 3, 2026Eric GanDeep Learning OptimizationEdge of Stability

    Independent Researcher.

  5. Plateaus, Optima, and Overfitting in Multi-Layer Perceptrons: A Saddle-Saddle-Attractor Scenario

    Apr 2, 2026Alex Alì Maleknia, Yuzuru SatoNeural Network Training DynamicsMultilayer Perceptrons

    Department of Mathematics, Hokkaido University, Kita 12 Nishi 7, Kita-ku, Sapporo, Hokkaido 060-0812, Japan · Present address: IMAG, IROKO, Univ Montpellier, Inria, CNRS · RIES, Hokkaido University, Kita 12 Nishi 7, Kita-ku, Sapporo, Hokkaido 060-0812, Japan

  6. GRASP: Gradient Realignment via Active Shared Perception for Multi-Agent Collaborative Optimization

    Apr 1, 2026Sihan Zhou, Tiantian He, Yifan Lu +2Cooperative MARLNon-Stationary RL

    College of Computer Science and Technology, and Key Laboratory of Social Computing and Cognitive Intelligence (Ministry of Education), Dalian University of Technology, China · Center for Frontier AI Research, Institute of High Performance Computing, Singapore Institute of Manufacturing Technology, Agency for Science, Technology and Research (A*STAR), Singapore · School of Computer Science and Engineering, Nanyang Technological University, Singapore

  7. Immediate Derivatives Suffice for Online Recurrent Adaptation

    Mar 30, 2026Aur Shalev MerinRecurrent Neural NetworksGradient Descent Dynamics

    Independent Researcher

  8. Rigorous Asymptotics for First-Order Algorithms Through the Dynamical Cavity Method

    Mar 15, 2026Yatin Dandi, David Gamarnik, Francisco Pernice +1Approximate Message PassingDynamical Mean-Field Theory

    Statistical Physics of Computation Laboratory, École polytechnique fédérale de Lausanne (EPFL) CH-1015 Lausanne. · Sloan School of Management, Operations Research Center and Institute of Data, Systems and Society (IDSS), MIT. · CSAIL and LIDS, MIT.

  9. Spectral Edge Dynamics of Training Trajectories: Signal--Noise Geometry Across Scales

    Mar 14, 2026Yongzhong XuTransformer Training DynamicsNeural Network Training Dynamics

  10. Large Spikes in Stochastic Gradient Descent: A Large-Deviations View

    Mar 10, 2026Benjamin Gess, Daniel HeydeckerLarge Deviation PrinciplesGradient Descent Dynamics

    Technische Universität Berlin, Fakultät II 10623 Berlin, Germany · Max Planck Institute for Mathematics in the Sciences 04103 Leipzig, Germany · University of Oslo, Blindern 0316 Oslo, Norway

  11. Gradient Flow Polarizes Softmax Outputs towards Low-Entropy Solutions

    Mar 6, 2026Aditya Varre, Mark Rofin, Nicolas FlammarionAttention EntropyGradient Descent Dynamics

    Theory of Machine Learning Lab, EPFL.

  12. The Norm-Separation Delay Law of Grokking: A First-Principles Theory of Delayed Generalization

    Mar 5, 2026Truong Xuan Khanh, Truong Quynh Hoa, Luu Duc Trung +1GrokkingNeural Network Generalization

    H&K Research Studio, Clevix LLC, Hanoi, Vietnam · Banking Academy of Vietnam, Hanoi, Vietnam

  13. Non-Euclidean Gradient Descent Operates at the Edge of Stability

    Mar 5, 2026Rustem Islamov, Michael Crawshaw, Jeremy Cohen +1Edge of StabilityGradient Descent Dynamics

    University of Basel, Switzerland · George Mason University, USA · Flatiron Institute, USA

  14. On the Rate of Convergence of GD in Non-linear Neural Networks: An Adversarial Robustness Perspective

    Mar 2, 2026Guy Smorodinsky, Sveta Gimpleson, Itay SafranNeural Network Training DynamicsGradient Descent Dynamics

    Faculty of Computer and Information Science, Ben-Gurion University of the Negev, Israel · Department of Computer Science, Reichman University, Israel

  15. Never Saddle for Reparameterized Steepest Descent as Mirror Flow

    Mar 2, 2026Tom Jacobs, Chao Zhou, Rebekka BurkholzNeural Network Training DynamicsGradient Descent Dynamics

    CISPA Helmholtz Center for Information Security, Saarbrücken, Germany

  16. Training Dynamics of Softmax Self-Attention: Fast Global Convergence via Preconditioning

    Mar 2, 2026Gautam Goel, Mahdi Soltanolkotabi, Peter BartlettSelf-AttentionSoftmax Attention

    Simons Institute, UC Berkeley · Department of Electrical and Computer Engineering, University of Southern California

  17. On the Convergence of Stochastic Gradient Descent with Perturbed Forward-Backward Passes

    Feb 24, 2026Boao Kong, Hengrui Zhang, Kun YuanStochastic Optimization ConvergenceStochastic Gradient Descent

    Peking University. · Sichuan University.

  18. Low-Dimensional and Transversely Curved Optimization Dynamics in Grokking

    Feb 18, 2026Yongzhong XuGrokkingNeural Network Training Dynamics

  19. On the Stability of Nonlinear Dynamics in GD and SGD: Beyond Quadratic Potentials

    Feb 16, 2026Rotem Mulayoff, Sebastian U. StichGradient Descent DynamicsStochastic Gradient Descent

    CISPA Helmholtz Center for Information Security, Saarbrücken, Germany

  20. On the Dynamics of Delayed Generalization in Linear Models

    Feb 9, 2026Atreya Vedantam, Nataraj Das, Chandrashekar LakshminarayananLinear ClassificationNeural Network Generalization

    Indian Institute of Technology Madras, Chennai.

  21. Gradient Flow Through Diagram Expansions: Learning Regimes and Explicit Solutions

    Feb 4, 2026Dmitry Yarotsky, Eugene Golikov, Yaroslav GusevTensor DecompositionMean-Field Theory

    Applied AI Institute, Moscow, Russia · Steklov Mathematical Institute of Russian Academy of Sciences, Moscow, Russia.

  22. Not All Preferences Deserve Gradients: Understanding Gradient Utility in Offline Reasoning Alignment

    Feb 1, 2026Hui Wu, Hengyi Cai, Jinman Zhao +6Pairwise Preference LearningTraining Data Selection

    Baidu Inc. · University of Toronto · School of Intelligence Science and Technology, Peking University +1

  23. Spectral Gradient Descent Mitigates Anisotropy-Driven Misalignment: A Case Study in Phase Retrieval

    Jan 30, 2026Guillaume Braun, Han Bao, Wei Huang +1Gradient DescentGradient Descent Dynamics

    RIKEN AIP · The Institute of Statistical Mathematics · Tohoku University +2

  24. Riemannian Lyapunov Framework: Optimization as Closed Loop Control on Riemannian Manifolds

    Jan 29, 2026Yixuan Wang, Omkar Sudhir Patil, Warren E. DixonStochastic Optimization ConvergenceGradient Descent Dynamics

    Department of Mechanical and Aerospace Engineering University of Florida

  25. Why β1=β2β_1 = β_2 Is Dynamically Special in Adam

    Jan 29, 2026Alberto Fernández-Hernández, Cristian Pérez-Corral, Jose I. Mestre +2Momentum MethodsAdaptive Gradient Methods

    Universitat Politècnica de València Valencia, Spain · Universitat Jaume I Castelló de la Plana, Spain

  26. To Grok Grokking: Provable Grokking in Ridge Regression

    Jan 27, 2026Mingyue Xu, Gal Vardi, Itay SafranNeural Network GeneralizationRidge Regression

    Department of Computer Science, Purdue University, West Lafayette, IN, USA · Department of Computer Science and Applied Mathematics, Weizmann Institute of Science, Israel · Stein Faculty of Computer and Information Science, Ben-Gurion University of the Negev, Israel

  27. Gradient Descent as Implicit EM in Distance-Based Neural Models

    Dec 31, 2025Alan OurslandExpectation-MaximizationNeural Network Training Dynamics

  28. Exact Dynamics of Multi-class Stochastic Gradient Descent

    Oct 15, 2025Elizabeth Collins-Woodfin, Inbar SeroussiLogistic RegressionHigh-Dimensional Asymptotics

  29. Modified Loss of Momentum Gradient Descent: Fine-Grained Analysis

    Sep 10, 2025Matias D. Cattaneo, Boris ShigidaGradient Descent Dynamics

    Princeton University

  30. Gradient Descent on Logistic Regression: Do Large Step-Sizes Work with Data on the Sphere?

    Jul 15, 2025Si Yi Meng, Baptiste Goujaud, Antonio Orvieto +1Logistic RegressionGradient Descent

    Inria Paris, Paris, France · Inria Saclay, Palaiseau, France · ELLIS Institute Tübingen; MPI for Intelligent Systems; Tübingen AI Center, Germany +1