Gradient Descent Dynamics

Ask about this topic

Overview

This field studies how parameters and predictions change during training, and how learning rates, initialization, data structure, and optimizer choice affect convergence and generalization. Researchers use gradient-flow and ODE approximations, discrete dynamical-systems tools such as Lyapunov and bifurcation analysis, and stochastic or spectral methods to study settings from linear regression and matrix factorization to neural networks. A major theme is the edge of stability, where large steps can produce oscillations that alter sharpness, feature learning, and implicit bias; related work examines early stopping, grokking, and the effects of minibatch noise and momentum. An open tension is that continuous-time and simplified-model theories often give clear guarantees, while finite-step training in modern networks can show cycles, transient amplification, and optimizer-specific behavior that those approximations miss.

Latest papers

CardsList

157 papers23 papers in the last four weeks, up 109% on the four weeks before. 0.1% of all new papers.

  1. On the Effectiveness of the z-Transform Method in Quadratic Optimization

    Jul 4, 2025Francis BachOptimization Convergence AnalysisGradient Descent Dynamics

    SIERRA · Inria - Ecole Normale Sup´erieure · PSL Research University

  2. Negative Stepsizes Make Gradient-Descent-Ascent Converge

    May 2, 2025Henry Shugart, Jason M. AltschulerLearning Rate SchedulingOptimization Convergence Analysis

    UPenn

  3. The Dynamics of Generalization in Deep Learning

    Apr 23, 2025Rubing Yang, Pratik ChaudhariNeural Network GeneralizationNeural Network Training Dynamics

    Department of Applied Mathematics and Computational Sciences, University of Pennsylvania Philadelphia, PA, 19104, USA · Department of Electrical and Systems Engineering, University of Pennsylvania Philadelphia, PA, 19104, USA

  4. Derivation of effective gradient flow equations and dynamical truncation of training data in Deep Learning

    Jan 13, 2025Thomas ChenNeural Network InterpretabilityNeural Network Training Dynamics

  5. How Biological Synapses Self-Assemble Gradient Learning

    Dec 28, 2024Qianli Liao, Liu Ziyin, Yulu Gan +3Biologically Plausible LearningNeural Processes

    Center for Brains, Minds, and Machines, MIT · McGovern Institute, MIT · Research Laboratory of Electronics, MIT +3

  6. Gradient Descent on Logistic Regression with Non-Separable Data and Large Step Sizes

    Jun 7, 2024Si Yi Meng, Antonio Orvieto, Daniel Yiming Cao +1Dynamical SystemsLogistic Regression

    Department of Computer Science Cornell University Ithaca, NY, USA · ELLIS Institute Tübingen MPI for Intelligent Systems Tübingen AI Center, Germany

  7. On Regularization via Early Stopping for Least Squares Regression

    Jun 6, 2024Rishi Sonthalia, Jackie Lok, Elizaveta RebrovaEarly StoppingLinear Regression

    Department of Mathematics, Boston College · ORFE Department, Princeton University