Overview

This field studies how parameters and predictions change during training, and how learning rates, initialization, data structure, and optimizer choice affect convergence and generalization. Researchers use gradient-flow and ODE approximations, discrete dynamical-systems tools such as Lyapunov and bifurcation analysis, and stochastic or spectral methods to study settings from linear regression and matrix factorization to neural networks. A major theme is the edge of stability, where large steps can produce oscillations that alter sharpness, feature learning, and implicit bias; related work examines early stopping, grokking, and the effects of minibatch noise and momentum. An open tension is that continuous-time and simplified-model theories often give clear guarantees, while finite-step training in modern networks can show cycles, transient amplification, and optimizer-specific behavior that those approximations miss.

Latest papers

CardsList

171 papers20 papers in the last four weeks, down 9% on the four weeks before. 0.2% of all new papers.

  1. On the Convergence of Continuous-Time AdamW

    Nov 16, 2026Bangjie Wang, Ning NingOptimization Convergence AnalysisAdaptive Gradient Methods

  2. Classification with Large-Step Gradient Descent: Unifying Self-Stabilization and Convergence Rates

    Nov 12, 2026Salma Tarmoun, Lachlan Ewen MacDonald, Konstantinos Emmanouilidis +2Large-Step-Size Gradient DescentGradient Descent Dynamics

  3. Learning to Restart: Differentiable Restart Policies for Optimization Dynamics

    Nov 6, 2026Yiman Fong, Zhonglin Xie, Zaiwen WenLearning to OptimizeDifferentiable Optimization

  4. Projection-Free Gradient Tracking Neurodynamic for Distributed Nonnegative Sparse Optimization

    Nov 5, 2026Juntao Li, Qihang Guo, Deyuan MengDistributed OptimizationConstrained Optimization

  5. Truncated backpropagation through time induces a two-sided gradient turnpike property

    Nov 5, 2026Julian D. Schiller, Matthias A. MüllerTruncated Backpropagation Through TimeRecurrent Neural Networks

  6. Symmetry Breaking at the Edge of Stability via Central Flow Analysis

    Nov 4, 2026Naima Elosegui Borras, Georgios ArvanitidisNeural Network Training DynamicsGradient Descent Dynamics

  7. Gradient Descent on Two ReLU Neurons: Global Landscape and Bifurcation Dynamics

    Oct 30, 2026Binghua Li, Mengzhe Li, Denny Wu +1Neural Network Training DynamicsNeural Network Optimization

  8. Infinite ResNets are Gaussian Mean ODEs

    Oct 30, 2026Louis-Pierre Chaintron, Lénaïc Chizat, Javier MaassNeural Network Training DynamicsGradient Descent Dynamics

  9. General Frequency Principle for Linear Approximation Classes

    Oct 30, 2026Yujun Wang, Yiqi GuFrequency-Domain Feature LearningGradient Descent Dynamics

  10. Rethinking GFlowNet Training: Negative Reinforce Dynamics

    Oct 28, 2026Pedro Dall'Antonia, Daniel Csillag, Diego MesquitaGenerative Flow NetworksGradient Descent Dynamics

  11. Silent Alignment in Two-Layer Linear Networks: A High-Dimensional Perspective

    Oct 21, 2026Simon Martin, Francis Bach, Giulio BiroliTwo-Layer Neural NetworksNeural Network Training Dynamics

  12. Phase Transitions in Learning Driven by Extremal Data

    Oct 21, 2026Ethan Levien, Trevor GrandPreMultiple Instance LearningPhase Transitions in Neural Networks

  13. When Parameter Gradient Flows Close in Representation Space

    Oct 20, 2026Shengye TaoGradient Descent Dynamics

  14. Gradient descent reaches arbitrary accuracy in a fixed number of large steps on multiclass logistic regression

    Oct 19, 2026Andreea-Alexandra Musat, Nicolas BoumalLarge-Step-Size Gradient DescentGradient Descent

  15. The Accuracy Well: Nonmonotone Accuracy Dynamics at Vanishing Initialization

    Oct 14, 2026Yaroslav Gusev, Eugene Golikov, Dmitry YarotskyNeural Network Training DynamicsGradient Descent Dynamics

  16. From Spectral to Frobenius: When Muon Loses Its Spectral Implicit Bias

    Oct 14, 2026Yongtao Wu, Mahdi Mansouri, Zhenyu ZhuMuon OptimizerImplicit Regularization

  17. Randomized Momentum Transport with Cheap Predictors

    Oct 9, 2026El Mahdi Chayti, Martin JaggiMomentum MethodsStochastic Optimization Convergence

  18. Controlled-Minimax: Stabilizing Adversarial Optimization via Control Theory

    Oct 9, 2026Tianjiao Luo, Jianfei Chen, Jun ZhuMin-Max OptimizationOptimization Convergence Analysis

  19. PRAXIS: Learning Dynamics of Self-Improving Models with Symbolic Archives

    Oct 8, 2026Venkat Margapuri, Mustafa TeberSelf-Improving AgentsAdaptive Learning

  20. Rare Gate Disagreements Can Limit Plasticity: When Gradient Flow Mispredicts Finite-Batch SGD

    Oct 8, 2026Ruoyu Zhao, Mingxuan Zhang, Jianbo Dai +3Neural Network Training DynamicsStochastic Gradient Descent

  21. Finite-Sample Approximation of Hessian-Guided Perturbed Wasserstein Gradient Flows

    Oct 7, 2026Ryotaro Kawata, Atsushi Nitanda, Taiji SuzukiWasserstein Gradient FlowsGradient Descent Dynamics