Gradient Descent Dynamics

This field studies how parameters and predictions change during training, and how learning rates, initialization, data structure, and optimizer choice affect convergence and generalization. Researchers use gradient-flow and ODE approximations, discrete dynamical-systems tools such as Lyapunov and bifurcation analysis, and stochastic or spectral methods to study settings from linear regression and matrix factorization to neural networks. A major theme is the edge of stability, where large steps can produce oscillations that alter sharpness, feature learning, and implicit bias; related work examines early stopping, grokking, and the effects of minibatch noise and momentum. An open tension is that continuous-time and simplified-model theories often give clear guarantees, while finite-step training in modern networks can show cycles, transient amplification, and optimizer-specific behavior that those approximations miss.