Gradient Descent Trajectory

Gradient descent trajectory analysis focuses on understanding the path taken by optimization algorithms, like stochastic gradient descent (SGD) and Adam, as they search for optimal model parameters. Current research investigates the trajectory's relationship to the loss function's landscape, particularly focusing on how the trajectory interacts with the Hessian and gradient matrices, especially in high-dimensional settings and with models such as neural networks (including ResNets). These studies aim to explain phenomena like implicit regularization and the "edge of stability," ultimately seeking to improve algorithm design and convergence guarantees for various machine learning tasks. This work has implications for both theoretical understanding of optimization and practical improvements in training efficiency and generalization performance.

Papers

February 20, 2024

On the Convergence of Gradient Descent for Large Learning Rates
Alexandru Crăciun, Debarghya Ghoshdastidar
Gradient Descent Stochastic Gradient Descent Core Stability Flat Minimum Large Learning Rate Gradient Descent Trajectory

October 4, 2023

High-dimensional SGD aligns with emerging outlier eigenspaces
Gerard Ben Arous, Reza Gheissari, Jiaoyang Huang, Aukosh Jagannath
Gradient Descent Stochastic Gradient Descent Gradient Descent Trajectory

August 31, 2023

On the Implicit Bias of Adam
Matias D. Cattaneo, Jason M. Klusowski, Boris Shigida
Strong Generalization Implicit Bias Loss Gradient Gradient Descent Trajectory

July 21, 2023

Convergence of SGD for Training Neural Networks with Sliced Wasserstein Losses
Eloi Tanguy
Neural Network Stochastic Gradient Descent Early Stage Convergence Gradient Flow Wasserstein Distance Sliced Wasserstein Gradient Descent Trajectory

July 9, 2023

Trajectory Alignment: Understanding the Edge of Stability Phenomenon via Bifurcation Theory
Minhak Song, Chulhee Yun
Neural Network Gradient Descent Extreme Edge Hessian Matrix Bifurcation Diagram Gradient Descent Trajectory Trajectory Alignment

January 5, 2023

Training trajectories, mini-batch losses and the curious role of the learning rate
Mark Sandler, Andrey Zhmoginov, Max Vladymyrov, Nolan Miller
Loss Function Gradient Descent Stochastic Gradient Descent Integral Role Learning Rate Mini Batch Training Trajectory Stochastic Weight Averaging Gradient Descent Trajectory

November 5, 2021

Sharp Bounds for Federated Averaging (Local SGD) and Continuous Perspective
Margalit Glasgow, Honglin Yuan, Tengyu Ma
Federated Averaging Continuous Time Local SGD Sharp Bound Gradient Descent Trajectory