Gradient Flow

Gradient flow, the continuous-time limit of gradient descent, is a powerful tool for analyzing the training dynamics of machine learning models, particularly deep neural networks. Current research focuses on understanding gradient flow's behavior in various architectures (e.g., ResNets, transformers) and optimization algorithms (e.g., stochastic gradient descent, mirror descent), investigating phenomena like oversmoothing and the impact of different regularizations. This analysis helps explain implicit biases, convergence rates, and the effectiveness of various training techniques, ultimately leading to improved model design and training strategies. The insights gained are crucial for enhancing the performance and stability of machine learning algorithms across diverse applications.

Papers

March 6, 2023

Critical Points and Convergence Analysis of Generative Deep Linear Networks Trained with Bures-Wasserstein Loss
Pierre Bréchet, Katerina Papagiannouli, Jing An, Guido Montúfar
Gradient Descent Matrix Factorization Gradient Flow Convergence Analysis Low Rank Matrix Deep Generative Critical Point

March 2, 2023

The Double-Edged Sword of Implicit Bias: Generalization vs. Robustness in ReLU Networks
Spencer Frei, Gal Vardi, Peter L. Bartlett, Nathan Srebro
Native Robustness Strong Generalization Adversarial Example Adversarial Robustness Gradient Flow Implicit Bias ReLU Network Two Layer ReLU

February 25, 2023

Achieving High Accuracy with PINNs via Energy Natural Gradients
Johannes Müller, Marius Zeinhofer
Physic Informed Neural Network Natural Gradient Gradient Flow Hessian Matrix Natural Gradient Descent Enhancing Accuracy

February 20, 2023

Over-Parameterization Exponentially Slows Down Gradient Descent for Learning a Single Neuron
Weihang Xu, Simon S. Du
Gradient Descent Gradient Flow Global Convergence Benign Over Parameterization Single Neuron Gradient Descent Dynamic

February 11, 2023

A Policy Gradient Framework for Stochastic Optimal Control Problems with Global Convergence Guarantee
Mo Zhou, Jianfeng Lu
Policy Gradient Gradient Flow Stochastic Optimal Control Global Convergence Guarantee

February 2, 2023

January 31, 2023

Dynamic Flows on Curved Space Generated by Labeled Data
Xinru Hua, Truyen Nguyen, Tam Le, Jose Blanchet, Viet Anh Nguyen
Gradient Flow Riemannian Gradient Labeled Data

January 25, 2023

MLPGradientFlow: going with the flow of multilayer perceptrons (and finding minima fast and accurately)
Johanni Brea, Flavio Martinelli, Berfin Şimşek, Wulfram Gerstner
Gradient Descent Multi Layer Natural Gradient Gradient Flow Flow Mood

January 19, 2023

Convergence beyond the over-parameterized regime using Rayleigh quotients
David A. R. Robin, Kevin Scaman, Marc Lelarge
Loss Function Early Stage Convergence Gradient Flow Loss Minimization Parameterized Regime Parametric Learning Rayleigh Regression

December 13, 2022

Gradient flow in the gaussian covariate model: exact solution of learning curves and multiple descent structures
Antoine Bodin, Nicolas Macris
Gradient Flow Generalization Error Double Descent Aggregated Curve Multivariate Gaussian Epoch Wise Double Descent

November 25, 2022

Particle-based Variational Inference with Preconditioned Functional Gradient Flow
Hanze Dong, Xi Wang, Yong Lin, Tong Zhang
Variational Inference Gradient Flow Gradient Estimation Particle Based Variational Inference

November 16, 2022

On the symmetries in the dynamics of wide two-layer neural networks
Karl Hajjar, Lenaic Chizat
Neural Network Gradient Flow Approximate Symmetry Two Layer Neural Network Two Layer ReLU

November 15, 2022

Regularized Stein Variational Gradient Flow
Ye He, Krishnakumar Balasubramanian, Bharath K. Sriperumbudur, Jianfeng Lu
Gradient Flow Wasserstein Gradient Flow Stein Variational Gradient Descent

October 31, 2022

October 29, 2022

Perturbation Analysis of Neural Collapse
Tom Tirer, Haoxiang Huang, Jonathan Niles-Weed
Deep Neural Network Gradient Flow Neural Collapse Deep Layer Perturbation Analysis Practical Deep

October 28, 2022

Toward Equation of Motion for Deep Neural Networks: Continuous-time Gradient Descent and Discretization Error Analysis
Taiki Miyagawa
Deep Neural Network Gradient Descent Motion Information Gradient Flow Learning Dynamic Discrete Time Diverse Equation Discretization Error

October 27, 2022

Gradient Flow

Papers

Critical Points and Convergence Analysis of Generative Deep Linear Networks Trained with Bures-Wasserstein Loss

The Double-Edged Sword of Implicit Bias: Generalization vs. Robustness in ReLU Networks

Achieving High Accuracy with PINNs via Energy Natural Gradients

Over-Parameterization Exponentially Slows Down Gradient Descent for Learning a Single Neuron

A Policy Gradient Framework for Stochastic Optimal Control Problems with Global Convergence Guarantee

MonoFlow: Rethinking Divergence GANs via the Perspective of Wasserstein Gradient Flows

Over-parameterised Shallow Neural Networks with Asymmetrical Node Scaling: Global Convergence Guarantees and Feature Learning

Dynamic Flows on Curved Space Generated by Labeled Data

MLPGradientFlow: going with the flow of multilayer perceptrons (and finding minima fast and accurately)

Convergence beyond the over-parameterized regime using Rayleigh quotients

Gradient flow in the gaussian covariate model: exact solution of learning curves and multiple descent structures

Particle-based Variational Inference with Preconditioned Functional Gradient Flow

On the symmetries in the dynamics of wide two-layer neural networks

Regularized Stein Variational Gradient Flow

Lipschitz-regularized gradient flows and generative particle algorithms for high-dimensional scarce data

Symmetries, flat minima, and the conserved quantities of gradient flow

Perturbation Analysis of Neural Collapse

Toward Equation of Motion for Deep Neural Networks: Continuous-time Gradient Descent and Discretization Error Analysis

Learning Single-Index Models with Shallow Neural Networks

Stochastic Mirror Descent in Average Ensemble Models