Symmetry Breaking at the Edge of Stability via Central Flow Analysis
Organizations: Technical University of Denmark
Abstract
Continuous symmetries in neural networks lead to conserved quantities that are invariant under gradient flow, such as layer balancedness in linear and ReLU networks. Finite-step gradient descent need not preserve these laws, particularly at the edge of stability (EOS), where the iterates oscillate around a smoother trajectory, which can be described by the central flow dynamics (Cohen et al., 2025). We study how gradient flow conservation laws evolve under the central flow description of this regime. We show that the curvature correction induces a finite-dimensional drift determined by the oscillation covariance along critical parameter directions. We thus take the first steps to analyse which conservation laws remain and are broken at the EOS according to the central flow dynamics by analysing a scalar two-layer model, two-layer matrix factorisation, and a two-layer ReLU network, where the known conservation laws are shown to be complete (Marcotte et al., 2023).
Figures & tables
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
| model | gradient flow laws | ||
|---|---|---|---|
| scalar | |||
| matrix factorisation | |||
| two-layer ReLU |
| scalar | matrix fact. | ReLU (local) | ReLU (trajectory) | |
|---|---|---|---|---|
| dimensions | , | , , | , , | |
| data | rank- target | , | as local | |
| initialisation | , | , | , | as local |
| n_seeds | ||||
Neural Network Training Dynamics
Neural network training dynamics studies how parameters and representations change during optimization and how those changes shape learning, stability, and generalization.
See all 549 papers