Vanishing gradients and overfitting are central problems in machine learning, yet are typically analyzed in asymptotic regimes that obscure their dynamical origins. Here we provide a dynamical description of learning in multi-layer perceptrons (MLPs) via a minimal model inspired by Fukumizu and Amari. We show that training dynamics traverse plateau and near-optimal regions, both organized by saddle structures, before converging to an overfitting regime. Under suitable conditions on the data, this regime collapses to a single attractor modulo symmetry. Furthermore, for finite noisy datasets, convergence to the theoretical optimum is impossible, and the dynamics necessarily settle into an overfitting solution.
Figures & tables
Figure 1: A multi-layer perceptron with two hidden layers of arbitrary size.
Figure 2: A schematic representation of the saddle-saddle-attractor scenario in MLP gradient descent learning. Empirically, the number of positive eigenvalues is smaller near the optimal region than in the plateau region. The overfitting is a stable attractor.
Figure 4: Graphs obtained after training the minimal model for 2 million iterations. In the first column we represent the learning curve; on the right there is a representation of the parameters’ orbits during the training, where red stars are the overfitting points and grey lines model the singular region (dark grey) and optimal region (light gray). Plots in (a) represent the training without observational noise (i.e., τ=0 ), while in (b) we added some small noise with τ=0.2 .
Neural network training dynamics studies how parameters and representations change during optimization and how those changes shape learning, stability, and generalization.
Department of Electrical and Computer Engineering University of Southern California Los Angeles, CA, USA · Department of Computer Science University of Southern California Los Angeles, CA, USA
Faculty of Computer and Information Science, Ben-Gurion University · Schools of Electrical and Computer Engineering, and Industrial and Systems Engineering, Georgia Institute of Technology · Electrical and Computer Engineering Department, Rice University