Emergent Low-Rank Training Dynamics in MLPs with Smooth Activations
Organizations: University of Michigan
Abstract
Recent empirical evidence has demonstrated that the training dynamics of large-scale deep neural networks occur within low-dimensional subspaces. While this has inspired new research into low-rank training, compression, and adaptation, theoretical justification for these dynamics in nonlinear networks remains limited. %compared to deep linear settings. To address this gap, this paper analyzes the learning dynamics of multi-layer perceptrons (MLPs) under gradient descent (GD). We demonstrate that the weight dynamics concentrate within invariant low-dimensional subspaces throughout training. Theoretically, we precisely characterize these invariant subspaces for two-layer networks with smooth nonlinear activations, providing insight into their emergence. Experimentally, we validate that this phenomenon extends beyond our theoretical assumptions. Leveraging these insights, we empirically show there exists a low-rank MLP parameterization that, when initialized within the appropriate subspaces, matches the classification performance of fully-parameterized counterparts on a variety of classification tasks.
Figures & tables
| Model | Trainable params | Init. FLOPs | Per-iter. FLOPs |
|---|---|---|---|
| Dense | N/A | ||
| Low-rank MLP (this paper, width ) | |||
| LTH (sparsity ) | |||
| DLRT (rank ) |
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Neural Network Training Dynamics
Neural network training dynamics studies how parameters and representations change during optimization and how those changes shape learning, stability, and generalization.
See all 547 papers