cs.LGFeb 5, 2026

Emergent Low-Rank Training Dynamics in MLPs with Smooth Activations

Authors: Alec S. Xu, Can Yaras, Matthew Asato, Qing Qu, Laura Balzano

Organizations: University of Michigan

Abstract

Recent empirical evidence has demonstrated that the training dynamics of large-scale deep neural networks occur within low-dimensional subspaces. While this has inspired new research into low-rank training, compression, and adaptation, theoretical justification for these dynamics in nonlinear networks remains limited. %compared to deep linear settings. To address this gap, this paper analyzes the learning dynamics of multi-layer perceptrons (MLPs) under gradient descent (GD). We demonstrate that the weight dynamics concentrate within invariant low-dimensional subspaces throughout training. Theoretically, we precisely characterize these invariant subspaces for two-layer networks with smooth nonlinear activations, providing insight into their emergence. Experimentally, we validate that this phenomenon extends beyond our theoretical assumptions. Leveraging these insights, we empirically show there exists a low-rank MLP parameterization that, when initialized within the appropriate subspaces, matches the classification performance of fully-parameterized counterparts on a variety of classification tasks.

Figures & tables

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Rationally Linearized MLPs

    Oct 9, 2026Arijit Sehanobish, Byeongchan Kim, Avinava Dubey +2Multilayer PerceptronsEfficient Neural Network Inference

  2. Stabilizing the Dynamic Low-Rank Training

    Sep 26, 2026Zhonghan Xu, Ling Wang, Junhao Chen +2Low-Rank ApproximationNeural Network Training Dynamics