Recent empirical evidence has demonstrated that the training dynamics of large-scale deep neural networks occur within low-dimensional subspaces. While this has inspired new research into low-rank training, compression, and adaptation, theoretical justification for these dynamics in nonlinear networks remains limited. %compared to deep linear settings. To address this gap, this paper analyzes the learning dynamics of multi-layer perceptrons (MLPs) under gradient descent (GD). We demonstrate that the weight dynamics concentrate within invariant low-dimensional subspaces throughout training. Theoretically, we precisely characterize these invariant subspaces for two-layer networks with smooth nonlinear activations, providing insight into their emergence. Experimentally, we validate that this phenomenon extends beyond our theoretical assumptions. Leveraging these insights, we empirically show there exists a low-rank MLP parameterization that, when initialized within the appropriate subspaces, matches the classification performance of fully-parameterized counterparts on a variety of classification tasks.
Figures & tables
Figure 1: Low rank updates in MLPs with smooth activation functions. Each plot shows how the singular values of the first layer (out of four layers total) evolve throughout training in MLPs with ELU , GELU , and SiLU activation functions, which are all smooth. We trained each MLP on synthetic data and squared-error loss using gradient descent. Specific experimental details and additional plots of the deeper layer singular values are in Section A.1 .
Figure 2: The middle singular subspace of the first-layer weight matrix in the ELU network evolves noticeably slower than that in the ReLU network, and the corresponding singular values remain closer to their initialization. The right singular subspaces exhibit similar dynamics in both cases (see Section A.1 ).
Figure 3: Under our exact theoretical setting, W1,1(t) accounts for almost all of the change in W1(t) , and thus W1(t) .
Figure 4: For every l∈[L−1] in deep MLPs with smooth activations, Wl,1(t) accounts for almost all of the change in Wl(t) .
Figure 5: Training deep MLPs with SGD plus momentum (top row), or with Adam (bottom row), on unwhitened input data using cross-entropy loss approximately maintains the previously observed low-rank training dynamics.
Figure 6: The properly-initialized low-rank MLP (orange) achieves nearly-identical test loss and accuracy trajectories as the fully parameterized MLP (blue). Meanwhile, the low-rank MLP with random subspace initialization (green) gets stuck at a noticeably worse local minimum.
Figure 7: The properly-initialized low-rank MLP classifier head (orange) achieves similar test accuracies to the fully parameterized version (blue) in VGG-16 on CIFAR-10, while the random-subspace initialized low-rank MLP (green) is stuck in the random guessing stage.
Model
Trainable params
Init. FLOPs
Per-iter. FLOPs
Dense
1.2M
N/A
1.59×109
Low-rank MLP (this paper, width =20 )
40K
1.99×1010
5.35×107
LTH (sparsity =0.968 )
40K
T×1.59×109
5.13×107
DLRT (rank =10 )
39.4K
2.12×1010
9.92×107
Table 1: Initialization and per-iteration FLOP comparison. T is the number of dense training iterations before LTH pruning.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: For networks with smooth activation functions, the middle singular subspace of the first layer weights evolves noticeably slower than in networks with non-smooth activation functions.
Figure 9: The change in the middle singular subspaces of the second layer weights in networks with smooth vs. nonsmooth activation functions mimic those in the first layer weights ( Figure 8 ).
Figure 10: The change in the middle singular subspaces of the third layer weights in networks with smooth vs. nonsmooth activation functions mimic those in the first and second layer weights ( Figures 8 and 9 ).
Figure 11: Top: even when W2 is trained, nearly all of the change in W1(t) is from W1,1(t) . Bottom: repeat of Figure 3 as reference.
Figure 12: The gradient norm and top- K singular values appear to monotonically decay from their initial values, despite the objective being non-convex.
Figure 13: Throughout training, σK+1(G1(t)) remains significantly smaller than σK(W2⊤YX⊤) .
Figure 14: The bottom d−K singular values of G1(0) scale with ϵ when ϕ is smooth ( GELU ), but do not change with ϵ when ϕ is nonsmooth ( ReLU ). The top- K singular values of G1(0) overlap for both smooth and nonsmooth ϕ .
Figure 15: For all low-rank MLPs with U and V initialized close to ( ψ≤45 degrees) Ubig and Vbig , GD reaches similar solutions. Otherwise, GD gets stuck in progressively worse local minima as ψ increases.
Figure 16: Increasing the width r of the low-rank MLP initially leads to a noticeable increase in test accuracy, but further increasing r leads to diminishing returns in test accuracy gains. Here, we only trained the classifier head in VGG-16, and kept the convolutional layers frozen at their ImageNet pre-trained weights.
Neural network training dynamics studies how parameters and representations change during optimization and how those changes shape learning, stability, and generalization.