Broken scale symmetries in undercomplete linear autoencoders

Farhad Pashakhanloo; Jacob A. Zavatone-Veth

cs.LGOct 2, 2026

Broken scale symmetries in undercomplete linear autoencoders

Authors: Farhad Pashakhanloo, Jacob A. Zavatone-Veth

Organizations: Center for Brain Science Harvard University · Center for Brain Science and Society of Fellows Harvard University

Abstract

Neural network loss landscapes have many symmetries, which are preserved by gradient flow but broken by finite-stepsize stochastic gradient descent (SGD). A canonical example of such a symmetry is scale in homogeneous networks: one can scale up the parameters in one layer and down in the next without changing the network output. Previous work has documented cases in which SGD breaks this symmetry in favor of balancing gradient noise or minimizing fluctuations. Here, we show that the solution geometry of undercomplete linear autoencoders instead selects a preferred sign for scale drift: on the PCA solution manifold, SGD favors large decoder weights. This directed scale drift occurs on a slow timescale, and its dynamics admit an analytically-tractable effective description. However, it cannot continue indefinitely: increasing scale eventually drives the dynamics towards a finite-stepsize stability boundary. The resulting solutions are sharper than a balanced baseline in the sense of the maximum eigenvalue of the loss Hessian, but different sharpness measures can move in opposing directions. Thus, undercomplete autoencoders give a concrete illustration of how loss geometry can convert residual gradient noise into directed motion along a manifold of functionally-equivalent solutions.

Figures & tables

Figure 1: (A) Continual minibatch training drives scale imbalance in linear autoencoders. Frobenius norms of decoder and encoder weights, $\|W\|_{F}$ and $\|U\|_{F}$ , for contractive and expansive autoencoders, respectively (see App. F and Fig. S1 for additional details). (B) Scale growth in a contractive linear autoencoder for different learning rates ( $\eta$ ) and minibatch size ( $B$ ). $s_{1}=(W^{\top}W)_{11}$ is the dominant scale. Theory predictions are based on Eq. 6 . (C) Anisotropic scale growth in a contractive autoencoder. Different scale modes, $s_{i}=(W^{\top}W)_{ii}$ , grow at different rates. Theory predictions are based on Eq. 6 . (D-F) Edge of Stability phenomenon driven by imbalancing dynamics. (D) Scale growth and saturation for different modes. (E) Sequential saturation of eigenvalues. (F) Saturation as a function of the inverse of the learning rate and comparison to theory.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure S1: Continual minibatch training drives scale imbalance in linear autoencoders. (A) Validation loss for contractive and expansive linear autoencoders during online training with random initialization at the onset. (B) Scale imbalance is measured by $q=\|W\|_{F}^{2}-\|U\|_{F}^{2}$ . (C,D) Frobenius norms of decoder and encoder weights, $\|W\|_{F}$ and $\|U\|_{F}$ , for contractive and expansive autoencoders, respectively (see § F for experimental details).

Figure S2: Continual minibatch training drives scale imbalance in homogeneous autoencoders. (A) Validation loss for contractive and expansive ReLU autoencoders during online training with random initialization at the onset. (B) Scale imbalance is measured by $q$ as defined in Eq. ( E.3 ). In this case we have a decoder weight matrix $W$ and an encoder weight matrix $U$ , so $q=\|W\|_{F}^{2}-\|U\|_{F}^{2}$ . (C,D) Frobenius norms of decoder and encoder weights, $\|W\|_{F}$ and $\|U\|_{F}$ , for expansive and contractive ReLU autoencoders, respectively (see § F for experimental details).

Neural Network Training Dynamics

Neural network training dynamics studies how parameters and representations change during optimization and how those changes shape learning, stability, and generalization.

See all 519 papers

Learning the identity: a case study of how SGD selects among functional decompositions
Sep 30, 2026Andy Arditi, Weian Xie, David Bau +1Deep Linear Networks Implicit Bias
Northeastern University · Massachusetts Institute of Technology
A prism hierarchy of learning regimes in large linear autoencoders
Jun 3, 2026Eugene Golikov, Yaroslav Gusev, Dmitry YarotskyAutoencoders Neural Network Training Dynamics
Applied AI Institute Moscow, Russia · Steklov Mathematical Institute of Russsian Academy of Sciences Moscow, Russia
The Geometric Wall: Manifold Structure Predicts Layerwise Sparse Autoencoder Scaling Laws
May 11, 2026Eslam Zaher, Maciej Trzaskowski, Quan Nguyen +1Neural Representation Geometry Sparse Autoencoders
ARC Training Centre for Information Resilience (CIRES) · School of Mathematics and Physics, University of Queensland · Institute for Molecular Bioscience, University of Queensland +2

Sep 30, 2026J/K move · Enter open · S save

Learning the identity: a case study of how SGD selects among functional decompositions

Andy Arditi, Weian Xie, David Bau, Liu Ziyin

Northeastern University · Massachusetts Institute of Technology

Deep Linear Networks Implicit Bias Stochastic Gradient Descent

Open paper AskPDF arXiv

One might think that learning the identity function with a deep linear residual network is trivial - the path along residual connections already implements the identity, and so the network need only drive its weights to zero. However, this zero-weight solution is just one point on an entire manifold of population-loss minimizers, each corresponding to a different decomposition of the identity across the network's layers. Although the population loss does not distinguish among these solutions, stochastic gradient descent (SGD) reproducibly favors particular ones. For instance, under anisotropic label noise, the learned layers exhibit a noise-dependent spectrum; even with weight decay, SGD does not generally recover the zero-weight solution. Changing only the parametrization, while leaving the set of realizable functions unchanged, yields different behavior: factoring each weight matrix as a product of two matrices causes the weights to collapse to zero, even without explicit weight decay. While perhaps mysterious and unintuitive at first, these phenomena can be understood through the lens of entropic loss, which augments the population loss with a term proportional to the expected squared norm of the minibatch gradient (Ziyin et al., 2025). On the identity manifold, the population loss is constant, while the entropic term distinguishes among these decompositions. We characterize its minimizers analytically and use them to derive predictions for the structure of solutions favored by SGD. Networks trained with SGD closely match these predictions. Overall, the identity learning task studied here serves as a clean and simple case study of how the lens of entropic loss can clarify why SGD favors particular decompositions of the same input-output function.

Abstract

Neural Network Training Dynamics

Explore similar work

Learning the identity: a case study of how SGD selects among functional decompositions

A prism hierarchy of learning regimes in large linear autoencoders

The Geometric Wall: Manifold Structure Predicts Layerwise Sparse Autoencoder Scaling Laws