Overview

Deep linear networks are compositions of linear transformations used to study how depth changes learning, even when the overall input-output function remains linear. Researchers analyze gradient flow and gradient descent, matrix factorization, spectral dynamics, and convex reformulations across settings such as regression, classification, and matrix completion. These models help explain depth-dependent implicit biases, including preferences for low-rank or sparse solutions, as well as feature compression, convergence, and loss-landscape geometry. A central open tension is how reliably these exact results transfer to nonlinear networks, since behavior can depend strongly on initialization, data structure, optimizer, and finite-time dynamics.

Latest papers

34 papers6 papers in the last four weeks, level with the four weeks before. 0.0% of all new papers.

Jan 11, 2026

The Impact of Anisotropic Covariance Structure on the Training Dynamics and Generalization Error of Linear Networks

The success of deep neural networks largely depends on the statistical structure of the training data. While learning dynamics and generalization on isotropic data are well-established, the impact of pronounced anisotropy on these crucial aspects is not yet fully understood. We examine the impact of data anisotropy, represented by a spiked covariance structure, a canonical yet tractable model, on the learning dynamics and generalization error of a two-layer linear network in a linear regression setting. Our analysis reveals that the learning dynamics proceed in two distinct phases, governed initially by the input-output correlation and subsequently by other principal directions of the data structure. Furthermore, we derive an analytical expression for the generalization error, quantifying how the alignment of the spike structure of the data with the learning task improves performance. Our findings offer deep theoretical insights into how data anisotropy shapes the learning trajectory and final performance, providing a foundation for understanding complex interactions in more advanced network architectures.
Apr 23, 2024

The Geometry of the Set of Equivalent Linear Neural Networks

We characterize the geometry of the set of all weight vectors for which a fully-connected linear neural network computes the same linear transformation WW. Said another way, the set of all sequences of matrices (of predetermined sizes) whose product is WW. This set of weight vectors is an algebraic variety in the Euclidean weight space, called the fiber of WW under the matrix multiplication map. Sometimes this variety is a manifold, but usually not. The rank stratification of a fiber is a natural partition of the fiber into smooth manifolds of various dimensions called strata. We derive the dimensions of these strata and characterize how they are connected to each other. They satisfy the frontier condition: if a stratum intersects the closure of another stratum, then the former stratum is a subset of the closure of the latter stratum. This subset relationship can be expressed as a partial order with a single minimal element, a stratum that is included in the closure of every other stratum. Hence the strata can be arranged in a hierarchy represented by a directed acyclic graph that represents that partial order; we give an algorithm that computes that graph. A key result describes the relationship between this partial order and the ranks of certain matrices in the network. We derive the tangent and normal subspaces of the strata, and we construct elegant bases for those subspaces. Each stratum represents a different pattern of information flow through the network, expressed as a barcode, borrowed from the study of persistent homology. The topology of a stratum depends solely on its barcode. So does its geometry, up to a linear transformation in weight space. Connections among the strata are best understood through "barcode moves," which can be expressed as simple modifications of the barcodes.
Nov 6, 2023

Understanding Deep Representation Learning via Layerwise Feature Compression and Discrimination

Over the past decade, deep learning has proven to be a highly effective tool for learning meaningful features from raw data. However, it remains an open question how deep networks perform hierarchical feature learning across layers. In this work, we attempt to unveil this mystery by investigating the structures of intermediate features. Motivated by our empirical findings that linear layers mimic the roles of deep layers in nonlinear networks for feature learning, we explore how deep linear networks transform input data into output by investigating the output (i.e., features) of each layer after training in the context of multi-class classification problems. Toward this goal, we first define metrics to measure within-class compression and between-class discrimination of intermediate features, respectively. Through theoretical analysis of these two metrics, we show that the evolution of features follows a simple and quantitative pattern from shallow to deep layers when the input data is nearly orthogonal and the network weights are minimum-norm, balanced, and approximate low-rank: Each layer of the linear network progressively compresses within-class features at a geometric rate and discriminates between-class features at a linear rate with respect to the number of layers that data have passed through. To the best of our knowledge, this is the first quantitative characterization of feature evolution in hierarchical representations of deep linear networks. Empirically, our extensive experiments not only validate our theoretical results numerically but also reveal a similar pattern in deep nonlinear networks which aligns well with recent empirical studies. Moreover, we demonstrate the practical implications of our results in transfer learning. Our code is available at https://github.com/Heimine/PNC_DLN.