math.DGOct 31, 2026

The Continuum Limit of the Deep Linear Network

Authors: Catherine Eva Pfaff, Karen Uhlenbeck

Abstract

The Deep Linear Network (DLN) is a simple version of a Multilayer Perceptron (MLP) in which the activation function is the identity and the bias vector is the zero vector. We examine the behavior of the model when the number of weight matrices, which is finite in the original model, goes to infinity. We show that a plausible continuum limit is a dimensionally reduced version of the Yang-Mills equation on the torus, revealing an unexpected geometric interpretation of infinite-depth DLNs. While we do not prove this rigorously, it is a very pretty calculation. It also gives some insight into several features of the construction. We note that the choices of boundary conditions on the ends of the discrete series survive better in the continuum limit. Also, the regularizing functional which uses the moment map survives nicely in the continuum limit, whereas the regularizing functional using the size of approximating terms has no sensible limit. This surely is useful information for the choices we make in the finite-dimensional theory, and might also be useful in the more complicated MLP setting.

Explore similar work

May 6, 2026

Why Geometric Continuity Emerges in Deep Neural Networks: Residual Connections and Rotational Symmetry Breaking

Weight matrices in deep networks exhibit geometric continuity---principal singular vectors of adjacent layers point in similar directions. While this property has been widely observed, its origin remains unexplained. Through experiments on toy MLPs and small transformers, we identify two mechanisms: residual connections create cross-layer gradient coherence that aligns weight updates across layers, and symmetry-breaking nonlinearities constrain all layers to a shared coordinate frame, preventing the rotation drift that would otherwise destabilize weight structure. Crucially, a nonlinear but rotation-equivariant activation reaches less than half of ReLU's continuity, isolating symmetry breaking---not nonlinearity alone---as the active ingredient. Activation and normalization play distinct roles: activation concentrates continuity in the leading singular direction, while normalization distributes it across multiple directions. In transformers, continuity is projection-specific: Q, K, Gate, and Up (which read from the residual stream) develop input-space (v1\mathbf{v}_1) continuity; O and Down (which write to it) develop output-space (u1\mathbf{u}_1) continuity; V alone, lacking an adjacent nonlinearity, develops only low continuity.
Jun 16, 2025

Random Matrix Theory for Deep Learning: Beyond Eigenvalues of Linear Models

Modern Machine Learning (ML) and Deep Neural Networks (DNNs) often operate on high-dimensional data and rely on overparameterized models, where classical low-dimensional intuitions break down. In particular, the proportional regime where the data dimension, sample size, and number of model parameters are all large and comparable, gives rise to novel and sometimes counterintuitive behaviors. This paper extends traditional Random Matrix Theory (RMT) beyond eigenvalue-based analysis of linear models to address the challenges posed by nonlinear ML models such as DNNs in this regime. We introduce the concept of High-dimensional Equivalent, which unifies and generalizes both Deterministic Equivalent and Linear Equivalent, to systematically address three technical challenges: high dimensionality, nonlinearity, and the need to analyze generic eigenspectral functionals. Leveraging this framework, we provide precise characterizations of the training and generalization performance of linear models, nonlinear shallow networks, and deep networks. Our results capture rich phenomena, including scaling laws, double descent, and nonlinear learning dynamics, offering a unified perspective on the theoretical understanding of deep learning in high dimensions.
May 4, 2026

Universality in Deep Neural Networks: An approach via the Lindeberg exchange principle

We consider the infinite-width limit of a fully connected deep neural network with general weights, and we prove quantitative general bounds on the 22-Wasserstein distance between the network and its infinite-width Gaussian limit, under appropriate regularity assumptions on the activation function. Our main tool is a Lindeberg principle for Deep Neural Networks, which we use to successively replace the weights on each layer by Gaussian random variables.