A Closed Form Depth-Width Law for Neural Collapse in Deep Linear Networks

Pankhil Gawade; Kirsten Fischer; Guy Wolf; Ewa Szczurek

cs.LGOct 15, 2026

A Closed Form Depth-Width Law for Neural Collapse in Deep Linear Networks

Authors: Pankhil Gawade, Kirsten Fischer, Guy Wolf, Ewa Szczurek

Organizations: Institute of AI for Health, Helmholtz Munich · School of Computation, Information and Technology, Technical University of Munich · Université de Montréal · Mila – Quebec AI Institute ·

Abstract

How depth and width of neural networks shape learned representations is usually studied only after training. We show that, in a tractable feature-learning setting, this dependence can instead be obtained in closed form. Using neural collapse as a probe of representation geometry, we express NC1--NC3 directly in terms of the learned feature kernel, extending an identity known for NC1 to the angular part of NC2 and to NC3 under a ridge readout. For deep linear networks with exchangeable class structure, the resulting kernel equations yield a geometric law, exact within the theory, for the layerwise evolution of within-class variability. At large width, the accumulated compression becomes a function of the depth-to-width ratio $L/N$ , with its rate determined self-consistently by the data, prior and architecture. Numerical experiments support the derived dependence on depth, width and signal-to-noise ratio. The result is a representation-level scaling law derived from feature-learning theory rather than fitted after training.

Figures & tables

Figure 1: Verification of the exchangeable linear theory. (a) Geometric depth law for $C=5$ and $g_{\ell}=1$ : normalized NC1, $\mathrm{NC}_{1}^{(l)}/\mathrm{NC}_{1}^{(0)}$ , versus layer for $N=64,128,256,512$ , comparing Theorem 2 with the numerical fixed-point solution. (b) Data dependence of the contraction rate for $C=5$ , $g_{\ell}=1$ , $N=256$ , and $L=16$ : $\log(t_{W}/t_{B})$ versus SNR. (c) Depth–width scaling for $C=5$ and $g_{\ell}=1$ : terminal normalized NC1 versus $\tau=L/N$ , showing the predicted leading dependence on $L/N$ . (d) Layerwise NC1 for a sampled four-class Gaussian task and a contractive linear network ( $v_{w}=0.5$ ), comparing the closed-form prediction with an SGLD-trained finite-width network.

Figure 2: NC1, NC2 and NC3 on four-class XOR ( $N=256$ , $L=3$ ), computed from the self-consistent kernel theory and from trained finite-width networks. Error bars denote variation across seeds.

Neural Representation Geometry

Neural representation geometry studies how learned features are arranged and how that structure relates to model behavior and performance.

See all 253 papers

The Weight Gram Matrix Captures Sequential Feature Linearization in Deep Networks
May 7, 2026Taehun Cha, Daniel Beaglehole, Adityanarayanan Radhakrishnan +1Neural Network Training Dynamics Gradient Descent Dynamics
Department of Mathematics Korea University · Computer Science and Engineering UC San Diego · MIT Mathematics Broad Institute of MIT and Harvard
Understanding Deep Representation Learning via Layerwise Feature Compression and Discrimination
Nov 6, 2023Peng Wang, Xiao Li, Can Yaras +4Hierarchical Representation Learning Deep Linear Networks
University of Michigan · Ohio State University
Neural Collapse Dynamics: Depth, Activation, Regularisation, and Feature Norm Threshold
Mar 31, 2026Anamika Paul RupaNeural Collapse Neural Network Training Dynamics
Department of Electrical Engineering and Computer Science, Howard University, Washington, DC, 20059, USA

May 7, 2026J/K move · Enter open · S save

The Weight Gram Matrix Captures Sequential Feature Linearization in Deep Networks

Taehun Cha, Daniel Beaglehole, Adityanarayanan Radhakrishnan, Donghun Lee

Department of Mathematics Korea University · Computer Science and Engineering UC San Diego · MIT Mathematics Broad Institute of MIT and Harvard

Neural Network Training Dynamics Gradient Descent Dynamics

Open paper AskPDF arXiv

Understanding how deep neural networks learn representations remains a central challenge in machine learning theory. In this work, we propose a feature-centric framework for analyzing neural network training by relating weight updates to feature evolution. We introduce a simple identity, the Feature Learning Equation, which identifies the weight Gram matrix as the key object capturing feature dynamics. This enables us to interpret gradient descent as implicitly inducing a hypothetical evolution of features, whose covariance structure - termed the Virtual Covariance - characterizes how representations evolve during training. Building on this perspective, we introduce Target Linearity, a measure quantifying the linear alignment between features and targets. By analyzing the training and layer-wise dynamics, we show that deep networks learn to sequentially transform representations toward target-linear structure. This linearization perspective provides a unified interpretation of several empirical phenomena, including Neural Collapse and linear interpolation in generative models.

Abstract

Neural Representation Geometry

Explore similar work

The Weight Gram Matrix Captures Sequential Feature Linearization in Deep Networks

Understanding Deep Representation Learning via Layerwise Feature Compression and Discrimination

Neural Collapse Dynamics: Depth, Activation, Regularisation, and Feature Norm Threshold