Deep Neural Regression Collapse

Akshay Rangamani; Altay Unal

cs.LGMar 25, 2026

Deep Neural Regression Collapse

Authors: Akshay Rangamani, Altay Unal

Organizations: Department of Data Science, New Jersey Institute of Technology

Abstract

Neural Collapse is a phenomenon that helps identify sparse and low rank structures in deep classifiers. Recent work has extended the definition of neural collapse to regression problems, albeit only measuring the phenomenon at the last layer. In this paper, we establish that Neural Regression Collapse (NRC) also occurs below the last layer across different types of models. We show that in the collapsed layers of neural regression models, features lie in a subspace that corresponds to the target dimension, the feature covariance aligns with the target covariance, the input subspace of the layer weights aligns with the feature subspace, and the linear prediction error of the features is close to the overall prediction error of the model. In addition to establishing Deep NRC, we also show that models that exhibit Deep NRC learn the intrinsic dimension of low rank targets and explore the necessity of weight decay in inducing Deep NRC. This paper provides a more complete picture of the simple structure learned by deep networks in the context of regression.

Figures & tables

Collapse Condition	Classification (NC) [ 1 , 2 ]	Regression (NRC) [ 5 ] , This paper
NC1/NRC1: Noise Suppression	$\mathrm{Tr}(\Sigma_{W}^{\ell})/\mathrm{Tr}(\Sigma_{T}^{\ell})\rightarrow 0$	$1-\frac{\mathrm{Tr}(\boldsymbol{U}^{\ell\top}\boldsymbol{\Sigma}_{\boldsymbol{H}^{\ell}}\boldsymbol{U}^{\ell})}{\mathrm{Tr}(\boldsymbol{\Sigma}_{\boldsymbol{H}^{\ell}})}\rightarrow 0$
NC2/NRC2: Signal-Target Alignment	$\boldsymbol{M}^{\ell}\boldsymbol{M}^{\ell\top}\propto\boldsymbol{I}_{C}-\frac{1}{C}\boldsymbol{1}_{C}\boldsymbol{1}_{C}^{\top}$	$\textrm{CKA}(\boldsymbol{H}^{\ell},\boldsymbol{Y})\approx 1$
NC3/NRC3: Feature-Weight Alignment	$\tfrac{1}{C}\sum\cos\angle(\boldsymbol{M}^{\ell},\boldsymbol{W}^{\ell\top})\rightarrow 1$	$\tfrac{1}{t}\sum\cos\angle(\boldsymbol{U}^{\ell},\boldsymbol{W}^{\ell\top})\rightarrow 1$
NC4/NRC4: Linear Predictability	Acc (NCC $(\boldsymbol{H}^{\ell})$ ) $\approx$ Acc ( $f_{\boldsymbol{W}}$ )	$\frac{1}{N}\\|\boldsymbol{H}^{\ell}(\boldsymbol{H}^{\ell\dagger}\boldsymbol{Y})-\boldsymbol{Y}\\|_{F}^{2}\approx\frac{1}{N}\\|f_{\boldsymbol{W}}(\boldsymbol{X})-\boldsymbol{Y}\\|_{F}^{2}$

Table 1: A unified picture of Neural Collapse in classification and regression

Dataset	Input Dimension	Target Dimension	Number of Samples	Task	Architecture
Swimmer	8	2	1,000	Imitation Learning	8-layer MLP, hidden dimension $h=256$
Reacher	11	2	1,000		8-layer MLP, hidden dimension $h=512$
Hopper	11	3	5,000		8-layer MLP, hidden dimension $h=256$
Carla2D [ 37 ]	$288\times 200\times 3$	2	50,000	Driving Simulation	ResNet-18
UTKFace [ 38 ]	$200\times 200\times 3$	1	25,000	Age Regression	ResNet-34
SGEMM [ 39 ]	14	4	240,000	Matrix Multiplication	8-layer MLP, hidden dimension $h=512$

Table 2: Overview of datasets and models used in experiments

Panel of Figure 1 — Figure 1: Deep NRC in ResNets: NRC measurements from a ResNet34 trained on the age-regression task in UTKFace (left column) and a ResNet18 trained on Carla2D (right column). The vertical green line in all plots indicates the first collapsed layer. First row (NRC1) shows the noise component being a small fraction of the energy in the collapsed layer representations. The second row shows the CKA between layer features and the target (NRC2). The third row (NRC3) shows the alignment between the features and the weights in the collapsed layers and the final row (NRC4) shows the MSE of linearly predicting the targets from the features in each layer.

Panel of Figure 2 — Figure 2: Deep NRC in MLPs: NRC measurements from $8$ -layer MLPs trained on imitation learning tasks in the MuJoCo Swimmer (left column) and Hopper (right column) environments. The vertical green line in all plots indicates the first collapsed layer. Top row (NRC1) shows the noise component being a small fraction of the energy in the collapsed layer representations. The second row shows the CKA between layer features and the target (NRC2). The third row (NRC3) shows the alignment between the features and the weights in the collapsed layers and the bottom row (NRC4) shows the MSE of linearly predicting the targets from the features in each layer.

Panel of Figure 3 — Figure 3: Learning intrinsic dimension of low rank targets: Top row - $8$ -layer, $256$ -width MLP trained on SGEMM ( $4$ -dim target, rank- $1$ ). Bottom row - $10$ -layer, $1024$ -width MLP trained on synthetic low rank nonlinear dataset ( $10$ -dim target, rank- $2$ ). In the left column we plot the noise component using the NRC1 formula, but measured using the bottom ( $h-r$ ) dimensional subspace, not ( $h-t$ ). In the middle column, we plot the stable rank of the layer features and observe that it matches the target stable rank in the collapsed layers. In the right column, we plot the feature-weight alignment (NRC3) using the top- $r$ dimensional subspace in purple and the alignment between the features and the bottom ( $h-r$ ) dimensional subspace of the weights in salmon. These plots establish that collapsed layers learn the intrinsic dimension of the data.

Panel of Figure 4 — Figure 4: Effect of weight decay: Left Column - We train ResNet18s on CARLA2D with varying values of weight decay, and observe the effects on deep neural regression collapse. In the top and middle rows we plot the NRC1 and NRC3 metrics. The bottom row shows the stable rank of the weights. In each plot, the measurements with the right value of weight decay ( $\lambda=5e-3$ ) are shown in purple, while the measurements with smaller values of weight decay ( $\lambda\in[5e-4,5e-5,0]$ ) are in salmon. This shows that weight decay is necessary to achieve feature-weight alignment, which implies a low rank bias in the weights of the layers. In the right column we explore through an experiment on synthetic data, how larger values of weight decay ( $\lambda=1e-3$ ) can induce NRC1 and NRC3 (top and middle rows), but perform worse at prediction (bottom row). While weight decay is necessary for observing Deep NRC, too high a value can hinder learning.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: Loss Plots: The training and test loss plots for the collapsed models trained on SGEMM (top left), Swimmer (top right), Reacher (middle left), Hopper (middle right), Carla2D (bottom left), and UTKFace (bottom right) are shown. The vertical green line indicates the epoch at which the trained models first start experiencing Deep NRC. We observe that the collapsed models exhibit Deep NRC after both training and test losses are stabilized. We also observe that the generalization gap is small for the models experiencing Deep NRC, showing that Deep NRC promotes the generalization capabilities of the deep learning models.

Panel of Figure 6 — Figure 6: Results on Reacher dataset: Top row - Noise Suppression (NRC1) shown in the left plot and CKA alignment (NRC2) shown in the right plot . Bottom row - Feature-Weight Alignment (NRC3) shown in the left plot and the right plot shows Linear Predictability (NRC4) or the MSE of predicting the target from the layer features using the pseudo inverse method (overall train loss shown for reference). We observe that all four conditions of collapse occur not just in the last layer, but also in the previous layers.

Panel of Figure 7 — Figure 7: Results on SGEMM dataset: Top row - Noise Suppression (NRC1) shown in the left plot and CKA alignment (NRC2) shown in the right plot . Bottom row - Feature-Weight Alignment (NRC3) shown in the left plot and the right plot shows Linear Predictability (NRC4) or the MSE of predicting the target from the layer features using the pseudo inverse method (overall train loss shown for reference). We observe that all four conditions of collapse occur not just in the last layer, but also in the previous layers.

Panel of Figure 8 — Figure 8: Results on synthetic nonlinear dataset: Top row - Noise Suppression (NRC1) shown in the left two plots - noise component and stable rank $\boldsymbol{H}$ (target stable rank shown for reference) - and CKA alignment (NRC2) shown in the right most plot. Bottom row - Feature-Weight Alignment (NRC3) shown in the left most plot, middle plot measures the stable rank of the weight matrix $\boldsymbol{W}$ and the right most plot shows Linear Predictability (NRC4) or the MSE of predicting the target from the layer features using the pseudo inverse method (overall train loss shown for reference). We observe that all four conditions of collapse occur not just in the last layer, but also in the previous layers. Inputs are drawn from a $20$ -dimensional normal distribution. $t=3$ dimensional targets for this model were generated from a fully-connected neural network with 2 hidden layers of dimension $r=10$ .

Panel of Figure 9 — Figure 9: Weight decay experiment results on SGEMM dataset: We train MLPs on SGEMM dataset with varying values of weight decay, and observe the effects of deep NRC. From left to right, we plot the NRC1, NRC3 metrics, and the stable rank of the weights at each layer respectively. In each plot, the measurements with the right weight decay value ( $\lambda=1e-3$ ) are shown in purple, while the remaining weight decay values ( $\lambda=[1e-4,1e-5,0]$ ) are shown in salmon. From the top left figure, we observe that the noise component is progressing to 0 for the models trained with lower amounts of weight decay or without weight decay. However, we observe that the model trained with the right weight decay induces NRC1 in earlier layers compared to the other models. Additionally, we also observe that NRC3 is only induced with the right weight decay value while the models with lower weight decay cannot achieve feature-weight alignment. This shows that weight decay is a necessity for NRC3, and this implies a low-rank bias in the weights of the layers, which can also be observed with the bottom plot.

Representation Learning

Representation learning develops data transformations that expose useful structure for prediction, transfer, generation, and control.

See all 799 papers

Neural Collapse Dynamics: Depth, Activation, Regularisation, and Feature Norm Threshold
Mar 31, 2026Anamika Paul RupaNeural Collapse Neural Network Training Dynamics
Department of Electrical Engineering and Computer Science, Howard University, Washington, DC, 20059, USA
A Closed Form Depth-Width Law for Neural Collapse in Deep Linear Networks
Oct 15, 2026Pankhil Gawade, Kirsten Fischer, Guy Wolf +1Neural Representation Geometry Neural Scaling Laws
Institute of AI for Health, Helmholtz Munich · School of Computation, Information and Technology, Technical University of Munich · Université de Montréal +3
The Implicit Bias of Depth: From Neural Collapse to Softmax Codes
May 21, 2026Connall Garrod, Jonathan P. Keating, Christos ThrampoulidisDeep Linear Networks Implicit Bias
Mathematical Institute, University of Oxford · Department of Electrical and Computer Engineering, University of British Columbia

Mar 31, 2026J/K move · Enter open · S save

Neural Collapse Dynamics: Depth, Activation, Regularisation, and Feature Norm Threshold

Anamika Paul Rupa

Department of Electrical Engineering and Computer Science, Howard University, Washington, DC, 20059, USA

Neural Collapse Neural Network Training Dynamics

Open paper AskPDF arXiv

Neural collapse (NC) -- the convergence of penultimate-layer features to a simplex equiangular tight frame -- is well understood at equilibrium, but the dynamics governing its onset remain poorly characterised. We identify a simple and predictive regularity: NC occurs when the mean feature norm reaches a model-dataset-specific critical value, fn*, that is largely invariant to training conditions. This value concentrates tightly within each (model, dataset) pair (CV < 8%); training dynamics primarily affect the rate at which fn approaches fn*, rather than the value itself. In standard training trajectories, the crossing of fn below fn* consistently precedes NC onset, providing a practical predictor with a mean lead time of 62 epochs (MAE 24 epochs). A direct intervention experiment confirms fn* is a stable attractor of the gradient flow -- perturbations to feature scale are self-corrected during training, with convergence to the same value regardless of direction (p>0.2). Completing the (architecture)x(dataset) grid reveals the paper's strongest result: ResNet-20 on MNIST gives fn* = 5.867 -- a +458% architecture effect versus only +68% on CIFAR-10. The grid is strongly non-additive; fn* cannot be decomposed into independent architecture and dataset contributions. Four structural regularities emerge: (1) depth has a non-monotonic effect on collapse speed; (2) activation jointly determines both collapse speed and fn*; (3) weight decay defines a three-regime phase diagram -- too little slows, an optimal range is fastest, and too much prevents collapse; (4) width monotonically accelerates collapse while shifting fn* by at most 13%. These results establish feature-norm dynamics as an actionable diagnostic for predicting NC timing, suggesting that norm-threshold behaviour is a general mechanism underlying delayed representational reorganisation in deep networks.

Abstract

Representation Learning

Explore similar work

Neural Collapse Dynamics: Depth, Activation, Regularisation, and Feature Norm Threshold

A Closed Form Depth-Width Law for Neural Collapse in Deep Linear Networks

The Implicit Bias of Depth: From Neural Collapse to Softmax Codes