Neural Collapse is a phenomenon that helps identify sparse and low rank structures in deep classifiers. Recent work has extended the definition of neural collapse to regression problems, albeit only measuring the phenomenon at the last layer. In this paper, we establish that Neural Regression Collapse (NRC) also occurs below the last layer across different types of models. We show that in the collapsed layers of neural regression models, features lie in a subspace that corresponds to the target dimension, the feature covariance aligns with the target covariance, the input subspace of the layer weights aligns with the feature subspace, and the linear prediction error of the features is close to the overall prediction error of the model. In addition to establishing Deep NRC, we also show that models that exhibit Deep NRC learn the intrinsic dimension of low rank targets and explore the necessity of weight decay in inducing Deep NRC. This paper provides a more complete picture of the simple structure learned by deep networks in the context of regression.
Figures & tables
Collapse Condition
Classification (NC) [ 1 , 2 ]
Regression (NRC) [ 5 ] , This paper
NC1/NRC1: Noise Suppression
Tr(ΣWℓ)/Tr(ΣTℓ)→0
1−Tr(ΣHℓ)Tr(Uℓ⊤ΣHℓUℓ)→0
NC2/NRC2: Signal-Target Alignment
MℓMℓ⊤∝IC−C11C1C⊤
CKA(Hℓ,Y)≈1
NC3/NRC3: Feature-Weight Alignment
C1∑cos∠(Mℓ,Wℓ⊤)→1
t1∑cos∠(Uℓ,Wℓ⊤)→1
NC4/NRC4: Linear Predictability
Acc (NCC (Hℓ) ) ≈ Acc ( fW )
N1∥Hℓ(Hℓ†Y)−Y∥F2≈N1∥fW(X)−Y∥F2
Table 1: A unified picture of Neural Collapse in classification and regression
Dataset
Input Dimension
Target Dimension
Number of Samples
Task
Architecture
Swimmer
8
2
1,000
Imitation Learning
8-layer MLP, hidden dimension h=256
Reacher
11
2
1,000
8-layer MLP, hidden dimension h=512
Hopper
11
3
5,000
8-layer MLP, hidden dimension h=256
Carla2D [ 37 ]
288×200×3
2
50,000
Driving Simulation
ResNet-18
UTKFace [ 38 ]
200×200×3
1
25,000
Age Regression
ResNet-34
SGEMM [ 39 ]
14
4
240,000
Matrix Multiplication
8-layer MLP, hidden dimension h=512
Table 2: Overview of datasets and models used in experiments
Figure 1: Deep NRC in ResNets: NRC measurements from a ResNet34 trained on the age-regression task in UTKFace (left column) and a ResNet18 trained on Carla2D (right column). The vertical green line in all plots indicates the first collapsed layer. First row (NRC1) shows the noise component being a small fraction of the energy in the collapsed layer representations. The second row shows the CKA between layer features and the target (NRC2). The third row (NRC3) shows the alignment between the features and the weights in the collapsed layers and the final row (NRC4) shows the MSE of linearly predicting the targets from the features in each layer.
Figure 2: Deep NRC in MLPs: NRC measurements from 8 -layer MLPs trained on imitation learning tasks in the MuJoCo Swimmer (left column) and Hopper (right column) environments. The vertical green line in all plots indicates the first collapsed layer. Top row (NRC1) shows the noise component being a small fraction of the energy in the collapsed layer representations. The second row shows the CKA between layer features and the target (NRC2). The third row (NRC3) shows the alignment between the features and the weights in the collapsed layers and the bottom row (NRC4) shows the MSE of linearly predicting the targets from the features in each layer.
Figure 3: Learning intrinsic dimension of low rank targets: Top row - 8 -layer, 256 -width MLP trained on SGEMM ( 4 -dim target, rank- 1 ). Bottom row - 10 -layer, 1024 -width MLP trained on synthetic low rank nonlinear dataset ( 10 -dim target, rank- 2 ). In the left column we plot the noise component using the NRC1 formula, but measured using the bottom ( h−r ) dimensional subspace, not ( h−t ). In the middle column, we plot the stable rank of the layer features and observe that it matches the target stable rank in the collapsed layers. In the right column, we plot the feature-weight alignment (NRC3) using the top- r dimensional subspace in purple and the alignment between the features and the bottom ( h−r ) dimensional subspace of the weights in salmon. These plots establish that collapsed layers learn the intrinsic dimension of the data.
Figure 4: Effect of weight decay: Left Column - We train ResNet18s on CARLA2D with varying values of weight decay, and observe the effects on deep neural regression collapse. In the top and middle rows we plot the NRC1 and NRC3 metrics. The bottom row shows the stable rank of the weights. In each plot, the measurements with the right value of weight decay ( λ=5e−3 ) are shown in purple, while the measurements with smaller values of weight decay ( λ∈[5e−4,5e−5,0] ) are in salmon. This shows that weight decay is necessary to achieve feature-weight alignment, which implies a low rank bias in the weights of the layers. In the right column we explore through an experiment on synthetic data, how larger values of weight decay ( λ=1e−3 ) can induce NRC1 and NRC3 (top and middle rows), but perform worse at prediction (bottom row). While weight decay is necessary for observing Deep NRC, too high a value can hinder learning.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Loss Plots: The training and test loss plots for the collapsed models trained on SGEMM (top left), Swimmer (top right), Reacher (middle left), Hopper (middle right), Carla2D (bottom left), and UTKFace (bottom right) are shown. The vertical green line indicates the epoch at which the trained models first start experiencing Deep NRC. We observe that the collapsed models exhibit Deep NRC after both training and test losses are stabilized. We also observe that the generalization gap is small for the models experiencing Deep NRC, showing that Deep NRC promotes the generalization capabilities of the deep learning models.
Figure 6: Results on Reacher dataset: Top row - Noise Suppression (NRC1) shown in the left plot and CKA alignment (NRC2) shown in the right plot . Bottom row - Feature-Weight Alignment (NRC3) shown in the left plot and the right plot shows Linear Predictability (NRC4) or the MSE of predicting the target from the layer features using the pseudo inverse method (overall train loss shown for reference). We observe that all four conditions of collapse occur not just in the last layer, but also in the previous layers.
Figure 7: Results on SGEMM dataset: Top row - Noise Suppression (NRC1) shown in the left plot and CKA alignment (NRC2) shown in the right plot . Bottom row - Feature-Weight Alignment (NRC3) shown in the left plot and the right plot shows Linear Predictability (NRC4) or the MSE of predicting the target from the layer features using the pseudo inverse method (overall train loss shown for reference). We observe that all four conditions of collapse occur not just in the last layer, but also in the previous layers.
Figure 8: Results on synthetic nonlinear dataset: Top row - Noise Suppression (NRC1) shown in the left two plots - noise component and stable rank H (target stable rank shown for reference) - and CKA alignment (NRC2) shown in the right most plot. Bottom row - Feature-Weight Alignment (NRC3) shown in the left most plot, middle plot measures the stable rank of the weight matrix W and the right most plot shows Linear Predictability (NRC4) or the MSE of predicting the target from the layer features using the pseudo inverse method (overall train loss shown for reference). We observe that all four conditions of collapse occur not just in the last layer, but also in the previous layers. Inputs are drawn from a 20 -dimensional normal distribution. t=3 dimensional targets for this model were generated from a fully-connected neural network with 2 hidden layers of dimension r=10 .
Figure 9: Weight decay experiment results on SGEMM dataset: We train MLPs on SGEMM dataset with varying values of weight decay, and observe the effects of deep NRC. From left to right, we plot the NRC1, NRC3 metrics, and the stable rank of the weights at each layer respectively. In each plot, the measurements with the right weight decay value ( λ=1e−3 ) are shown in purple, while the remaining weight decay values ( λ=[1e−4,1e−5,0] ) are shown in salmon. From the top left figure, we observe that the noise component is progressing to 0 for the models trained with lower amounts of weight decay or without weight decay. However, we observe that the model trained with the right weight decay induces NRC1 in earlier layers compared to the other models. Additionally, we also observe that NRC3 is only induced with the right weight decay value while the models with lower weight decay cannot achieve feature-weight alignment. This shows that weight decay is a necessity for NRC3, and this implies a low-rank bias in the weights of the layers, which can also be observed with the bottom plot.
Neural collapse (NC) -- the convergence of penultimate-layer features to a simplex equiangular tight frame -- is well understood at equilibrium, but the dynamics governing its onset remain poorly characterised. We identify a simple and predictive regularity: NC occurs when the mean feature norm reaches a model-dataset-specific critical value, fn*, that is largely invariant to training conditions. This value concentrates tightly within each (model, dataset) pair (CV < 8%); training dynamics primarily affect the rate at which fn approaches fn*, rather than the value itself. In standard training trajectories, the crossing of fn below fn* consistently precedes NC onset, providing a practical predictor with a mean lead time of 62 epochs (MAE 24 epochs). A direct intervention experiment confirms fn* is a stable attractor of the gradient flow -- perturbations to feature scale are self-corrected during training, with convergence to the same value regardless of direction (p>0.2). Completing the (architecture)x(dataset) grid reveals the paper's strongest result: ResNet-20 on MNIST gives fn* = 5.867 -- a +458% architecture effect versus only +68% on CIFAR-10. The grid is strongly non-additive; fn* cannot be decomposed into independent architecture and dataset contributions. Four structural regularities emerge: (1) depth has a non-monotonic effect on collapse speed; (2) activation jointly determines both collapse speed and fn*; (3) weight decay defines a three-regime phase diagram -- too little slows, an optimal range is fastest, and too much prevents collapse; (4) width monotonically accelerates collapse while shifting fn* by at most 13%. These results establish feature-norm dynamics as an actionable diagnostic for predicting NC timing, suggesting that norm-threshold behaviour is a general mechanism underlying delayed representational reorganisation in deep networks.
Anamika Paul Rupa
Department of Electrical Engineering and Computer Science, Howard University, Washington, DC, 20059, USA
How depth and width of neural networks shape learned representations is usually studied only after training. We show that, in a tractable feature-learning setting, this dependence can instead be obtained in closed form. Using neural collapse as a probe of representation geometry, we express NC1--NC3 directly in terms of the learned feature kernel, extending an identity known for NC1 to the angular part of NC2 and to NC3 under a ridge readout. For deep linear networks with exchangeable class structure, the resulting kernel equations yield a geometric law, exact within the theory, for the layerwise evolution of within-class variability. At large width, the accumulated compression becomes a function of the depth-to-width ratio L/N, with its rate determined self-consistently by the data, prior and architecture. Numerical experiments support the derived dependence on depth, width and signal-to-noise ratio. The result is a representation-level scaling law derived from feature-learning theory rather than fitted after training.
Pankhil Gawade, Kirsten Fischer, Guy Wolf +1
Institute of AI for Health, Helmholtz Munich · School of Computation, Information and Technology, Technical University of Munich · Université de Montréal +3
Neural collapse (NC) describes the structured geometry that emerges in the features and weights of trained classifiers. Recent theory suggests NC can be suboptimal in deep architectures, attributing this to an explicit low-rank bias from L2 regularization. We study the deep unconstrained feature model (UFM)-equivalent to a deep linear network with orthogonal inputs-trained without regularization, to isolate how gradient descent and depth alone shape NC. We show that depth induces an implicit low-rank bias: low-rank matrices propagate norm more efficiently through successive multiplications, promoting low-rank alternatives to NC. These alternatives, we argue, correspond to softmax codes: max-margin solutions previously found in width-bottlenecked networks. Analyzing training dynamics under spectral initialization, we identify an early-time repulsion among singular values that drives low-rank emergence, and characterize how depth shrinks NC's basin of attraction. Finally, we show that some effects act in the opposite direction: for randomly initialized networks, increasing width biases training toward higher-rank solutions. Our results provide the first asymptotic and dynamic characterization of implicit bias in deep UFMs trained with unregularized multiclass cross-entropy.
Connall Garrod, Jonathan P. Keating, Christos Thrampoulidis
Mathematical Institute, University of Oxford · Department of Electrical and Computer Engineering, University of British Columbia