Neural collapse (NC) -- the convergence of penultimate-layer features to a simplex equiangular tight frame -- is well understood at equilibrium, but the dynamics governing its onset remain poorly characterised. We identify a simple and predictive regularity: NC occurs when the mean feature norm reaches a model-dataset-specific critical value, fn*, that is largely invariant to training conditions. This value concentrates tightly within each (model, dataset) pair (CV < 8%); training dynamics primarily affect the rate at which fn approaches fn*, rather than the value itself. In standard training trajectories, the crossing of fn below fn* consistently precedes NC onset, providing a practical predictor with a mean lead time of 62 epochs (MAE 24 epochs). A direct intervention experiment confirms fn* is a stable attractor of the gradient flow -- perturbations to feature scale are self-corrected during training, with convergence to the same value regardless of direction (p>0.2). Completing the (architecture)x(dataset) grid reveals the paper's strongest result: ResNet-20 on MNIST gives fn* = 5.867 -- a +458% architecture effect versus only +68% on CIFAR-10. The grid is strongly non-additive; fn* cannot be decomposed into independent architecture and dataset contributions. Four structural regularities emerge: (1) depth has a non-monotonic effect on collapse speed; (2) activation jointly determines both collapse speed and fn*; (3) weight decay defines a three-regime phase diagram -- too little slows, an optimal range is fastest, and too much prevents collapse; (4) width monotonically accelerates collapse while shifting fn* by at most 13%. These results establish feature-norm dynamics as an actionable diagnostic for predicting NC timing, suggesting that norm-threshold behaviour is a general mechanism underlying delayed representational reorganisation in deep networks.
Figures & tables
Condition
fn∗ @ NC1 < 0.01
fn∗ @ NC1 < 0.02
Ratio
Depth = 5
1.077
1.251
1.16
Depth = 7
1.112
0.953
0.86
Width = 128
1.241
1.276
1.03
Width = 256
1.125
1.222
1.09
λ=10−5
1.035
1.197
1.16
λ=5×10−5
0.983
1.150
1.17
Table 1: Threshold robustness: fn∗ at NC1 <0.01 vs. NC1 <0.02 (MLP-5, ReLU, MNIST; mean of 3 seeds per condition).
Work
Onset
Norm marker
Arch. × data
Interv.
Type
Papyan et al. [34]
No
No
No
No
Empirical
Zhu et al. [55]
No
No
No
No
Theory
Han et al. [6]
Partial
No
No
No
Theory
Pan and Cao [33]
Partial
No
No
No
Theory
Wang et al. [45]
Yes
No
No
No
Empirical
Zhao et al. [52]
Yes
No
No
No
Theory+Emp.
Table 2: Positioning relative to representative prior work on neural collapse; the four comparison axes are defined in the text.
Table 4: Activation sweep (MLP-5, λ=10−4 , MNIST; mean ± std). † SiLU plateaus above NC1 =0.05 ; fn is reported at the epoch of minimum Phase-2 NC1 (Section 5.3 ).
Figure 3: Activation comparison on MLP-5/MNIST ( λ=10−4 , 3 seeds): (a) NC1 (log scale), (b) feature norm (log scale), with per-activation fn∗ labelled at the right edge. Tanh NC1 is lightly smoothed (3-point rolling mean) for display.
Figure 4: SiLU plateau dynamics (MLP-5/MNIST, λ=10−4 , Phase-2 1000 epochs, 3 seeds): (a) NC1 plateau, (b) feature norm with the reported fn∗ (dashed), (c) agreement of three plateau estimators.
λ
N
Collapsed
TNC
fn at TNC
10−5
3
3/3
380±50
1.035±0.087
5×10−5
3
3/3
290±10
0.983±0.006
10−4
3
3/3
297±12
1.077±0.025
5×10−4
3
0/3
DNF
n/a
Table 5: Weight-decay sweep (MLP-5, ReLU, MNIST, NC1 <0.01 ; mean ± std). DNF (did not finish): NC1 stalled at ≈0.03 for all seeds.
Condition
Values varied
TNC range
fn∗ (mean ± std)
Depth sweep
depth ∈{2,3,5,7}
250–340
1.259±0.251
WD sweep
λ∈{10−5,5×10−5,10−4}
290–380
1.032±0.061
Width sweep
width ∈{128,256,512,1024}
257–383
1.135±0.085
All ReLU runs
(combined, N=21 )
250–383
1.093±0.092
Table 6: Feature-norm threshold fn∗ pooled across all MNIST/MLP-5/ReLU conditions; the pool mixes strict (NC1 <0.01 ) and relaxed (depths 2–3, NC1 <0.05 ) collapse criteria.
Dataset
Architecture
NC1
N
fn∗
95% t -CI
CV
MNIST
MLP-5
<0.01
12
1.052±0.066
[1.010,1.093]
6.2%
MNIST
ResNet-20
<0.01
3
5.867±0.034
[5.781,5.952]
0.6%
Fashion-MNIST
MLP-5
<0.05
3
1.367±0.077
[1.176,1.559]
5.6%
Fashion-MNIST
ResNet-20
<0.01
3
1.678±0.012
[1.648,1.708]
0.7%
CIFAR-10
MLP-5
<0.05
3
0.900±0.065
[0.738,1.063]
7.3%
CIFAR-10
ResNet-20
<0.01
3
1.515±0.008
[1.494,1.535]
0.5%
Table 7: Feature-norm threshold fn∗ by (model, dataset) pair (ReLU); mean ± std, with the NC1 collapse criterion and confirmed-collapse count N . RN-20: ResNet-20; F-MNIST: Fashion-MNIST.
Figure 5: Weight-decay sweep (MLP-5, ReLU, MNIST): (a) TNC vs. λ , (b) fn at TNC , (c) all confirmed values.
Figure 6: Summary of NC dynamics results: (a) depth effect on TNC , (b) activation effect, (c) weight-decay effect, (d) fn by (model, dataset) pair.
Figure 7: ResNet-20 on MNIST (3 seeds): (a) accuracy, (b) NC1, (c) feature norm per seed.
Architecture
Optimiser
fn∗
MLP-5
Adam (matched)
1.095±0.039 (reproduces main grid)
ResNet-20
Adam (matched)
7.468±0.129
ResNet-20
SGD (main grid)
5.867
MLP-5
SGD (lr =0.1 )
→0 (degenerate, non-ETF; excluded)
Table 8: Optimiser de-confound on MNIST (three seeds each, strict NC1 <0.01 ; values mean ± std).
Figure 8: Fashion-MNIST baseline dynamics (MLP-5, ReLU, λ=10−4 ; mean ± std, 3 seeds): (a) accuracy, (b) NC1, (c) NC2 and NC3, (d) feature norm (dashed line marks the reported fn∗ ).
Figure 9: Feature-norm threshold fn∗ by (model, dataset) pair across the six evaluated cells, grouped by dataset and coloured by architecture (mean ± std).
Neural Collapse is a phenomenon that helps identify sparse and low rank structures in deep classifiers. Recent work has extended the definition of neural collapse to regression problems, albeit only measuring the phenomenon at the last layer. In this paper, we establish that Neural Regression Collapse (NRC) also occurs below the last layer across different types of models. We show that in the collapsed layers of neural regression models, features lie in a subspace that corresponds to the target dimension, the feature covariance aligns with the target covariance, the input subspace of the layer weights aligns with the feature subspace, and the linear prediction error of the features is close to the overall prediction error of the model. In addition to establishing Deep NRC, we also show that models that exhibit Deep NRC learn the intrinsic dimension of low rank targets and explore the necessity of weight decay in inducing Deep NRC. This paper provides a more complete picture of the simple structure learned by deep networks in the context of regression.
Akshay Rangamani, Altay Unal
Department of Data Science, New Jersey Institute of Technology
How depth and width of neural networks shape learned representations is usually studied only after training. We show that, in a tractable feature-learning setting, this dependence can instead be obtained in closed form. Using neural collapse as a probe of representation geometry, we express NC1--NC3 directly in terms of the learned feature kernel, extending an identity known for NC1 to the angular part of NC2 and to NC3 under a ridge readout. For deep linear networks with exchangeable class structure, the resulting kernel equations yield a geometric law, exact within the theory, for the layerwise evolution of within-class variability. At large width, the accumulated compression becomes a function of the depth-to-width ratio L/N, with its rate determined self-consistently by the data, prior and architecture. Numerical experiments support the derived dependence on depth, width and signal-to-noise ratio. The result is a representation-level scaling law derived from feature-learning theory rather than fitted after training.
Pankhil Gawade, Kirsten Fischer, Guy Wolf +1
Institute of AI for Health, Helmholtz Munich · School of Computation, Information and Technology, Technical University of Munich · Université de Montréal +3
Neural collapse (NC) describes the structured geometry that emerges in the features and weights of trained classifiers. Recent theory suggests NC can be suboptimal in deep architectures, attributing this to an explicit low-rank bias from L2 regularization. We study the deep unconstrained feature model (UFM)-equivalent to a deep linear network with orthogonal inputs-trained without regularization, to isolate how gradient descent and depth alone shape NC. We show that depth induces an implicit low-rank bias: low-rank matrices propagate norm more efficiently through successive multiplications, promoting low-rank alternatives to NC. These alternatives, we argue, correspond to softmax codes: max-margin solutions previously found in width-bottlenecked networks. Analyzing training dynamics under spectral initialization, we identify an early-time repulsion among singular values that drives low-rank emergence, and characterize how depth shrinks NC's basin of attraction. Finally, we show that some effects act in the opposite direction: for randomly initialized networks, increasing width biases training toward higher-rank solutions. Our results provide the first asymptotic and dynamic characterization of implicit bias in deep UFMs trained with unregularized multiclass cross-entropy.
Connall Garrod, Jonathan P. Keating, Christos Thrampoulidis
Mathematical Institute, University of Oxford · Department of Electrical and Computer Engineering, University of British Columbia