Neural Collapse Dynamics: Depth, Activation, Regularisation, and Feature Norm Threshold

Anamika Paul Rupa

cs.LGMar 31, 2026

Neural Collapse Dynamics: Depth, Activation, Regularisation, and Feature Norm Threshold

Authors: Anamika Paul Rupa

Organizations: Department of Electrical Engineering and Computer Science, Howard University, Washington, DC, 20059, USA

Abstract

Neural collapse (NC) -- the convergence of penultimate-layer features to a simplex equiangular tight frame -- is well understood at equilibrium, but the dynamics governing its onset remain poorly characterised. We identify a simple and predictive regularity: NC occurs when the mean feature norm reaches a model-dataset-specific critical value, fn*, that is largely invariant to training conditions. This value concentrates tightly within each (model, dataset) pair (CV < 8%); training dynamics primarily affect the rate at which fn approaches fn*, rather than the value itself. In standard training trajectories, the crossing of fn below fn* consistently precedes NC onset, providing a practical predictor with a mean lead time of 62 epochs (MAE 24 epochs). A direct intervention experiment confirms fn* is a stable attractor of the gradient flow -- perturbations to feature scale are self-corrected during training, with convergence to the same value regardless of direction (p>0.2). Completing the (architecture)x(dataset) grid reveals the paper's strongest result: ResNet-20 on MNIST gives fn* = 5.867 -- a +458% architecture effect versus only +68% on CIFAR-10. The grid is strongly non-additive; fn* cannot be decomposed into independent architecture and dataset contributions. Four structural regularities emerge: (1) depth has a non-monotonic effect on collapse speed; (2) activation jointly determines both collapse speed and fn*; (3) weight decay defines a three-regime phase diagram -- too little slows, an optimal range is fastest, and too much prevents collapse; (4) width monotonically accelerates collapse while shifting fn* by at most 13%. These results establish feature-norm dynamics as an actionable diagnostic for predicting NC timing, suggesting that norm-threshold behaviour is a general mechanism underlying delayed representational reorganisation in deep networks.

Figures & tables

Condition	$\mathrm{fn}^{*}$ @ NC1 $<$ 0.01	$\mathrm{fn}^{*}$ @ NC1 $<$ 0.02	Ratio
Depth = 5	1.077	1.251	1.16
Depth = 7	1.112	0.953	0.86
Width = 128	1.241	1.276	1.03
Width = 256	1.125	1.222	1.09
$\lambda=10^{-5}$	1.035	1.197	1.16
$\lambda=5\times 10^{-5}$	0.983	1.150	1.17

Table 1: Threshold robustness:

\mathrm{fn}^{*}

at NC1

<0.01

vs. NC1

<0.02

(MLP-5, ReLU, MNIST; mean of 3 seeds per condition).

Work	Onset	Norm marker	Arch. $\times$ data	Interv.	Type
Papyan et al. [34]	No	No	No	No	Empirical
Zhu et al. [55]	No	No	No	No	Theory
Han et al. [6]	Partial	No	No	No	Theory
Pan and Cao [33]	Partial	No	No	No	Theory
Wang et al. [45]	Yes	No	No	No	Empirical
Zhao et al. [52]	Yes	No	No	No	Theory+Emp.

Table 2: Positioning relative to representative prior work on neural collapse; the four comparison axes are defined in the text.

Figure 1: MNIST baseline dynamics (MLP-5, ReLU,

\lambda=10^{-4}

): (a) accuracy, (b) NC1, (c) NC2 and NC3, (d) feature norm.

Figure 2: CIFAR-10 ResNet-20 dynamics (3 seeds, SGD,

\lambda=10^{-3}

): (a) accuracy, (b) NC1, (c) feature norm per seed.

Depth	N	NC1 thresh.	$T_{\mathrm{NC}}$	$\mathrm{fn}$ at $T_{\mathrm{NC}}$
2	3	$<0.05$	$273\pm 67$	$1.428\pm 0.323$
3	3	$<0.05$	$250\pm 35$	$1.417\pm 0.276$
5	3	$<0.01$	$297\pm 12$	$1.077\pm 0.025$
7	3	$<0.01$	$340\pm 10$	$1.112\pm 0.040$

Table 3: Depth sweep (MLP, ReLU,

\lambda=10^{-4}

, MNIST; mean

\pm

std,

N=3

Act.	N	NC1 thresh.	$T_{\mathrm{NC}}$	$\mathrm{fn}$ at $T_{\mathrm{NC}}$
ReLU	3	$<0.01$	$297\pm 12$	$1.077\pm 0.025$
LeakyReLU	3	$<0.01$	$280\pm 26$	$1.026\pm 0.029$
Tanh	3	$<0.05$	$220\pm 0$	$1.325\pm 0.040$
GELU	5	$<0.05$	$260\pm 23$	$2.118\pm 0.446$
SiLU	3	plateau †	$1027\pm 179$	$3.257\pm 0.188$

Table 4: Activation sweep (MLP-5,

\lambda=10^{-4}

, MNIST; mean

\pm

std). † SiLU plateaus above NC1

=0.05

;

\mathrm{fn}

is reported at the epoch of minimum Phase-2 NC1 (Section 5.3 ).

Figure 3: Activation comparison on MLP-5/MNIST (

\lambda=10^{-4}

, 3 seeds): (a) NC1 (log scale), (b) feature norm (log scale), with per-activation

\mathrm{fn}^{*}

labelled at the right edge. Tanh NC1 is lightly smoothed (3-point rolling mean) for display.

Figure 4: SiLU plateau dynamics (MLP-5/MNIST,

\lambda=10^{-4}

, Phase-2

1000

epochs, 3 seeds): (a) NC1 plateau, (b) feature norm with the reported

\mathrm{fn}^{*}

(dashed), (c) agreement of three plateau estimators.

$\lambda$	N	Collapsed	$T_{\mathrm{NC}}$	$\mathrm{fn}$ at $T_{\mathrm{NC}}$
$10^{-5}$	3	3/3	$380\pm 50$	$1.035\pm 0.087$
$5\times 10^{-5}$	3	3/3	$290\pm 10$	$0.983\pm 0.006$
$10^{-4}$	3	3/3	$297\pm 12$	$1.077\pm 0.025$
$5\times 10^{-4}$	3	0/3	DNF	n/a

Table 5: Weight-decay sweep (MLP-5, ReLU, MNIST, NC1

<0.01

; mean

\pm

std). DNF (did not finish): NC1 stalled at

{\approx}0.03

for all seeds.

Condition	Values varied	$T_{\mathrm{NC}}$ range	$\mathrm{fn}^{*}$ (mean $\pm$ std)
Depth sweep	depth $\in\{2,3,5,7\}$	250–340	$1.259\pm 0.251$
WD sweep	$\lambda\in\{10^{-5},\ 5\times 10^{-5},\ 10^{-4}\}$	290–380	$1.032\pm 0.061$
Width sweep	width $\in\{128,256,512,1024\}$	257–383	$1.135\pm 0.085$
All ReLU runs	(combined, $N=21$ )	250–383	$1.093\pm 0.092$

Table 6: Feature-norm threshold

\mathrm{fn}^{*}

pooled across all MNIST/MLP-5/ReLU conditions; the pool mixes strict (NC1

<0.01

) and relaxed (depths 2–3, NC1

<0.05

) collapse criteria.

Dataset	Architecture	NC1	$N$	$\mathrm{fn}^{*}$	95% $t$ -CI	CV
MNIST	MLP-5	$<0.01$	12	$1.052\pm 0.066$	$[1.010,\ 1.093]$	$6.2\%$
MNIST	ResNet-20	$<0.01$	3	$5.867\pm 0.034$	$[5.781,\ 5.952]$	$0.6\%$
Fashion-MNIST	MLP-5	$<0.05$	3	$1.367\pm 0.077$	$[1.176,\ 1.559]$	$5.6\%$
Fashion-MNIST	ResNet-20	$<0.01$	3	$1.678\pm 0.012$	$[1.648,\ 1.708]$	$0.7\%$
CIFAR-10	MLP-5	$<0.05$	3	$0.900\pm 0.065$	$[0.738,\ 1.063]$	$7.3\%$
CIFAR-10	ResNet-20	$<0.01$	3	$1.515\pm 0.008$	$[1.494,\ 1.535]$	$0.5\%$

Table 7: Feature-norm threshold

\mathrm{fn}^{*}

by (model, dataset) pair (ReLU); mean

\pm

std, with the NC1 collapse criterion and confirmed-collapse count

N

. RN-20: ResNet-20; F-MNIST: Fashion-MNIST.

Figure 5: Weight-decay sweep (MLP-5, ReLU, MNIST): (a)

T_{\mathrm{NC}}

vs.

\lambda

, (b)

\mathrm{fn}

T_{\mathrm{NC}}

, (c) all confirmed values.

Figure 6: Summary of NC dynamics results: (a) depth effect on

T_{\mathrm{NC}}

, (b) activation effect, (c) weight-decay effect, (d)

\mathrm{fn}

by (model, dataset) pair.

Figure 7: ResNet-20 on MNIST (3 seeds): (a) accuracy, (b) NC1, (c) feature norm per seed.

Architecture	Optimiser	$\mathrm{fn}^{*}$
MLP-5	Adam (matched)	$1.095\pm 0.039$ (reproduces main grid)
ResNet-20	Adam (matched)	$7.468\pm 0.129$
ResNet-20	SGD (main grid)	$5.867$
MLP-5	SGD (lr $=0.1$ )	$\to 0$ (degenerate, non-ETF; excluded)

Table 8: Optimiser de-confound on MNIST (three seeds each, strict NC1

<0.01

; values mean

\pm

std).

Figure 8: Fashion-MNIST baseline dynamics (MLP-5, ReLU,

\lambda=10^{-4}

; mean

\pm

std, 3 seeds): (a) accuracy, (b) NC1, (c) NC2 and NC3, (d) feature norm (dashed line marks the reported

\mathrm{fn}^{*}

Figure 9: Feature-norm threshold

\mathrm{fn}^{*}

by (model, dataset) pair across the six evaluated cells, grouped by dataset and coloured by architecture (mean

\pm

std).

Width	Params	$T_{\mathrm{NC}}$	$\mathrm{fn}$ at $T_{\mathrm{NC}}$	CV
128	0.15M	$383\pm 15$	$1.241\pm 0.105$	8.4%
256	0.53M	$327\pm 12$	$1.125\pm 0.041$	3.6%
512	2.10M	$310\pm 10$	$1.096\pm 0.017$	1.6%
1024	8.39M	$257\pm 12$	$1.080\pm 0.059$	5.4%

Table 9: Width sweep (MLP-5, ReLU,

\lambda=10^{-4}

, MNIST, NC1

<0.01

N=3

Figure 10: Width sweep (MLP-5, ReLU,

\lambda=10^{-4}

, MNIST): (a)

\mathrm{fn}

T_{\mathrm{NC}}

, (b)

T_{\mathrm{NC}}

vs. width, (c) fn crossing precedes

T_{\mathrm{NC}}

Condition	$\alpha$	fn after rescaling	$T_{\mathrm{NC}}$	$\mathrm{fn}^{*}$ at collapse
Control	1.0	$16.2$	$287\pm 35$	$1.103\pm 0.020$
Scale-down	0.3	$4.9$	$317\pm 6$	$1.093\pm 0.033$
Scale-up	3.0	$48.7$	$317\pm 46$	$1.054\pm 0.049$

Table 10: Intervention experiment (MLP-5, ReLU,

\lambda=10^{-4}

, MNIST,

N=3

Figure 11: Protocol validation, CE-only vs. two-phase training: (a) NC1, (b) feature norm.

Neural Collapse

Neural Collapse studies the class-structured geometry that can emerge in a network’s features and classifier during training.

See all 22 papers

Deep Neural Regression Collapse
Mar 25, 2026Akshay Rangamani, Altay UnalRepresentation Learning
Department of Data Science, New Jersey Institute of Technology
A Closed Form Depth-Width Law for Neural Collapse in Deep Linear Networks
Oct 15, 2026Pankhil Gawade, Kirsten Fischer, Guy Wolf +1Neural Representation Geometry Neural Scaling Laws
Institute of AI for Health, Helmholtz Munich · School of Computation, Information and Technology, Technical University of Munich · Université de Montréal +3
The Implicit Bias of Depth: From Neural Collapse to Softmax Codes
May 21, 2026Connall Garrod, Jonathan P. Keating, Christos ThrampoulidisDeep Linear Networks Implicit Bias
Mathematical Institute, University of Oxford · Department of Electrical and Computer Engineering, University of British Columbia

Mar 25, 2026J/K move · Enter open · S save

Deep Neural Regression Collapse

Akshay Rangamani, Altay Unal

Department of Data Science, New Jersey Institute of Technology

Representation Learning

Open paper AskPDF arXiv

Neural Collapse is a phenomenon that helps identify sparse and low rank structures in deep classifiers. Recent work has extended the definition of neural collapse to regression problems, albeit only measuring the phenomenon at the last layer. In this paper, we establish that Neural Regression Collapse (NRC) also occurs below the last layer across different types of models. We show that in the collapsed layers of neural regression models, features lie in a subspace that corresponds to the target dimension, the feature covariance aligns with the target covariance, the input subspace of the layer weights aligns with the feature subspace, and the linear prediction error of the features is close to the overall prediction error of the model. In addition to establishing Deep NRC, we also show that models that exhibit Deep NRC learn the intrinsic dimension of low rank targets and explore the necessity of weight decay in inducing Deep NRC. This paper provides a more complete picture of the simple structure learned by deep networks in the context of regression.

Abstract

Neural Collapse

Explore similar work

Deep Neural Regression Collapse

A Closed Form Depth-Width Law for Neural Collapse in Deep Linear Networks

The Implicit Bias of Depth: From Neural Collapse to Softmax Codes