Optimizer choice matters for the emergence of Neural Collapse

Jim Zhao; Tin Sum Cheng; Wojciech Masarczyk; Aurelien Lucchi

cs.LGFeb 18, 2026

Optimizer choice matters for the emergence of Neural Collapse

Authors: Jim Zhao, Tin Sum Cheng, Wojciech Masarczyk, Aurelien Lucchi

Organizations: University of Basel · Warsaw University of Technology IDEAS Research Institute

Abstract

Neural Collapse (NC) refers to the emergence of highly symmetric geometric structures in the representations of deep neural networks during the terminal phase of training. Despite its prevalence, the theoretical understanding of NC remains limited. Existing analyses largely ignore the role of the optimizer, thereby suggesting that NC is universal across optimization methods. In this work, we challenge this assumption and demonstrate that the choice of optimizer plays a critical role in the emergence of NC. The phenomenon is typically quantified through NC metrics, which, however, are difficult to track and analyze theoretically. To overcome this limitation, we introduce a novel diagnostic metric, NC0, whose convergence to zero is a necessary condition for NC. Using NC0, we provide theoretical evidence that NC cannot emerge under decoupled weight decay in adaptive optimizers, as implemented in AdamW. Concretely, we prove that SGD, SignGD with coupled weight decay (a special case of Adam), and SignGD with decoupled weight decay (a special case of AdamW) exhibit qualitatively different NC0 dynamics. Also, we show the accelerating effect of momentum on NC (beyond convergence of train loss) when trained with SGD, being the first result concerning momentum in the context of NC. Finally, we conduct extensive empirical experiments consisting of 3,900 training runs across various datasets, architectures, optimizers, and hyperparameters, confirming our theoretical results. This work provides the first theoretical explanation for optimizer-dependent emergence of NC and highlights the overlooked role of weight-decay coupling in shaping the implicit biases of optimizers.

Figures & tables

Figure 1: NC0 (left) and NC3 (right) metrics at the end of training. Lower values indicate stronger NC. AdamW shows consistently higher metrics than Adam. Averages computed over runs with varying learning rates and momentum; shaded regions show

\pm 1

standard deviation. X-axis is log-scaled. Note that there are no values for Adam for WD larger than 0.05 as the model did not train due to over regularization.

Result	Optimizers	Model	Convergence to 0?	learning rate
Theorem 3.1	SGD with DWD	Any	yes, exponential	constant
Theorem 3.2	SGD with CWD	Any	yes, exponential	constant
Theorem 3.3	SignGD with DWD	UFM	yes	step-wise decay
Theorem 3.4	SignGD with CWD	UFM	no	-

Table 1: Overview of our theoretical results on NC0.

Figure 2: NC0 weakly correlates with NC3 across different optimizers and learning rates. Details on the regression fit can be found in Section D.3

Optimizer	NC0 ↓	NC1 ↓	NC2 ↓	NC3 ↓
SGD	2.14e-04 ( $<-99.5\%$ )	0.05 ( $-99.3\%$ )	0.29 ( $-63.0\%$ )	0.35 ( $-75.1\%$ )
SGDW	0.55 ( $-68.9\%$ )	0.26 ( $-96.3\%$ )	0.46 ( $-42.4\%$ )	0.80 ( $-43.5\%$ )
Adam	0.34 ( $-80.6\%$ )	0.04 ( $-99.5\%$ )	0.29 ( $-63.9\%$ )	0.29 ( $-79.5\%$ )
AdamW	5.33 ( $\gg 100\%$ )	0.20 ( $-97.2\%$ )	0.54 ( $-32.4\%$ )	0.78 ( $-45.2\%$ )
Signum	0.78 ( $-55.3\%$ )	0.13 ( $-98.1\%$ )	0.50 ( $-36.8\%$ )	0.58 ( $-59.0\%$ )
SignumW	3185.69 ( $\gg 100\%$ )	0.30 ( $-95.7\%$ )	1.15 ( $+44.2\%$ )	1.40 ( $-1.2\%$ )

Table 2: Final NC metrics for the same setting as in Figure 6 , following the setup of 25 . Lower values (

\downarrow

) indicate stronger neural collapse. Values in parentheses represent percentages relative to the metric at initialization.

Figure 3: NC0 and NC3 metrics at the end of training for a ResNet9 trained on FashionMNIST for Signum and SignumW (left side) and SGD and SGDW (right side). Shaded area refers to one standard deviation across all trainings run with corresponding optimizer. Note that there are fewer values for Signum and SGD as the model did not train due to over regularization for too large WD.

Figure 4: Train loss, train accuracy and NC metrics for fixed WD=0.005 and mom=0.7 and 0.9. Although both runs converge to almost exactly the same train loss, the final NC metrics differ considerably. Plots including NC1 and NC2 can be found in Figure 24 .

Panel of Figure 5 — Figure 5: Heatmap of NC0, NC2 and NC3 for varying values of momentum and weight decay on ResNet9 trained on FashionMNIST with SGD.

Figure 6: NC metrics throughout training on a ResNet9 trained on FashionMNIST.

Figure 7: Singular values of last-layer weights

\mathbf{W}

(left) and centered class means

\mathbf{M}

(right) throughout training. The dotted line corresponds to the smallest singular value and the full line corresponds to the average singular value, excluding the smallest singular value. Singular values for SignumW are out-of-range and are shown in Figure 29 in the appendix.

Figure 8: Interpolating Adam and AdamW by varying the coupled and decoupled weight decay. Total weight decay was fixed to

0.0005

. Note that coupled weight decay = 0 is equivalent to AdamW and coupled weight decay = 0.0005 is equivalent to Adam. Experiments trained on ResNet9 with MNIST.

Figure 9: Training dynamic of NC0 with optimizers SGD, Adam, AdamW, Adam0 ( $\beta_{1}=\beta_{2}=0$ ), AdamW0 ( $\beta_{1}=\beta_{2}=0$ ). For AdamW and SignSGD the inlay shows the NC0 metric more detailed for the last 2000 steps. Note that 5 steps correspond to one training epoch.

Optimizer	NC0 ↓	NC1 ↓	NC2 ↓	NC3 ↓
SGD	1.53e-05 ( $<-99.5\%$ )	0.02 ( $<-99.5\%$ )	0.19 ( $-75.8\%$ )	0.13 ( $-90.9\%$ )
SGDW	1.54e-04 ( $<-99.5\%$ )	0.01 ( $<-99.5\%$ )	0.15 ( $-81.7\%$ )	0.10 ( $-92.7\%$ )
Adam	0.12 ( $<-93.2\%$ )	0.04 ( $-99.5\%$ )	0.23 ( $-71.6\%$ )	0.17 ( $-88.2\%$ )
AdamW	8.09 ( $\gg$ 100% )	0.01 ( $<-99.5\%$ )	0.14 ( $-82.1\%$ )	0.49 ( $-65.1\%$ )

Table 3: Final NC metrics for the run with the smallest absolute NC3 metric and

>99\%

training accuracy for each optimizer. Lower values (

\downarrow

) indicate stronger neural collapse. Values in parentheses represent percentages relative to the metric at initialization. Hyperparameters used for each optimizer can be found in Table 5 .

Appendix figures & tables52 assets

Supplementary material from the paper’s appendix.

Appendix

Architectures	Optimizers	Datasets
ResNet9, VGG9	SGD, SGDW, Adam, AdamW, Signum, SignumW	MNIST, FashionMNIST, CIFAR10

Appendix

Table 4: Overview of experiments conducted in this work.

Optimizer	Learning rate	Momentum/ $\beta_{1}$	Weight decay
SGD	0.01	0.9	0.05
SGDW	0.0679	0.5	0.05
Adam	0.005	0.98	0.05
AdamW	0.005	0.95	5
Signum	0.001	0.9	0.05
SignumW	0.001	0.98	10

Appendix

Table 5: Hyperparameters for each optimizer to achieve the smallest NC3 metric shown in Table 3 .

Figure 10: Figure 2 shown again for ease of reading. NC0 weakly correlates with NC3 across different optimizers and learning rates (here shown for ResNet9 trained on FashionMNIST).

Figure 11: NC0 correlates with NC3 even when considered across all learning rates together (here shown for ResNet9 trained on FashionMNIST).

Figure 12: NC0 vs. NC1 across different optimizers and learning rates (here shown for ResNet9 trained on FashionMNIST).

Figure 13: NC0 vs. NC2 across different optimizers and learning rates (here shown for ResNet9 trained on FashionMNIST).

Figure 14: ResNet9 trained on FashionMNIST with Adam and AdamW for more epochs.

Figure 15: Selected runs from Figure 5 trained for more number of epochs.

Figure 16: NC0 (left), NC3 (center left), NC1 (center right), and NC2 (right) for increasing weight decay.

Figure 17: Train loss (left) and train accuracy (right) for training runs with smallest final NC3 metric for different optimizers.

Figure 18: NC0 (left) and NC3 (right) for training runs with smallest final NC3 metric for different optimizers.

Figure 19: Singular values of last-layer weights

\mathbf{W}

(left) and centered class means

\mathbf{M}

(right) throughout training for runs corresponding to Table 5 . The dotted line corresponds to the smallest singular value and the full line corresponds to the average singular value, excluding the smallest singular value.

Figure 20: NC1 (left) and NC2 (right) for training runs with smallest final NC3 metric for different optimizers.

Figure 21: NC0 (left) and normalized NC0 (right) for training runs with smallest final NC3 metric for different optimizers.

Figure 22: NC0 (left) and normalized NC0 (right) for training runs with zero weight decay from the ablation study in Section D.4.6 . Note that the x-axis is in logarithmic scale and that the point at epoch -1 corresponds to the model at initialization.

Figure 23: Train loss, train accuracy and NC metrics for fixed WD=0.005 and different values of momentum on a ResNet9 trained with SGD with otherwise same hyperparameters as in Figure 5 .

Figure 24: Train loss, train accuracy and NC metrics for fixed WD=0.005 and mom=0.7 and 0.9. Although both runs converge to almost exactly the same train loss, the final NC metrics differ considerably.

SGD with …	NC1	NC2	NC3
no WD (2000 epochs)	$\approx 0.2$	$\approx 0.55$	$\approx 0.7$
WD (200 epochs)	$\approx 0.02$	$\approx 0.2$	$\approx 0.13$

Appendix

Table 7: Smallest NC metrics achieved with and without weight decay for training a ResNet9 on FashionMNIST.

Figure 25: Training loss and train accuracy (top row), NC0 and NC3 (middle row), and NC1 and NC2 (bottom row) for a ResNet9 trained on FashionMNIST with SGD without weight decay for varying values of momentum and number of epochs. Note that the x-axis is in log-scale to improve readability.

Figure 26: NC0 metric (top) and NC3 metric (bottom for different values of weight decay, momentum and LR for Adam vs. AdamW.

Figure 27: NC0 metric (top) and NC3 metric (bottom for different values of weight decay, momentum and LR for Signum vs. SignumW.

Figure 28: NC0 metric (top) and NC3 metric (bottom for different values of weight decay, momentum and LR for SGD vs. SGDW.

Figure 29: Singular values of last-layer weights

\mathbf{W}

(left) and feature matrix

\mathbf{M}

(right) throughout training for SignumW on ResNet9 trained on FashionMNIST. Dotted line corresponds do smallest singular value and full line corresponds to the average singular value excluding the smallest singular value.

Figure 30: NC0 (left) and NC3 (right) metrics plotted against weight decay on a ResNet50 trained on ImageNet1K for Adam and AdamW. Shaded area refers to one standard deviation across all trainings run with corresponding optimizer.

Figure 31: NC1 (left) and NC2 (right) metrics plotted against weight decay on a ResNet50 trained on ImageNet1K for Adam and AdamW. Shaded area refers to one standard deviation across all trainings run with corresponding optimizer.

Figure 32: NC0 (left) and NC3 (right) metrics plotted against weight decay on a VGG9 trained on FashionMNIST for Adam and AdamW. Shaded area refers to one standard deviation across all trainings run with corresponding optimizer.

Figure 33: NC0 and NC3 metrics plotted against weight decay on a VGG9 trained on FashionMNIST for Signum and SignumW (left side) and SGD and SGDW (right side). Shaded area refers to one standard deviation across all trainings run with corresponding optimizer.

Figure 34: NC0 vs. NC1 on VGG9 trained on FashionMNIST. Note that the x-axis is plotted in log-scale.

Figure 35: NC0 vs. NC2 on VGG9 trained on FashionMNIST. Note that the x-axis is plotted in log-scale.

Figure 36: NC0 vs. NC3 on VGG9 trained on FashionMNIST. Note that the x-axis is plotted in log-scale.

Figure 37: NC0 (left) and NC3 (right) metrics plotted against weight decay on a ResNet9 trained on Cifar10 for Adam and AdamW. Shaded area refers to one standard deviation across all trainings run with corresponding optimizer.

Figure 38: NC0 and NC3 metrics plotted against weight decay on a ResNet9 trained on Cifar10 for Signum and SignumW (left side) and SGD and SGDW (right side). Shaded area refers to one standard deviation across all trainings run with corresponding optimizer.

Figure 39: NC0 vs. NC1 on ResNet9 trained on Cifar10. Note that the x-axis is plotted in log-scale.

Figure 40: NC0 vs. NC2 on ResNet9 trained on Cifar10. Note that the x-axis is plotted in log-scale.

Figure 41: NC0 vs. NC3 on ResNet9 trained on Cifar10. Note that the x-axis is plotted in log-scale.

Figure 42: NC0 (left) and NC3 (right) metrics plotted against weight decay on a VGG9 trained on Cifar10 for Adam and AdamW. Shaded area refers to one standard deviation across all trainings run with corresponding optimizer.

Figure 43: NC0 and NC3 metrics plotted against weight decay on a VGG9 trained on Cifar10 for Signum and SignumW (left side) and SGD and SGDW (right side). Shaded area refers to one standard deviation across all trainings run with corresponding optimizer.

Figure 44: NC0 vs. NC1 on VGG9 trained on Cifar10. Note that the x-axis is plotted in log-scale.

Figure 45: NC0 vs. NC2 on VGG9 trained on Cifar10. Note that the x-axis is plotted in log-scale.

Figure 46: NC0 vs. NC3 on VGG9 trained on Cifar10. Note that the x-axis is plotted in log-scale.

Figure 47: NC0 (left) and NC3 (right) metrics plotted against weight decay on a ResNet9 trained on MNIST for Adam and AdamW. Shaded area refers to one standard deviation across all trainings run with corresponding optimizer.

Figure 48: NC0 and NC3 metrics plotted against weight decay on a ResNet9 trained on MNIST for Signum and SignumW (left side) and SGD and SGDW (right side). Shaded area refers to one standard deviation across all trainings run with corresponding optimizer.

Figure 49: NC0 vs. NC1 on ResNet9 trained on MNIST. Note that the x-axis is plotted in log-scale.

Figure 50: NC0 vs. NC2 on ResNet9 trained on MNIST. Note that the x-axis is plotted in log-scale.

Figure 51: NC0 vs. NC3 on ResNet9 trained on MNIST. Note that the x-axis is plotted in log-scale.

Figure 52: NC0 (left) and NC3 (right) metrics plotted against weight decay on a VGG9 trained on MNIST for Adam and AdamW. Shaded area refers to one standard deviation across all trainings run with corresponding optimizer.

Figure 53: NC0 and NC3 metrics plotted against weight decay on a VGG9 trained on MNIST for Signum and SignumW (left side) and SGD and SGDW (right side). Shaded area refers to one standard deviation across all trainings run with corresponding optimizer.

Figure 54: NC0 vs. NC1 on VGG9 trained on MNIST. Note that the x-axis is plotted in log-scale.

Figure 55: NC0 vs. NC2 on VGG9 trained on MNIST. Note that the x-axis is plotted in log-scale.

Figure 56: NC0 vs. NC3 on VGG9 trained on MNIST. Note that the x-axis is plotted in log-scale.

Figure 57: NC4 is largely uncorrelated with NC1-3 across different optimizers and learning rates.

Neural Collapse

Neural Collapse studies the class-structured geometry that can emerge in a network’s features and classifier during training.

See all 22 papers

Neural Collapse Dynamics: Depth, Activation, Regularisation, and Feature Norm Threshold
Mar 31, 2026Anamika Paul RupaNeural Collapse Neural Network Training Dynamics
Department of Electrical Engineering and Computer Science, Howard University, Washington, DC, 20059, USA
On the Principles Behind Neural Network Optimizers
Aug 17, 2026Yushun ZhangNeural Network Optimization Adaptive Gradient Methods
School of Data Science, The Chinese University of Hong Kong, Shenzhen, China
Representations Have Trajectories: Characterizing Temporal Representation Dynamics in Neural Networks
Oct 17, 2026Shivaprasad B. Gowda, Sparsh MittalLow-Rank Representation Learning Neural Representation Geometry
Indian Institute of Information Technology, Nagpur · Indian Institute of Technology, Roorkee

Mar 31, 2026J/K move · Enter open · S save

Neural Collapse Dynamics: Depth, Activation, Regularisation, and Feature Norm Threshold

Anamika Paul Rupa

Department of Electrical Engineering and Computer Science, Howard University, Washington, DC, 20059, USA

Neural Collapse Neural Network Training Dynamics

Open paper AskPDF arXiv

Neural collapse (NC) -- the convergence of penultimate-layer features to a simplex equiangular tight frame -- is well understood at equilibrium, but the dynamics governing its onset remain poorly characterised. We identify a simple and predictive regularity: NC occurs when the mean feature norm reaches a model-dataset-specific critical value, fn*, that is largely invariant to training conditions. This value concentrates tightly within each (model, dataset) pair (CV < 8%); training dynamics primarily affect the rate at which fn approaches fn*, rather than the value itself. In standard training trajectories, the crossing of fn below fn* consistently precedes NC onset, providing a practical predictor with a mean lead time of 62 epochs (MAE 24 epochs). A direct intervention experiment confirms fn* is a stable attractor of the gradient flow -- perturbations to feature scale are self-corrected during training, with convergence to the same value regardless of direction (p>0.2). Completing the (architecture)x(dataset) grid reveals the paper's strongest result: ResNet-20 on MNIST gives fn* = 5.867 -- a +458% architecture effect versus only +68% on CIFAR-10. The grid is strongly non-additive; fn* cannot be decomposed into independent architecture and dataset contributions. Four structural regularities emerge: (1) depth has a non-monotonic effect on collapse speed; (2) activation jointly determines both collapse speed and fn*; (3) weight decay defines a three-regime phase diagram -- too little slows, an optimal range is fastest, and too much prevents collapse; (4) width monotonically accelerates collapse while shifting fn* by at most 13%. These results establish feature-norm dynamics as an actionable diagnostic for predicting NC timing, suggesting that norm-threshold behaviour is a general mechanism underlying delayed representational reorganisation in deep networks.

Abstract

Neural Collapse

Explore similar work

Neural Collapse Dynamics: Depth, Activation, Regularisation, and Feature Norm Threshold

On the Principles Behind Neural Network Optimizers

Representations Have Trajectories: Characterizing Temporal Representation Dynamics in Neural Networks