Neural Collapse (NC) refers to the emergence of highly symmetric geometric structures in the representations of deep neural networks during the terminal phase of training. Despite its prevalence, the theoretical understanding of NC remains limited. Existing analyses largely ignore the role of the optimizer, thereby suggesting that NC is universal across optimization methods. In this work, we challenge this assumption and demonstrate that the choice of optimizer plays a critical role in the emergence of NC. The phenomenon is typically quantified through NC metrics, which, however, are difficult to track and analyze theoretically. To overcome this limitation, we introduce a novel diagnostic metric, NC0, whose convergence to zero is a necessary condition for NC. Using NC0, we provide theoretical evidence that NC cannot emerge under decoupled weight decay in adaptive optimizers, as implemented in AdamW. Concretely, we prove that SGD, SignGD with coupled weight decay (a special case of Adam), and SignGD with decoupled weight decay (a special case of AdamW) exhibit qualitatively different NC0 dynamics. Also, we show the accelerating effect of momentum on NC (beyond convergence of train loss) when trained with SGD, being the first result concerning momentum in the context of NC. Finally, we conduct extensive empirical experiments consisting of 3,900 training runs across various datasets, architectures, optimizers, and hyperparameters, confirming our theoretical results. This work provides the first theoretical explanation for optimizer-dependent emergence of NC and highlights the overlooked role of weight-decay coupling in shaping the implicit biases of optimizers.
Figures & tables
Figure 1: NC0 (left) and NC3 (right) metrics at the end of training. Lower values indicate stronger NC. AdamW shows consistently higher metrics than Adam. Averages computed over runs with varying learning rates and momentum; shaded regions show ±1 standard deviation. X-axis is log-scaled. Note that there are no values for Adam for WD larger than 0.05 as the model did not train due to over regularization.
Result
Optimizers
Model
Convergence to 0?
learning rate
Theorem 3.1
SGD with DWD
Any
yes, exponential
constant
Theorem 3.2
SGD with CWD
Any
yes, exponential
constant
Theorem 3.3
SignGD with DWD
UFM
yes
step-wise decay
Theorem 3.4
SignGD with CWD
UFM
no
-
Table 1: Overview of our theoretical results on NC0.
Figure 2: NC0 weakly correlates with NC3 across different optimizers and learning rates. Details on the regression fit can be found in Section D.3
Optimizer
NC0 ↓
NC1 ↓
NC2 ↓
NC3 ↓
SGD
2.14e-04 ( <−99.5% )
0.05 ( −99.3% )
0.29 ( −63.0% )
0.35 ( −75.1% )
SGDW
0.55 ( −68.9% )
0.26 ( −96.3% )
0.46 ( −42.4% )
0.80 ( −43.5% )
Adam
0.34 ( −80.6% )
0.04 ( −99.5% )
0.29 ( −63.9% )
0.29 ( −79.5% )
AdamW
5.33 ( ≫100% )
0.20 ( −97.2% )
0.54 ( −32.4% )
0.78 ( −45.2% )
Signum
0.78 ( −55.3% )
0.13 ( −98.1% )
0.50 ( −36.8% )
0.58 ( −59.0% )
SignumW
3185.69 ( ≫100% )
0.30 ( −95.7% )
1.15 ( +44.2% )
1.40 ( −1.2% )
Table 2: Final NC metrics for the same setting as in Figure 6 , following the setup of 25 . Lower values ( ↓ ) indicate stronger neural collapse. Values in parentheses represent percentages relative to the metric at initialization.
Figure 3: NC0 and NC3 metrics at the end of training for a ResNet9 trained on FashionMNIST for Signum and SignumW (left side) and SGD and SGDW (right side). Shaded area refers to one standard deviation across all trainings run with corresponding optimizer. Note that there are fewer values for Signum and SGD as the model did not train due to over regularization for too large WD.
Figure 4: Train loss, train accuracy and NC metrics for fixed WD=0.005 and mom=0.7 and 0.9. Although both runs converge to almost exactly the same train loss, the final NC metrics differ considerably. Plots including NC1 and NC2 can be found in Figure 24 .
Figure 5: Heatmap of NC0, NC2 and NC3 for varying values of momentum and weight decay on ResNet9 trained on FashionMNIST with SGD.
Figure 6: NC metrics throughout training on a ResNet9 trained on FashionMNIST.
Figure 7: Singular values of last-layer weights W (left) and centered class means M (right) throughout training. The dotted line corresponds to the smallest singular value and the full line corresponds to the average singular value, excluding the smallest singular value. Singular values for SignumW are out-of-range and are shown in Figure 29 in the appendix.
Figure 8: Interpolating Adam and AdamW by varying the coupled and decoupled weight decay. Total weight decay was fixed to 0.0005 . Note that coupled weight decay = 0 is equivalent to AdamW and coupled weight decay = 0.0005 is equivalent to Adam. Experiments trained on ResNet9 with MNIST.
Figure 9: Training dynamic of NC0 with optimizers SGD, Adam, AdamW, Adam0 ( β1=β2=0 ), AdamW0 ( β1=β2=0 ). For AdamW and SignSGD the inlay shows the NC0 metric more detailed for the last 2000 steps. Note that 5 steps correspond to one training epoch.
Optimizer
NC0 ↓
NC1 ↓
NC2 ↓
NC3 ↓
SGD
1.53e-05 ( <−99.5% )
0.02 ( <−99.5% )
0.19 ( −75.8% )
0.13 ( −90.9% )
SGDW
1.54e-04 ( <−99.5% )
0.01 ( <−99.5% )
0.15 ( −81.7% )
0.10 ( −92.7% )
Adam
0.12 ( <−93.2% )
0.04 ( −99.5% )
0.23 ( −71.6% )
0.17 ( −88.2% )
AdamW
8.09 ( ≫ 100% )
0.01 ( <−99.5% )
0.14 ( −82.1% )
0.49 ( −65.1% )
Table 3: Final NC metrics for the run with the smallest absolute NC3 metric and >99% training accuracy for each optimizer. Lower values ( ↓ ) indicate stronger neural collapse. Values in parentheses represent percentages relative to the metric at initialization. Hyperparameters used for each optimizer can be found in Table 5 .
Appendix figures & tables52 assets
Supplementary material from the paper’s appendix.
Appendix
Architectures
Optimizers
Datasets
ResNet9, VGG9
SGD, SGDW, Adam, AdamW, Signum, SignumW
MNIST, FashionMNIST, CIFAR10
Appendix
Table 4: Overview of experiments conducted in this work.
Optimizer
Learning rate
Momentum/ β1
Weight decay
SGD
0.01
0.9
0.05
SGDW
0.0679
0.5
0.05
Adam
0.005
0.98
0.05
AdamW
0.005
0.95
5
Signum
0.001
0.9
0.05
SignumW
0.001
0.98
10
Appendix
Table 5: Hyperparameters for each optimizer to achieve the smallest NC3 metric shown in Table 3 .
Figure 10: Figure 2 shown again for ease of reading. NC0 weakly correlates with NC3 across different optimizers and learning rates (here shown for ResNet9 trained on FashionMNIST).
Figure 11: NC0 correlates with NC3 even when considered across all learning rates together (here shown for ResNet9 trained on FashionMNIST).
Figure 12: NC0 vs. NC1 across different optimizers and learning rates (here shown for ResNet9 trained on FashionMNIST).
Figure 13: NC0 vs. NC2 across different optimizers and learning rates (here shown for ResNet9 trained on FashionMNIST).
Figure 14: ResNet9 trained on FashionMNIST with Adam and AdamW for more epochs.
Figure 15: Selected runs from Figure 5 trained for more number of epochs.
Figure 16: NC0 (left), NC3 (center left), NC1 (center right), and NC2 (right) for increasing weight decay.
Figure 17: Train loss (left) and train accuracy (right) for training runs with smallest final NC3 metric for different optimizers.
Figure 18: NC0 (left) and NC3 (right) for training runs with smallest final NC3 metric for different optimizers.
Figure 19: Singular values of last-layer weights W (left) and centered class means M (right) throughout training for runs corresponding to Table 5 . The dotted line corresponds to the smallest singular value and the full line corresponds to the average singular value, excluding the smallest singular value.
Figure 20: NC1 (left) and NC2 (right) for training runs with smallest final NC3 metric for different optimizers.
Figure 21: NC0 (left) and normalized NC0 (right) for training runs with smallest final NC3 metric for different optimizers.
Figure 22: NC0 (left) and normalized NC0 (right) for training runs with zero weight decay from the ablation study in Section D.4.6 . Note that the x-axis is in logarithmic scale and that the point at epoch -1 corresponds to the model at initialization.
Figure 23: Train loss, train accuracy and NC metrics for fixed WD=0.005 and different values of momentum on a ResNet9 trained with SGD with otherwise same hyperparameters as in Figure 5 .
Figure 24: Train loss, train accuracy and NC metrics for fixed WD=0.005 and mom=0.7 and 0.9. Although both runs converge to almost exactly the same train loss, the final NC metrics differ considerably.
SGD with …
NC1
NC2
NC3
no WD (2000 epochs)
≈0.2
≈0.55
≈0.7
WD (200 epochs)
≈0.02
≈0.2
≈0.13
Appendix
Table 7: Smallest NC metrics achieved with and without weight decay for training a ResNet9 on FashionMNIST.
Figure 25: Training loss and train accuracy (top row), NC0 and NC3 (middle row), and NC1 and NC2 (bottom row) for a ResNet9 trained on FashionMNIST with SGD without weight decay for varying values of momentum and number of epochs. Note that the x-axis is in log-scale to improve readability.
Figure 26: NC0 metric (top) and NC3 metric (bottom for different values of weight decay, momentum and LR for Adam vs. AdamW.
Figure 27: NC0 metric (top) and NC3 metric (bottom for different values of weight decay, momentum and LR for Signum vs. SignumW.
Figure 28: NC0 metric (top) and NC3 metric (bottom for different values of weight decay, momentum and LR for SGD vs. SGDW.
Figure 29: Singular values of last-layer weights W (left) and feature matrix M (right) throughout training for SignumW on ResNet9 trained on FashionMNIST. Dotted line corresponds do smallest singular value and full line corresponds to the average singular value excluding the smallest singular value.
Figure 30: NC0 (left) and NC3 (right) metrics plotted against weight decay on a ResNet50 trained on ImageNet1K for Adam and AdamW. Shaded area refers to one standard deviation across all trainings run with corresponding optimizer.
Figure 31: NC1 (left) and NC2 (right) metrics plotted against weight decay on a ResNet50 trained on ImageNet1K for Adam and AdamW. Shaded area refers to one standard deviation across all trainings run with corresponding optimizer.
Figure 32: NC0 (left) and NC3 (right) metrics plotted against weight decay on a VGG9 trained on FashionMNIST for Adam and AdamW. Shaded area refers to one standard deviation across all trainings run with corresponding optimizer.
Figure 33: NC0 and NC3 metrics plotted against weight decay on a VGG9 trained on FashionMNIST for Signum and SignumW (left side) and SGD and SGDW (right side). Shaded area refers to one standard deviation across all trainings run with corresponding optimizer.
Figure 34: NC0 vs. NC1 on VGG9 trained on FashionMNIST. Note that the x-axis is plotted in log-scale.
Figure 35: NC0 vs. NC2 on VGG9 trained on FashionMNIST. Note that the x-axis is plotted in log-scale.
Figure 36: NC0 vs. NC3 on VGG9 trained on FashionMNIST. Note that the x-axis is plotted in log-scale.
Figure 37: NC0 (left) and NC3 (right) metrics plotted against weight decay on a ResNet9 trained on Cifar10 for Adam and AdamW. Shaded area refers to one standard deviation across all trainings run with corresponding optimizer.
Figure 38: NC0 and NC3 metrics plotted against weight decay on a ResNet9 trained on Cifar10 for Signum and SignumW (left side) and SGD and SGDW (right side). Shaded area refers to one standard deviation across all trainings run with corresponding optimizer.
Figure 39: NC0 vs. NC1 on ResNet9 trained on Cifar10. Note that the x-axis is plotted in log-scale.
Figure 40: NC0 vs. NC2 on ResNet9 trained on Cifar10. Note that the x-axis is plotted in log-scale.
Figure 41: NC0 vs. NC3 on ResNet9 trained on Cifar10. Note that the x-axis is plotted in log-scale.
Figure 42: NC0 (left) and NC3 (right) metrics plotted against weight decay on a VGG9 trained on Cifar10 for Adam and AdamW. Shaded area refers to one standard deviation across all trainings run with corresponding optimizer.
Figure 43: NC0 and NC3 metrics plotted against weight decay on a VGG9 trained on Cifar10 for Signum and SignumW (left side) and SGD and SGDW (right side). Shaded area refers to one standard deviation across all trainings run with corresponding optimizer.
Figure 44: NC0 vs. NC1 on VGG9 trained on Cifar10. Note that the x-axis is plotted in log-scale.
Figure 45: NC0 vs. NC2 on VGG9 trained on Cifar10. Note that the x-axis is plotted in log-scale.
Figure 46: NC0 vs. NC3 on VGG9 trained on Cifar10. Note that the x-axis is plotted in log-scale.
Figure 47: NC0 (left) and NC3 (right) metrics plotted against weight decay on a ResNet9 trained on MNIST for Adam and AdamW. Shaded area refers to one standard deviation across all trainings run with corresponding optimizer.
Figure 48: NC0 and NC3 metrics plotted against weight decay on a ResNet9 trained on MNIST for Signum and SignumW (left side) and SGD and SGDW (right side). Shaded area refers to one standard deviation across all trainings run with corresponding optimizer.
Figure 49: NC0 vs. NC1 on ResNet9 trained on MNIST. Note that the x-axis is plotted in log-scale.
Figure 50: NC0 vs. NC2 on ResNet9 trained on MNIST. Note that the x-axis is plotted in log-scale.
Figure 51: NC0 vs. NC3 on ResNet9 trained on MNIST. Note that the x-axis is plotted in log-scale.
Figure 52: NC0 (left) and NC3 (right) metrics plotted against weight decay on a VGG9 trained on MNIST for Adam and AdamW. Shaded area refers to one standard deviation across all trainings run with corresponding optimizer.
Figure 53: NC0 and NC3 metrics plotted against weight decay on a VGG9 trained on MNIST for Signum and SignumW (left side) and SGD and SGDW (right side). Shaded area refers to one standard deviation across all trainings run with corresponding optimizer.
Figure 54: NC0 vs. NC1 on VGG9 trained on MNIST. Note that the x-axis is plotted in log-scale.
Figure 55: NC0 vs. NC2 on VGG9 trained on MNIST. Note that the x-axis is plotted in log-scale.
Figure 56: NC0 vs. NC3 on VGG9 trained on MNIST. Note that the x-axis is plotted in log-scale.
Figure 57: NC4 is largely uncorrelated with NC1-3 across different optimizers and learning rates.
Neural collapse (NC) -- the convergence of penultimate-layer features to a simplex equiangular tight frame -- is well understood at equilibrium, but the dynamics governing its onset remain poorly characterised. We identify a simple and predictive regularity: NC occurs when the mean feature norm reaches a model-dataset-specific critical value, fn*, that is largely invariant to training conditions. This value concentrates tightly within each (model, dataset) pair (CV < 8%); training dynamics primarily affect the rate at which fn approaches fn*, rather than the value itself. In standard training trajectories, the crossing of fn below fn* consistently precedes NC onset, providing a practical predictor with a mean lead time of 62 epochs (MAE 24 epochs). A direct intervention experiment confirms fn* is a stable attractor of the gradient flow -- perturbations to feature scale are self-corrected during training, with convergence to the same value regardless of direction (p>0.2). Completing the (architecture)x(dataset) grid reveals the paper's strongest result: ResNet-20 on MNIST gives fn* = 5.867 -- a +458% architecture effect versus only +68% on CIFAR-10. The grid is strongly non-additive; fn* cannot be decomposed into independent architecture and dataset contributions. Four structural regularities emerge: (1) depth has a non-monotonic effect on collapse speed; (2) activation jointly determines both collapse speed and fn*; (3) weight decay defines a three-regime phase diagram -- too little slows, an optimal range is fastest, and too much prevents collapse; (4) width monotonically accelerates collapse while shifting fn* by at most 13%. These results establish feature-norm dynamics as an actionable diagnostic for predicting NC timing, suggesting that norm-threshold behaviour is a general mechanism underlying delayed representational reorganisation in deep networks.
Anamika Paul Rupa
Department of Electrical Engineering and Computer Science, Howard University, Washington, DC, 20059, USA
Reliable optimization is central to neural network (NN) training, yet Adam, the default optimizer for modern LLMs, rests on a fragile foundation. This thesis develops a principled grounding for Adam and motivates new designs. First, we revisit Adam's divergence--convergence debate and show the existence of a problem-dependent phase transition: with properly chosen, batch-size-dependent hyperparameters, Adam converges, whereas under small-β2 regimes it can diverge. Second, we investigate why Adam substantially outperforms SGD on Transformers through Hessian structure. We find that the Hessian evolves toward a near-block-diagonal form along training, accompanied by strong block heterogeneity. We prove that this structure makes Adam's diagonal preconditioner effective. We further show that this special Hessian structure originates from consecutive multiplications of large matrix variables, and we provide a rigorous analysis based on random matrix theory. Finally, these insights motivate Adam-mini, a new optimizer that reduces Adam's memory footprint by 50% while preserving its performance. Our results also have broader implications beyond Adam: they reveal new local structures in matrix-based nonconvex problems, and also help understand and improve recent NN optimizers, such as Muon.
Yushun Zhang
School of Data Science, The Chinese University of Hong Kong, Shenzhen, China
Recent work has shown that deep neural networks exhibit an implicit low-rank simplicity bias. In this paper, we investigate the questions that naturally arise from this observation. What are the temporal consequences of low-rank simplicity bias during optimization? How does representation geometry evolve throughout optimization rather than only at convergence? Do different neural architectures exhibit distinct geometric optimization behaviors? To answer these questions, we conduct a large-scale empirical study tracking representational geometry throughout training. We find that neural networks follow three recurring geometric trajectories: persistent high-rank, collapse-and-recovery, and persistent low-rank. We further identify Destructive Representation Collapse (DRC) as a pathological regime in which representations collapse to rank-one and fail to recover, leading to a sustained loss of representational capacity. While collapse-and-recovery is consistently associated with successful optimization in controlled settings, experiments on real-world models show that the relationship between geometric trajectories and optimization outcomes is configuration dependent. Overall, our findings suggest that representation geometry evolves dynamically during training and provide a new perspective on how deep neural networks learn.
Shivaprasad B. Gowda, Sparsh Mittal
Indian Institute of Information Technology, Nagpur · Indian Institute of Technology, Roorkee