Neural Collapse

Neural collapse (NC) describes a surprising geometric structure consistently observed in the final layers of deep neural networks during training, where features cluster and align with classifier weights in a highly symmetric manner. Current research focuses on understanding NC's emergence across various architectures and tasks, including classification, regression, and even language modeling, often employing unconstrained feature models for theoretical analysis. This phenomenon has implications for improving model generalization, fairness, and robustness, as well as for developing novel training strategies and out-of-distribution detection methods.