Authors: James Alcala, Rayna Andreeva, Vladimir A. Kobzar, Dustin G. Mixon, Sanghoon Na, Shashank Sule, Yangxinyu Xie
Organizations: Department of Mathematics, University of Southern California, Los Angeles, CA · School of Informatics, University of Edinburgh, Edinburgh, Scotland · Department of Mathematics, The Ohio State University, Columbus, OH · Translational Data Analytics Institute, The Ohio State University, Columbus, OH · Department of Mathematics, University of Maryland, College Park, College Park, MD · Department of Statistics and Data Science, University of Pennsylvania, Philadelphia, PA
When training a neural network for classification, the feature vectors of the training set are known to collapse to the vertices of a regular simplex, provided the dimension d of the feature space and the number n of classes satisfies n≤d+1. This phenomenon is known as neural collapse. For other applications like language models, one instead takes n≫d. Here, the neural collapse phenomenon still occurs, but with different emergent geometric figures. We characterize these geometric figures in the orthoplex regime where d+2≤n≤2d. The techniques in our analysis primarily involve Radon's theorem and convexity.
Figures & tables
Figure 1: Schematic of our main results on neural collapse in the orthoplex regime. Theorems 2 , 5 , and 12 appear in Sections 2 , 3 , and 4 , respectively.
Figure 2: Smallest examples of low- and high-entropy softmax codes.
Figure 3: Fix n=10 . The first plot reports when fn,τ is concave or convex. The next three plots report the best choice of (d1,…,dl) as a function of temperature τ . Here, we take d to be 6 , 7 , and 8 , respectively. We omit the d=5 case since the corresponding softmax code is unique up to rotation.