Organizations: School of Artificial Intelligence, Shenzhen University, Shenzhen, 518060, China. · National Engineering Laboratory for Big Data System Computing Technology, Shenzhen University, Shenzhen, 518060, China. · Guangdong Provincial Key Laboratory of Intelligent Information Processing, Shenzhen University, Shenzhen, 518060, China. · College of Computer Science and Software Engineering, Shenzhen University, Shenzhen, 518060, China. · Big Data Institute, University of Oxford, Oxford, OX3 7LF, United Kingdom. · Institute of Image Understanding Research, North Minzu University, Yinchuan, 750021, China.
Classification and feature learning are two closely interrelated fundamental tasks. Feature learning in supervised settings is typically achieved by training classification models, under the common assumption that higher classification accuracy implies better feature quality. In this paper, we first point out that classification accuracy reflects the sample-wise local properties of a model on a dataset, whereas feature learning aims to learn features with good dataset-level global properties, such as intra-class compactness and inter-class separability. From the first principle of feature learning, we redesign the binary cross-entropy (BCE) loss, a counterpart of the cross-entropy (CE) loss used in classification. Then, in normalized feature space, we analyze the minima of the BCE and CE losses and show that both can lead to neural collapse, where sample features from different classes become maximally compact and maximally separated. More importantly, by investigating the convergence behaviors of the two losses as the scale factor γ varies in the normalized space, we reveal how classification accuracy and feature quality change differently across spaces of different sizes. Under a fixed training strategy, a very large γ achieves perfect classification accuracy but slows convergence toward the ideal feature geometry, whereas a very small γ makes classification highly sensitive to small perturbations. A moderate γ is therefore necessary for simultaneously achieving reliable classification and well-structured features. Extensive experiments validate these findings and demonstrate that BCE loss exhibits advantages in long-tail and open-set recognitions, as it better aligns with the global requirements of feature learning.
Figures & tables
Figure 1: The intuitive difference between classification and feature learning. Classification (left) requires that the feature vector of each sample be closer to its own class center/classifier vector than to those of other classes, and classification accuracy reflects the model’s local properties at each sample. In contrast, intra-class compactness and inter-class separability of sample features are the model’s global properties at the dataset level, and feature learning (right) drives the feature vectors of all samples to be as close as possible to their respective class centers while simultaneously being far from other class centers
Figure 2: The evolution of the three NC metrics in the training of ResNet18, ResNet50, DenseNet121, and ViT on the training data of CIFAR10 driven by the normalized CE and BCE losses with γ=8 using SGD and AdamW, respectively
γ
2
3
4
5
6∗
6.5
7∗
7.5
8∗
8.5
9∗
9.5∗
10∗
11
12∗
13
14∗
15
16∗
CE
28
31
64
31
77
145
180
308
435
741
1448
2083
-
-
-
-
-
-
-
BCE
24
22
22
14
15
-
21
-
37
-
53
-
83
137
243
411
737
1165
2076
Table 1: The number of training epochs required by ResNet18 to reach the stopping criterion of Eqs. ( 22 ) and ( 23 ) on MNIST trained by the normalized CE and BCE with different γ>1 . “ ∗ ” indicating that the data is used for fitting the functions of epoch number on γ . When γ≥10 , training ResNet18 with the CE loss requires more than 4,000 epochs to converge, so the experiment was not performed
Loss
Fitted function
Fitting data
Predicted data
R2↑
RMSE ↓
R2↑
RMSE ↓
CE
568.39γ−3645.68
0.8556
299.08
−0.5945
317.58
236.01γ2−3098.39γ+10205.49
0.9939
61.59
0.7463
126.69
40.65γ3−706.28γ2+4076.80γ−7729.46
0.9969
43.85
0.9038
77.99
0.54exp(0.87γ)−61.83
0.9932
64.98
0.9253
68.72
4.22log(0.32γ)−3.01
0.8094
343.40
−0.9326
349.63
Table 2: Fitted functions and their evaluation results for the epoch number used by the CE and BCE with γ>5
Figure 3: The distribution of the final classifier biases and the positive/negative unbiased decision scores for 56 ResNet18s (top, without the final ReLU activation) and 56 ViTs (bottom) trained on CIFAR10, with various scale factors γ . The maximum number of training epochs is fixed at 200. The initial mean ( bˉ=K1∑j=1Kbj ) of the biases is 0 for ResNet18 and 5 for ViT, and the initial variance is both 0 . The solid line represents the theoretical extremum when achieving NC
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Neural Collapse
Classification
setting-1
setting-2
setting-3
setting-4
setting-5
setting-6
Hyper-parameter
epochs
200
200
200
200
200
200
optimizer
SGD
AdamW
SGD
AdamW
SGD
AdamW
batch size
128
128
128
128
128
128
learning rate
0.01
0.001
0.03
0.0003
0.1
0.005
learning rate decay
step
cosine
step
cosine
step
cosine
Appendix
Table 9.1: Detailed experimental settings
Figure 9.1: The evolution of the three NC metrics in the training of ResNet18 (top), ResNet50 (middle) and DenseNet121 (bottom) on MNIST with CE and BCE losses using SGD and AdamW, respectively
Figure 9.2: The evolution of the three NC metrics in the training of ResNet18 (top), ResNet50 (middle) and DenseNet121 (bottom) on CIFAR10 with CE and BCE losses using SGD and AdamW, respectively
Figure 9.3: The evolution of the three NC metrics in the training of ResNet18 (top), ResNet50 (middle) and DenseNet121 (bottom) on CIFAR100 with CE and BCE losses using SGD and AdamW, respectively
Figure 9.4: The evolution of the three NC metrics in the training of ViT on CIFAR10 (top) and CIFAR100 (bottom) with CE and BCE losses using SGD and AdamW, respectively
Figure 9.5: The distribution of the final classifier biases and positive/negative unbiased decision scores for ResNet18 trained on MNIST with varying mean of initial classifier biases, while the variance of initial classifier biases is 0 and γ=8 . The final classifier biases of CE-trained models are determined by their initial values, while that of BCE-trained models always converge to the same value
Figure 9.6: The distribution of the final classifier biases and positive/negative unbiased decision scores for ViT trained on CIFAR10 with varying mean of initial classifier biases, while the variance of initial classifier biases is 0 and γ=8 . The final classifier biases of CE-trained models are determined by their initial values, while that of BCE-trained models always converge to the same value
Figure 9.7: The distribution of the final classifier biases and positive/negative unbiased decision scores of ResNet18 trained on MNIST with varying initial variance of the classifier biases. The different scale factors, γ=8 (1-st row), γ=16 (2-nd row), γ=32 (3-rd row), and γ=64 (4-th row), also significantly impact the convergence of decision scores and the final classifier biases
Figure 9.8: The distribution of the final classifier bias and unbiased positive/negative decision scores of ViT trained on CIFAR10 with varying initial variance of the classifier bias. The different scale factors, γ=8 (1-st row), γ=16 (2-nd row), and γ=32 (3-rd row), also significantly impact the convergence of decision scores and the final classifier bias
Figure 9.9: The distribution of the final classifier biases and positive/negative unbiased decision scores of ResNet18 trained on MNIST by using various scale factors γ . The variance of initial classifier biases is set to 0, and their initial mean takes different values, 0,10,20 , and 30 , respectively
Figure 9.10: The distribution of the final classifier biases and positive/negative unbiased decision scores of ResNet18 trained on MNIST by using various scale factors γ . The variance of the initial classifier biases is set to 0
Figure 9.11: The distribution of the final classifier biases and positive/negative unbiased decision scores of ResNet18 (without the final ReLU) trained on CIFAR10 by using various scale factors γ . The mean and variance of the initial classifier biases is both set to 0
Figure 9.12: The distribution of the final classifier biases and positive/negative unbiased decision scores of ViT trained on CIFAR10 by using various scale factors γ . The variance of the initial classifier biases is set to 0
Figure 9.13: The fitted functions describing the variation of epoch number with respect to γ for CE (top) and BCE (bottom) losses. The epoch numbers in the figure indicate the training duration required to achieve minimum values (i.e., the criterion in Eq. ( 9.1 )) of the two losses for training ResNet18 on MNIST, with different scale factors γ . The blue dots represent that the data was used for function fitting, the red dots represent the predicted values, and the green triangles represent the observed data in experiments
Loss
Fitted function
Fitting data
Predicted data
R2↑
RMSE ↓
R2↑
RMSE ↓
CE
27.14/γ+44.41
0.9987
32.93
0.9908
52.08
−0.06/γ2+33.05/γ−7.52
0.9999
7.92
0.9897
55.16
−0.002/γ3+0.16/γ2+2.83/γ+1.20
1.0000
3.39
0.9638
103.11
4002.21exp(−45.08γ)+185.11
0.9943
69.52
0.8388
217.72
−710.69log(5.63γ)+175.75
0.8213
389.56
0.3936
422.27
Appendix
Table 9.2: Fitted functions and their evaluation results for the epoch number used by the CE and BCE with γ<0.5
Loss
Fitted function
Fitting data
Predicted data
R2↑
RMSE ↓
R2↑
RMSE ↓
CE
568.39γ−3645.68
0.8556
299.08
−0.5945
317.58
236.01γ2−3098.39γ+10205.49
0.9939
61.59
0.7463
126.69
40.65γ3−706.28γ2+4076.80γ−7729.46
0.9969
43.85
0.9038
77.99
0.54exp(0.87γ)−61.83
0.9932
64.98
0.9253
68.72
4.22log(0.32γ)−3.01
0.8094
343.40
−0.9326
349.63
Appendix
Table 9.3: Fitted functions and their evaluation results for the epoch number used by the CE and BCE with γ>5
M
Loss
CIFAR10
CIFAR100
Tiny-ImageNet
SGD
AdamW
SGD
AdamW
SGD
AdamW
ResNet18
CE
95.90
96.24
78.39
78.05
68.10
67.13
BCE
95.97
95.96
78.46
78.38
69.10
69.12
ResNet50
CE
96.30
96.51
80.89
78.06
71.65
69.49
BCE
96.40
96.24
80.87
79.89
72.95
70.82
ViT
CE
79.51
87.00
64.82
63.19
50.42
54.83
Appendix
Table 9.4: The classification accuracy (%) on the test datasets of CIFAR10, CIFAR100 and Tiny-ImageNet. We conducted learning rate optimization for each experimental combination while keeping other hyperparameters fixed, and the γ=32
Bias Mean
Metrics
MNIST
SVHN
CIFAR10
CIFAR+50
CE
BCE
CE
BCE
CE
BCE
CE
BCE
–
AUROC
96.74
90.22
94.44
66.60
74.34
68.14
78.34
62.17
OSCR
96.59
90.04
92.56
71.00
60.10
67.00
61.22
63.87
[2pt/2pt] 0
AUROC
98.63
99.29
94.42
95.21
83.86
85.70
87.82
90.06
OSCR
98.50
99.09
92.53
93.49
81.54
83.50
85.81
88.43
[2pt/2pt] 5
AUROC
98.58
99.15
94.30
95.07
84.17
85.78
87.76
90.15
Appendix
Table 9.5: The results of open-set recognition. Results are averaged among five randomized trials and the scale factor γ=32
Neural collapse (NC) describes the elegant geometric structure that sample features and classifier vectors exhibit when a classification model's loss reaches its minimum. Class imbalance, however, generally distorts standard NC. In this paper, we theoretically reformulate NC on imbalanced datasets and in open-set settings. Our analysis is based on an optimization problem over class-wise averaged losses, rather than the global averaged losses used in prior NC studies. We prove that when class-wise averaged losses attain their minima, standard NC is achieved on balanced or imbalanced datasets, for both cross-entropy (CE) and binary cross-entropy (BCE) losses. We further extend the analysis to an open-set-inspired scenario, where some classes contain no training samples yet are still assigned classifier vectors in the losses. We find that in this scenario, when class-wise averaged losses reach their minima, standard NC still emerges on the known classes with training samples, while the classifier vectors of unknown classes without training samples overlap with each other and become orthogonal to those of the known classes. Finally, we analyze Softmax- and Sigmoid-based contrastive losses re-balanced by inverse sample frequency, proving that at their minima on balanced or imbalanced datasets, sample features of different classes likewise form standard NC.
Qiufu Li, Huibing Xiao, Linlin Shen +1
School of Artificial Intelligence; National Engineering Laboratory for Big Data System Computing Technology; Guangdong Provincial Key Laboratory of Intelligent Information Processing; Intelligent Computing Center, Shenzhen University, Shenzhen 518060, China · Institute of Image Understanding Research, North Minzu University, Yinchuan 750021, China
Supervised classification has a theoretical optimum, Neural Collapse (NC), yet neither of its two dominant paradigms reaches it in practice. Cross entropy (CE) leaves radial degrees of freedom unconstrained and converges to a degenerate geometry, while supervised contrastive learning (SCL) drives features toward NC during pretraining but discards this structure in a post hoc linear probing phase. We show that both paradigms are different appearances of the same method that contrasts prototypes on the unit hypersphere, and that closing the gap requires fixing each at its point of failure. From the CE side, we propose NTCE and NONL, two normalized losses that import contrastive optimization's missing ingredients into classifier learning: a large effective negative set and decoupled alignment and uniformity terms. From the SCL side, we prove that SCL's objective already optimizes throughout training for a principled classifier whose weights are the class mean embeddings, making linear probing both redundant and harmful. Empirically, on four benchmarks including ImageNet-1K, NTCE and NONL surpass CE accuracy, closely approximate NC (≥95%), and match CE's converged NC on 4/5 metrics in under 7.5% of its iterations, while SCL with fixed prototypes matches linear probing without the hours-long classifier training phase. The learned geometry yields +5.5% mean relative improvement in transfer learning, up to +8.7% under severe class imbalance, and improved robustness to corruptions on ImageNet-C. Our work recasts supervised learning as prototype learning on the hypersphere, with NC reached by design.
Panagiotis Koromilas, Theodoros Giannakopoulos, Mihalis A. Nicolaou +1
University of Athens · NSCR Demokritos · University of Cyprus +2
Classification heads built on conic functions give each class a bounded region of the feature space, and such regions are used to reject samples of unseen classes. We ask whether these regions also keep the features bounded when the head is trained with softmax cross-entropy. For a head whose class scores combine a linear term with a Euclidean norm, we first show that the region of each class is bounded by a quadric of revolution, whose eccentricity is the reciprocal of one learned ratio. We then attach to each class an asymptotic margin, which compares the growth of its score with the growth of all the other scores and can be computed by a second-order cone program. The loss of a class has infimum zero exactly when its margin is positive, and it then approaches zero only along diverging features. When the margin is negative, the loss is coercive and its minimizers lie in an explicit ball. Since softmax depends only on the differences between scores, a class with a bounded region can still have a positive margin, so bounded regions do not confine the features. The result extends to other norms, including the polyhedral conic functions. On CIFAR-10 and CIFAR-100, all 1210 cells of 67 trained conic heads have positive margins, whereas a prototype head keeps nine of its ten cells bounded, as the theory predicts.
Omer Nezih Gerek
Department of Electrical and Electronics Engineering, Eskişehir Technical University, Eskişehir, Turkey.