Organizations: School of Artificial Intelligence, Shenzhen University, Shenzhen, 518060, China. · National Engineering Laboratory for Big Data System Computing Technology, Shenzhen University, Shenzhen, 518060, China. · Guangdong Provincial Key Laboratory of Intelligent Information Processing, Shenzhen University, Shenzhen, 518060, China. · College of Computer Science and Software Engineering, Shenzhen University, Shenzhen, 518060, China. · Big Data Institute, University of Oxford, Oxford, OX3 7LF, United Kingdom. · Institute of Image Understanding Research, North Minzu University, Yinchuan, 750021, China.
Classification and feature learning are two closely interrelated fundamental tasks. Feature learning in supervised settings is typically achieved by training classification models, under the common assumption that higher classification accuracy implies better feature quality. In this paper, we first point out that classification accuracy reflects the sample-wise local properties of a model on a dataset, whereas feature learning aims to learn features with good dataset-level global properties, such as intra-class compactness and inter-class separability. From the first principle of feature learning, we redesign the binary cross-entropy (BCE) loss, a counterpart of the cross-entropy (CE) loss used in classification. Then, in normalized feature space, we analyze the minima of the BCE and CE losses and show that both can lead to neural collapse, where sample features from different classes become maximally compact and maximally separated. More importantly, by investigating the convergence behaviors of the two losses as the scale factor γ varies in the normalized space, we reveal how classification accuracy and feature quality change differently across spaces of different sizes. Under a fixed training strategy, a very large γ achieves perfect classification accuracy but slows convergence toward the ideal feature geometry, whereas a very small γ makes classification highly sensitive to small perturbations. A moderate γ is therefore necessary for simultaneously achieving reliable classification and well-structured features. Extensive experiments validate these findings and demonstrate that BCE loss exhibits advantages in long-tail and open-set recognitions, as it better aligns with the global requirements of feature learning.
Figures & tables
Figure 1: The intuitive difference between classification and feature learning. Classification (left) requires that the feature vector of each sample be closer to its own class center/classifier vector than to those of other classes, and classification accuracy reflects the model’s local properties at each sample. In contrast, intra-class compactness and inter-class separability of sample features are the model’s global properties at the dataset level, and feature learning (right) drives the feature vectors of all samples to be as close as possible to their respective class centers while simultaneously being far from other class centers
Figure 2: The evolution of the three NC metrics in the training of ResNet18, ResNet50, DenseNet121, and ViT on the training data of CIFAR10 driven by the normalized CE and BCE losses with γ=8 using SGD and AdamW, respectively
γ
2
3
4
5
6∗
6.5
7∗
7.5
8∗
8.5
9∗
9.5∗
10∗
11
12∗
13
14∗
15
16∗
CE
28
31
64
31
77
145
180
308
435
741
1448
2083
-
-
-
-
-
-
-
BCE
24
22
22
14
15
-
21
-
37
-
53
-
83
137
243
411
737
1165
2076
Table 1: The number of training epochs required by ResNet18 to reach the stopping criterion of Eqs. ( 22 ) and ( 23 ) on MNIST trained by the normalized CE and BCE with different γ>1 . “ ∗ ” indicating that the data is used for fitting the functions of epoch number on γ . When γ≥10 , training ResNet18 with the CE loss requires more than 4,000 epochs to converge, so the experiment was not performed
Loss
Fitted function
Fitting data
Predicted data
R2↑
RMSE ↓
R2↑
RMSE ↓
CE
568.39γ−3645.68
0.8556
299.08
−0.5945
317.58
236.01γ2−3098.39γ+10205.49
0.9939
61.59
0.7463
126.69
40.65γ3−706.28γ2+4076.80γ−7729.46
0.9969
43.85
0.9038
77.99
0.54exp(0.87γ)−61.83
0.9932
64.98
0.9253
68.72
4.22log(0.32γ)−3.01
0.8094
343.40
−0.9326
349.63
Table 2: Fitted functions and their evaluation results for the epoch number used by the CE and BCE with γ>5
Figure 3: The distribution of the final classifier biases and the positive/negative unbiased decision scores for 56 ResNet18s (top, without the final ReLU activation) and 56 ViTs (bottom) trained on CIFAR10, with various scale factors γ . The maximum number of training epochs is fixed at 200. The initial mean ( bˉ=K1∑j=1Kbj ) of the biases is 0 for ResNet18 and 5 for ViT, and the initial variance is both 0 . The solid line represents the theoretical extremum when achieving NC
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Neural Collapse
Classification
setting-1
setting-2
setting-3
setting-4
setting-5
setting-6
Hyper-parameter
epochs
200
200
200
200
200
200
optimizer
SGD
AdamW
SGD
AdamW
SGD
AdamW
batch size
128
128
128
128
128
128
learning rate
0.01
0.001
0.03
0.0003
0.1
0.005
learning rate decay
step
cosine
step
cosine
step
cosine
Appendix
Table 9.1: Detailed experimental settings
Figure 9.1: The evolution of the three NC metrics in the training of ResNet18 (top), ResNet50 (middle) and DenseNet121 (bottom) on MNIST with CE and BCE losses using SGD and AdamW, respectively
Figure 9.2: The evolution of the three NC metrics in the training of ResNet18 (top), ResNet50 (middle) and DenseNet121 (bottom) on CIFAR10 with CE and BCE losses using SGD and AdamW, respectively
Figure 9.3: The evolution of the three NC metrics in the training of ResNet18 (top), ResNet50 (middle) and DenseNet121 (bottom) on CIFAR100 with CE and BCE losses using SGD and AdamW, respectively
Figure 9.4: The evolution of the three NC metrics in the training of ViT on CIFAR10 (top) and CIFAR100 (bottom) with CE and BCE losses using SGD and AdamW, respectively
Figure 9.5: The distribution of the final classifier biases and positive/negative unbiased decision scores for ResNet18 trained on MNIST with varying mean of initial classifier biases, while the variance of initial classifier biases is 0 and γ=8 . The final classifier biases of CE-trained models are determined by their initial values, while that of BCE-trained models always converge to the same value
Figure 9.6: The distribution of the final classifier biases and positive/negative unbiased decision scores for ViT trained on CIFAR10 with varying mean of initial classifier biases, while the variance of initial classifier biases is 0 and γ=8 . The final classifier biases of CE-trained models are determined by their initial values, while that of BCE-trained models always converge to the same value
Figure 9.7: The distribution of the final classifier biases and positive/negative unbiased decision scores of ResNet18 trained on MNIST with varying initial variance of the classifier biases. The different scale factors, γ=8 (1-st row), γ=16 (2-nd row), γ=32 (3-rd row), and γ=64 (4-th row), also significantly impact the convergence of decision scores and the final classifier biases
Figure 9.8: The distribution of the final classifier bias and unbiased positive/negative decision scores of ViT trained on CIFAR10 with varying initial variance of the classifier bias. The different scale factors, γ=8 (1-st row), γ=16 (2-nd row), and γ=32 (3-rd row), also significantly impact the convergence of decision scores and the final classifier bias
Figure 9.9: The distribution of the final classifier biases and positive/negative unbiased decision scores of ResNet18 trained on MNIST by using various scale factors γ . The variance of initial classifier biases is set to 0, and their initial mean takes different values, 0,10,20 , and 30 , respectively
Figure 9.10: The distribution of the final classifier biases and positive/negative unbiased decision scores of ResNet18 trained on MNIST by using various scale factors γ . The variance of the initial classifier biases is set to 0
Figure 9.11: The distribution of the final classifier biases and positive/negative unbiased decision scores of ResNet18 (without the final ReLU) trained on CIFAR10 by using various scale factors γ . The mean and variance of the initial classifier biases is both set to 0
Figure 9.12: The distribution of the final classifier biases and positive/negative unbiased decision scores of ViT trained on CIFAR10 by using various scale factors γ . The variance of the initial classifier biases is set to 0
Figure 9.13: The fitted functions describing the variation of epoch number with respect to γ for CE (top) and BCE (bottom) losses. The epoch numbers in the figure indicate the training duration required to achieve minimum values (i.e., the criterion in Eq. ( 9.1 )) of the two losses for training ResNet18 on MNIST, with different scale factors γ . The blue dots represent that the data was used for function fitting, the red dots represent the predicted values, and the green triangles represent the observed data in experiments
Loss
Fitted function
Fitting data
Predicted data
R2↑
RMSE ↓
R2↑
RMSE ↓
CE
27.14/γ+44.41
0.9987
32.93
0.9908
52.08
−0.06/γ2+33.05/γ−7.52
0.9999
7.92
0.9897
55.16
−0.002/γ3+0.16/γ2+2.83/γ+1.20
1.0000
3.39
0.9638
103.11
4002.21exp(−45.08γ)+185.11
0.9943
69.52
0.8388
217.72
−710.69log(5.63γ)+175.75
0.8213
389.56
0.3936
422.27
Appendix
Table 9.2: Fitted functions and their evaluation results for the epoch number used by the CE and BCE with γ<0.5
Loss
Fitted function
Fitting data
Predicted data
R2↑
RMSE ↓
R2↑
RMSE ↓
CE
568.39γ−3645.68
0.8556
299.08
−0.5945
317.58
236.01γ2−3098.39γ+10205.49
0.9939
61.59
0.7463
126.69
40.65γ3−706.28γ2+4076.80γ−7729.46
0.9969
43.85
0.9038
77.99
0.54exp(0.87γ)−61.83
0.9932
64.98
0.9253
68.72
4.22log(0.32γ)−3.01
0.8094
343.40
−0.9326
349.63
Appendix
Table 9.3: Fitted functions and their evaluation results for the epoch number used by the CE and BCE with γ>5
M
Loss
CIFAR10
CIFAR100
Tiny-ImageNet
SGD
AdamW
SGD
AdamW
SGD
AdamW
ResNet18
CE
95.90
96.24
78.39
78.05
68.10
67.13
BCE
95.97
95.96
78.46
78.38
69.10
69.12
ResNet50
CE
96.30
96.51
80.89
78.06
71.65
69.49
BCE
96.40
96.24
80.87
79.89
72.95
70.82
ViT
CE
79.51
87.00
64.82
63.19
50.42
54.83
Appendix
Table 9.4: The classification accuracy (%) on the test datasets of CIFAR10, CIFAR100 and Tiny-ImageNet. We conducted learning rate optimization for each experimental combination while keeping other hyperparameters fixed, and the γ=32
Bias Mean
Metrics
MNIST
SVHN
CIFAR10
CIFAR+50
CE
BCE
CE
BCE
CE
BCE
CE
BCE
–
AUROC
96.74
90.22
94.44
66.60
74.34
68.14
78.34
62.17
OSCR
96.59
90.04
92.56
71.00
60.10
67.00
61.22
63.87
[2pt/2pt] 0
AUROC
98.63
99.29
94.42
95.21
83.86
85.70
87.82
90.06
OSCR
98.50
99.09
92.53
93.49
81.54
83.50
85.81
88.43
[2pt/2pt] 5
AUROC
98.58
99.15
94.30
95.07
84.17
85.78
87.76
90.15
Appendix
Table 9.5: The results of open-set recognition. Results are averaged among five randomized trials and the scale factor γ=32
School of Artificial Intelligence; National Engineering Laboratory for Big Data System Computing Technology; Guangdong Provincial Key Laboratory of Intelligent Information Processing; Intelligent Computing Center, Shenzhen University, Shenzhen 518060, China · Institute of Image Understanding Research, North Minzu University, Yinchuan 750021, China