Organizations: School of Artificial Intelligence; National Engineering Laboratory for Big Data System Computing Technology; Guangdong Provincial Key Laboratory of Intelligent Information Processing; Intelligent Computing Center, Shenzhen University, Shenzhen 518060, China · Institute of Image Understanding Research, North Minzu University, Yinchuan 750021, China
Neural collapse (NC) describes the elegant geometric structure that sample features and classifier vectors exhibit when a classification model's loss reaches its minimum. Class imbalance, however, generally distorts standard NC. In this paper, we theoretically reformulate NC on imbalanced datasets and in open-set settings. Our analysis is based on an optimization problem over class-wise averaged losses, rather than the global averaged losses used in prior NC studies. We prove that when class-wise averaged losses attain their minima, standard NC is achieved on balanced or imbalanced datasets, for both cross-entropy (CE) and binary cross-entropy (BCE) losses. We further extend the analysis to an open-set-inspired scenario, where some classes contain no training samples yet are still assigned classifier vectors in the losses. We find that in this scenario, when class-wise averaged losses reach their minima, standard NC still emerges on the known classes with training samples, while the classifier vectors of unknown classes without training samples overlap with each other and become orthogonal to those of the known classes. Finally, we analyze Softmax- and Sigmoid-based contrastive losses re-balanced by inverse sample frequency, proving that at their minima on balanced or imbalanced datasets, sample features of different classes likewise form standard NC.
Figures & tables
Loss
Dataset
IF
ResNet18
sposxx
sposwx
snegwx
b
CE
CIFAR10
1
.9799±.0029
.9895±.0015
−.1099±.0079
.0000±.0221
10
.9778±.0056
.9886±.0028
−.1098±.0081
.0000±.0131
50
.9709±.0113
.9850±.0056
−.1093±.0105
.0000±.0246
100
.9697±.0134
.9843±.0067
−.1092±.0110
.0000±.0347
NC theor. value
1.0
1.0
−1/9≈−.1111
—
Table 1: Statistical results (mean ± std) for positive/negative similarities ( spos / sneg ) and biases ( b ) of different models trained by CE and BCE on CIFAR10-LT and CIFAR100-LT with IF=1,10,50 , and 100 . The CE and BCE are class-wise averaged using Eq. ( 7 ). In the table, when the absolute value of a number if less than 1 , we omit the ‘0’ in its integer part to save space.
Loss
Dataset
IF
ViT-Tiny
sposxx
sposwx
snegwx
b
CE
CIFAR10
1
.9997±.0000
.9999±.0000
−.1111±.0004
.0011±.0012
10
.9993±.0001
.9997±.0001
−.1111±.0003
−.0004±.0013
50
.9992±.0001
.9996±.0000
−.1111±.0004
−.0002±.0012
100
.9985±.0002
.9993±.0001
−.1110±.0003
−.0008±.0008
NC theor. value
1.0
1.0
−1/9≈−.1111
—
Table 2
Figure 1: Distribution of positive and negative scaled similarities ( γwk⊤xˉj ) and biases of ResNet18 trained by class-wise (CW) and globally (G.) averaged CE and BCE.
Figure 2: Similarity matrices for class centers {xˉk}k=1K and classifier vectors {wk}k=1K of ResNet18 trained by different BCE losses on CIFAR10-LT with IF = 100.
D
K1
ResNet18 trained using CE
sposxx
snegxx
sposwx
snegwx
snegww
CIFAR10
9
.9938±.0005
−.1241±.0051
.9968±.0002
−.1246±.0034
−.1250±.0004
7
.9556±.0067
−.1221±.0175
.9649±.0022
−.1558±.0137
−.1659±.0102
5
.9532±.0060
−.1155±.0220
.9429±.0063
−.2095±.0189
−.2442±.0111
3
.9385±.0073
−.1476±.0145
.9271±.0050
−.3155±.0188
−.4307±.0162
CIFAR100
90
.9948±.0010
−.0108±.0541
.9971±.0006
−.0112±.0541
−.0112±.0541
Table 2: Statistical results (mean ± std) for different metrics of ResNet18 trained by CE and BCE on CIFAR10 and CIFAR100 with K−K1 empty classes. The CE and BCE are class-wise averaged using Eqs. ( 12 ) and ( 14 ).
D
K1
ResNet18 trained using CE
sempwx
sempww
semp2ww
∥x~∥
CIFAR10
9
−.1668±.0000
—
.0000±.0028
0.0114
7
−.9261±.0083
.8431±.0171
−.0220±.0170
0.1784
5
−.8663±.0092
.7383±.0214
−.0583±.0103
0.3135
3
−.7569±.0068
.5581±.0234
−.1603±.0281
0.4631
CIFAR100
90
−.2541±.0156
.4836±.0309
−.0001±.0158
0.0204
Table 6
epoch
ViT-Tiny trained by BCE
sposxx
snegxx
sposwx
snegwx
snegww
50
.9726±.0256
.4947±.0079
.0783±.0199
−.4608±.0076
.3899±.0066
100
.9902±.0045
.4096±.0041
.2305±.0053
−.3929±.0044
.3302±.0044
200
.9949±.0052
.2601±.0026
.5001±.0047
−.2629±.0032
.2074±.0034
300
.9937±.0038
.1203±.0037
.7573±.0040
−.1368±.0041
.0844±.0036
400
.9954±.0059
.0052±.0035
.9688±.0037
−.0329±.0034
−.0136±.0026
Table 3: Statistical results (mean ± std) for different metrics of ViT-Tiny trained on CIFAR100 with K1=50 .
Loss
Dataset
IF
ResNet18
sposxx
snegxx
b
Softmax-based
CIFAR10
1
0.9996±0.0001
−0.1111±0.0036
0.0000±0.0002
10
0.9996±0.0002
−0.1110±0.0031
0.0000±0.0004
50
0.9996±0.0002
−0.1111±0.0040
0.0000±0.0024
100
0.9995±0.0003
−0.1104±0.0054
0.0000±0.0044
NC theo. value
1.0
−1/9≈−0.1111
—
Table 4: Statistical results (mean ± std) of different models trained by Softmax-based and Sigmoid-based losses on CIFAR10-LT and CIFAR100-LT with IF=1,10,50 , and 100 .
Loss
Dataset
IF
ViT-Tiny
sposxx
snegxx
b
Softmax-based
CIFAR10
1
0.9998±0.0001
−0.1111±0.0036
0.0002±0.0002
10
0.9997±0.0001
−0.1111±0.0040
0.0000±0.0007
50
0.9996±0.0002
−0.1110±0.0057
0.0000±0.0024
100
0.9997±0.0001
−0.1109±0.0053
0.0000±0.0046
NC theo. value
1.0
−1/9≈−0.1111
—
Table 9
Classification Training
Open-set Scenario
Supervised Contrastive Learning
ResNet18
ViT-Tiny
ResNet18
ViT-Tiny
ResNet18
ViT-Tiny
Hyper-parameter
epochs
800
800
1000
1000
5000
5000
optimizer
SGD
AdamW
SGD
AdamW
SGD
AdamW
batch size
128
128
128
128
8192
8192
learning rate
0.01
0.0001
0.01
0.0001
0.5
0.0002
LR schedule
step
step
step
step
step
cosine
Table S.1: Detailed experimental settings.
Dataset
nmax for classification
nmax for contrastive learning
IF = 10
IF = 50
IF = 100
IF = 10
IF = 50
IF = 100
CIFAR10-LT
5000
5000
5000
2005
2927
3302
CIFAR100-LT
500
500
500
210
325
378
Table S.2: The number of samples, nmax , of the largest class in CIFAR10-LT and CIFAR100-LT for classification and contrastive learning.
D
K1
ViT-Tiny trained using CE
sposxx
snegxx
sposwx
snegwx
snegww
CIFAR10
9
1.000±.0000
−.1250±.0001
1.000±.0000
−.1250±.0019
−.1250±.0029
7
.9909±.0023
−.1646±.0207
.9948±.0014
−.1658±.0170
−.1666±.0117
5
.9996±.0000
−.2499±.0004
.9998±.0000
−.2499±.0020
−.2500±.0031
3
.9979±.0002
−.4977±.0020
.9984±.0001
−.4992±.0029
−.4998±.0046
CIFAR100
90
.9881±.0060
−.0107±.0067
.9935±.0032
−.0112±.0067
−.0112±.0060
Table S.3: Statistical results (mean ± std) for different metrics of ViT-Tiny trained by CE and BCE on CIFAR10 and CIFAR100 with K−K1 empty classes. The CE and BCE are class-wise averaged using Eqs. ( 12 ) and ( 14 ).
D
K1
ViT-Tiny trained using CE
sempwx
sempww
semp2ww
∥x~∥
CIFAR10
9
.1177±.0000
—
.0001±.0015
0.0002
7
−.7712±.0043
.9997±.0002
−.0004±.0072
0.0217
5
−.9965±.0000
1.000±.0000
.0000±.0009
0.0057
3
−.9113±.0016
.9999±.0001
.0000±.0053
0.0292
CIFAR100
90
−.0011±.0000
.9997±.0004
.0000±.0048
0.0211
Table 13
Figure S.1: Distributions of positive and negative scaled similarities and biases for ResNet18 trained on CIFAR10-LT. “CW” and “G.” denote class-wise and globally averaged losses, respectively.
Figure S.2: Distributions of positive and negative scaled similarities and biases for ResNet18 trained on CIFAR100-LT. “CW” and “G.” denote class-wise and globally averaged losses, respectively.
Figure S.3: Distributions of positive and negative scaled similarities and biases for ViT-Tiny trained on CIFAR10-LT. “CW” and “G.” denote class-wise and globally averaged losses, respectively.
Figure S.4: Distributions of positive and negative scaled similarities and biases for ViT-Tiny trained on CIFAR100-LT. “CW” and “G.” denote class-wise and globally averaged losses, respectively.
Figure S.5: Similarity matrices for class centers {xˉk}k=1K and classifier vectors {wk}k=1K of ResNet18 trained with BCE on CIFAR10-LT with IF =10 .
Figure S.6: Similarity matrices for class centers {xˉk}k=1K and classifier vectors {wk}k=1K of ResNet18 trained with BCE on CIFAR10-LT with IF =50 .
Figure S.7: Similarity matrices for class centers {xˉk}k=1K and classifier vectors {wk}k=1K of ResNet18 trained with BCE on CIFAR10-LT with IF =100 .
Figure S.8: Similarity matrices for class centers {xˉk}k=1K and classifier vectors {wk}k=1K of ResNet18 trained with CE on CIFAR10-LT with IF =10 .
Figure S.9: Similarity matrices for class centers {xˉk}k=1K and classifier vectors {wk}k=1K of ResNet18 trained with CE on CIFAR10-LT with IF =50 .
Figure S.10: Similarity matrices for class centers {xˉk}k=1K and classifier vectors {wk}k=1K of ResNet18 trained with CE on CIFAR10-LT with IF =100 .
Figure S.11: Similarity matrices for class centers {xˉk}k=1K and classifier vectors {wk}k=1K of ResNet18 trained with BCE on CIFAR100-LT with IF =10 .
Figure S.12: Similarity matrices for class centers {xˉk}k=1K and classifier vectors {wk}k=1K of ResNet18 trained with BCE on CIFAR100-LT with IF =50 .
Figure S.13: Similarity matrices for class centers {xˉk}k=1K and classifier vectors {wk}k=1K of ResNet18 trained with BCE on CIFAR100-LT with IF =100 .
Department of Engineering, Engineering Technology East Tennessee State University · Department of Electrical and Computer Engineering Tufts University · Department of Electrical and Computer Engineering Worcester Polytechnic Institute +1