In real-world applications of multiclass classification models, misclassification in an important class (e.g., stop sign) can be significantly more harmful than in other classes (e.g., no parking). Thus, it is crucial to improve the recall of an important class while maintaining overall accuracy. For this problem, we found that improving the separation of important classes relative to other classes in the feature space is effective. Existing methods that give a class-sensitive penalty for cross-entropy loss do not improve the separation. Moreover, the methods designed to improve separations between all classes are unsuitable for our purpose because they do not consider the important classes. To achieve the separation, we propose a loss function that explicitly gives loss for the feature space, called class-sensitive additive angular margin (CAMRI) loss. CAMRI loss is expected to reduce the variance of an important class due to the addition of a penalty to the angle between the important class features and the corresponding weight vectors in the feature space. In addition, concentrating the penalty on only the important class hardly sacrifices separating the other classes. Experiments on CIFAR-10, GTSRB, and AwA2 showed that CAMRI loss could improve the recall of a specific class without sacrificing accuracy. In particular, compared with GTSRB's second-worst class recall when trained with cross-entropy loss, CAMRI loss improved recall by 9%.
Figures & tables
Figure 1: The contour plots of the loss values formed by each loss function: (a), (b), (c), and (d) CE loss, WCE loss, L2-constrained softmax loss, and ArcFace or CAMRI loss, respectively. The vertical and horizontal axes represent the elements z1,z2 of z , respectively, and the contour plots represent the loss values, with yellow and blue indicating high and low loss values, respectively. W1,W2 , and W3 are the weight vectors of the last FC layer’s weight. The ground-truth (GT) weight vector W1 is shown in the red arrow. The non-GT weight vectors W2 and W3 are shown in black arrows. The gray lines are the decision boundary.
Figure 2: The mean and standard deviation of the recall of the important class and accuracy over ten trials, obtained from the criteria described in Section 4.2.1 , are shown for each important class setting, as indicated in their subcaptions, for example, (a) indicates the result when the cat class, which is the worst 1 recall when learning CIFAR-10 with CE loss, is set as the important class. Settings (b) through (i) also correspond to their subcaptions. In each setting from (a) to (i), the left figures represent recall and accuracy, respectively. The horizontal axes are the compared methods, and the vertical axes are the recall or accuracy values. The error bars mean standard deviation.
Figure 3: Normalized feature vectors of MNIST z′ (represented as dots) and weight vectors (represented as solid lines) in the two-dimensional feature space: (a), (b), and (c) L2-constrained softmax loss, ArcFace, and CAMRI loss, respectively. The vertical and horizontal axes represent the elements z1′,z2′ of z′ , respectively. In the results, class ”3” (shown in red) was set as the important class.
class
CAMRI
ArcFace
WCE ( w=4 )
ArcFace ( m=0 )
CE
cat (important)
0.0695
0.0384
0.0603
0.0928
0.0842
min excl. cat
0.0948
0.0168
0.0540
0.0937
0.0570
median excl. cat
0.1143
0.0403
0.0766
0.1009
0.0806
dog (important)
0.0717
0.0256
0.0667
0.1009
0.0815
min excl. dog
0.0919
0.0158
0.0602
0.0928
0.0570
median excl. dog
0.1116
0.0328
0.0835
0.1009
0.0806
Table 1: The standard deviation of the radian angle between the feature vectors and the corresponding GT weight vector for each method is shown. The results are cases when each of the cat, dog, and airplane classes of CIFAR-10 are set to the important class. In each case, the standard deviation of the important class, the smallest standard deviation among those excluding the important class, and the median standard deviation among those excluding the important class are displayed.
Figure 4: Changes in recall of the important class (blue line) and accuracy (orange line) by varying penalties are shown. The lines show the mean value in ten trials, and the bands show the standard deviation. The left columns are WCE loss, and the right columns are CAMRI loss. (a)-(c) are the cat, dog, and airplane classes of CIFAR-10, respectively, as the important class. The horizontal axis is the value of the penalty ( wκ for WCE loss and mκ for CAMRI loss), whereas the vertical axis is the value of recall of the important class and accuracy.
Figure 5: Comparison of recall improved by CAMRI loss on (a)-(c) CIFAR-10, GTSRB, and AwA2, respectively. The horizontal axis shows baseline recalls obtained by CE loss, averaged over ten trials. The vertical axis shows changes in recall when trained with CAMRI loss for ten trials. The color of the dots corresponds to the class, and the results of ten trials are shown in the figure.
Figure 6: The left (a) is the baseline confusion matrix of CIFAR-10 obtained from trained using CE loss. (b), (c), and (d) Differences between the confusion matrix with CE loss (i.e., (a)) and that with the CAMRI loss, where the cat, dog, and airplane classes are set as the important class, respectively. Each result is the average of ten trials and corresponds to the result shown in Fig. 2 .
(a) CIFAR-10
Two important class setting
Three important class setting
method
cat
airplane
cat
deer
truck
Recall
0.802
0.936
0.815
0.911
0.939
CAMRI Loss
mk
0.274
0.170
0.234
0.160
0.025
Accuracy
0.8823
0.8842
CE Loss
Recall
0.738 ± 0.041
0.888 ± 0.022
0.738 ± 0.041
0.875 ± 0.023
0.916 ± 0.021
Table 2: For CAMRI loss, the accuracy, recall of each important class, and mk obtained from the hyperparameter optimization are shown for each dataset and each multiple important class setting. For CE loss, the mean and standard deviation of the accuracy and recall of each important class over ten trials are shown on (a)-(c) CIFAR-10, GTSRB, and AwA2, respectively. mk means the class-sensitive angular margin of class k of CAMRI loss for each important class.
Figure 7: Five random examples of correctly classified and misclassified examples are shown for each class of the worst and best recall when training AwA2 with CE loss, respectively. (a) and (b) represent correctly classified examples, and (c) and (d) represent misclassified examples. Also, (a) and (c) represent the worst recall class, and (b) and (d) represent the best recall class.
Metric
Base model shown in Table 4
EfficientNet [ 28 ]
Accuracy
0.651 ± 0.013
0.717 ± 0.005
Recall of mouse (worst1)
0.118 ± 0.051
0.153 ± 0.023
Recall of beaver (worst2)
0.145 ± 0.045
0.243 ± 0.043
Recall of moose (median)
0.559 ± 0.066
0.613 ± 0.024
Table 3: Comparison of accuracy, the worst recall, the second-worst recall, and median recall when training AwA2 with CE loss, respectively, using the model shown in Table 4 and EfficientNetB2. Each value represents the mean and standard deviation of ten trials.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: The contour plots of the loss values formed by each loss function are shown. From left to right are WCE (the same as Fig. 1 (c)) and CRWWCE loss. The styles of this figure are the same as in Fig. 1 .
Layer type
Output Shape
Activation
Input
32×32×3
-
Conv
32×32×64
ReLU
Conv
32×32×64
ReLU
BN
32×32×64
-
Conv
32×32×64
ReLU
MP
16×16×64
-
Appendix
Table 4: Summary of the model structure taking CIFAR-10 images as input. The activation function ( ∗ ) of the last FC layer depends on the loss function used. The linear activation function passes through the output of a layer to the next layer, i.e., it does not activate.
CIFAR-10
GTSRB
AwA2
cat
dog
airplane
limit-80-end
no-passing-end
limit-80
mouse
beaver
moose
method
(worst1)
(worst2)
(median)
(worst1)
(worst2)
(median)
(worst1)
(worst2)
(median)
Recall change
0.054
0.016
0.029
0.075
0.09
0.006
0.011
0.093
0.086
CAMRI
s
16
16
8
64
4
64
16
16
16
mk
4π/64
5π/64
4π/64
π/64
8π/64
π/64
8π/64
6π/64
5π/64
Recall change
0.027
0.008
0.022
0.048
0.071
0.004
-0.017
0.042
0.053
Appendix
Table 5: Summary of recall changes and hyperparameter values of loss functions corresponding to the values obtained from the performance evaluation in Table 2 . Recall changes differ from the baseline results (bottom row of Table 2 ). Hyperparameters are s and mk in CAMRI loss, s , and m in ArcFace, ckfn , and Ck′,kfp in CRWWCE loss, and wk in WCE loss. Recall change and hyperparameter values are shown in each cell from the top row.
CIFAR-10
GTSRB
AwA2
cat
dog
airplane
limit-80-end
no-passing-end
limit-80
mouse
beaver
moose
method
metric
(worst1)
(worst2)
(median)
(worst1)
(worst2)
(median)
(worst1)
(worst2)
(median)
CAMRI
Recall
0.865 ± 0.023
0.920 ± 0.014
0.920 ± 0.018
0.843 ± 0.055
0.967 ± 0.028
0.995 ± 0.003
0.130 ± 0.046
0.238 ± 0.053
0.646 ± 0.054
(proposal)
Accuracy
0.720 ± 0.006
0.727 ± 0.004
0.650 ± 0.036
0.874 ± 0.030
0.890 ± 0.022
0.981 ± 0.001
0.667 ± 0.014
0.664 ± 0.015
0.656 ± 0.025
ArcFace
Recall
0.766 ± 0.041
0.843 ± 0.032
0.911 ± 0.029
0.843 ± 0.043
0.942 ± 0.070
0.994 ± 0.003
0.300 ± 0.458
0.188 ± 0.052
0.612 ± 0.046
Accuracy
0.880 ± 0.004
0.880 ± 0.006
0.880 ± 0.005
0.528 ± 0.063
0.982 ± 0.002
0.982 ± 0.003
0.008 ± 0.006
0.666 ± 0.009
0.677 ± 0.009
Appendix
Table 6: Mean and standard deviation of recall of the important class, the upper row, and accuracy, the lower row, over ten trials. The results presented here show the cases with the highest recall for each method.