Authors: Kartheek Bondugula, Santiago Mazuelas, Aritz Pérez, Anqi Liu
Organizations: Basque Center for Applied Mathematics, Bilbao, Spain · Ikerbasque, Basque Foundation for Science · Johns Hopkins University, Baltimore, USA
Loss functions play a central role in supervised classification. Cross-entropy (CE) is widely used, whereas the mean absolute error (MAE) loss can offer robustness but is difficult to optimize. Interpolating between the CE and MAE losses, generalized cross-entropy (GCE) has recently been introduced to provide a trade-off between optimization difficulty and robustness. Existing formulations of GCE result in a non-convex optimization over classification margins that is prone to underfitting, leading to poor performances with complex datasets. In this paper, we propose a minimax formulation of generalized cross-entropy (MGCE) that results in a convex optimization over classification margins. Moreover, we show that MGCEs can provide an upper bound on the classification error. The proposed bilevel convex optimization can be efficiently implemented using stochastic gradient computed via implicit differentiation. Using benchmark datasets, we show that MGCE achieves strong accuracy, faster convergence, and better calibration, especially in the presence of label noise.
Figures & tables
Figure 1: Relation between β and the resulting loss function. For β=1 , the loss corresponds to the MAE while for β=∞ , it corresponds to CE. For β∈(1,∞) , the loss interpolates between the MAE and CE.
Figure 2: Relation between the minimax classifier hβ(x)y and the worst-case probability pβ(y∣x) corresponding with 2 classes. For β∈(1,∞) , the worst-case probabilities take a cautious stance, avoiding the extremes of MAE ( β=1 ) and CE ( β=∞ ) losses.
Input:
Instance x , model parameters μ ,
loss parameter β , tolerance ϵ
Output:
Gradient ∂μ∂φβ(x,μ)
Algorithm 1 Implicit gradient for MGCE via bisection.
Figure 3: Average test accuracy under clean training data obtained for multiple complex datasets. The value of loss parameter β is set to 1.4. The figure shows the fast convergence of the proposed MGCE in comparison to the GCE.
Dataset
Method
Test Accuracy (%)
SCE (%)
η =0.0
η =0.2
η =0.4
η =0.0
η =0.2
η =0.4
FashionMNIST
MGCE
92.94 ± 0.17
90.11 ± 0.38
88.65 ± 0.15
05.20 ± 0.23
02.44 ± 0.28
02.54 ± 1.30
GCE
92.93 ± 0.34
91.11 ± 0.22
89.74 ± 0.29
05.52 ± 0.31
07.14 ± 0.22
08.63 ± 0.35
CE
92.73 ± 0.32
89.62 ± 0.15
87.40 ± 0.13
05.33 ± 0.24
17.32 ± 0.85
33.52 ± 1.86
MAE
91.84 ± 0.24
89.86 ± 0.26
88.44 ± 0.38
73.47 ± 0.57
74.88 ± 0.46
74.23 ± 0.28
CIFAR10
MGCE
91.01 ± 0.43
86.91 ± 0.36
83.49 ± 0.45
05.53 ± 0.31
10.26 ± 0.37
12.15 ± 0.5
Table 1: Average test accuracy and SCE for clean and noisy data. The ratio η represents the fraction of samples corrupted by symmetric label noise. We report accuracies of the epoch where validation accuracy is maximum, and the loss parameter β is selected using cross-validation. Our proposed MGCE method obtains strong accuracy together with improved calibration.
Figure 4: Top-1 validation accuracy on the real-world noisy dataset WebVision. The figure shows that the proposed MGCE outperforms GCE, which significantly underfits on this complex dataset due to its non-convexity.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Top-1 test accuracy on the real-world noisy dataset Clothing-1M. The figure shows that the proposed MGCE outperforms GCE, which underfits on this complex dataset with 1 million training samples with noisy labels.
Dataset
λ0=10−3
λ0=10−4
λ0=10−5
MGCE
GCE
MGCE
GCE
MGCE
GCE
SVHN
80.42 ± 0.57
85.00 ± 1.06
92.41 ± 0.28
93.53 ± 0.13
94.58 ± 0.02
93.32 ± 0.15
CIFAR10
67.40 ± 1.27
69.32 ± 0.93
87.98 ± 0.53
88.06 ± 0.39
91.49 ± 0.53
91.11 ± 0.09
FashionMNIST
89.56 ± 0.53
89.51 ± 0.33
93.09 ± 0.55
92.74 ± 0.43
92.79 ± 0.50
92.71 ± 0.31
Appendix
Table 2: Average validation accuracies for different λ0
Figure 6: Average test accuracy under clean data training data obtained for multiple tabular datasets. The value of loss parameter β is set to 1.4 for both MGCE and GCE. In addition, the figure reports the objective values V1.40 , V1.18 , V1.05 , and VMAE corresponding with β=1.40 , β=1.18 , β=1.05 , and β=1.00 , respectively, that provide a lower bound on the test accuracy. The figure shows the fast convergence of the proposed MGCE in comparison to GCE, and that the bounds can provide a minimum estimate on the classification accuracy.
Dataset
Loss
Test Accuracy (%)
SCE (%)
η=0.0
η=0.2
η=0.4
η=0.0
η=0.2
η=0.4
MNIST
MGCE
97.28 ± 0.07
96.15 ± 0.05
94.99 ± 0.05
0.79 ± 0.06
6.52 ± 0.14
10.53 ± 0.15
GCE
97.14 ± 0.04
95.74 ± 0.07
94.19 ± 0.23
0.37 ± 0.03
17.3 ± 0.3
1.43 ± 0.07
CE
97.17 ± 0.04
95.59 ± 0.1
94.02 ± 0.09
0.4 ± 0.07
21.18 ± 0.76
39.46 ± 0.87
MAE
97.24 ± 0.03
96.1 ± 0.06
94.95 ± 0.13
80.03 ± 0.12
81.18 ± 0.22
80.55 ± 0.23
covertype
MGCE
77.24 ± 0.09
75.83 ± 0.21
74.08 ± 0.07
1.74 ± 0.17
12.58 ± 0.48
25.55 ± 0.2
Appendix
Table 3: Average test accuracy and static calibration error (SCE) for clean and noisy tabular datasets. The ratio η represents the fraction of samples corrupted by symmetric label noise. We report accuracies of the epoch where validation accuracy is maximum. Our proposed MGCE method obtains strong accuracy.
Cross-entropy (CE) is the default training loss for supervised classification, but its sample efficiency is limited when labels are scarce. Existing remedies primarily act on the data side, via augmentation, synthesis, or transfer from pretrained models; the training objective itself is rarely revisited. We revisit it here. Drawing on the classical observation that generative classifiers reach their asymptotic error with fewer samples than discriminative ones, we propose Generative Cross-Entropy (GenCE), a drop-in replacement for CE that introduces a generative learning principle into a standard discriminative network without altering the architecture or fitting a separate density model. GenCE follows from a Bayesian rewrite of the class-conditional likelihood and, in the mini-batch approximation, reduces to normalizing each sample's softmax score against the model's predictions on the batch, coupling the training signal across examples sharing a class. We extend the proper-scoring-rule framework to such non-local losses and prove that GenCE is strictly proper under a mild completeness condition: its population risk is uniquely minimized at the true posterior. Across three datasets, on two architectures and in both balanced small-data and class-imbalanced regimes, GenCE outperforms CE and other widely used losses, while also producing better-calibrated probabilities and stronger out-of-distribution detection.
Labeling a training set is often expensive and susceptible to errors, making the design of robust loss functions for label noise an important problem. The symmetry condition provides theoretical guarantees for robustness to such noise. In this work, we study a symmetrization method arising from the unique decomposition of any multi-class loss function into a symmetric component and a class-insensitive term. In particular, symmetrizing the cross-entropy loss leads to a linear multi-class extension of the unhinged loss. Unlike in the binary case, the multi-class version must have specific coefficients in order to satisfy the symmetry condition. Under suitable assumptions, we show that this multi-class unhinged loss is the unique convex multi-class symmetric loss. We also show that it has a fundamental local role: the linear approximation of any symmetric loss around score vectors with equal components is equivalent to the multi-class unhinged loss. We then introduce SGCE and alpha-MAE, two loss functions that interpolate between the multi-class unhinged loss and the Mean Absolute Error while allowing control of the beta-smoothness of the loss. Experiments on standard noisy-label benchmarks show competitive performance compared with existing robust loss functions.
Alexandre Lemire Paquin, Brahim Chaib-Draa, Philippe Giguère
Department of Computer Science and Software Engineering Université Laval, Québec, Canada
This paper investigates the deep learning optimization problem with softmax cross-entropy loss. We propose a layer separation strategy to alleviate the strong nonconvexity encountered during training deep networks. For cross-entropy models with fully connected and convolutional neural networks, we introduce auxiliary variables associated with hidden layer outputs and construct corresponding layer separation models, which decompose the original deeply nested optimization problem into a sequence of more manageable subproblems. We also conduct theoretical analyses, proving that the new layer separation loss provides an upper bound for the original cross-entropy loss. Moreover, we design alternating minimization algorithms and prove that, under appropriate conditions, these algorithms exhibit decreasing properties of the loss function. Numerical experiments validate the effectiveness of the proposed methods and indicate improved optimization behavior, especially for fully connected and convolutional neural networks.
Yaru Liu, Michael K. Ng, Yiqi Gu
School of Mathematical Sciences, University of Electronic Science and Technology of China, Sichuan 611731, China · Department of Mathematics, Hong Kong Baptist University, Kowloon Tong, Hong Kong, China