Minimax Generalized Cross-Entropy

Kartheek Bondugula; Santiago Mazuelas; Aritz Pérez; Anqi Liu

stat.MLMar 20, 2026

Minimax Generalized Cross-Entropy

Authors: Kartheek Bondugula, Santiago Mazuelas, Aritz Pérez, Anqi Liu

Organizations: Basque Center for Applied Mathematics, Bilbao, Spain · Ikerbasque, Basque Foundation for Science · Johns Hopkins University, Baltimore, USA

Abstract

Loss functions play a central role in supervised classification. Cross-entropy (CE) is widely used, whereas the mean absolute error (MAE) loss can offer robustness but is difficult to optimize. Interpolating between the CE and MAE losses, generalized cross-entropy (GCE) has recently been introduced to provide a trade-off between optimization difficulty and robustness. Existing formulations of GCE result in a non-convex optimization over classification margins that is prone to underfitting, leading to poor performances with complex datasets. In this paper, we propose a minimax formulation of generalized cross-entropy (MGCE) that results in a convex optimization over classification margins. Moreover, we show that MGCEs can provide an upper bound on the classification error. The proposed bilevel convex optimization can be efficiently implemented using stochastic gradient computed via implicit differentiation. Using benchmark datasets, we show that MGCE achieves strong accuracy, faster convergence, and better calibration, especially in the presence of label noise.

Figures & tables

Figure 1: Relation between $\beta$ and the resulting loss function. For $\beta=1$ , the loss corresponds to the MAE while for $\beta=\infty$ , it corresponds to CE. For $\beta\in(1,\infty)$ , the loss interpolates between the MAE and CE.

Figure 2: Relation between the minimax classifier $\mathrm{h}_{\beta}(x)_{y}$ and the worst-case probability $\mathrm{p}_{\beta}(y|x)$ corresponding with 2 classes. For $\beta\in(1,\infty)$ , the worst-case probabilities take a cautious stance, avoiding the extremes of MAE ( $\beta=1$ ) and CE ( $\beta=\infty$ ) losses.

Input:	Instance $x$ , model parameters $\boldsymbol{\mu}$ ,
	loss parameter $\beta$ , tolerance $\epsilon$
Output:	Gradient $\frac{\partial\varphi_{\beta}(x,\boldsymbol{\mu})}{\partial\boldsymbol{\mu}}$

Algorithm 1 Implicit gradient for MGCE via bisection.

Panel of Figure 3 — Figure 3: Average test accuracy under clean training data obtained for multiple complex datasets. The value of loss parameter $\beta$ is set to 1.4. The figure shows the fast convergence of the proposed MGCE in comparison to the GCE.

Dataset	Method	Test Accuracy (%)			SCE (%)
Dataset	Method	$\eta$ =0.0	$\eta$ =0.2	$\eta$ =0.4	$\eta$ =0.0	$\eta$ =0.2	$\eta$ =0.4
FashionMNIST	MGCE	92.94 $\pm$ 0.17	90.11 $\pm$ 0.38	88.65 $\pm$ 0.15	05.20 $\pm$ 0.23	02.44 $\pm$ 0.28	02.54 $\pm$ 1.30
	GCE	92.93 $\pm$ 0.34	91.11 $\pm$ 0.22	89.74 $\pm$ 0.29	05.52 $\pm$ 0.31	07.14 $\pm$ 0.22	08.63 $\pm$ 0.35
	CE	92.73 $\pm$ 0.32	89.62 $\pm$ 0.15	87.40 $\pm$ 0.13	05.33 $\pm$ 0.24	17.32 $\pm$ 0.85	33.52 $\pm$ 1.86
	MAE	91.84 $\pm$ 0.24	89.86 $\pm$ 0.26	88.44 $\pm$ 0.38	73.47 $\pm$ 0.57	74.88 $\pm$ 0.46	74.23 $\pm$ 0.28
CIFAR10	MGCE	91.01 $\pm$ 0.43	86.91 $\pm$ 0.36	83.49 $\pm$ 0.45	05.53 $\pm$ 0.31	10.26 $\pm$ 0.37	12.15 $\pm$ 0.5

Table 1: Average test accuracy and SCE for clean and noisy data. The ratio

\eta

represents the fraction of samples corrupted by symmetric label noise. We report accuracies of the epoch where validation accuracy is maximum, and the loss parameter

\beta

is selected using cross-validation. Our proposed MGCE method obtains strong accuracy together with improved calibration.

Figure 4: Top-1 validation accuracy on the real-world noisy dataset WebVision. The figure shows that the proposed MGCE outperforms GCE, which significantly underfits on this complex dataset due to its non-convexity.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Top-1 test accuracy on the real-world noisy dataset Clothing-1M. The figure shows that the proposed MGCE outperforms GCE, which underfits on this complex dataset with 1 million training samples with noisy labels.

Dataset	$\lambda_{0}=10^{-3}$		$\lambda_{0}=10^{-4}$		$\lambda_{0}=10^{-5}$
Dataset	MGCE	GCE	MGCE	GCE	MGCE	GCE
SVHN	80.42 $\pm$ 0.57	85.00 $\pm$ 1.06	92.41 $\pm$ 0.28	93.53 $\pm$ 0.13	94.58 $\pm$ 0.02	93.32 $\pm$ 0.15
CIFAR10	67.40 $\pm$ 1.27	69.32 $\pm$ 0.93	87.98 $\pm$ 0.53	88.06 $\pm$ 0.39	91.49 $\pm$ 0.53	91.11 $\pm$ 0.09
FashionMNIST	89.56 $\pm$ 0.53	89.51 $\pm$ 0.33	93.09 $\pm$ 0.55	92.74 $\pm$ 0.43	92.79 $\pm$ 0.50	92.71 $\pm$ 0.31

Appendix

Table 2: Average validation accuracies for different

\lambda_{0}

Panel of Figure 6 — Figure 6: Average test accuracy under clean data training data obtained for multiple tabular datasets. The value of loss parameter $\beta$ is set to 1.4 for both MGCE and GCE. In addition, the figure reports the objective values $\mathrm{V}_{1.40}$ , $\mathrm{V}_{1.18}$ , $\mathrm{V}_{1.05}$ , and $\mathrm{V}_{\mathrm{MAE}}$ corresponding with $\beta=1.40$ , $\beta=1.18$ , $\beta=1.05$ , and $\beta=1.00$ , respectively, that provide a lower bound on the test accuracy. The figure shows the fast convergence of the proposed MGCE in comparison to GCE, and that the bounds can provide a minimum estimate on the classification accuracy.

Dataset	Loss	Test Accuracy (%)			SCE (%)
Dataset	Loss	$\eta=0.0$	$\eta=0.2$	$\eta=0.4$	$\eta=0.0$	$\eta=0.2$	$\eta=0.4$
MNIST	MGCE	97.28 $\pm$ 0.07	96.15 $\pm$ 0.05	94.99 $\pm$ 0.05	0.79 $\pm$ 0.06	6.52 $\pm$ 0.14	10.53 $\pm$ 0.15
	GCE	97.14 $\pm$ 0.04	95.74 $\pm$ 0.07	94.19 $\pm$ 0.23	0.37 $\pm$ 0.03	17.3 $\pm$ 0.3	1.43 $\pm$ 0.07
	CE	97.17 $\pm$ 0.04	95.59 $\pm$ 0.1	94.02 $\pm$ 0.09	0.4 $\pm$ 0.07	21.18 $\pm$ 0.76	39.46 $\pm$ 0.87
	MAE	97.24 $\pm$ 0.03	96.1 $\pm$ 0.06	94.95 $\pm$ 0.13	80.03 $\pm$ 0.12	81.18 $\pm$ 0.22	80.55 $\pm$ 0.23
covertype	MGCE	77.24 $\pm$ 0.09	75.83 $\pm$ 0.21	74.08 $\pm$ 0.07	1.74 $\pm$ 0.17	12.58 $\pm$ 0.48	25.55 $\pm$ 0.2

Appendix

Table 3: Average test accuracy and static calibration error (SCE) for clean and noisy tabular datasets. The ratio

\eta

represents the fraction of samples corrupted by symmetric label noise. We report accuracies of the epoch where validation accuracy is maximum. Our proposed MGCE method obtains strong accuracy.

Robust Loss Functions

Robust loss functions limit the influence of outliers, noisy labels, and corrupted observations during model training.

See all 46 papers

Generative Cross-Entropy: A Strictly Proper Loss for Data-Efficient Classification
Apr 8, 2026Qipeng Zhan, Zhuoping Zhou, Li ShenData-Efficient Learning Proper Scoring Rules
University of Pennsylvania
Symmetrization of Loss Functions for Robust Training of Neural Networks in the Presence of Noisy Labels
May 19, 2026Alexandre Lemire Paquin, Brahim Chaib-Draa, Philippe GiguèreNeural Network Robustness Noisy-Label Learning
Department of Computer Science and Software Engineering Université Laval, Québec, Canada
A Layer Separation Optimization Framework for Cross-Entropy Training in Deep Learning
Apr 25, 2026Yaru Liu, Michael K. Ng, Yiqi GuDeep Learning Optimization Nonconvex Optimization
School of Mathematical Sciences, University of Electronic Science and Technology of China, Sichuan 611731, China · Department of Mathematics, Hong Kong Baptist University, Kowloon Tong, Hong Kong, China

Apr 8, 2026J/K move · Enter open · S save

Generative Cross-Entropy: A Strictly Proper Loss for Data-Efficient Classification

Qipeng Zhan, Zhuoping Zhou, Li Shen

University of Pennsylvania

Data-Efficient Learning Proper Scoring Rules

Open paper AskPDF arXiv

Cross-entropy (CE) is the default training loss for supervised classification, but its sample efficiency is limited when labels are scarce. Existing remedies primarily act on the data side, via augmentation, synthesis, or transfer from pretrained models; the training objective itself is rarely revisited. We revisit it here. Drawing on the classical observation that generative classifiers reach their asymptotic error with fewer samples than discriminative ones, we propose Generative Cross-Entropy (GenCE), a drop-in replacement for CE that introduces a generative learning principle into a standard discriminative network without altering the architecture or fitting a separate density model. GenCE follows from a Bayesian rewrite of the class-conditional likelihood and, in the mini-batch approximation, reduces to normalizing each sample's softmax score against the model's predictions on the batch, coupling the training signal across examples sharing a class. We extend the proper-scoring-rule framework to such non-local losses and prove that GenCE is strictly proper under a mild completeness condition: its population risk is uniquely minimized at the true posterior. Across three datasets, on two architectures and in both balanced small-data and class-imbalanced regimes, GenCE outperforms CE and other widely used losses, while also producing better-calibrated probabilities and stronger out-of-distribution detection.

Abstract

Robust Loss Functions

Explore similar work

Generative Cross-Entropy: A Strictly Proper Loss for Data-Efficient Classification

Symmetrization of Loss Functions for Robust Training of Neural Networks in the Presence of Noisy Labels

A Layer Separation Optimization Framework for Cross-Entropy Training in Deep Learning