Generative Cross-Entropy: A Strictly Proper Loss for Data-Efficient Classification

Qipeng Zhan; Zhuoping Zhou; Li Shen

cs.LGApr 8, 2026

Generative Cross-Entropy: A Strictly Proper Loss for Data-Efficient Classification

Authors: Qipeng Zhan, Zhuoping Zhou, Li Shen

Organizations: University of Pennsylvania

Abstract

Cross-entropy (CE) is the default training loss for supervised classification, but its sample efficiency is limited when labels are scarce. Existing remedies primarily act on the data side, via augmentation, synthesis, or transfer from pretrained models; the training objective itself is rarely revisited. We revisit it here. Drawing on the classical observation that generative classifiers reach their asymptotic error with fewer samples than discriminative ones, we propose Generative Cross-Entropy (GenCE), a drop-in replacement for CE that introduces a generative learning principle into a standard discriminative network without altering the architecture or fitting a separate density model. GenCE follows from a Bayesian rewrite of the class-conditional likelihood and, in the mini-batch approximation, reduces to normalizing each sample's softmax score against the model's predictions on the batch, coupling the training signal across examples sharing a class. We extend the proper-scoring-rule framework to such non-local losses and prove that GenCE is strictly proper under a mild completeness condition: its population risk is uniquely minimized at the true posterior. Across three datasets, on two architectures and in both balanced small-data and class-imbalanced regimes, GenCE outperforms CE and other widely used losses, while also producing better-calibrated probabilities and stronger out-of-distribution detection.

Figures & tables

Dataset	$N$	Model	GenCE	CE	GCE	Focal Loss	MAE	Brier Loss
CIFAR-10	5000	ResNet	$\mathbf{85.20_{\pm 0.35}}$	$84.05_{\pm 0.53}$	$82.73_{\pm 0.35}$	$83.96_{\pm 0.54}$	$83.66_{\pm 0.17}$	$83.82_{\pm 0.20}$
	5000	VGG	$\mathbf{83.54_{\pm 0.26}}$	$83.00_{\pm 0.25}$	$82.48_{\pm 0.35}$	$82.38_{\pm 0.39}$	$82.70_{\pm 0.57}$	$82.71_{\pm 0.32}$
	2000	ResNet	$\mathbf{76.61_{\pm 0.68}}$	$72.51_{\pm 0.94}$	$71.45_{\pm 1.90}$	$74.13_{\pm 0.95}$	$74.17_{\pm 1.01}$	$72.94_{\pm 1.02}$
	2000	VGG	$\mathbf{74.62_{\pm 0.65}}$	$70.38_{\pm 1.93}$	$69.20_{\pm 0.72}$	$69.74_{\pm 0.75}$	$71.14_{\pm 1.54}$	$71.06_{\pm 1.28}$
CIFAR-100	5000	ResNet	$\mathbf{51.74_{\pm 0.29}}$	$50.35_{\pm 0.16}$	$48.10_{\pm 0.81}$	$50.07_{\pm 0.35}$	$49.97_{\pm 0.21}$	$49.30_{\pm 0.44}$
	5000	VGG	$\mathbf{41.30_{\pm 0.81}}$	$35.64_{\pm 2.20}$	$36.33_{\pm 1.26}$	$37.29_{\pm 1.51}$	$37.48_{\pm 1.71}$	$37.73_{\pm 0.76}$

Table 1: Classification accuracy (%) on balanced small-data subsets across datasets and architectures. Each entry reports mean

{}_{\pm\text{std}}

over 5 random seeds. The best result is highlighted in bold .

Dataset	$\rho$	GenCE	CE	GCE	Focal Loss	MAE	Brier Loss
CIFAR-10	10	$\mathbf{87.00_{\pm 0.23}}$	$86.31_{\pm 0.23}$	$86.67_{\pm 0.19}$	$86.44_{\pm 0.32}$	$86.46_{\pm 0.34}$	$86.37_{\pm 0.30}$
CIFAR-10	100	$\mathbf{70.84_{\pm 0.55}}$	$69.12_{\pm 0.52}$	$68.38_{\pm 0.11}$	$68.35_{\pm 0.56}$	$68.70_{\pm 0.20}$	$68.34_{\pm 0.41}$
CIFAR-100	10	$\mathbf{59.13_{\pm 0.28}}$	$57.69_{\pm 0.30}$	$56.65_{\pm 0.77}$	$57.64_{\pm 0.25}$	$58.10_{\pm 0.42}$	$57.96_{\pm 0.53}$
CIFAR-100	100	$\mathbf{40.31_{\pm 0.36}}$	$38.42_{\pm 0.05}$	$37.36_{\pm 0.51}$	$37.94_{\pm 0.20}$	$37.98_{\pm 0.18}$	$38.42_{\pm 0.52}$

Table 2: Classification accuracy (%) on class-imbalanced CIFAR-10 and CIFAR-100 across different methods. All experiments use ResNet-50. The best result in each row is highlighted in bold .

Dataset	$N$	GenCE	CE	GCE	Focal Loss	MAE	Brier Loss
CIFAR-10	5000	$6.82_{\pm 0.55}$	$7.82_{\pm 0.67}$	$9.61_{\pm 0.43}$	$\mathbf{4.48_{\pm 0.33}}$	$9.17_{\pm 0.19}$	$7.65_{\pm 0.59}$
CIFAR-10	2000	$\mathbf{10.64_{\pm 0.39}}$	$14.82_{\pm 0.74}$	$17.77_{\pm 1.24}$	$11.05_{\pm 0.60}$	$16.00_{\pm 0.55}$	$14.38_{\pm 1.28}$
CIFAR-100	5000	$\mathbf{10.25_{\pm 0.69}}$	$15.14_{\pm 0.44}$	$19.72_{\pm 0.67}$	$10.98_{\pm 0.66}$	$17.65_{\pm 0.43}$	$14.26_{\pm 0.77}$
CIFAR-100	2000	$\mathbf{22.32_{\pm 0.80}}$	$28.34_{\pm 0.74}$	$35.31_{\pm 1.16}$	$25.88_{\pm 0.29}$	$31.18_{\pm 1.01}$	$27.58_{\pm 0.41}$
Mini-ImageNet	5000	$\mathbf{13.24_{\pm 0.92}}$	$23.00_{\pm 0.71}$	$28.88_{\pm 0.26}$	$18.23_{\pm 1.02}$	$25.04_{\pm 0.61}$	$22.07_{\pm 0.13}$
Mini-ImageNet	2000	$\mathbf{23.82_{\pm 0.17}}$	$32.94_{\pm 0.93}$	$35.83_{\pm 0.73}$	$30.39_{\pm 0.48}$	$34.49_{\pm 0.87}$	$32.55_{\pm 0.59}$

Table 3: Expected Calibration Error (ECE, %) across datasets and training-set sizes (ResNet). Lower is better. The best result in each row is in bold .

Figure 1: Reliability diagrams computed on CIFAR-10 with 15 bins, the dashed line (

y=x

) marks perfect calibration. Bars below (above) the diagonal indicate over- (under-) confidence; the gap between each bar and the diagonal contributes to ECE.

Dataset	Metric	GenCE	CE	GCE	Focal Loss	MAE	Brier Loss
Near-OOD	$\text{AUC}_{\text{entropy}}$	$\mathbf{87.22_{\pm 0.11}}$	$86.24_{\pm 0.47}$	$86.60_{\pm 0.62}$	$86.36_{\pm 0.62}$	$86.18_{\pm 0.20}$	$86.79_{\pm 0.17}$
Near-OOD	$\text{AUC}_{\text{confidence}}$	$\mathbf{86.98_{\pm 0.09}}$	$86.06_{\pm 0.46}$	$86.40_{\pm 0.60}$	$86.14_{\pm 0.58}$	$86.03_{\pm 0.20}$	$86.56_{\pm 0.17}$
Far-OOD	$\text{AUC}_{\text{entropy}}$	$\mathbf{94.87_{\pm 1.33}}$	$88.40_{\pm 3.43}$	$90.41_{\pm 2.09}$	$87.57_{\pm 3.20}$	$91.61_{\pm 1.00}$	$93.84_{\pm 0.50}$
Far-OOD	$\text{AUC}_{\text{confidence}}$	$\mathbf{94.33_{\pm 1.32}}$	$88.16_{\pm 3.19}$	$90.00_{\pm 2.04}$	$87.31_{\pm 3.95}$	$91.34_{\pm 0.89}$	$93.33_{\pm 0.48}$

Table 4: AUROC (%) of models trained on CIFAR-10 as in-distribution data and evaluated on Near-OOD (CIFAR-100) and Far-OOD (SVHN) regimes. Each entry reports mean

{}_{\pm\text{std}}

over 5 random seeds. Higher is better. The best value in each row is in bold .

Figure 2: Accuracy vs. batch size

B

for all losses on

N{=}2000

CIFAR-10.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Dataset	GenCE	GenCE w/ Label Smoothing	CE	CE w/ Label Smoothing	SupCon
CIFAR-10	$\mathbf{76.61_{\pm 0.68}}$	$76.48_{\pm 0.50}$	$72.51_{\pm 0.94}$	$73.14_{\pm 0.15}$	$69.48_{\pm 0.43}$
CIFAR-100	$32.06_{\pm 0.47}$	$\mathbf{32.60_{\pm 0.50}}$	$31.15_{\pm 1.11}$	$30.80_{\pm 1.04}$	$23.44_{\pm 0.89}$
Mini-ImageNet	$27.75_{\pm 0.44}$	$\mathbf{27.98_{\pm 0.51}}$	$24.91_{\pm 0.38}$	$26.97_{\pm 0.71}$	$20.50_{\pm 0.43}$

Appendix

Table 5: Classification accuracy (%) on balanced small-data subsets (

N{=}2000

, ResNet-50) across datasets. Each entry reports mean

{}_{\pm\text{std}}

over 5 random seeds. Note that SupCon does not support soft targets and is therefore not evaluated under the Label Smoothing setting.

Data-Efficient Learning

Data-efficient learning develops methods that achieve accurate, robust models with fewer labeled examples, training samples, or interactions.

See all 236 papers

Minimax Generalized Cross-Entropy
Mar 20, 2026Kartheek Bondugula, Santiago Mazuelas, Aritz Pérez +1Robust Loss Functions Min-Max Optimization
Basque Center for Applied Mathematics, Bilbao, Spain · Ikerbasque, Basque Foundation for Science · Johns Hopkins University, Baltimore, USA
V-ECE: Estimating General Expected Calibration Errors
Feb 27, 2026Eugène Berta, Sacha Braun, Francis Bach +2Probability Calibration Uncertainty Calibration
INRIA · Ecole Normale Sup´erieure, PSL Research University · UC Berkeley
Structured Proper Loss Geometries for Multiclass Classification: Theory and Controlled Empirical Evaluation
Jun 28, 2026Soumyadip SarkarProper Scoring Rules Multiclass Classification
Independent Researcher

Mar 20, 2026J/K move · Enter open · S save

Minimax Generalized Cross-Entropy

Kartheek Bondugula, Santiago Mazuelas, Aritz Pérez, Anqi Liu

Basque Center for Applied Mathematics, Bilbao, Spain · Ikerbasque, Basque Foundation for Science · Johns Hopkins University, Baltimore, USA

Robust Loss Functions Min-Max Optimization

Open paper AskPDF arXiv

Loss functions play a central role in supervised classification. Cross-entropy (CE) is widely used, whereas the mean absolute error (MAE) loss can offer robustness but is difficult to optimize. Interpolating between the CE and MAE losses, generalized cross-entropy (GCE) has recently been introduced to provide a trade-off between optimization difficulty and robustness. Existing formulations of GCE result in a non-convex optimization over classification margins that is prone to underfitting, leading to poor performances with complex datasets. In this paper, we propose a minimax formulation of generalized cross-entropy (MGCE) that results in a convex optimization over classification margins. Moreover, we show that MGCEs can provide an upper bound on the classification error. The proposed bilevel convex optimization can be efficiently implemented using stochastic gradient computed via implicit differentiation. Using benchmark datasets, we show that MGCE achieves strong accuracy, faster convergence, and better calibration, especially in the presence of label noise.

Abstract

Data-Efficient Learning

Explore similar work

Minimax Generalized Cross-Entropy

V-ECE: Estimating General Expected Calibration Errors

Structured Proper Loss Geometries for Multiclass Classification: Theory and Controlled Empirical Evaluation