Generative Cross-Entropy: A Strictly Proper Loss for Data-Efficient Classification

Qipeng Zhan; Zhuoping Zhou; Li Shen

cs.LGApr 8, 2026

Generative Cross-Entropy: A Strictly Proper Loss for Data-Efficient Classification

Authors: Qipeng Zhan, Zhuoping Zhou, Li Shen

Organizations: University of Pennsylvania

Abstract

Cross-entropy (CE) is the default training loss for supervised classification, but its sample efficiency is limited when labels are scarce. Existing remedies primarily act on the data side, via augmentation, synthesis, or transfer from pretrained models; the training objective itself is rarely revisited. We revisit it here. Drawing on the classical observation that generative classifiers reach their asymptotic error with fewer samples than discriminative ones, we propose Generative Cross-Entropy (GenCE), a drop-in replacement for CE that introduces a generative learning principle into a standard discriminative network without altering the architecture or fitting a separate density model. GenCE follows from a Bayesian rewrite of the class-conditional likelihood and, in the mini-batch approximation, reduces to normalizing each sample's softmax score against the model's predictions on the batch, coupling the training signal across examples sharing a class. We extend the proper-scoring-rule framework to such non-local losses and prove that GenCE is strictly proper under a mild completeness condition: its population risk is uniquely minimized at the true posterior. Across three datasets, on two architectures and in both balanced small-data and class-imbalanced regimes, GenCE outperforms CE and other widely used losses, while also producing better-calibrated probabilities and stronger out-of-distribution detection.

Figures & tables

Dataset	$N$	Model	GenCE	CE	GCE	Focal Loss	MAE	Brier Loss
CIFAR-10	5000	ResNet	$\mathbf{85.20_{\pm 0.35}}$	$84.05_{\pm 0.53}$	$82.73_{\pm 0.35}$	$83.96_{\pm 0.54}$	$83.66_{\pm 0.17}$	$83.82_{\pm 0.20}$
	5000	VGG	$\mathbf{83.54_{\pm 0.26}}$	$83.00_{\pm 0.25}$	$82.48_{\pm 0.35}$	$82.38_{\pm 0.39}$	$82.70_{\pm 0.57}$	$82.71_{\pm 0.32}$
	2000	ResNet	$\mathbf{76.61_{\pm 0.68}}$	$72.51_{\pm 0.94}$	$71.45_{\pm 1.90}$	$74.13_{\pm 0.95}$	$74.17_{\pm 1.01}$	$72.94_{\pm 1.02}$
	2000	VGG	$\mathbf{74.62_{\pm 0.65}}$	$70.38_{\pm 1.93}$	$69.20_{\pm 0.72}$	$69.74_{\pm 0.75}$	$71.14_{\pm 1.54}$	$71.06_{\pm 1.28}$
CIFAR-100	5000	ResNet	$\mathbf{51.74_{\pm 0.29}}$	$50.35_{\pm 0.16}$	$48.10_{\pm 0.81}$	$50.07_{\pm 0.35}$	$49.97_{\pm 0.21}$	$49.30_{\pm 0.44}$
	5000	VGG	$\mathbf{41.30_{\pm 0.81}}$	$35.64_{\pm 2.20}$	$36.33_{\pm 1.26}$	$37.29_{\pm 1.51}$	$37.48_{\pm 1.71}$	$37.73_{\pm 0.76}$

Table 1: Classification accuracy (%) on balanced small-data subsets across datasets and architectures. Each entry reports mean

{}_{\pm\text{std}}

over 5 random seeds. The best result is highlighted in bold .

Dataset	$\rho$	GenCE	CE	GCE	Focal Loss	MAE	Brier Loss
CIFAR-10	10	$\mathbf{87.00_{\pm 0.23}}$	$86.31_{\pm 0.23}$	$86.67_{\pm 0.19}$	$86.44_{\pm 0.32}$	$86.46_{\pm 0.34}$	$86.37_{\pm 0.30}$
CIFAR-10	100	$\mathbf{70.84_{\pm 0.55}}$	$69.12_{\pm 0.52}$	$68.38_{\pm 0.11}$	$68.35_{\pm 0.56}$	$68.70_{\pm 0.20}$	$68.34_{\pm 0.41}$
CIFAR-100	10	$\mathbf{59.13_{\pm 0.28}}$	$57.69_{\pm 0.30}$	$56.65_{\pm 0.77}$	$57.64_{\pm 0.25}$	$58.10_{\pm 0.42}$	$57.96_{\pm 0.53}$
CIFAR-100	100	$\mathbf{40.31_{\pm 0.36}}$	$38.42_{\pm 0.05}$	$37.36_{\pm 0.51}$	$37.94_{\pm 0.20}$	$37.98_{\pm 0.18}$	$38.42_{\pm 0.52}$

Table 2: Classification accuracy (%) on class-imbalanced CIFAR-10 and CIFAR-100 across different methods. All experiments use ResNet-50. The best result in each row is highlighted in bold .

Dataset	$N$	GenCE	CE	GCE	Focal Loss	MAE	Brier Loss
CIFAR-10	5000	$6.82_{\pm 0.55}$	$7.82_{\pm 0.67}$	$9.61_{\pm 0.43}$	$\mathbf{4.48_{\pm 0.33}}$	$9.17_{\pm 0.19}$	$7.65_{\pm 0.59}$
CIFAR-10	2000	$\mathbf{10.64_{\pm 0.39}}$	$14.82_{\pm 0.74}$	$17.77_{\pm 1.24}$	$11.05_{\pm 0.60}$	$16.00_{\pm 0.55}$	$14.38_{\pm 1.28}$
CIFAR-100	5000	$\mathbf{10.25_{\pm 0.69}}$	$15.14_{\pm 0.44}$	$19.72_{\pm 0.67}$	$10.98_{\pm 0.66}$	$17.65_{\pm 0.43}$	$14.26_{\pm 0.77}$
CIFAR-100	2000	$\mathbf{22.32_{\pm 0.80}}$	$28.34_{\pm 0.74}$	$35.31_{\pm 1.16}$	$25.88_{\pm 0.29}$	$31.18_{\pm 1.01}$	$27.58_{\pm 0.41}$
Mini-ImageNet	5000	$\mathbf{13.24_{\pm 0.92}}$	$23.00_{\pm 0.71}$	$28.88_{\pm 0.26}$	$18.23_{\pm 1.02}$	$25.04_{\pm 0.61}$	$22.07_{\pm 0.13}$
Mini-ImageNet	2000	$\mathbf{23.82_{\pm 0.17}}$	$32.94_{\pm 0.93}$	$35.83_{\pm 0.73}$	$30.39_{\pm 0.48}$	$34.49_{\pm 0.87}$	$32.55_{\pm 0.59}$

Table 3: Expected Calibration Error (ECE, %) across datasets and training-set sizes (ResNet). Lower is better. The best result in each row is in bold .

Figure 1: Reliability diagrams computed on CIFAR-10 with 15 bins, the dashed line (

y=x

) marks perfect calibration. Bars below (above) the diagonal indicate over- (under-) confidence; the gap between each bar and the diagonal contributes to ECE.

Dataset	Metric	GenCE	CE	GCE	Focal Loss	MAE	Brier Loss
Near-OOD	$\text{AUC}_{\text{entropy}}$	$\mathbf{87.22_{\pm 0.11}}$	$86.24_{\pm 0.47}$	$86.60_{\pm 0.62}$	$86.36_{\pm 0.62}$	$86.18_{\pm 0.20}$	$86.79_{\pm 0.17}$
Near-OOD	$\text{AUC}_{\text{confidence}}$	$\mathbf{86.98_{\pm 0.09}}$	$86.06_{\pm 0.46}$	$86.40_{\pm 0.60}$	$86.14_{\pm 0.58}$	$86.03_{\pm 0.20}$	$86.56_{\pm 0.17}$
Far-OOD	$\text{AUC}_{\text{entropy}}$	$\mathbf{94.87_{\pm 1.33}}$	$88.40_{\pm 3.43}$	$90.41_{\pm 2.09}$	$87.57_{\pm 3.20}$	$91.61_{\pm 1.00}$	$93.84_{\pm 0.50}$
Far-OOD	$\text{AUC}_{\text{confidence}}$	$\mathbf{94.33_{\pm 1.32}}$	$88.16_{\pm 3.19}$	$90.00_{\pm 2.04}$	$87.31_{\pm 3.95}$	$91.34_{\pm 0.89}$	$93.33_{\pm 0.48}$

Table 4: AUROC (%) of models trained on CIFAR-10 as in-distribution data and evaluated on Near-OOD (CIFAR-100) and Far-OOD (SVHN) regimes. Each entry reports mean

{}_{\pm\text{std}}

over 5 random seeds. Higher is better. The best value in each row is in bold .

Figure 2: Accuracy vs. batch size

B

for all losses on

N{=}2000

CIFAR-10.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Dataset	GenCE	GenCE w/ Label Smoothing	CE	CE w/ Label Smoothing	SupCon
CIFAR-10	$\mathbf{76.61_{\pm 0.68}}$	$76.48_{\pm 0.50}$	$72.51_{\pm 0.94}$	$73.14_{\pm 0.15}$	$69.48_{\pm 0.43}$
CIFAR-100	$32.06_{\pm 0.47}$	$\mathbf{32.60_{\pm 0.50}}$	$31.15_{\pm 1.11}$	$30.80_{\pm 1.04}$	$23.44_{\pm 0.89}$
Mini-ImageNet	$27.75_{\pm 0.44}$	$\mathbf{27.98_{\pm 0.51}}$	$24.91_{\pm 0.38}$	$26.97_{\pm 0.71}$	$20.50_{\pm 0.43}$

Appendix

Table 5: Classification accuracy (%) on balanced small-data subsets (

N{=}2000

, ResNet-50) across datasets. Each entry reports mean

{}_{\pm\text{std}}

over 5 random seeds. Note that SupCon does not support soft targets and is therefore not evaluated under the Label Smoothing setting.

Data-Efficient Learning

Data-efficient learning develops methods that achieve accurate, robust models with fewer labeled examples, training samples, or interactions.

See all 236 papers

Mar 20, 2026

Minimax Generalized Cross-Entropy

Loss functions play a central role in supervised classification. Cross-entropy (CE) is widely used, whereas the mean absolute error (MAE) loss can offer robustness but is difficult to optimize. Interpolating between the CE and MAE losses, generalized cross-entropy (GCE) has recently been introduced to provide a trade-off between optimization difficulty and robustness. Existing formulations of GCE result in a non-convex optimization over classification margins that is prone to underfitting, leading to poor performances with complex datasets. In this paper, we propose a minimax formulation of generalized cross-entropy (MGCE) that results in a convex optimization over classification margins. Moreover, we show that MGCEs can provide an upper bound on the classification error. The proposed bilevel convex optimization can be efficiently implemented using stochastic gradient computed via implicit differentiation. Using benchmark datasets, we show that MGCE achieves strong accuracy, faster convergence, and better calibration, especially in the presence of label noise.

Kartheek Bondugula, Santiago Mazuelas, Aritz Pérez +1

Basque Center for Applied Mathematics, Bilbao, Spain · Ikerbasque, Basque Foundation for Science · Johns Hopkins University, Baltimore, USA

Robust Loss Functions Min-Max Optimization

Ask PDF

Feb 27, 2026

V-ECE: Estimating General Expected Calibration Errors

In probabilistic classification, calibration error (CE) measures the average divergence of predicted probabilities

f(X)

from

\mathbb{P}(Y|f(X))

, the true class distribution for that predicted probability. While being a useful diagnostic tool, it is hard to estimate: popular binning-based estimators are often inconsistent and scale poorly beyond two classes. Recent work rewrites the CE as the excess risk of a model compared to the best recalibration of its own predictions, measured with a proper loss. However, this only works for Bregman-divergence-based calibration errors like the squared error, excluding the more popular

L_1

-distance-based CE. We show that using prediction-dependent proper scores can alleviate this restriction, allowing us to estimate CEs with general convex divergences, including

L_p

distances with closed-form losses in the binary and multiclass settings. To estimate the excess risk, we introduce a more accurate recalibrator that fits a residual to temperature scaling with gradient boosting. The resulting variational estimator, V-ECE, needs no bins or clusters and lower-bounds the true calibration error in expectation. On a benchmark of semi-synthetic tasks built from real classifiers, with known true CE, V-ECE is among the most accurate binary estimators for every calibration error and significantly outperforms all multiclass estimators. Our results are accompanied by additional theory on

L_p

CE, estimator bias, and over- or under-confidence estimation.

Eugène Berta, Sacha Braun, Francis Bach +2

INRIA · Ecole Normale Sup´erieure, PSL Research University · UC Berkeley

Probability Calibration Uncertainty Calibration Post-Hoc Calibration

Ask PDF

Jun 28, 2026

Structured Proper Loss Geometries for Multiclass Classification: Theory and Controlled Empirical Evaluation

Strictly proper scoring rules identify the true conditional class distribution at population level, but their curvature can alter optimization and finite-sample behavior. We study three multiclass objectives: a class-aware quadratic Bregman score (CAPM), a strongly convex generator with constrained log-cosh ridges (HPG), and an HPG objective with an annealed probability-margin penalty (APMS). CAPM is treated as a structured instance of established quadratic scoring-rule theory. We derive conditional-regret, curvature, range, and logit-gradient bounds for CAPM and HPG, and prove exact penalty-range and conditional-target displacement bounds for APMS. Controlled five-seed experiments use Digits, Wisconsin breast cancer, and synthetic confusion and long-tail problems under clean labels, symmetric and pair-flip corruption, class imbalance, calibration evaluation, input corruption, and first-order adversarial perturbations. The candidates are close to cross-entropy on clean data and show descriptive gains in some noisy-label cells, but the five-seed comparisons are interpreted descriptively rather than as significance evidence. The selected noisy-label baselines perform better on Digits with 40% symmetric label noise, and explicit prior-adjustment methods perform better in the 30:1 synthetic long-tail experiment. Ablations do not show a consistent benefit from the candidate-specific graph, ridge, or margin components. The mathematical analysis establishes the stated properties, and the experiments delimit the empirical evidence; together they do not support a claim of general superiority.

Soumyadip Sarkar

Independent Researcher

Proper Scoring Rules Multiclass Classification Classification Statistical Learning Theory

Ask PDF

Abstract

Data-Efficient Learning

Explore similar work

Minimax Generalized Cross-Entropy

V-ECE: Estimating General Expected Calibration Errors

Structured Proper Loss Geometries for Multiclass Classification: Theory and Controlled Empirical Evaluation