Compact set-valued deep ensembling in multi-class classification

Kim-Dung Tran; Dang-Man Nguyen; Vu-Linh Nguyen; Xuan-Truong Hoang; Sébastien Destercke; Van-Nam Huynh

cs.LGOct 4, 2026

Compact set-valued deep ensembling in multi-class classification

Authors: Kim-Dung Tran, Dang-Man Nguyen, Vu-Linh Nguyen, Xuan-Truong Hoang, Sébastien Destercke, Van-Nam Huynh

Organizations: UMR CNRS 7253, Heudiasyc, Universit´e de Technologie de Compi`egne, Compi`egne, France. · School of Knowledge Science, Japan Advanced Institute of Science and Technology, Nomi, Japan.

Abstract

This paper tackles visible challenges in deep ensemble learning, where deep neural networks serve as ensemble members: training and storage burdens, and robustness of cautious (set-valued) predictions targeting multiple utilities, which may involve reward-sensitivity. To mitigate the training and storage burdens, we propose to employ compact ensembles, such as Bayesian Neural Networks and Convolutional Neural Networks with the Monte-Carlo dropout prediction option, to produce probabilistic predictions. For each query instance, these probabilistic predictions are then used to define a representative distribution optimizing some statistical distance. The representative distribution is then employed to define the Bayes-optimal prediction (BOP) of any utility. To address the potential unrobustness of singleton prediction making, we propose a family of set-utilities satisfying some desirable properties and whose set-valued BOPs can be found efficiently. Empirical evidence is then given to illustrate the potential (dis)advantages of the proposed ensemble learning framework.

Figures & tables

Panel of Figure 1 — Figure 1 : Illustration of the conventional ensemble learning framework

Panel of Figure 2 — Figure 2 : Illustration of compact ensembles.

#	Name	# of instances	image type	# of classes
1	CIFAR-10	60,000	$32\times 32$ color images	10
2	Fashion-MNIST	70,000	$28\times 28$ gray images	10
3	Plant Leaf Diseases	61,486	$256\times 256$ color images	39

Table 1 : Data sets.

Panel of Figure 3 — Figure 3 : BNNs with clean training data. Results with $U_{\alpha}$ on clean (noisy) test data are plotted with dashed (thick) lines. Classifiers are color-coded: clf_sE , clf_L1 , and clf_KL .

Panel of Figure 4 — Figure 4 : BNNs with noisy training data. Results with $U_{\alpha}$ on clean (noisy) test data are plotted with dashed (thick) lines. Classifiers are color-coded: clf_sE , clf_L1 , and clf_KL .

Panel of Figure 5 — Figure 5 : The probabilistic prediction $\boldsymbol{p}(\mathcal{Y}\,|\,\boldsymbol{x})$ and set-valued BOP $\hat{Y}_{U_{1.63}}$ ( 4 ) of a few cat images from the CIFAR-10 data set. The chosen value of $\alpha$ is the median reward level.

Panel of Figure 6 — Figure 6 : The probabilistic prediction $\boldsymbol{p}(\mathcal{Y}\,|\,\boldsymbol{x})$ and set-valued BOPs $\hat{Y}_{U^{0/1}_{2.05}}$ and $\hat{Y}_{U_{1.63}}$ ( 4 ) of a clean cat image from the CIFAR-10 data set. The ensembles are trained using clean training data $\mathbf{D}$ . The chosen values of $\alpha$ are the median reward levels.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol/Acronym	Meaning
$\mathcal{X}$ , $\boldsymbol{x}$	Instance space, instance
$\mathcal{Y}$ , $Y$ , $\hat{y},\hat{y}_{u}$	Output space, set prediction, BOP, BOP with utility $u$
$\hat{Y}_{U^{0/1}_{\alpha}}$	Set-valued BOP produced by utility-discounted accuracy function $U^{0/1}_{\alpha}$
$X^{p}$ , $Y^{k}$	Feature, class variable
$\mathcal{Y}^{k}$	Output space of $Y^{k}$
$\mathcal{Y}^{k}_{R}\mathrel{\vbox{\hbox{\footnotesize.}\hbox{\footnotesize.}}}=\mathcal{Y}^{k}\cup\{\mathcal{Y}^{k}\}$	Set of predictions with rejection

Appendix

Table 2 : Notation and acronyms

Panel of Figure 7 — Figure 7 : BNNs with clean training data. Results with $U_{\alpha}^{0/1}$ on clean (noisy) test data are plotted with dashed (thick) lines. Classifiers are color-coded: clf_sE , clf_L1 , and clf_KL .

Panel of Figure 8 — Figure 8 : BNNs with noisy training data. Results with $U_{\alpha}^{0/1}$ on clean (noisy) test data are plotted with dashed (thick) lines. Classifiers are color-coded: clf_sE , clf_L1 , and clf_KL .

Panel of Figure 9 — Figure 9 : CNNs with clean training data. Results with $U_{\alpha}$ on clean (noisy) test data are plotted with dashed (thick) lines. Classifiers are color-coded: clf_sE , clf_L1 , and clf_KL .

Panel of Figure 10 — Figure 10 : CNNs with noisy training data. Results with $U_{\alpha}$ on clean (noisy) test data are plotted with dashed (thick) lines. Classifiers are color-coded: clf_sE , clf_L1 , and clf_KL .

Panel of Figure 11 — Figure 11 : CNNs with clean training data. Results with $U_{\alpha}^{0/1}$ on clean (noisy) test data are plotted with dashed (thick) lines. Classifiers are color-coded: clf_sE , clf_L1 , and clf_KL .

Panel of Figure 12 — Figure 12 : CNNs with noisy training data. Results with $U_{\alpha}^{0/1}$ on clean (noisy) test data are plotted with dashed (thick) lines. Classifiers are color-coded: clf_sE , clf_L1 , and clf_KL .

		CIFAR-10		Fashion-MNIST		Plan Leaf Diseases
		BNNs	CNNs	BNNs	CNNS	BNN	CNNs
c-n	$\mathrm{ECE}_{\mathrm{cl}}$	7.99 $\pm$ 0.38	8.75 $\pm$ 0.86	2.82 $\pm$ 0.04	6.62 $\pm$ 0.06	2.45 $\pm$ 0.14	3.68 $\pm$ 0.13
	$\mathrm{MCE}_{\mathrm{cl}}$	71.11 $\pm$ 3.55	70.80 $\pm$ 5.51	59.30 $\pm$ 11.09	77.87 $\pm$ 1.42	97.02 $\pm$ 0.34	95.74 $\pm$ 0.62
	$\mathrm{ECE}_{\mathrm{co}}$	26.53 $\pm$ 5.06	31.46 $\pm$ 5.69	9.50 $\pm$ 0.51	5.33 $\pm$ 0.62	11.53 $\pm$ 2.69	33.20 $\pm$ 3.91
	$\mathrm{MCE}_{\mathrm{co}}$	43.17 $\pm$ 10.86	51.74 $\pm$ 8.57	50.80 $\pm$ 25.43	21.52 $\pm$ 6.76	18.59 $\pm$ 3.86	61.42 $\pm$ 2.99
c-c	$\mathrm{ECE}_{\mathrm{cl}}$	0.90 $\pm$ 0.04	1.29 $\pm$ 0.03	0.40 $\pm$ 0.02	3.48 $\pm$ 0.08	0.50 $\pm$ 0.01	1.52 $\pm$ 0.03

Appendix

Table 3 : Calibration errors of the representative distributions ( 8 ) produced by

d_{\mathrm{sE}}

( 9 ): The shorthand notations

\mathrm{ECE}_{\mathrm{cl}}

\mathrm{MCE}_{\mathrm{cl}}

\mathrm{ECE}_{\mathrm{co}}

, and

\mathrm{MCE}_{\mathrm{co}}

are used instead of

\mathrm{ECE}_{\mathrm{classwise}}

\mathrm{MCE}_{\mathrm{classwise}}

\mathrm{ECE}_{\mathrm{confidence}}

, and

\mathrm{MCE}_{\mathrm{confidence}}

. Results on (clean train data, clean test data), (clean train data, noisy test data), (noisy train data, clean test data), and (noisy train data, noisy test data) are marked as c-c, c-n, n-c, and n-n, respectively.

		CIFAR-10		Fashion-MNIST		Plan Leaf Diseases
		BNNs	CNNs	BNNs	CNNS	BNN	CNNs
c-n	$\mathrm{ECE}_{\mathrm{cl}}$	9.35 $\pm$ 0.64	11.57 $\pm$ 0.88	2.59 $\pm$ 0.04	5.79 $\pm$ 0.13	2.49 $\pm$ 0.16	4.23 $\pm$ 0.08
	$\mathrm{MCE}_{\mathrm{cl}}$	81.42 $\pm$ 4.37	88.76 $\pm$ 2.33	68.29 $\pm$ 3.99	81.41 $\pm$ 0.89	95.52 $\pm$ 0.29	98.22 $\pm$ 1.14
	$\mathrm{ECE}_{\mathrm{co}}$	38.59 $\pm$ 5.24	51.72 $\pm$ 5.25	8.03 $\pm$ 0.54	5.06 $\pm$ 0.87	37.21 $\pm$ 2.55	62.66 $\pm$ 3.21
	$\mathrm{MCE}_{\mathrm{co}}$	61.86 $\pm$ 14.26	68.27 $\pm$ 4.53	42.55 $\pm$ 29.52	24.81 $\pm$ 16.83	50.94 $\pm$ 2.12	82.51 $\pm$ 3.14
c-c	$\mathrm{ECE}_{\mathrm{cl}}$	0.78 $\pm$ 0.01	1.07 $\pm$ 0.07	0.46 $\pm$ 0.02	2.08 $\pm$ 0.09	0.07 $\pm$ 0.00	0.30 $\pm$ 0.00

Appendix

Table 4 : Calibration errors of the representative distributions ( 8 ) produced by

d_{\mathrm{L1}}

( 10 ): The shorthand notations

\mathrm{ECE}_{\mathrm{cl}}

\mathrm{MCE}_{\mathrm{cl}}

\mathrm{ECE}_{\mathrm{co}}

, and

\mathrm{MCE}_{\mathrm{co}}

are used instead of

\mathrm{ECE}_{\mathrm{classwise}}

\mathrm{MCE}_{\mathrm{classwise}}

\mathrm{ECE}_{\mathrm{confidence}}

, and

\mathrm{MCE}_{\mathrm{confidence}}

		CIFAR-10		Fashion-MNIST		Plan Leaf Diseases
		BNNs	CNNs	BNNs	CNNS	BNN	CNNs
c-n	$\mathrm{ECE}_{\mathrm{cl}}$	9.39 $\pm$ 0.62	11.67 $\pm$ 0.87	2.46 $\pm$ 0.05	5.97 $\pm$ 0.12	2.62 $\pm$ 0.17	4.25 $\pm$ 0.08
	$\mathrm{MCE}_{\mathrm{cl}}$	82.07 $\pm$ 5.07	90.11 $\pm$ 1.77	67.95 $\pm$ 4.12	82.45 $\pm$ 0.81	97.10 $\pm$ 2.05	99.27 $\pm$ 0.49
	$\mathrm{ECE}_{\mathrm{co}}$	39.58 $\pm$ 5.04	52.73 $\pm$ 5.14	7.35 $\pm$ 0.51	6.68 $\pm$ 0.69	41.77 $\pm$ 2.87	65.71 $\pm$ 3.71
	$\mathrm{MCE}_{\mathrm{co}}$	56.61 $\pm$ 6.75	68.20 $\pm$ 3.29	41.64 $\pm$ 30.06	17.77 $\pm$ 1.82	57.19 $\pm$ 1.94	83.65 $\pm$ 2.30
c-c	$\mathrm{ECE}_{\mathrm{cl}}$	0.88 $\pm$ 0.02	1.15 $\pm$ 0.09	0.36 $\pm$ 0.02	2.16 $\pm$ 0.09	0.06 $\pm$ 0.00	0.26 $\pm$ 0.01

Appendix

Table 5 : Calibration errors of the representative distributions ( 8 ) produced by

d_{\mathrm{KL}}

( 11 ): The shorthand notations

\mathrm{ECE}_{\mathrm{cl}}

\mathrm{MCE}_{\mathrm{cl}}

\mathrm{ECE}_{\mathrm{co}}

, and

\mathrm{MCE}_{\mathrm{co}}

are used instead of

\mathrm{ECE}_{\mathrm{classwise}}

\mathrm{MCE}_{\mathrm{classwise}}

\mathrm{ECE}_{\mathrm{confidence}}

, and

\mathrm{MCE}_{\mathrm{confidence}}

Explore similar work

Efficient Bayesian Deep Ensembles via Analytic Predictive Inference
Jul 7, 2026Sina Aghaee Dabaghan Fard, Marie Maros, Jaesung LeeBayesian Neural Networks Ensemble
Wm Michael Barnes ’64 Department of Industrial and Systems Engineering Texas A&M University College Station, TX, USA
Do Deep Ensembles Actually Capture Uncertainty in Graph Neural Networks?
May 21, 2026Pedro C. Vieira, Pedro Ribeiro, Viacheslav BorovitskiyGraph Neural Networks Ensemble
University of Edinburgh · DCC/FCUP, University of Porto
Choosing a parallel heterogeneous ensemble method for tabular classification
Jul 6, 2026Vassili Maillet, Gustavo, Angulo +1Ensemble Tabular Learning
Jesus · Mines Paris, PSL University, France

Jul 7, 2026, cs.LGJ/K move · Enter open · S save

Efficient Bayesian Deep Ensembles via Analytic Predictive Inference

Sina Aghaee Dabaghan Fard, Marie Maros, Jaesung Lee

Wm Michael Barnes ’64 Department of Industrial and Systems Engineering Texas A&M University College Station, TX, USA

Bayesian Neural Networks Ensemble Posterior

Open paper AskPDF arXiv

We introduce an efficient Bayesian deep ensemble method for predictive regression designed to enhance interpretability while maintaining competitive predictive performance and computational efficiency. Our method combines the statistical rigor of Bayesian inference with the scalability of deep ensembles, providing calibrated uncertainty estimates that enable its use not only for standalone prediction but also as a component within broader learning systems. To achieve these goals, our work relies on three key design components: (i) low-dimensional ensemble representation: predictions are expressed as a combination of a small number of trained neural predictors, enabling scalable inference whose cost depends on ensemble size rather than dataset size; (ii) closed-form Bayesian aggregation: ensemble predictions are combined using Bayesian linear regression, yielding interpretable posterior weights and calibrated uncertainty without approximate inference; and (iii) Independent ensemble training: multiple neural networks are trained separately, producing diverse predictive representations that improve robustness and uncertainty calibration. Empirical results on standard regression benchmarks demonstrate that the proposed approach achieves competitive predictive performance while maintaining reliable uncertainty estimates across settings.

Abstract

Explore similar work

Efficient Bayesian Deep Ensembles via Analytic Predictive Inference

Do Deep Ensembles Actually Capture Uncertainty in Graph Neural Networks?

Choosing a parallel heterogeneous ensemble method for tabular classification