Is BatchEnsemble a Single Model? On Calibration and Diversity of Efficient Ensembles

Anton Zamyatin; Patrick Indri; Sagar Malhotra; Thomas Gärtner

cs.LGJan 23, 2026

Is BatchEnsemble a Single Model? On Calibration and Diversity of Efficient Ensembles

Authors: Anton Zamyatin, Patrick Indri, Sagar Malhotra, Thomas Gärtner

Organizations: TU Wien

Abstract

In resource-constrained and low-latency settings, uncertainty estimates must be efficiently obtained. Deep Ensembles provide robust epistemic uncertainty (EU) but require training multiple full-size models. BatchEnsemble aims to deliver ensemble-like EU at far lower parameter and memory cost by applying learned rank-1 perturbations to a shared base network. We show that BatchEnsemble not only underperforms Deep Ensembles but closely tracks a single model baseline in terms of accuracy, calibration and out-of-distribution (OOD) detection on CIFAR10/10C/SVHN. A controlled study on MNIST finds members are near-identical in function and parameter space, indicating limited capacity to realize distinct predictive modes. Thus, BatchEnsemble behaves more like a single model than a true ensemble.

Figures & tables

Method	Acc↑	NLL↓	ECE↓	H	JSD
Single	0.941 ±0.004	0.237 ±0.003	0.034 ±0.003	0.073 ±0.005	N/A
MC Dropout	0.936 ±0.001	0.210 ±0.007	0.021 ±0.002	0.120 ±0.005	0.022 ±0.001
Deep Ensemble	0.952 ±0.001	0.152 ±0.003	0.007 ±0.002	0.136 ±0.001	0.037 ±0.001
BatchEnsemble	0.938 ±0.001	0.230 ±0.006	0.032 ±0.001	0.088 ±0.003	0.002 ±0.000

Table 1: In-Distribution Benchmark (CIFAR-10)

Method	Acc↑	NLL↓	ECE↓	H	JSD
Single	0.558 ±0.008	2.545 ±0.142	0.323 ±0.014	0.327 ±0.026	N/A
MC Dropout	0.578 ±0.002	2.054 ±0.084	0.251 ±0.009	0.449 ±0.031	0.088 ±0.005
Deep Ensemble	0.575 ±0.006	1.682 ±0.045	0.206 ±0.009	0.593 ±0.018	0.176 ±0.005
BatchEnsemble	0.547 ±0.015	2.357 ±0.164	0.308 ±0.012	0.400 ±0.011	0.008 ±0.002

Table 2: Distribution Shift Benchmark (CIFAR-10C, shift intensity 5)

Method	AUPR↑	AUROC↑	FPR95↓	H	JSD
Single	0.938 ±0.017	0.893 ±0.024	0.318 ±0.071	0.480 ±0.096	N/A
MC Dropout	0.932 ±0.008	0.884 ±0.009	0.324 ±0.019	0.608 ±0.056	0.122 ±0.016
Deep Ensemble	0.953 ±0.006	0.924 ±0.011	0.199 ±0.024	0.829 ±0.067	0.277 ±0.032
BatchEnsemble	0.932 ±0.015	0.888 ±0.020	0.297 ±0.061	0.495 ±0.100	0.010 ±0.006

Table 3: OOD Benchmark (SVHN)

Panel of Figure 1 — Figure 1: BatchEnsemble lacks functional and parametric diversity (MNIST MLP). (a) Pairwise prediction disagreement across ID, distribution-shifted, and OOD test sets (higher is more diverse). (b) Cosine similarity of members’ weights (higher is more similar). Deep Ensembles are diverse in both spaces; BatchEnsemble members cluster tightly.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Single	MC Dropout	Deep Ensemble	BatchEnsemble
#Params	11.17 Mio.	11.17 Mio.	44.71 Mio.	11.22 Mio.
Training Time	15min	18min	1h15min	1h1min

Appendix

Table 4: Comparison of Computational Efficiency

Method	Acc↑	NLL↓	ECE↓	H	JSD
Standard	0.941 ±0.004	0.237 ±0.003	0.034 ±0.003	0.073 ±0.005	N/A
MC Dropout	0.936 ±0.001	0.210 ±0.007	0.021 ±0.002	0.120 ±0.005	0.022 ±0.001
Deep Ensemble	0.952 ±0.001	0.152 ±0.003	0.007 ±0.002	0.136 ±0.001	0.037 ±0.001
BatchEnsemble (C)	0.938 ±0.001	0.230 ±0.006	0.032 ±0.001	0.088 ±0.003	0.002 ±0.000
BatchEnsemble (K)	0.931 ±0.002	0.231 ±0.010	0.025 ±0.002	0.124 ±0.003	0.010 ±0.001

Appendix

Table 5: In-Distribution Benchmark (CIFAR-10)

Name	Acc↑	NLL↓	ECE↓	H	JSD
Standard	0.558 ±0.008	2.545 ±0.142	0.323 ±0.014	0.327 ±0.026	N/A
MC Dropout	0.578 ±0.002	2.054 ±0.084	0.251 ±0.009	0.449 ±0.031	0.088 ±0.005
Deep Ensemble	0.575 ±0.006	1.682 ±0.045	0.206 ±0.009	0.593 ±0.018	0.176 ±0.005
BatchEnsemble (C)	0.547 ±0.015	2.357 ±0.164	0.308 ±0.012	0.400 ±0.011	0.008 ±0.002
BatchEnsemble (K)	0.521 ±0.008	2.266 ±0.110	0.275 ±0.012	0.544 ±0.023	0.039 ±0.004

Appendix

Table 6: Distribution Shift Benchmark (CIFAR-10C, shift intensity 5)

Name	AUPR↑	AUROC↑	FPR95↓	H	JSD
Standard	0.938 ±0.017	0.893 ±0.024	0.318 ±0.071	0.480 ±0.096	N/A
MC Dropout	0.932 ±0.008	0.884 ±0.009	0.324 ±0.019	0.608 ±0.056	0.122 ±0.016
Deep Ensemble	0.953 ±0.006	0.924 ±0.011	0.199 ±0.024	0.829 ±0.067	0.277 ±0.032
BatchEnsemble (C)	0.932 ±0.015	0.888 ±0.020	0.297 ±0.061	0.495 ±0.100	0.010 ±0.006
BatchEnsemble (K)	0.944 ±0.008	0.911 ±0.013	0.231 ±0.025	0.692 ±0.064	0.043 ±0.008

Appendix

Table 7: OOD Benchmark (SVHN)

Ensemble Learning

Ensemble learning combines multiple predictors to improve accuracy, robustness, calibration, or uncertainty estimates.

See all 272 papers

Controllable Diversity in Normalization-Based Implicit Ensembles via Softmax-Temperature Modulation
Jul 26, 2026Mihai Suteu, Ovidiu SerbanEnsemble Learning Ensemble Diversity
Data Science Institute, Imperial College London
Optimizable Model Averaging for Ensembling Heterogeneous Model Outputs
May 28, 2025Mijung ParkEnsemble Learning Model Calibration
Department of Computer Science University of British Columbia, Vancouver
Covariance Last-Layer Ensembles: Function-Space Diversity for Efficient Uncertainty Quantification
Jul 26, 2026H. Martin Gillis, Isaac Xu, Gabriel Spadon +1Uncertainty Quantification Ensemble Learning
Dalhousie University Halifax, Nova Scotia, Canada

Jul 26, 2026J/K move · Enter open · S save

Controllable Diversity in Normalization-Based Implicit Ensembles via Softmax-Temperature Modulation

Mihai Suteu, Ovidiu Serban

Data Science Institute, Imperial College London

Ensemble Learning Ensemble Diversity Uncertainty Calibration Epistemic Uncertainty

Open paper AskPDF arXiv

Deep ensembles provide the most reliable uncertainty estimates in deep learning, but their cost grows linearly with the number of members. Implicit ensembles lower this cost by sharing a single backbone across members. Member diversity is a primary determinant of ensemble quality, yet no implicit ensemble can shape it during training; existing methods fix it at initialisation or build it into the architecture. We introduce

σ

N-Ens, a normalisation-based implicit ensemble that treats each member as a task in a multi-task architecture and modulates the shared backbone through sigmoid-bounded scalers. We also introduce a softmax-temperature regulariser, which shapes the equilibrium level of sharing between members and traces the accuracy-calibration frontier. Because only normalisation layers are replicated, the mechanism can wrap convolutional and transformer backbones alike, also allowing pretrained models to be adapted through a short fine-tune. We frame the epistemic uncertainty such an ensemble expresses as modulation uncertainty, and explain why its calibration holds under input corruption, and why its out-of-distribution detection is weaker. Our method is evaluated across ResNets and transformers on CIFAR-10/100, ImageNet and SST-2.

σ

N-Ens matches or outperforms deep ensembles at a fraction of their parameter cost, scales with ensemble size where partitioning methods collapse, and maintains calibration under distribution shift.

Abstract

Ensemble Learning

Explore similar work

Controllable Diversity in Normalization-Based Implicit Ensembles via Softmax-Temperature Modulation

Optimizable Model Averaging for Ensembling Heterogeneous Model Outputs

Covariance Last-Layer Ensembles: Function-Space Diversity for Efficient Uncertainty Quantification