Optimizable Model Averaging for Ensembling Heterogeneous Model Outputs

Mijung Park

cs.LGMay 28, 2025

Optimizable Model Averaging for Ensembling Heterogeneous Model Outputs

Authors: Mijung Park

Organizations: Department of Computer Science University of British Columbia, Vancouver

Abstract

Foundation model ensembles can improve accuracy and robustness, but full fine-tuning and parameter-space merging are often expensive or impossible when candidate models have different architectures. We propose Optimizable Model Averaging (OMA), a black-box approach that learns nonnegative model-combination weights, one scalar coefficient per model, to average the candidate models' predictive probabilities, leaving all model parameters frozen, by minimizing regularized predictive entropy. A central challenge is that minimizing predictive entropy can favour confidently incorrect models: increased confidence need not imply improved accuracy or calibration. We therefore examine regularization as a mechanism for controlling adaptation and empirically evaluate its effect on calibration, observing lower expected calibration error in selected configurations, alongside better performance-calibration trade-offs than model output averaging. As foundation models continue to advance, these approaches will enable the incorporation of future, potentially much better models to improve performance on challenging text and image classification tasks.

Figures & tables

Dataset	Method	Accuracy ( $\uparrow$ )	ECE ( $\downarrow$ )
SST-2	Output averaging	95.30	0.099
	OMA (best accuracy)	95.71 $\pm$ 0.62	0.027 $\pm$ 0.003
	OMA (best ECE)	94.95 $\pm$ 0.35	0.011 $\pm$ 0.007
CoLA	Output averaging	70.17	0.064
	OMA (best accuracy)	71.06 $\pm$ 0.40	0.058 $\pm$ 0.004
	OMA (best ECE)	69.63 $\pm$ 0.98	0.052 $\pm$ 0.005

Table 1: Accuracy 1 1 1 Following ( Wortsman et al., 2022a ) , we mean Accuracy as the average of classification accuracy and

F_{1}

score for MRPC, classification accuracy for RTE and SST-2, and Matthews correlation for CoLA ( Matthews, 1975 ) . (%) and ECE of output averaging and OMA at the best-accuracy and best-ECE hyperparameter settings (mean

\pm

std over 5 seeds) 2 2 2 These configurations were selected retrospectively using evaluation labels and characterize the attainable performance–calibration trade-off within the search grid. They are not used as evidence for label-free hyperparameter selection. . Best values per dataset are in bold.

	MRPC	RTE	CoLA	SST-2
bert-base	86.44	65.34	59.05	92.43
bert-large	90.61	61.73	64.55	93.58
funnel (xlarge-base)	92.01	87	69.6	95.99
funnel (xlarge)	89.69	87	68.06	54.93
MS (Bert, best)	88.3	61	59.1	92.5
MS (Bert, greedy)	88.3	61.7	59.1	93

Table 2: Performance of other methods on GLUE benchmark. Top : individual model performance when fine-tuned on each dataset. Bottom : Performance of existing methods. Model soups (MS) on two different architectures (Bert-base-uncased and T5-base ( Raffel et al., 2020 ) ). Fisher merging (Fisher) on Bert-base architecture.

Figure 2: OMA weights at the best ECE, learned by Algorithm 1 and individual model’s performance

	MRPC	RTE	CoLA	SST-2
BERT-Base	74.80	53.07	43.57	88.19
BERT-Large	74.98	50.90	53.91	91.63
funnel(xlarge-base)	74.38	52.71	13.44	92.89
funnel(xlarge)	74.80	49.10	42.68	94.50
LoRA (output avg)	74.80	59.21	51.48	92.54
Bayesian LoRA	74.98	50.90	53.91	91.74

Table 3: Performance of LoRA and Bayesian LoRA on GLUE benchmark. Top : individual model performance when fine-tuned on each dataset using LoRA. Bottom : Performance of output averaging and Bayesian LoRA

		Img-1K	Img-V2	Img-R	Img-sketch	Img-A	ObjNet
	Output avg	84.90	79.14	96.22	75.85	85.08	77.13
	OMA	85.11	78.91	96.17	76.23	86.68	77.64
Zeroshot		(+0.25%)	(-0.29%)	(-0.05%)	(+0.50%)	(+1.88%)	(+0.66%)
	Output avg	88.98	81.53	94.36	72.76	83.61	73.90
	OMA	89.19	81.90	94.64	72.68	85.47	74.02
MAP		(+0.24%)	(+0.45%)	(+0.30%)	(-0.11%)	(+2.22%)	(+0.16%)

Table 4: Top-1 accuracy (%) of output averaging and OMA for different candidate model pools. Numbers in parentheses are the relative improvement of OMA over output averaging; the best accuracy in each column is in bold. These numbers are highest accuracy from Fig. 3 .

Method	Img-1K	V2	R	Sketch	A	ObjNet
Accuracy-weighted	88.98	81.83	94.38	73.95	83.89	74.05
Single best model	88.27	80.30	95.80	74.66	86.06	76.42
Model Soups, greedy	90.94	84.22	95.46	74.23	92.67	78.52
CoCa, frozen	90.6	–	–	–	–	–
EVA-02-L	90.0	82.4	89.9	70.1	87.7	62.8

Table 5: TOP: Other baselines using the same candidate models in Table 4 . Bottom: External reference numbers from prior methods. These methods differ in architecture, training data, and fine-tuning protocol, so we include them for context, rather than as direct comparisons. Accuracy values are percentages.

Figure 3: ECE versus Accuracy across six datasets (each column) and three candidate pools (each row) (error bars showing the standard deviation over seeds)

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Name in Paper	OpenCLIP Name	Pretraining Data	Params (M)	FLOPs (B)	Memory (G)	Avg. Perf. (38 sets)
fe-1	ViT-H-14-378-quickgelu	dfn5b	986.71	1054.05	4.4	0.7079
fe-2	ViT-H-14-quickgelu	dfn5b	986.11	381.68	—	0.6961
fe-3	EVA02-E-14-plus	laion2b_s9b_b144k	5044.89	2362.19	20.2	0.6930
fe-4	ViT-SO400M-14-SigLIP-384	webli	877.96	723.48	4.1	0.6921
fe-5	ViT-bigG-14-CLIPA-336	datacomp1b	2517.76	2271.58	10.35	0.6842
—	ViT-bigG-14-CLIPA	datacomp1b	2517.22	1007.93	—	0.6822

Appendix

Table 6: We select

8

pre-trained open-clip models ( Ilharco et al., 2021 ) based on their performance of zero-shot classifiers evaluated on

38

datasets and the diversity of the pretraining datasets.

Panel of Figure 4 — Figure 4: The diagonal elements are orders of magnitude larger than off-diagonal elements (with the MAP estimates trained for ImageNet-1K. Performance in Table 4 ). Showing the magnitude of a subset of the Hessian for a [400 × 400] subset of the Hessian, which is originally roughly [1 million by 1 million], sampling 20 classes log-uniformly and 20 input dimensions uniformly.

Name in Paper	OpenCLIP Name	Pretraining Data	Params (M)	FLOPs (B)	Memory (G)	Avg. Perf. (38 sets)
fe-1	ViT-H-14-378-quickgelu	dfn5b	986.71	1054.05	4.4	0.7079
fe-2	ViT-H-14-quickgelu	dfn5b	986.11	381.68	—	0.6961
fe-3	EVA02-E-14-plus	laion2b_s9b_b144k	5044.89	2362.19	20.2	0.6930
fe-4	ViT-SO400M-14-SigLIP-384	webli	877.96	723.48	4.1	0.6921
fe-5	ViT-bigG-14-CLIPA-336	datacomp1b	2517.76	2271.58	10.35	0.6842
—	ViT-bigG-14-CLIPA	datacomp1b	2517.22	1007.93	—	0.6822

Appendix

Table 7: We select

8

pre-trained open-clip models ( Ilharco et al., 2021 ) based on their performance of zero-shot classifiers evaluated on

38

datasets and the diversity of the pretraining datasets.

		fe-1	fe-2	fe-3	fe-4	fe-5	fe-6	fe-7	fe-8
imagenet	map	88.05	87.11	87.98	88.15	88.27	87.86	86.06	85.49
	zeroshot	84.51	83.57	82.52	83.05	83.19	82.1	80.64	78.92
imagenet-v2	map	80.29	78.89	80.12	80.31	80.13	79.95	77.84	76.08
	zeroshot	78.41	77.34	76.34	77.16	77.34	75.78	74.23	71.51
imagenet-r	map	91.04	90.33	91.77	93.66	93.31	91.41	90.72	87.58
	zeroshot	93.79	92.94	94.83	95.81	95.36	94.28	93.54	91.03

Appendix

Table 8: For ImageNet-1K and ImageNet-V2 datasets, the trained linear classifiers (trained with ImageNet-1K) outperform zeroshot models. However, for datasets such as ImageNet-A, ImageNet-R, ImageNet-Sketch, and ObjectNet, Zeroshot models consistently outperform trained classifiers, implying that these datasets exhibit larger domain shifts from ImageNet-1K.

	Imagenet-1K	V2	R	Sketch	A
training subset 1	88.402	81.21	94.74	73.478	85.743
training subset 2	88.634	81.12	94.847	73.308	86.39 —
training subset 3	88.542	81.36	94.883	73.625	85.943
variance	0.0090	0.0098	0.0036	0.0167	0.0734

Appendix

Table 9: Across three training subsets, the accuracy of our method does not change much and the variance across different subsets is relatively small (last row), indicating the results we showed in Table 4 are not due to random chance.

Name in Paper	OpenCLIP Name	Pretraining Data	Params (M)	FLOPs (B)	Memory (G)	Avg. Perf. (38 sets)
fe-1	ViT-H-14-378-quickgelu	dfn5b	986.71	1054.05	4.4	0.7079
fe-2	ViT-SO400M-14-SigLIP-384	webli	877.96	723.48	4.1	0.6921
fe-3	ViT-bigG-14-CLIPA-336	datacomp1b	2517.76	2271.58	10.35	0.6842
fe-4	EVA02-E-14	laion2b_s4b_b115k	4704.59	2311.42	18.8	0.6690
fe-5	ViT-H-14-quickgelu	metaclip_fullcc	986.11	381.68	4.4	0.6671

Appendix

Table 10: We select

5

pre-trained open-clip models from ( Ilharco et al., 2021 ) as feature extractors for Camelyon17, Flowers102, and Sun397 datasets.

	Camelyon17		Flowers102		Sun397
Name in Paper	map	zeroshot	map	zeroshot	map	zeroshot
fe-1	91.48	65.71	99.8	90.59	85.49	77.09
fe-2	91.73	51.65	99.61	92.35	85.08	75.41
fe-3	92.22	51.81	99.8	87.84	85.06	76.32
fe-4	91.54	50.81	99.71	83.14	84.54	76.57
fe-5	91.49	65.53	99.61	85.59	83.43	76.53

Appendix

Table 11: Individual models’ performance tested on Camelyon17, Flowers102, and Sun397 datasets. MAP estimates (training with each data’s training set) always outperform zeroshot models in these datasets.

		Cam17	Flw102	Sun397
Zeroshot	Output avg	51.78	90.69	79.61
	OMA	70.42	92.94	79.8
MAP	Output avg	93.63	99.80	85.82
	BMA	93.64	99.80	85.84
Zeroshot&MAP	Output avg	93.63	99.80	85.82
	OMA	93.70	99.83	86.39

Appendix

Table 12: BMA and OMA performances tested on Camelyon17, Flowers102, and Sun397 datasets. Using both MAP and zeroshot models together with our OMA paradigm improves the performance.

batch size	learning rate	epochs	$\alpha$	test accuracy
1000	0.01	200	0.1	90.15
1000	0.01	200	1.0	92.21
1000	0.01	200	10.0	92.43
1000	0.01	200	50.0	92.18
1000	0.01	200	100.0	92.15

Appendix

Table 13: Hyperparameter search for the prior variance

\alpha

for Camelyon17

batch size	learning rate	epochs	$\lambda$	test accuracy
1200	0.001	400	0.1	77.086
1200	0.001	400	1.0	77.121
1200	0.001	400	10.0	77.064
1200	0.001	400	100.0	78.035
1200	0.001	400	1000.0	78.069
1200	0.001	400	10000.0	77.143

Appendix

Table 14: Hyperparameter search for the regularization constant

\lambda

for ObjectNet

Figure 8: Ablation over the number of epochs.

Figure 9: Ablation over the regularization constant $\lambda$ .

number of samples	runtime (seconds)	log-marginal-likelihood
200,000	26.4	-79.68
400,000	54.97	-79.02
800,000	106.62	-78.31
1,600,000	214.26	-77.55
3,200,000	436.79	-76.88
6,400,000	862.89	-76.18

Appendix

Table 15: When variance is large

number of samples	runtime	log-marginal-likelihood
1,000	0.13	-93.57
10,000	1.23	-94.86
100,000	13.23	-94.82
1,000,000	128.62	-94.83
10,000,000	1181	-94.82

Appendix

Table 16: When variance is small

Ensemble Learning

Ensemble learning combines multiple predictors to improve accuracy, robustness, calibration, or uncertainty estimates.

See all 272 papers

May 18, 2026

Ensembling Tabular Foundation Models - A Diversity Ceiling And A Calibration Trap

Tabular foundation models (TFMs) now match or beat tuned gradient-boosted trees on a growing fraction of tabular tasks, but no single TFM wins on every dataset. Ensembling is the go to fix here, and it works less well than expected. Six modern TFMs form a near-redundant pool: their mean pairwise Q-statistic is

0.961

, close enough to

1

that any convex combination is bounded above. We benchmark six ensemble strategies over six TFMs on 153 OpenML classification tasks. The best ensemble, two-level cascade stacking, buys

+0.18\%

accuracy over the strongest single TFM at

253\times

the compute. A Friedman and Nemenyi analysis places three ensembles and the best base TFM in a single equivalence group; three other ensembles are significantly worse than the best base. Stacking with a logistic-regression meta-learner is the most striking case: competitive accuracy and ROC-AUC, the worst log-loss rank among the ensembles. The meta-learner improves accuracy by sharpening class boundaries, which destroys calibration. We recommend greedy selection as the practical default.

Aditya Tanna, Yash Desai, Pratinav Seth +3

Lexsi Labs

Tabular Foundation Models Ensemble Learning Tabular Classification Ensemble Diversity Stacked Ensemble Learning

Ask PDF

Jan 23, 2026

Is BatchEnsemble a Single Model? On Calibration and Diversity of Efficient Ensembles

In resource-constrained and low-latency settings, uncertainty estimates must be efficiently obtained. Deep Ensembles provide robust epistemic uncertainty (EU) but require training multiple full-size models. BatchEnsemble aims to deliver ensemble-like EU at far lower parameter and memory cost by applying learned rank-1 perturbations to a shared base network. We show that BatchEnsemble not only underperforms Deep Ensembles but closely tracks a single model baseline in terms of accuracy, calibration and out-of-distribution (OOD) detection on CIFAR10/10C/SVHN. A controlled study on MNIST finds members are near-identical in function and parameter space, indicating limited capacity to realize distinct predictive modes. Thus, BatchEnsemble behaves more like a single model than a true ensemble.

Anton Zamyatin, Patrick Indri, Sagar Malhotra +1

TU Wien

Ensemble Learning Ensemble Diversity Deep Ensembles Model Calibration

Ask PDF

Oct 20, 2025

Navigating the Alignment-Calibration Trade-off: A Pareto-Superior Frontier via Model Merging

The "alignment tax" of post-training is typically framed as a drop in task accuracy. We show it also involves a severe loss of calibration, making models overconfident, less reliable, and model outputs less diverse. We show that this trade-off can be navigated effectively via a simple post-hoc intervention: interpolating between a model's weights before and after alignment. Crucially, this is not a strict trade-off. We find that the process consistently reveals Pareto-optimal interpolations - models that improve accuracy beyond both parents while substantially recovering the calibration lost during alignment. Our work demonstrates that simple model merging provides a computationally efficient method for mitigating the full scope of the alignment tax, yielding models that are more capable and more reliable.

Tiancheng Hu, Benjamin Minixhofer, Nigel Collier

University of Cambridge

LLM Alignment Model Calibration Multi-Objective Language Model Alignment Language Model Merging

Ask PDF

Abstract

Ensemble Learning

Explore similar work

Ensembling Tabular Foundation Models - A Diversity Ceiling And A Calibration Trap

Is BatchEnsemble a Single Model? On Calibration and Diversity of Efficient Ensembles

Navigating the Alignment-Calibration Trade-off: A Pareto-Superior Frontier via Model Merging