Structured Basis Function Networks: Loss-Centric Multi-Hypothesis Ensembles with Controllable Diversity
Authors: Alejandro Rodriguez Dominguez, Muhammad Shahzad, Xia Hong
Organizations: University of Reading, Department of Computer Science, Whiteknights House, Reading, RG6 6UR, United Kingdom · Miralta Finance Bank S.A.,28020, Madrid, Spain
Ensemble learning and multi-hypothesis prediction are two complementary paradigms for improving predictive performance and robustness. This work introduces the Structured Basis Function Network (s-BFN), a loss-aware ensemble framework that unifies these paradigms through a supervised aggregation mechanism motivated by the loss-induced Bregman-centroid geometry of prediction space. The approach constructs a structured dataset of base predictions and trains a parametric combiner directly against the observed labels, with the centroidal formulation serving as a geometric reference rather than an exact centroid solver, while a diversity-control parameter regulates specialization among predictors. The proposed framework is evaluated across regression and classification tasks, including tabular data, image recognition, and large-scale temporally structured video classification. Empirical results show that s-BFN provides strong performance across the evaluated regimes and is competitive with the aggregation baselines considered in the study. The strongest gains are observed in higher-dimensional and temporally structured settings, where aggregation capacity and controlled diversity are jointly beneficial, while simpler tabular regimes exhibit earlier saturation. These results highlight a capacity-complexity-diversity trade-off rather than a strictly monotonic relationship between diversity, ensemble size, and accuracy.
Figures & tables
Symbol
Meaning
N
Number of observations.
M
Number of base predictors (hypotheses).
C
Output dimension; number of classes in classification.
fθj
Base predictor j , parameterized by θj .
pij
Class-probability vector produced by base predictor j for observation i .
Di
Structured prediction vector obtained by concatenating the M base-predictor outputs for observation i .
Table 1: Main notation used in the methodology.
Geometry
Loss Li
Prediction (generic)
Prediction (s-BFN)
Euclidean (squared loss)
21∥y^i−yi∥2
y^i=G(Di,α)
y^i=Φ(Di)α
Simplex (cross-entropy)
−logy^i,yi
y^i=softmax(G(Di,α))
y^i=softmax(Φ(Di)α)
Table 2: Geometry, loss, generic-combiner prediction, and s-BFN prediction.
Figure 1: Workflow: base predictors produce a structured dataset D . A suitable loss geometry is selected, inducing a Bregman divergence Bϕ (e.g., Euclidean/squared loss, KL/cross-entropy, or Itakura–Saito). The corresponding Bregman centroid z^=argminzM1∑j=1MBϕ(z,zj) provides the geometric reference for the aggregation rule. We instantiate G as an s-BFN with a radial map Φ(⋅;ϑ) and K units, and train it as a supervised combiner on the observed labels rather than as an exact solver of the centroid optimization problem. Optimization is closed-form in Euclidean settings and gradient-based for non-Euclidean losses.
Regime
Datasets
Task type
Prediction space
Tabular regression
Air Quality; Appliances Energy
Multivariate regression
Low-dimensional continuous output
Image classification
MNIST; Fashion-MNIST; CIFAR-10
Multi-class classification
10-class probability simplex
Video classification
UCF101
101-class action recognition
High-dimensional probability simplex with temporal structure
Table 3: Experimental regimes considered in the evaluation.
Variant
Centre Cj
Scale γj
Basis function φ
G1
i1∑q=1ifθj(xq)
i−11∑q=1i(fθj(xq)−Cj)2
exp(−2γj2∥fθj(xi)−Cj∥2)
G2
k1∑q=i−k+1ifθj(xq)
k−11∑q=i−k+1i(fθj(xq)−Cj)2
exp(−2γj2∥fθj(xi)−Cj∥2)
G3
k1∑q=i−k+1ifθj(xq)
Mmaxp=jd(fθj(xi),fθp(xi))
γj∥fθj(xi)−Cj∥2
Table 4: Iterative s-BFN variants with different centre ( Cj ) and scale ( γj ) definitions. i=1,…,N denotes training iterations, j=1,…,M the predictors, and k a number of recent predictions.
Hyperparameter
Values
Number of hypotheses ( M )
2, 5, 10, 20, 35
Neurons per layer ( υ )
20, 50, 200, 2000
Learning rates ( ηθ , ηα )
0.03, 0.3, 1.0
Initialization scale ( χ )
0.0001, 0.01, 0.1, 1.0
Diversity-control parameter ( ε )
0, 0.1, 0.35, 0.5
Regularization (predictors, λ1 )
0, 0.0001, 0.01, 0.07
Table 5: Hyperparameter search space used for tuning. M is the number of hypotheses, υ the number of neurons per layer, ηθ and ηα the learning rates for base learners and s-BFN respectively, χ a scaling factor for initializing predictor parameters Θ , ε the diversity control parameter, and λ1 , λ2 the base learners and ensemble regularization coefficients respectively.
Model
Best mean RMSE
Std. dev.
1st Q
3rd Q
SVM-RBF
29.83
1.99
34.80
37.65
Random Forest
55.66
15.47
69.00
91.55
Gradient Boosting
55.76
38.73
93.92
151.58
Arithmetic Combiner
39.19
0.15
41.75
43.93
s-BFN least-squares
22.46
9.14
38.98
54.71
Table 6: Absolute humidity prediction: statistics of the 10-fold validation test RMSE across all hyperparameter combinations. For each configuration, RMSE values are averaged across the 10 folds; the table reports the best configuration-level mean RMSE and dispersion statistics across configurations (Q: quartiles).
Model
Best mean RMSE
Std. dev.
1st Q
3rd Q
SVM-RBF
104.68
1.27
107.26
109.01
Random Forest
298.46
29.35
328.48
373.81
Gradient Boosting
292.08
67.26
389.10
476.89
Arithmetic Combiner
115.17
0.11
128.54
144.83
s-BFN least-squares
101.12
8.26
102.25
103.64
Table 7: Energy appliance prediction: statistics of the 10-fold validation test RMSE across all hyperparameter combinations. For each configuration, RMSE values are averaged across the 10 folds; the table reports the best configuration-level mean RMSE and dispersion statistics across configurations (Q: quartiles).
Figure 2: Mean RMSE as a function of the diversity-control parameter ε for different ensemble sizes M on the air quality dataset. For each (M,ε) pair, results are aggregated by averaging over all remaining hyperparameter settings, ensuring a clear and controlled assessment of the effect of ε .
Figure 3: Mean RMSE as a function of the diversity-control parameter ε for different ensemble sizes M on the energy appliances dataset. For each (M,ε) pair, results are aggregated by averaging over all remaining hyperparameter settings, ensuring a clear and controlled assessment of the effect of ε .
Name in figs/tables
Base learners
Combiners
"Het. Mixed"
SimpleCNN, ResNetTiny1, ResNetTiny2
s-BFN, MoE, Mean Agg.
"Homo. ResNet"
ResNetTiny2 (or ResNetTiny1)
s-BFN, MoE, Mean Agg.
"Homo. CNN"
SimpleCNN
s-BFN, MoE, Mean Agg.
"Arithmetic Mean"
Same as corresponding ensemble
Accuracy avg. (baseline)
Table 8: Ensemble variants used in the architecture, diversity, and computational efficiency analyses. s-BFN: probability inputs → RBF features → linear logits → softmax with temperature T (learned, loss-aware combiner). MoE: gated weighted sum of base probabilities. Mean Aggregation: logit averaging (average logits, then a single softmax). Arithmetic Mean: arithmetic mean of base accuracies ( Rupprecht et al., 2017 ) .
MNIST ( n=16 )
Fashion-MNIST ( n=16 )
CIFAR-10 ( n=12 )
Baseline
Δ
W/T/L
pH
Δ
W/T/L
pH
Δ
W/T/L
pH
SingleBest
+13.72
16/0/0
2.75×10−4
+19.26
16/0/0
2.75×10−4
+11.72
12/0/0
0.0044
MeanProb
+10.36
16/0/0
2.75×10−4
+15.76
16/0/0
2.75×10−4
+7.59
12/0/0
0.0044
LogitAverage
+9.56
16/0/0
2.75×10−4
+14.59
16/0/0
2.75×10−4
+7.09
12/0/0
0.0044
LinearStack
-1.33
1/1/14
0.0228
-0.74
2/0/14
0.0126
-1.30
3/0/9
0.3296
MLPStack-Matched
-0.26
7/0/9
0.8603
+0.64
11/0/5
0.1928
-0.59
5/0/7
0.4072
Table 9: Configuration-level comparison of s - BFN against competing aggregation rules over the complete controlled grid. The competing baselines are a validation-selected single expert (SingleBest), probability averaging (MeanProb) ( Lakshminarayanan et al., 2017 ) , logit averaging (LogitAverage) ( Tassi et al., 2022 ) , linear stacking (LinearStack) and a capacity-matched nonlinear stacker (MLPStack-Matched) ( Wolpert, 1992 ) , Mixture-of-Experts (MoE) ( Jacobs et al., 1991 ) , the stacking and model-averaging variants of Regularized Neural Ensemblers (NE-Stack and NE-MA) ( Arango et al., 2025 ) , and greedy ensemble selection (GES) ( Caruana et al., 2004 ) . Δ denotes the mean accuracy difference (s - BFN minus baseline), in percentage points, across the evaluated (M,ε) configurations. W/T/L denotes the number of configurations in which s - BFN obtains higher/equal/lower mean accuracy. pH is the paired two-sided Wilcoxon signed-rank p -value after Holm correction over the nine comparisons within each dataset. The complete configuration-level accuracies and 95% confidence intervals over the five seeds are reported in D . Bold values indicate pH<0.05 .
Figure 4: Classification accuracy for different ensemble configurations on MNIST. Ensemble labels indicate base-learner composition (homogeneous or heterogeneous), while aggregation rules (s-BFN, MoE, Mean Aggregation, Arithmetic Mean) are reported separately.
Figure 5: Classification accuracy for different ensemble configurations on Fashion-MNIST.
Figure 6: Classification accuracy for different ensemble configurations on CIFAR-10.
Figure 7: Gibbs risk RG(ε) and expected disagreement d(ε) (left axis), together with majority-vote error RMV(ε) and its PAC-Bayes upper bound (right axis), for varying diversity-control parameter ε . The curves illustrate an intermediate ε where the bound and empirical error are minimized.
M
ε
K
Val Acc.
Test Acc.
5
0.1
100
0.795
0.792
5
0.3
100
0.827
0.822
5
0.6
100
0.843
0.843
10
0.1
200
0.821
0.823
10
0.3
200
0.855
0.843
10
0.6
200
0.841
0.843
Table 10: UCF101 performance of s-BFN under controlled variation of ensemble size M and diversity-control parameter ε .
M
Mean
MoE
Fusion
s-BFN
5
0.803
0.774
0.364
0.843
10
0.786
0.732
0.277
0.843
20
0.766
0.692
0.326
0.864
Table 11: Test accuracy comparison of aggregation strategies on UCF101, using the validation-selected ε per ensemble size M for s-BFN.
Figure 8: Validation accuracy of s-BFN on UCF101 as a function of the diversity-control parameter ε for different ensemble sizes M .
Figure 9: Test accuracy of s-BFN on UCF101 as a function of the diversity-control parameter ε for different ensemble sizes M .
Figure 10: Heatmap of validation accuracy over ensemble size M and the diversity-control parameter ε .
Figure 11: Learned aggregation temperature as a function of the diversity-control parameter ε for different ensemble sizes M .
Figure 12: Computational efficiency evaluation for different ensemble configurations on CIFAR-10. sRBF corresponds to the s-BFN combiner and Base Avg refers to the arithmetic mean of the individual base learners’ accuracies.
Figure 13: Computational efficiency results on Fashion-MNIST. sRBF corresponds to the s-BFN combiner and Base Avg refers to the arithmetic mean of the individual base learners’ accuracies.
Dataset
Ensemble
s-BFN_Acc
MoE_Acc
Base_Acc
Δ s-BFN
Δ MoE
MNIST
Heterogeneous
48.47
47.56
39.65
8.82
7.91
MNIST
Homogeneous
32.97
34.70
30.40
2.56
4.29
FashionMNIST
Heterogeneous
69.81
63.10
54.19
15.62
8.90
FashionMNIST
Homogeneous
63.70
61.57
53.02
10.69
8.56
CIFAR-10
Heterogeneous
49.10
47.96
39.71
9.39
8.25
CIFAR-10
Homogeneous
32.70
33.45
30.38
2.32
3.06
Table 12: Overall accuracy (mean across settings) per dataset and ensemble. Values are reported in %, with gains in percentage points (pp) relative to the average base-learner accuracy.
Dataset
Ensemble
Architecture
M
s-BFN (%)
MoE (%)
Base Avg (%)
Train (s)
Eval (s)
MNIST
Het.
Mixed
5
70.45
66.48
59.34
–
–
MNIST
Homo.
ResNetTiny2
5
71.66
70.03
68.20
–
–
Fashion-MNIST
Het.
Mixed
5
83.54
82.38
75.99
38.90
2.73
Fashion-MNIST
Homo.
CNN
10
83.01
81.02
80.25
29.50
2.10
CIFAR-10
Het.
Mixed
10
68.94
66.99
60.28
313.10
4.22
CIFAR-10
Homo.
ResNetTiny2
5
71.72
66.27
64.33
258.30
4.05
Table 13: Best s-BFN configuration per dataset and ensemble. All runs used temperature T=3 and learning rate ηα=0.1 ; ε values follow the original configurations and are omitted for brevity. Accuracies in %.
Dataset
Ensemble
Train (s)
Eval (s)
FLOPs
Params
FashionMNIST
Heterogeneous
125.8
4.09
5.8M
1.24M
FashionMNIST
Homogeneous
72.2
2.98
1.7M
0.80M
CIFAR-10
Heterogeneous
741.0
6.06
7.8M
1.25M
CIFAR-10
Homogeneous
2712.4
6.84
14.4M
7.32M
Table 14: Efficiency metrics for FashionMNIST and CIFAR-10: mean train/eval time, average FLOPs per base model, and total base-model parameters.
ε
Dataset
Heterogeneous
Homogeneous
0.0
CIFAR-10
26.59
16.86
0.5
CIFAR-10
59.84
42.89
0.8
CIFAR-10
60.87
38.36
0.0
FashionMNIST
51.25
40.25
0.5
FashionMNIST
79.48
75.00
0.8
FashionMNIST
78.71
75.86
Table 15: Sensitivity of s-BFN accuracy (%) to ε across datasets for homogeneous and heterogeneous s-BFN configurations.
Model
Best mean RMSE
Std. dev.
1st Q
3rd Q
s-BFN LS
22.46
9.14
38.98
54.71
G1
37.32
2.92
37.94
39.66
G2
37.28
3.03
38.72
40.87
G3
37.35
2.51
39.02
39.67
Table 16: Statistics of 10-fold validation RMSE across all hyperparameter combinations. For each configuration, RMSE values are averaged across the 10 folds; the table reports the best configuration-level mean RMSE and dispersion statistics across configurations (Q: quartiles).
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 14: Average RMSE on test set for s-BFN using Gaussian basis functions (GBF, blue) and RBF-KMeans (red) across increasing basis sizes M . Error bars denote standard deviation across 5 runs.
Figure 15: Basis function center locations for GBF (left, red crosses) and RBF-KMeans (right, green crosses), with M∈{2,5,10,20,50} . RBF-KMeans adapts to data structure, GBF maintains even spread.
M
ε
Entropy
Val Acc.
Test Acc.
5
0.1
3.887
0.795
0.792
5
0.6
4.014
0.843
0.843
10
0.1
4.036
0.821
0.823
10
0.6
4.145
0.841
0.843
20
0.1
4.121
0.815
0.822
20
0.6
4.147
0.858
0.864
Appendix
Table 17: Predictive entropy and performance of s-BFN on UCF101 across ensemble sizes and values of the diversity-control parameter.
Figure 16: Validation accuracy of s-BFN on UCF101 as a function of the diversity-control parameter and aggregation capacity. Higher values indicate better performance.
M
ε
SB
MP
LA
LS
MLP
MoE
NES
NEMA
GES
s-BFN
5
0.0
28.91 ± 9.51
49.58 ± 17.60
51.98 ± 5.27
69.78 ± 6.60
66.69 ± 7.86
75.48 ± 7.28
46.36 ± 11.03
43.03 ± 11.82
49.17 ± 15.99
72.27 ± 7.99
5
0.3
95.63 ± 1.62
96.85 ± 0.41
97.10 ± 0.45
97.77 ± 0.14
97.66 ± 0.14
97.12 ± 0.42
96.71 ± 0.75
95.85 ± 1.83
96.43 ± 0.87
97.69 ± 0.14
5
0.5
96.98 ± 0.77
97.48 ± 0.13
97.51 ± 0.17
98.08 ± 0.17
98.04 ± 0.17
97.95 ± 0.20
97.35 ± 0.67
97.14 ± 0.83
97.75 ± 0.41
98.07 ± 0.23
5
0.8
97.22 ± 0.43
97.01 ± 0.70
96.91 ± 0.77
97.96 ± 0.15
97.92 ± 0.23
97.71 ± 0.39
97.23 ± 0.63
97.28 ± 0.78
97.56 ± 0.46
97.91 ± 0.14
10
0.0
9.12 ± 1.19
16.07 ± 6.10
19.49 ± 6.77
44.75 ± 6.89
41.14 ± 5.12
45.85 ± 7.07
22.32 ± 9.51
19.86 ± 7.35
22.47 ± 4.84
44.32 ± 5.65
10
0.3
94.73 ± 0.65
96.38 ± 0.52
96.49 ± 0.41
97.63 ± 0.19
97.54 ± 0.20
97.14 ± 0.41
96.11 ± 1.52
95.85 ± 0.80
95.74 ± 0.31
97.61 ± 0.13
Appendix
Table 18: Complete controlled aggregation grid for MNIST. Accuracy is reported as mean ± 95% confidence-interval half-width (%) over five seeds.
M
ε
SB
MP
LA
LS
MLP
MoE
NES
NEMA
GES
s-BFN
5
0.0
26.37 ± 8.38
32.29 ± 6.95
33.86 ± 3.42
57.68 ± 1.81
55.02 ± 3.93
58.76 ± 2.42
36.86 ± 5.56
36.26 ± 7.28
34.24 ± 6.95
58.58 ± 2.62
5
0.3
66.84 ± 3.46
73.67 ± 3.16
74.20 ± 2.70
79.06 ± 0.54
77.37 ± 0.21
77.94 ± 1.48
74.29 ± 1.09
71.77 ± 5.32
71.30 ± 2.23
79.05 ± 0.17
5
0.5
74.85 ± 2.22
76.53 ± 0.91
76.51 ± 0.75
79.86 ± 0.56
79.01 ± 0.73
78.81 ± 0.40
76.41 ± 2.00
75.89 ± 2.75
77.18 ± 1.17
79.81 ± 0.34
5
0.8
76.62 ± 1.48
76.78 ± 1.13
76.72 ± 1.10
79.88 ± 0.68
78.97 ± 0.78
78.34 ± 1.17
77.32 ± 1.55
76.65 ± 2.44
77.40 ± 1.21
80.18 ± 0.88
10
0.0
8.40 ± 3.07
19.54 ± 7.41
27.99 ± 4.40
55.21 ± 3.20
50.25 ± 2.56
55.33 ± 2.50
27.14 ± 3.66
21.48 ± 6.68
19.37 ± 3.64
54.45 ± 2.96
10
0.3
69.16 ± 2.02
73.60 ± 1.37
74.08 ± 1.05
79.19 ± 0.75
77.64 ± 0.66
78.05 ± 1.39
72.64 ± 3.24
71.42 ± 5.29
73.44 ± 1.00
79.08 ± 0.83
Appendix
Table 19: Complete controlled aggregation grid for Fashion-MNIST. Accuracy is reported as mean ± 95% confidence-interval half-width (%) over five seeds.
M
ε
SB
MP
LA
LS
MLP
MoE
NES
NEMA
GES
s-BFN
5
0.0
19.81 ± 5.62
33.78 ± 5.21
28.36 ± 2.07
39.64 ± 2.23
38.71 ± 2.49
40.49 ± 2.55
29.96 ± 3.79
26.68 ± 5.65
29.76 ± 6.24
40.39 ± 2.33
5
0.5
65.36 ± 2.89
68.68 ± 1.55
69.51 ± 1.66
71.91 ± 1.10
71.75 ± 1.05
71.00 ± 0.94
68.20 ± 3.43
67.65 ± 2.67
69.67 ± 1.46
71.80 ± 0.91
5
0.8
65.82 ± 4.00
68.07 ± 1.43
68.25 ± 1.31
71.36 ± 1.27
71.32 ± 1.16
70.49 ± 1.22
68.43 ± 2.76
68.19 ± 1.82
69.78 ± 1.43
71.58 ± 1.23
10
0.0
10.00 ± 0.00
14.34 ± 3.05
15.17 ± 0.65
22.01 ± 1.56
16.75 ± 5.41
16.48 ± 5.44
12.39 ± 3.87
10.21 ± 0.58
14.34 ± 3.05
16.73 ± 6.18
10
0.5
58.27 ± 1.15
65.12 ± 0.61
65.65 ± 0.77
68.90 ± 0.46
68.96 ± 0.47
67.63 ± 0.43
64.78 ± 4.23
65.55 ± 1.63
66.85 ± 0.76
68.89 ± 0.49
10
0.8
65.46 ± 1.44
67.69 ± 0.95
67.88 ± 1.02
71.65 ± 0.40
71.91 ± 0.38
70.26 ± 0.51
68.60 ± 3.69
69.21 ± 1.25
69.86 ± 0.36
71.91 ± 0.40
Appendix
Table 20: Complete controlled aggregation grid for CIFAR-10. Accuracy is reported as mean ± 95% confidence-interval half-width (%) over five seeds.
This paper tackles visible challenges in deep ensemble learning, where deep neural networks serve as ensemble members: training and storage burdens, and robustness of cautious (set-valued) predictions targeting multiple utilities, which may involve reward-sensitivity. To mitigate the training and storage burdens, we propose to employ compact ensembles, such as Bayesian Neural Networks and Convolutional Neural Networks with the Monte-Carlo dropout prediction option, to produce probabilistic predictions. For each query instance, these probabilistic predictions are then used to define a representative distribution optimizing some statistical distance. The representative distribution is then employed to define the Bayes-optimal prediction (BOP) of any utility. To address the potential unrobustness of singleton prediction making, we propose a family of set-utilities satisfying some desirable properties and whose set-valued BOPs can be found efficiently. Empirical evidence is then given to illustrate the potential (dis)advantages of the proposed ensemble learning framework.
Kim-Dung Tran, Dang-Man Nguyen, Vu-Linh Nguyen +3
UMR CNRS 7253, Heudiasyc, Universit´e de Technologie de Compi`egne, Compi`egne, France. · School of Knowledge Science, Japan Advanced Institute of Science and Technology, Nomi, Japan.
In resource-constrained and low-latency settings, uncertainty estimates must be efficiently obtained. Deep Ensembles provide robust epistemic uncertainty (EU) but require training multiple full-size models. BatchEnsemble aims to deliver ensemble-like EU at far lower parameter and memory cost by applying learned rank-1 perturbations to a shared base network. We show that BatchEnsemble not only underperforms Deep Ensembles but closely tracks a single model baseline in terms of accuracy, calibration and out-of-distribution (OOD) detection on CIFAR10/10C/SVHN. A controlled study on MNIST finds members are near-identical in function and parameter space, indicating limited capacity to realize distinct predictive modes. Thus, BatchEnsemble behaves more like a single model than a true ensemble.
We present EARCP (Ensemble Auto-Régulé par Cohérence et Performance), a novel ensemble architecture that dynamically weights heterogeneous expert models based on both their individual performance and inter-model coherence. Unlike traditional ensemble methods that rely on static or offline-learned combinations, EARCP continuously adapts model weights through a principled online learning mechanism that balances exploitation of high-performing models with exploration guided by consensus signals. The architecture combines theoretical foundations from multiplicative weight update algorithms with a novel coherence-based regularization term, providing both theoretical guarantees through regret bounds and practical robustness in non-stationary environments. We formalize the EARCP framework, prove sublinear regret bounds of O(sqrt(T log M)) under standard assumptions, and demonstrate its effectiveness through empirical evaluation on sequential prediction tasks including time series forecasting, activity recognition, and financial prediction. The architecture is designed as a general-purpose framework applicable to any domain requiring ensemble learning with temporal dependencies. An open-source implementation is available at https://github.com/Volgat/earcp and via PyPI (pip install earcp).