Structured Basis Function Networks: Loss-Centric Multi-Hypothesis Ensembles with Controllable Diversity
Organizations: University of Reading, Department of Computer Science, Whiteknights House, Reading, RG6 6UR, United Kingdom · Miralta Finance Bank S.A.,28020, Madrid, Spain
Abstract
Ensemble learning and multi-hypothesis prediction are two complementary paradigms for improving predictive performance and robustness. This work introduces the Structured Basis Function Network (s-BFN), a loss-aware ensemble framework that unifies these paradigms through a supervised aggregation mechanism motivated by the loss-induced Bregman-centroid geometry of prediction space. The approach constructs a structured dataset of base predictions and trains a parametric combiner directly against the observed labels, with the centroidal formulation serving as a geometric reference rather than an exact centroid solver, while a diversity-control parameter regulates specialization among predictors. The proposed framework is evaluated across regression and classification tasks, including tabular data, image recognition, and large-scale temporally structured video classification. Empirical results show that s-BFN provides strong performance across the evaluated regimes and is competitive with the aggregation baselines considered in the study. The strongest gains are observed in higher-dimensional and temporally structured settings, where aggregation capacity and controlled diversity are jointly beneficial, while simpler tabular regimes exhibit earlier saturation. These results highlight a capacity-complexity-diversity trade-off rather than a strictly monotonic relationship between diversity, ensemble size, and accuracy.
Figures & tables
| Symbol | Meaning |
|---|---|
| Number of observations. | |
| Number of base predictors (hypotheses). | |
| Output dimension; number of classes in classification. | |
| Base predictor , parameterized by . | |
| Class-probability vector produced by base predictor for observation . | |
| Structured prediction vector obtained by concatenating the base-predictor outputs for observation . |
| Geometry | Loss | Prediction (generic) | Prediction (s-BFN) |
|---|---|---|---|
| Euclidean (squared loss) | |||
| Simplex (cross-entropy) |
| Regime | Datasets | Task type | Prediction space |
|---|---|---|---|
| Tabular regression | Air Quality; Appliances Energy | Multivariate regression | Low-dimensional continuous output |
| Image classification | MNIST; Fashion-MNIST; CIFAR-10 | Multi-class classification | 10-class probability simplex |
| Video classification | UCF101 | 101-class action recognition | High-dimensional probability simplex with temporal structure |
| Variant | Centre | Scale | Basis function |
|---|---|---|---|
| Hyperparameter | Values |
|---|---|
| Number of hypotheses ( ) | 2, 5, 10, 20, 35 |
| Neurons per layer ( ) | 20, 50, 200, 2000 |
| Learning rates ( , ) | 0.03, 0.3, 1.0 |
| Initialization scale ( ) | 0.0001, 0.01, 0.1, 1.0 |
| Diversity-control parameter ( ) | 0, 0.1, 0.35, 0.5 |
| Regularization (predictors, ) | 0, 0.0001, 0.01, 0.07 |
| Model | Best mean RMSE | Std. dev. | 1st Q | 3rd Q |
|---|---|---|---|---|
| SVM-RBF | 29.83 | 1.99 | 34.80 | 37.65 |
| Random Forest | 55.66 | 15.47 | 69.00 | 91.55 |
| Gradient Boosting | 55.76 | 38.73 | 93.92 | 151.58 |
| Arithmetic Combiner | 39.19 | 0.15 | 41.75 | 43.93 |
| s-BFN least-squares | 22.46 | 9.14 | 38.98 | 54.71 |
| Model | Best mean RMSE | Std. dev. | 1st Q | 3rd Q |
|---|---|---|---|---|
| SVM-RBF | 104.68 | 1.27 | 107.26 | 109.01 |
| Random Forest | 298.46 | 29.35 | 328.48 | 373.81 |
| Gradient Boosting | 292.08 | 67.26 | 389.10 | 476.89 |
| Arithmetic Combiner | 115.17 | 0.11 | 128.54 | 144.83 |
| s-BFN least-squares | 101.12 | 8.26 | 102.25 | 103.64 |
| Name in figs/tables | Base learners | Combiners |
|---|---|---|
| "Het. Mixed" | SimpleCNN, ResNetTiny1, ResNetTiny2 | s-BFN, MoE, Mean Agg. |
| "Homo. ResNet" | ResNetTiny2 (or ResNetTiny1) | s-BFN, MoE, Mean Agg. |
| "Homo. CNN" | SimpleCNN | s-BFN, MoE, Mean Agg. |
| "Arithmetic Mean" | Same as corresponding ensemble | Accuracy avg. (baseline) |
| MNIST ( ) | Fashion-MNIST ( ) | CIFAR-10 ( ) | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Baseline | W/T/L | W/T/L | W/T/L | ||||||
| SingleBest | +13.72 | 16/0/0 | +19.26 | 16/0/0 | +11.72 | 12/0/0 | 0.0044 | ||
| MeanProb | +10.36 | 16/0/0 | +15.76 | 16/0/0 | +7.59 | 12/0/0 | 0.0044 | ||
| LogitAverage | +9.56 | 16/0/0 | +14.59 | 16/0/0 | +7.09 | 12/0/0 | 0.0044 | ||
| LinearStack | -1.33 | 1/1/14 | 0.0228 | -0.74 | 2/0/14 | 0.0126 | -1.30 | 3/0/9 | 0.3296 |
| MLPStack-Matched | -0.26 | 7/0/9 | 0.8603 | +0.64 | 11/0/5 | 0.1928 | -0.59 | 5/0/7 | 0.4072 |
| Val Acc. | Test Acc. | |||
|---|---|---|---|---|
| 5 | 0.1 | 100 | 0.795 | 0.792 |
| 5 | 0.3 | 100 | 0.827 | 0.822 |
| 5 | 0.6 | 100 | 0.843 | 0.843 |
| 10 | 0.1 | 200 | 0.821 | 0.823 |
| 10 | 0.3 | 200 | 0.855 | 0.843 |
| 10 | 0.6 | 200 | 0.841 | 0.843 |
| Mean | MoE | Fusion | s-BFN | |
|---|---|---|---|---|
| 5 | 0.803 | 0.774 | 0.364 | 0.843 |
| 10 | 0.786 | 0.732 | 0.277 | 0.843 |
| 20 | 0.766 | 0.692 | 0.326 | 0.864 |
| Dataset | Ensemble | s-BFN_Acc | MoE_Acc | Base_Acc | s-BFN | MoE |
|---|---|---|---|---|---|---|
| MNIST | Heterogeneous | 48.47 | 47.56 | 39.65 | 8.82 | 7.91 |
| MNIST | Homogeneous | 32.97 | 34.70 | 30.40 | 2.56 | 4.29 |
| FashionMNIST | Heterogeneous | 69.81 | 63.10 | 54.19 | 15.62 | 8.90 |
| FashionMNIST | Homogeneous | 63.70 | 61.57 | 53.02 | 10.69 | 8.56 |
| CIFAR-10 | Heterogeneous | 49.10 | 47.96 | 39.71 | 9.39 | 8.25 |
| CIFAR-10 | Homogeneous | 32.70 | 33.45 | 30.38 | 2.32 | 3.06 |
| Dataset | Ensemble | Architecture | s-BFN (%) | MoE (%) | Base Avg (%) | Train (s) | Eval (s) | |
|---|---|---|---|---|---|---|---|---|
| MNIST | Het. | Mixed | 5 | 70.45 | 66.48 | 59.34 | – | – |
| MNIST | Homo. | ResNetTiny2 | 5 | 71.66 | 70.03 | 68.20 | – | – |
| Fashion-MNIST | Het. | Mixed | 5 | 83.54 | 82.38 | 75.99 | 38.90 | 2.73 |
| Fashion-MNIST | Homo. | CNN | 10 | 83.01 | 81.02 | 80.25 | 29.50 | 2.10 |
| CIFAR-10 | Het. | Mixed | 10 | 68.94 | 66.99 | 60.28 | 313.10 | 4.22 |
| CIFAR-10 | Homo. | ResNetTiny2 | 5 | 71.72 | 66.27 | 64.33 | 258.30 | 4.05 |
| Dataset | Ensemble | Train (s) | Eval (s) | FLOPs | Params |
|---|---|---|---|---|---|
| FashionMNIST | Heterogeneous | 125.8 | 4.09 | 5.8M | 1.24M |
| FashionMNIST | Homogeneous | 72.2 | 2.98 | 1.7M | 0.80M |
| CIFAR-10 | Heterogeneous | 741.0 | 6.06 | 7.8M | 1.25M |
| CIFAR-10 | Homogeneous | 2712.4 | 6.84 | 14.4M | 7.32M |
| Dataset | Heterogeneous | Homogeneous | |
|---|---|---|---|
| 0.0 | CIFAR-10 | 26.59 | 16.86 |
| 0.5 | CIFAR-10 | 59.84 | 42.89 |
| 0.8 | CIFAR-10 | 60.87 | 38.36 |
| 0.0 | FashionMNIST | 51.25 | 40.25 |
| 0.5 | FashionMNIST | 79.48 | 75.00 |
| 0.8 | FashionMNIST | 78.71 | 75.86 |
| Model | Best mean RMSE | Std. dev. | 1st Q | 3rd Q |
|---|---|---|---|---|
| s-BFN LS | 22.46 | 9.14 | 38.98 | 54.71 |
| 37.32 | 2.92 | 37.94 | 39.66 | |
| 37.28 | 3.03 | 38.72 | 40.87 | |
| 37.35 | 2.51 | 39.02 | 39.67 |
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Entropy | Val Acc. | Test Acc. | ||
|---|---|---|---|---|
| 5 | 0.1 | 3.887 | 0.795 | 0.792 |
| 5 | 0.6 | 4.014 | 0.843 | 0.843 |
| 10 | 0.1 | 4.036 | 0.821 | 0.823 |
| 10 | 0.6 | 4.145 | 0.841 | 0.843 |
| 20 | 0.1 | 4.121 | 0.815 | 0.822 |
| 20 | 0.6 | 4.147 | 0.858 | 0.864 |
| SB | MP | LA | LS | MLP | MoE | NES | NEMA | GES | s-BFN | ||
|---|---|---|---|---|---|---|---|---|---|---|---|
| 5 | 0.0 | 28.91 9.51 | 49.58 17.60 | 51.98 5.27 | 69.78 6.60 | 66.69 7.86 | 75.48 7.28 | 46.36 11.03 | 43.03 11.82 | 49.17 15.99 | 72.27 7.99 |
| 5 | 0.3 | 95.63 1.62 | 96.85 0.41 | 97.10 0.45 | 97.77 0.14 | 97.66 0.14 | 97.12 0.42 | 96.71 0.75 | 95.85 1.83 | 96.43 0.87 | 97.69 0.14 |
| 5 | 0.5 | 96.98 0.77 | 97.48 0.13 | 97.51 0.17 | 98.08 0.17 | 98.04 0.17 | 97.95 0.20 | 97.35 0.67 | 97.14 0.83 | 97.75 0.41 | 98.07 0.23 |
| 5 | 0.8 | 97.22 0.43 | 97.01 0.70 | 96.91 0.77 | 97.96 0.15 | 97.92 0.23 | 97.71 0.39 | 97.23 0.63 | 97.28 0.78 | 97.56 0.46 | 97.91 0.14 |
| 10 | 0.0 | 9.12 1.19 | 16.07 6.10 | 19.49 6.77 | 44.75 6.89 | 41.14 5.12 | 45.85 7.07 | 22.32 9.51 | 19.86 7.35 | 22.47 4.84 | 44.32 5.65 |
| 10 | 0.3 | 94.73 0.65 | 96.38 0.52 | 96.49 0.41 | 97.63 0.19 | 97.54 0.20 | 97.14 0.41 | 96.11 1.52 | 95.85 0.80 | 95.74 0.31 | 97.61 0.13 |
| SB | MP | LA | LS | MLP | MoE | NES | NEMA | GES | s-BFN | ||
|---|---|---|---|---|---|---|---|---|---|---|---|
| 5 | 0.0 | 26.37 8.38 | 32.29 6.95 | 33.86 3.42 | 57.68 1.81 | 55.02 3.93 | 58.76 2.42 | 36.86 5.56 | 36.26 7.28 | 34.24 6.95 | 58.58 2.62 |
| 5 | 0.3 | 66.84 3.46 | 73.67 3.16 | 74.20 2.70 | 79.06 0.54 | 77.37 0.21 | 77.94 1.48 | 74.29 1.09 | 71.77 5.32 | 71.30 2.23 | 79.05 0.17 |
| 5 | 0.5 | 74.85 2.22 | 76.53 0.91 | 76.51 0.75 | 79.86 0.56 | 79.01 0.73 | 78.81 0.40 | 76.41 2.00 | 75.89 2.75 | 77.18 1.17 | 79.81 0.34 |
| 5 | 0.8 | 76.62 1.48 | 76.78 1.13 | 76.72 1.10 | 79.88 0.68 | 78.97 0.78 | 78.34 1.17 | 77.32 1.55 | 76.65 2.44 | 77.40 1.21 | 80.18 0.88 |
| 10 | 0.0 | 8.40 3.07 | 19.54 7.41 | 27.99 4.40 | 55.21 3.20 | 50.25 2.56 | 55.33 2.50 | 27.14 3.66 | 21.48 6.68 | 19.37 3.64 | 54.45 2.96 |
| 10 | 0.3 | 69.16 2.02 | 73.60 1.37 | 74.08 1.05 | 79.19 0.75 | 77.64 0.66 | 78.05 1.39 | 72.64 3.24 | 71.42 5.29 | 73.44 1.00 | 79.08 0.83 |
| SB | MP | LA | LS | MLP | MoE | NES | NEMA | GES | s-BFN | ||
|---|---|---|---|---|---|---|---|---|---|---|---|
| 5 | 0.0 | 19.81 5.62 | 33.78 5.21 | 28.36 2.07 | 39.64 2.23 | 38.71 2.49 | 40.49 2.55 | 29.96 3.79 | 26.68 5.65 | 29.76 6.24 | 40.39 2.33 |
| 5 | 0.5 | 65.36 2.89 | 68.68 1.55 | 69.51 1.66 | 71.91 1.10 | 71.75 1.05 | 71.00 0.94 | 68.20 3.43 | 67.65 2.67 | 69.67 1.46 | 71.80 0.91 |
| 5 | 0.8 | 65.82 4.00 | 68.07 1.43 | 68.25 1.31 | 71.36 1.27 | 71.32 1.16 | 70.49 1.22 | 68.43 2.76 | 68.19 1.82 | 69.78 1.43 | 71.58 1.23 |
| 10 | 0.0 | 10.00 0.00 | 14.34 3.05 | 15.17 0.65 | 22.01 1.56 | 16.75 5.41 | 16.48 5.44 | 12.39 3.87 | 10.21 0.58 | 14.34 3.05 | 16.73 6.18 |
| 10 | 0.5 | 58.27 1.15 | 65.12 0.61 | 65.65 0.77 | 68.90 0.46 | 68.96 0.47 | 67.63 0.43 | 64.78 4.23 | 65.55 1.63 | 66.85 0.76 | 68.89 0.49 |
| 10 | 0.8 | 65.46 1.44 | 67.69 0.95 | 67.88 1.02 | 71.65 0.40 | 71.91 0.38 | 70.26 0.51 | 68.60 3.69 | 69.21 1.25 | 69.86 0.36 | 71.91 0.40 |
Ensemble Learning
Ensemble learning combines multiple predictors to improve accuracy, robustness, calibration, or uncertainty estimates.
See all 302 papers