Bidirectional Information Flow (BIF) - A Sample Efficient Hierarchical Gaussian Process for Bayesian Optimization
Authors: Juan D. Guerra, Thomas Garbay, Numa Dancause, Guillaume Lajoie, Marco Bonizzato
Organizations: Polytechnique Montréal Department of Computer and Software Engineering · Mila - Québec Artificial Intelligence Institute · Polytechnique Montréal Department of Electrical Engineering · Université de Montréal Department of Neuroscience · Université de Montréal Department of Mathematics and Statistics · CIFAR AI Chair
Hierarchical Gaussian Process (H-GP) models divide problems into different subtasks, allowing for different components to address each part, making them well-suited for problems with inherent compositional structure. However, existing H-GP frameworks typically employ one-way information sharing - either top-down or bottom-up - which limits sample efficiency and slows convergence. We propose Bidirectional Information Flow (BIF), which establishes continuous two-way communication. BIF retains the modular structure of hierarchical models - the parent conditions its own posterior on child summaries, treating them as structured priors - while introducing top-down feedback to softly decompose environment observations from the parent into sub-responses. This mutual exchange improves sample efficiency, enables robust training, and allows modular reuse of learned subtask models. We prove analytically the regret of a GP with a learned kernel scales linearly with the mismatch to the true kernel, tightening in the hierarchical case to the sum of child-level errors. Ablation shows removing the downward pathway collapses child R2 by up to 58%. Across synthetic, neurostimulation, and HPO benchmarks, BIF achieves up to 4× higher parent R2 and roughly 100% AUC improvement over vanilla GPBO, and outperforms or is the runner-up in 38/40 tests compared to the state-of-the-art baselines while supporting modular child transfer to novel composite tasks.
Figures & tables
Figure 1: Bidirectional Information Flow framework: Colour coding demonstrates each child’s responsibility for a parent dimension. Arrows indicate the direction of information flow: red for prior map creation (child to parent), and blue for sub-response inference via the contribution system (parent to child). Steps (1–7) summarize Alg. 1 .
Figure 2: Baseline GPBO models vs. BIF UCB on 2D (a–c) and 3D (d–g) synthetic tasks. (a, d) shows all model performance for the 2D and 3D tasks with shaded regions depicting standard error; (b-c, e-g) show BIF child environment reconstructions (green: BIF; orange: Laferrière et al. (15) , black: ground truth). See Section 5 for details. Models selected based on presence of direct sub-task modeling for comparison. Child RO metric is omitted for ease of reading. All results presented in Tables 1 & 2 . Parent Visualizations for Neural Stimulation available in Appendix F .
Dataset
Model
Parent RO ↑
Parent R² ↑
Child RO ↑
Child R² ↑
AUC ↑
Michalewicz
Vanilla GP
97.98 ± 1.52
55.29 ± 1.67
-
-
10,138.74
Deep GP
28.46 ± 7.64
25.19 ± 6.38
-
-
3,356.45
ADD GP
97.27 ± 0.71
75.17 ± 1.80
93.25 ± 2.96
75.74 ± 2.34
18,716.24
Laferrière
97.85 ± 0.71
70.81 ± 1.93
74.79 ± 14.21
18.29 ± 3.71
14,992.09
BIF UCB
98.87 ± 0.60
86.10 ± 0.88
94.25 ± 2.66
61.88 ± 3.25
18,587.90
BIF EI
98.94 ± 0.52
84.69 ± 0.99
94.03 ± 6.25
69.63 ± 2.42
19,226.67
Table 1: Summary of synthetic performance after 100 queries of training. Vanilla GPBO and Deep GP do not contain direct subtask mapping, rendering them unable to do the child RO and R2 tasks.
Dataset
Model
Parent RO ↑
Parent R² ↑
Child RO ↑
Child R² ↑
AUC ↑
Neural Stimulation
Vanilla GP
98.46 ± 0.07
70.13 ± 0.11
-
-
14,119.99
Deep GP
78.18 ± 3.33
69.58 ± 2.61
-
-
11,728.75
ADD GP
86.92 ± 3.22
56.03 ± 1.66
86.61 ± 17.27
49.31 ± 4.05
17,087
Laferrière
96.37 ± 0.11
45.45 ± 0.15
55.34 ± 26.14
17.84 ± 0.27
13,799.99
BIF UCB
81.83 ± 0.25
79.70 ± 0.12
100.00 ± 0.00
63.98 ± 0.15
18,299.35
BIF EI
93.20 ± 2.71
60.59 ± 2.20
88.88 ± 15.66
37.22 ± 3.92
16,566.67
Table 2: Summary of real-world performance after 100 queries of training. Vanilla GPBO and Deep GP do not contain direct subtask mapping, rendering them unable to do the child RO and R2 tasks.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Vanilla
Laferrière
BIF
Deep GP
ADD GP-UCB
Clock Time
0:07
0:57
1:14
5:11
0:15
Appendix
Table 3: Empirical wall clock speeds of all tested models. Optimization done over 100 queries on the Michalewicz dataset. Times given as minutes : seconds.
Figure 3: Hyper Parameter Search Results for Max Seen Normalization. Due to time constraints and the number of hyperparameters, search is run one by one for each parameter. Baseline parameters are kappa 7.5, gamma 3, initialization set size of 3, and 10 training iterations. (a-d) represent hyperparameter search for BIF in the Neural Dataset and (e-h) for Laferrière on the neural dataset. (i-l) shows the hyperparameter search for BIF and (m-p) for Laferrière on the 2D synthetic dataset. Finally, (q-t) present the results for BIF on the 3D dataset while (u-x) present Laferrièree
Normalization
Z-Score
Maximum Observed Division
Dataset
Hyper Parameter
2D
3D
Neural
Subtract
2D
3D
Neural
Subtract
BIF
Kappa κ
9
8
8
7.5
4
7
7
8
Gamma γ
4
5
4
2
2
2
1
5
Child Initialization r
3
2
1
1
5
3
2
1
Training Iterations t
10
20
15
10
5
20
5
20
Appendix
Table 4: Optimal Hyperparameters for all models, datasets, and normalization types. Note that 2D and 3D datasets refer to the synthetic 2D and synthetic 3D datasets.
Figure 4: Parent Environment Reconstructions for Neural Dataset (a-d). True heatmap denotes the ground truth and all other heatmaps correspond to a specific model type. The more the heatmaps resemble the ground truth, the higher the R2 score.
Parent RO
Parent R 2
Average Child RO
Average Child R 2
AUC
Neural Stimulation
↑
↑
↑
↑
Michalewicz
↓
↑
↑
↑
↑
HPO - GAN
↑
HPO - Lasso
↓
↑
↑
Synthetic 3D
↑
Holder 2D
↑
Appendix
Table 5: Paired t-test across 30 repetitions significance difference testing. Results with ↑ indicate significantly higher scores for BIF against the next best method, ↓ indicate significantly lower scores for BIF, and no arrow denotes no significant difference
Figure 5: Visual Diagram for Modularity Experiment. BIF hierarchy are trained on initial datasets with their children being placed into a new parent to boost performance.
Figure 6: Effect of swapping pretrained children on BIF performance. Good pretraining denotes switching to the same child task and bad denotes using the pretrained child on a different task. Baseline performance is regular BIF training as presented in Algorithm 1 .
Figure 7: Results of Nonlinearity Experiments described in Appendix I.1 . We aim to show the inherent biases of the model due to the assumption of linear interactions between submodels. Higher α values signify an increase in nonlinearity as shown in the description of the experiments in Appendix I.1 . The alpha values scale exponentially with a base of 2.
Dataset
Model
Relative Optimum
Parent R2
Child R2
AUC
Subtraction
Vanilla GPBO
99.72
96.91
-
14,110.62
Laferrière
98.31
73.17
7.62
15,467.07
BIF
99.16
93.25
81.65
22,650.28
Appendix
Table 6: Performance on Subtractive interaction task. Note that Vanilla GPBO denotes a non-hierarchical GPBO method consequently removing its ability to represent the child R2 task. Performance metrics are described in Section 5
Figure 8: Performance of Bidirectional Information Flow with Scaling Noise as detailed in Appendix I.2 . Higher values of noise mean higher values of β in the description of the experiment in Appendix I.2 . Relative Optimum is more affected than other metrics with scaling noise, followed by average coefficient of determination for the children, and parent coefficient of determination not significantly affected.
Figure 9: Performance of BIF - UCB on Rosenbrock nonlinearity dataset. α acts as a scaling parameter, where higher values increase the degree of nonlinear interactions shown in the bar graph. Line plots show BIF performance as nonlinearity increases.
No Up Module
No Down Module
Full BIF
Sub 2D
RO
98.43±0.29
98.67±0.34
99.16±0.22
Parent R2
92.63±1.08
92.79±1.62
93.25±0.63
Child R2
74.76±2.55
10.99±1.67
81.65±1.94
AUC
20,251.86
13,813.10
22,650.28
Michalewicz
RO
98.91±0.60
99.89±0.06
98.87±0.60
Parent R2
78.37±3.71
79.24±1.16
86.10±0.88
Appendix
Table 7: Ablation study on the upward and downward information flow modules presented in BIF. No Up Module signifies the removal of the prior maps from children to parent. No Down Module removes the contribution system continually training the children post-initialization.
Figure 10: Performance of BIF on Neurostimulation based on different optimization objectives. Different colours denote the optimal setup to attain the desired goal. RO + AUC represents a combined optimization setup that prioritizes an average score of RO and AUC. κ values represent the optimal kappa for the optimization criteria