The use of collections of independent local Gaussian processes (GP-ind) combined with trust regions is a widely adopted approach for large-scale and high-dimensional Bayesian optimisation (BO). However, this approach suffers from several limitations, including discontinuities at the boundaries of disjoint regions, an inability to capture long-range spatial correlations, potential information loss, and abrupt increases in computational overhead. To address these limitations, this paper introduces a novel BO algorithm, termed BO-pro-c-tr. The proposed method employs a product-of-experts Gaussian process model with uncertainty calibration (GP-pro-c) as its surrogate model and incorporates a one-to-many association strategy for integrating GP-pro-c with trust regions. Empirical results suggest that, when combined with the GP-LCB acquisition function, BO-pro-c-tr achieves an average reduction of 2.6% in simple regret and 34.9% in computational overhead relative to BO algorithms based on independent local GP models coupled with trust regions.
Figure 1: Illustration of the GP-pro-c model with trust regions in BO-pro-c-tr for a one-dimensional optimisation problem. The green and red curves (with shaded uncertainty bands) represent the posterior predictions of two local GP models, while the blue curve (with shaded uncertainty band) denotes the aggregated GP-pro-c prediction. The figure also illustrates the association between two local GP models and three trust regions. The green and red hyper-rectangles represent newly created trust regions centred on the best observations of their respective local models, whereas the grey hyper-rectangle represents an active trust region carried over from a previous optimisation iteration. The dotted vertical lines indicate the centres of the trust regions.
Algorithm
GP model (number of data
Association
Data
Weighting
Acquisition
points per local GP)
of GP-TR
assignment
scheme
function
glo-ei
single global GP
EI
glo-lcb
single global GP
GP-LCB
glo-ts
single global GP
TS
turbo-2
2 independent local GPs
one-to-one
pro-c-ei
product-of-experts GP (200)
ball-tree
entropy
EI
Table 1: Summary of the BO algorithms used in the experiments for evaluating and comparing BO-pro-c-tr, BO-pro-c, TuRBO, and BO-glo.
Function
Number of
Input
Minimiser
Global minimum
variables
domain
x∗
f(x∗)
Ackley
10
[−10,10]10
(0, … , 0)
0
Levy
10
[−10,10]10
(1, … , 1)
0
Rastrigin
10
[−5.12,5.12]10
(0, … , 0)
0
Rosenbrock
10
[−5,10]10
(1, … , 1)
0
Ackley
20
[−10,10]20
(0, … , 0)
0
Table 2: Summary information of the 16 synthetic benchmark functions used in the BO experiments for comparing and evaluating the BO-pro-c-tr, BO-pro-c, TuRBO, and BO-glo algorithms.
Figure 2: Experiments on four synthetic benchmark functions: Ackley, Levy, Rastrigin, and Rosenbrock, with d=10 . All variants of BO-pro-c-tr-ei and BO-pro-c-tr-lcb algorithms achieved better performance than the other algorithms, while incurring only slightly higher computational overhead than TuRBO.
Figure 3: Experiments on four synthetic benchmark functions: Ackley, Levy, Rastrigin, and Rosenbrock, with d=20 . All variants of BO-pro-c-tr-ei and BO-pro-c-tr-lcb algorithms achieved better performance than the other algorithms, while incurring only slightly higher computational overhead than TuRBO.
Figure 4: Experiments on four synthetic benchmark functions: Ackley, Levy, Rastrigin, and Rosenbrock, with d=50 . BO-pro-c-tr-ei and BO-pro-c-tr-lcb algorithms using 400 data points per local GP achieved the best performance while maintaining relatively low computational overheads. Note that TuRBO incurred an abrupt increase in computational overhead during the later stages of optimisation because one of the local GP models accumulated a large number of training data points while the associated trust region had not yet met the threshold criteria for discarding the local GP model and trust region.
Figure 5: Experiments on four synthetic benchmark functions: Ackley, Levy, Rastrigin, and Rosenbrock, with d=100 . BO-pro-c-tr-ei and BO-pro-c-tr-lcb algorithms using 400 data points per local GP achieved the best performance while maintaining lower computational overheads. Note that TuRBO incurred an abrupt increase in computational overhead during the later stages of optimisation because one of the local GP models accumulated a large number of training data points while the associated trust region had not yet met the threshold criteria for discarding the local GP model and trust region.
Figure 6: Experiments on the robot pushing task ( d=14 , B=14 ). The figure compares the optimisation performance and computational overhead of BO-pro-c-tr (pro-c-tr) and TuRBO (turbo-2). BO-pro-c-tr using the EI acquisition function with 400 data points per local GP model (pro-c-tr-ei-400) achieved the highest rewards while incurring only a modest increase in computational overhead relative to TuRBO.
Figure 7: Experiments on the rover trajectory planning task ( d=60 , B=60 ). The figure compares the optimisation performance and computational overhead of BO-pro-c-tr (pro-c-tr) and TuRBO (turbo-2). BO-pro-c-tr using the GP-LCB acquisition function with 200 data points per local GP model (pro-c-tr-lcb-200) achieved the highest rewards, while all BO-pro-c-tr variants maintained lower computational overhead than TuRBO.
Function
BO-pro-c-tr-ei-400
BO-pro-c-tr-lcb-400
BO-pro-c-tr-ts-400
TuRBO
10d Ackley
0.988±0.183
0.987±0.115
1.239±0.253
1.448±0.109
10d Levy
0.992±0.246
1.030±0.151
1.243±0.443
1.243±0.242
10d Rastrigin
3.128±0.625
2.999±0.326
3.519±0.217
3.337±0.413
10d Rosenbrock
5.863±0.151
5.815±0.208
5.032±0.909
4.524±0.401
20d Ackley
1.389±0.181
1.499±0.144
1.573±0.110
1.734±0.107
20d Levy
1.386±0.408
1.397±0.268
1.664±0.410
1.775±0.288
Table 3: Average log-scale simple regrets achieved by BO-pro-c-tr-ei-400, BO-pro-c-tr-lcb-400, BO-pro-c-tr-ts-400, and TuRBO across 16 synthetic benchmark functions with input dimensionalities of 10, 20, 50, and 100.
Function
BO-pro-c-tr-ei-400
BO-pro-c-tr-lcb-400
BO-pro-c-tr-ts-400
TuRBO
10d Ackley
0.333±0.014
0.330±0.013
0.374±0.015
0.185±0.017
10d Levy
0.329±0.021
0.322±0.020
0.375±0.021
0.238±0.026
10d Rastrigin
0.330±0.012
0.327±0.011
0.377±0.010
0.195±0.023
10d Rosenbrock
0.325±0.014
0.327±0.021
0.368±0.025
0.348±0.022
20d Ackley
0.721±0.028
0.716±0.027
0.808±0.021
0.443±0.036
20d Levy
0.694±0.024
0.694±0.029
0.791±0.029
0.586±0.123
Table 4: Average computational overhead per optimisation iteration (seconds) incurred by BO-pro-c-tr-ei-400, BO-pro-c-tr-lcb-400, BO-pro-c-tr-ts-400, and TuRBO across 16 synthetic benchmark functions with input dimensionalities of 10, 20, 50, and 100.