The use of collections of independent local Gaussian processes (GP-ind) combined with trust regions is a widely adopted approach for large-scale and high-dimensional Bayesian optimisation (BO). However, this approach suffers from several limitations, including discontinuities at the boundaries of disjoint regions, an inability to capture long-range spatial correlations, potential information loss, and abrupt increases in computational overhead. To address these limitations, this paper introduces a novel BO algorithm, termed BO-pro-c-tr. The proposed method employs a product-of-experts Gaussian process model with uncertainty calibration (GP-pro-c) as its surrogate model and incorporates a one-to-many association strategy for integrating GP-pro-c with trust regions. Empirical results suggest that, when combined with the GP-LCB acquisition function, BO-pro-c-tr achieves an average reduction of 2.6% in simple regret and 34.9% in computational overhead relative to BO algorithms based on independent local GP models coupled with trust regions.
Figure 1: Illustration of the GP-pro-c model with trust regions in BO-pro-c-tr for a one-dimensional optimisation problem. The green and red curves (with shaded uncertainty bands) represent the posterior predictions of two local GP models, while the blue curve (with shaded uncertainty band) denotes the aggregated GP-pro-c prediction. The figure also illustrates the association between two local GP models and three trust regions. The green and red hyper-rectangles represent newly created trust regions centred on the best observations of their respective local models, whereas the grey hyper-rectangle represents an active trust region carried over from a previous optimisation iteration. The dotted vertical lines indicate the centres of the trust regions.
Algorithm
GP model (number of data
Association
Data
Weighting
Acquisition
points per local GP)
of GP-TR
assignment
scheme
function
glo-ei
single global GP
EI
glo-lcb
single global GP
GP-LCB
glo-ts
single global GP
TS
turbo-2
2 independent local GPs
one-to-one
pro-c-ei
product-of-experts GP (200)
ball-tree
entropy
EI
Table 1: Summary of the BO algorithms used in the experiments for evaluating and comparing BO-pro-c-tr, BO-pro-c, TuRBO, and BO-glo.
Function
Number of
Input
Minimiser
Global minimum
variables
domain
x∗
f(x∗)
Ackley
10
[−10,10]10
(0, … , 0)
0
Levy
10
[−10,10]10
(1, … , 1)
0
Rastrigin
10
[−5.12,5.12]10
(0, … , 0)
0
Rosenbrock
10
[−5,10]10
(1, … , 1)
0
Ackley
20
[−10,10]20
(0, … , 0)
0
Table 2: Summary information of the 16 synthetic benchmark functions used in the BO experiments for comparing and evaluating the BO-pro-c-tr, BO-pro-c, TuRBO, and BO-glo algorithms.
Figure 2: Experiments on four synthetic benchmark functions: Ackley, Levy, Rastrigin, and Rosenbrock, with d=10 . All variants of BO-pro-c-tr-ei and BO-pro-c-tr-lcb algorithms achieved better performance than the other algorithms, while incurring only slightly higher computational overhead than TuRBO.
Figure 3: Experiments on four synthetic benchmark functions: Ackley, Levy, Rastrigin, and Rosenbrock, with d=20 . All variants of BO-pro-c-tr-ei and BO-pro-c-tr-lcb algorithms achieved better performance than the other algorithms, while incurring only slightly higher computational overhead than TuRBO.
Figure 4: Experiments on four synthetic benchmark functions: Ackley, Levy, Rastrigin, and Rosenbrock, with d=50 . BO-pro-c-tr-ei and BO-pro-c-tr-lcb algorithms using 400 data points per local GP achieved the best performance while maintaining relatively low computational overheads. Note that TuRBO incurred an abrupt increase in computational overhead during the later stages of optimisation because one of the local GP models accumulated a large number of training data points while the associated trust region had not yet met the threshold criteria for discarding the local GP model and trust region.
Figure 5: Experiments on four synthetic benchmark functions: Ackley, Levy, Rastrigin, and Rosenbrock, with d=100 . BO-pro-c-tr-ei and BO-pro-c-tr-lcb algorithms using 400 data points per local GP achieved the best performance while maintaining lower computational overheads. Note that TuRBO incurred an abrupt increase in computational overhead during the later stages of optimisation because one of the local GP models accumulated a large number of training data points while the associated trust region had not yet met the threshold criteria for discarding the local GP model and trust region.
Figure 6: Experiments on the robot pushing task ( d=14 , B=14 ). The figure compares the optimisation performance and computational overhead of BO-pro-c-tr (pro-c-tr) and TuRBO (turbo-2). BO-pro-c-tr using the EI acquisition function with 400 data points per local GP model (pro-c-tr-ei-400) achieved the highest rewards while incurring only a modest increase in computational overhead relative to TuRBO.
Figure 7: Experiments on the rover trajectory planning task ( d=60 , B=60 ). The figure compares the optimisation performance and computational overhead of BO-pro-c-tr (pro-c-tr) and TuRBO (turbo-2). BO-pro-c-tr using the GP-LCB acquisition function with 200 data points per local GP model (pro-c-tr-lcb-200) achieved the highest rewards, while all BO-pro-c-tr variants maintained lower computational overhead than TuRBO.
Function
BO-pro-c-tr-ei-400
BO-pro-c-tr-lcb-400
BO-pro-c-tr-ts-400
TuRBO
10d Ackley
0.988±0.183
0.987±0.115
1.239±0.253
1.448±0.109
10d Levy
0.992±0.246
1.030±0.151
1.243±0.443
1.243±0.242
10d Rastrigin
3.128±0.625
2.999±0.326
3.519±0.217
3.337±0.413
10d Rosenbrock
5.863±0.151
5.815±0.208
5.032±0.909
4.524±0.401
20d Ackley
1.389±0.181
1.499±0.144
1.573±0.110
1.734±0.107
20d Levy
1.386±0.408
1.397±0.268
1.664±0.410
1.775±0.288
Table 3: Average log-scale simple regrets achieved by BO-pro-c-tr-ei-400, BO-pro-c-tr-lcb-400, BO-pro-c-tr-ts-400, and TuRBO across 16 synthetic benchmark functions with input dimensionalities of 10, 20, 50, and 100.
Function
BO-pro-c-tr-ei-400
BO-pro-c-tr-lcb-400
BO-pro-c-tr-ts-400
TuRBO
10d Ackley
0.333±0.014
0.330±0.013
0.374±0.015
0.185±0.017
10d Levy
0.329±0.021
0.322±0.020
0.375±0.021
0.238±0.026
10d Rastrigin
0.330±0.012
0.327±0.011
0.377±0.010
0.195±0.023
10d Rosenbrock
0.325±0.014
0.327±0.021
0.368±0.025
0.348±0.022
20d Ackley
0.721±0.028
0.716±0.027
0.808±0.021
0.443±0.036
20d Levy
0.694±0.024
0.694±0.029
0.791±0.029
0.586±0.123
Table 4: Average computational overhead per optimisation iteration (seconds) incurred by BO-pro-c-tr-ei-400, BO-pro-c-tr-lcb-400, BO-pro-c-tr-ts-400, and TuRBO across 16 synthetic benchmark functions with input dimensionalities of 10, 20, 50, and 100.
Bayesian optimisation (BO) typically relies on a single global Gaussian process (GP) model as its surrogate model. However, GP regression has cubic computational complexity in the number of training data points, limiting its applicability to large-scale optimisation problems. The product-of-experts Gaussian process model with uncertainty calibration (GP-pro-c) mitigates this limitation by combining multiple local GP experts, enabling improved uncertainty quantification, reduced computational cost, and preservation of global correlations. Despite these desirable properties, the use of GP-pro-c in BO has not been thoroughly studied. This paper introduces BO-pro-c, a Bayesian optimisation algorithm that uses GP-pro-c as its surrogate model, and evaluate its performance across a diverse range of BO settings. Experimental results suggest that BO-pro-c maintains competitive optimisation performance while achieving a 0.9% reduction in simple regret and a 39.4% reduction in computational overhead relative to a BO algorithm based on a single global GP model.
Gaussian process (GP) regression with a single global GP (GP-glo) incurs cubic computational cost, limiting scalability to large datasets. Product-of-experts GP models (GP-pro), which combine local GP models to capture global correlations, alleviate this computational burden. However, training local experts on disjoint data subsets can lead to overestimated posterior variances. We propose GP-pro-c, a product-of-experts GP model that calibrates these variances using an information-based method. The method exploits the monotonicity and submodularity of information gain in GPs to define a calibration ratio that reduces the posterior variance of individual local GP models. We evaluate GP-pro-c using negative log-likelihood (NLL), root mean squared error (RMSE), and expected normalised calibration error (ENCE). Experiments on four synthetic functions and six regression datasets show that GP-pro-c achieves average reductions of 2.3% in NLL and 12.0% in ENCE compared with the uncalibrated GP-pro model. The proposed method mitigates posterior variance overestimation while maintaining predictive accuracy and reducing computational complexity. GP-pro-c provides a promising approach for uncertainty estimation in scalable GP models and may serve as a useful surrogate model for Bayesian optimisation with high-dimensional and large-scale data.
Yean Hoon Ong, Paolo Barucca, Wei Pan +1
University College London, London, UK · The University of Manchester, Manchester, UK
Bayesian optimization (BO) is a powerful paradigm for optimizing expensive black-box functions. Traditional BO methods typically rely on separate hand-crafted acquisition functions and surrogate models for the underlying function, and often operate in a myopic manner. In this paper, we propose a novel direct regret optimization approach that jointly learns the optimal model and non-myopic acquisition by distilling from a set of candidate models and acquisitions, and explicitly targets minimizing the multi-step regret. Our framework leverages an ensemble of Gaussian Processes (GPs) with varying hyperparameters to generate simulated BO trajectories, each guided by an acquisition function drawn from a pool of conventional choices and terminated by a Bayesian early stop criterion. These trajectories train an end-to-end Decision Transformer that selects the next query so as to improve the ultimate objective, following a dense training sparse learning paradigm: the transformer is trained on abundant simulated data, while a limited number of real evaluations refine the GPs online. On synthetic and real-world benchmarks, our method attains the best or near-best final simple regret against standard, lookahead, trust-region and amortized BO baselines, with the largest gains in high-dimensional settings. Ablations attribute the gains jointly to region-of-interest filtering and the learned policy, and matched-budget comparisons against explicit two-step lookahead acquisitions show that the advantage is not shared by lookahead alone.
Fengxue Zhang, Yuxin Chen
Department of Computer Science University of Chicago Chicago, IL 60637