Many real-world optimization problems rely on expensive simulations or experiments, making the efficient use of available data essential. Multi-fidelity optimization of high-dimensional black-box functions subject to black-box constraints is increasingly relevant as the cost of objective evaluations continues to rise in applications such as machine learning, engineering, and control. To our knowledge, no existing method simultaneously addresses high-dimensionality, black-box constraints, an arbitrary number of fidelity levels, and non-nested sampling. In this work, we extend the Scalable Constrained Bayesian Optimization method to the multi-fidelity setting, resulting in the MF-SCBO method. The proposed approach is evaluated on standard benchmark functions as well as challenging problems. The experimental results demonstrate that MF-SCBO generally achieves better convergence than both the single-fidelity SCBO and the other multi-fidelity method considered in this high-dimensional and constrained settings.
Figures & tables
xnext∈argmaxx∈Ωa(x)
Algorithm 1 Bayesian Optimization
Figure 1: Multi-fidelity Gaussian process regression on a toy function. Multi-fidelity GP (red) constructed from a small number of high-fidelity data points and a larger set of low-fidelity data points (red). The nested case is considered, where the high-fidelity samples are a subset of the low-fidelity points. The standard GP built using only high-fidelity data is displayed in blue. The 95% confidence intervals predicted by the GPs are shown as shaded regions.
Figure 2: MF-SCBO representation for two fidelity levels (green for the low fidelity and blue for the high fidelity): (1) Initial points are taken from Ω (represented by crosses for the low fidelity and circle for the high fidelity), and the trust region is defined (grid with central black circle). (2) The different Gaussian process models are fitted. (3a) r random points (represented by triangles) are sampled in the trust region. (3b) A batch of q realizations is computed and selected via Thompson sampling on these r designs. For each realization, the x axis corresponds to the x points in Ω and is discretized by the {xdiscr,i}i=1r (the triangles) chosen previously. The y axis shows the realizations. For simplicity, a single graph is shown for the objective and the constraints. For each realization, we keep the best point (completely filled triangle). (3c) For the q best points {xnext,i}i=1q , the fidelity level is choose by computing ( 16 ). (4) The real value of the objectives and the constraints are computed and added to the previous observations. (5) The trust region is readjusted by modifying its center and radius.
Figure 3: Convergence curves for the synthetic experiments: Hartmann6D, Borehole8D, Ackley20D, Rosenbrock40D, and Rosenbrock100D. Blue corresponds to the MF-SCBO- predicted method, green to MF-SCBO- evaluated , black to SCBO, and red to MFMFES. The first row shows the best value as a function of high-fidelity evaluations, the second row shows the best value versus total cost, and the third row displays the number of points used at each fidelity level s for each iteration over the total cost.
Table 1: Parameters and optimization results for the synthetic functions: Hartmann6D, Borehole8D, Ackley20D, Rosenbrock40D, and Rosenbrock100. The table lists the problem dimension, number of fidelities, associated cost, batch size, and number of initial points for each fidelity. The mean of the best value after 4000 objective function evaluations is reported for each method, along with the 1.96× standard deviation. The best value across methods is highlighted in bold.
Figure 4: Convergence curves for the realistic experiments: Solar2, Solar7 and Airfoil. Blue corresponds to the MF-SCBO- predicted method, green to MF-SCBO- evaluated , black to SCBO, and red to MFMFES. The first row shows the best value as a function of high-fidelity evaluations, the second row shows the best value versus total cost, and the third row displays the number of points used at each fidelity level s for each iteration over the total cost.
Table 2: Parameters and optimization results for the realistic functions: Solar2, Solar7 and Airfoil. The table lists the problem dimension, number of fidelities, associated cost, batch size, and number of initial points for each fidelity. The mean of the best value after 4000 objective function evaluations is reported for each method, along with the 1.96× standard deviation. The best value across methods is highlighted in bold.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Two dimensional Ackley function at two different fidelity levels, 0 for the low fidelity function and 1 for the high fidelity function.
Figure 6: Two dimensional Rosenbrok function at three different fidelity levels, from 0 (lowest) to 2 (highest).
Instance
# of variables
# of constraints
# of stoch. outpus
cont.
discr.
fixed
simu.
a priori
(obj. or constr.)
solar2
12
2
(x6,x11)=(2650,36)
7
5
4
solar7
6
1
x4=40
4
2
3
Appendix
Table 3: Characteristics of the solar2 and solar7 instances, including the number of variables (continuous and discrete), the fixed values of the discrete variables, the number of constraints (simulated and a priori), and the number of stochastic outputs.
Figure 7: Illustration of the PARSEC airfoil parameterization.
Coefficient
Bound
Description
x1
[0.005,0.06]
Leading-edge radius
x2
[0.25,0.5]
Upper crest position in horizontal coordinates
x3
[0.05,0.15]
Upper crest position in vertical coordinates
x4
[−1,−0.4]
Upper crest curvature
x5
[0.35,0.5]
Lower crest position in horizontal coordinates
x6
[−0.12,−0.04]
Lower crest position in vertical coordinates
Appendix
Table 4: Description of the shape variables for the PARSEC parameterization of airfoils. The trailing-edge thickness, x9 , is not optimized and is fixed to 0 .
Index
Reynolds
Mach
Angle of attack (deg.)
Weights ωi
1
6.3×106
0.5
2
0.4
2
6.3×106
0.55
2.2
0.2
3
6.3×106
0.5
2.5
0.2
4
6.3×106
0.55
2
0.2
Appendix
Table 5: Operating conditions used in the multi-point design strategy, together with the associated weights ωi .
Figure 8: Convergence curves for different number of fidelity level for Hartmann6D and Rosenbrock100D. Green, blue and red correspond respectively to 2 , 3 et 4 number of fidelity levels. Circles correspond to the evaluated method and squares to the predicted method.
Figure 9: Convergence curves for different batch size for Hartmann6D, Borehole8D, Ackley20D and Rosenbrock100D. Green, blue and red correspond respectively to 4 , 8 et 16 batch sizes. Circles correspond to the evaluated method and squares to the predicted method.
Figure 10: Convergence curves for different costs for Hartmann6D, Borehole8D, Ackley20D and Rosenbrock100D. Green and blue correspond respectively to (1,20) and (1,50) costs for two fidelities and (1,10,20) , (1,20,50) for three fidelities. Circles correspond to the evaluated method and squares to the predicted method.
Black-box optimization is a ubiquitous problem in science and engineering, often dealing with expensive objective functions with cheaper lower-fidelity proxies available. Multi-fidelity Bayesian optimization (MF-BO) is a principled approach to this problem, leveraging correlations across different fidelities when querying the objective. However, for many important MF-BO tasks, the true highest-fidelity function is prohibitively expensive to be part of the optimization loop. Nevertheless, practitioners often have gold standard data (observations of the highest-fidelity function) obtained from previous experiments that might provide information for the current task. For instance, in molecular optimization, chemists often pick the top-k candidate molecules using various computer simulations, and later reveal their true objective function values. In this work, we demonstrate the suboptimality of standard MF-BO algorithms in the real-world scenarios above, even under ideal assumptions. Next, we mitigate this problem by incorporating historical high-fidelity data accompanied by task descriptors---which can be explicitly given or extracted from unstructured metadata. We demonstrate the effectiveness of our methods on synthetic functions, as well as real-world problems in chemistry and hyperparameter optimization.
Gustavo Sutter, Hao Wang, Luis Ricardez-Sandoval +2
Cheriton School of Computer Science, University of Waterloo, Waterloo, ON, Canada · Vector Institute, Toronto, ON, Canada · Department of Chemical Engineering, University of Waterloo, Waterloo, ON, Canada +2
Optimizing industrial process flowsheets is often computationally prohibitive due to the high cost of rigorous simulations and the curse of dimensionality inherent in complex design spaces. To address these challenges, we present a reduced-space multi-fidelity Bayesian optimization (RS-MFBO) framework designed for high-dimensional, expensive black-box functions. The approach integrates Global Sensitivity Analysis (GSA) for dimensionality reduction with a fidelity-augmented Gaussian process that captures correlations between low-cost approximations and expensive high-fidelity evaluations. A cost-aware acquisition strategy, augmented with cooldown and promotion mechanisms, adaptively guides the allocation of samples across fidelities. The framework is validated on two distinct industrial process simulators: a plasmid DNA bioprocess in SuperPro Designer and a green fuel synthesis plant in Aspen HYSYS. Results across diverse economic and physical objectives demonstrate that the proposed method substantially reduces the number of high-fidelity simulator evaluations while maintaining competitive optimization performance compared to single-fidelity baselines. These results highlight RS-MFBO as a scalable, simulator-agnostic approach for cost-constrained black-box optimization.
Niki Triantafyllou, Andrea Bernardi, Maria M. Papathanasiou
Sargent Centre for Process Systems Engineering, Department of Chemical Engineering, Imperial College London, UK
Bayesian optimization (BO) for high-dimensional constrained problems remains a significant challenge due to the curse of dimensionality. We propose Local Constrained Bayesian Optimization (LCBO), a novel framework tailored for such settings. Unlike trust-region methods that are prone to premature shrinking when confronting tight or complex constraints, LCBO leverages the differentiable landscape of constraint-penalized surrogates to alternate between rapid local descent and uncertainty-driven exploration. Theoretically, we prove that LCBO achieves a convergence rate for the Karush-Kuhn-Tucker (KKT) residual that depends polynomially on the dimension d for common kernels under mild assumptions, offering a rigorous alternative to global BO where regret bounds typically scale exponentially. Extensive evaluations on high-dimensional benchmarks (up to 100D) demonstrate that LCBO consistently outperforms state-of-the-art baselines.
Jing Jingzhe, Fan Zheyi, Szu Hui Ng +1
State Key Laboratory of Mathematical Sciences, Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing, China · School of Mathematical Sciences, University of Chinese Academy of Sciences, Beijing, China · Department of Industrial Systems Engineering & Management, National University of Singapore, Singapore.