Investigating Composite Modeling in Multi-Objective Bayesian Optimization for Scientific Design
Authors: Ricky Ye, Sahas Goli, Muhammad Ali, Poompol Buathong, Peter I. Frazier
Organizations: Department of Computer Science, Cornell University · Center for Applied Mathematics, Cornell University · School of Operations Research and Information Engineering, Cornell University
Scientific design often requires balancing competing objectives that are expensive to evaluate through experiments or simulations. Multi-objective Bayesian optimization (MOBO) addresses this challenge by using probabilistic models to guide the selection of promising designs. In many applications, tunable inputs map to observable intermediate responses, which are then combined through known functions to compute the final objectives. Prior work on Bayesian optimization of composite functions shows that learning from these intermediate responses, which we call Composite modeling, can substantially improve sampling efficiency in single-objective optimization, compared to methods that condition only on black-box observations of the objective function. The benefits of Composite modeling in MOBO, however, remain understudied. In this work, we study the benefits of Composite modeling in MOBO and investigate the factors driving variation in its gains. We first establish single-objective conditions under which intermediate observations provide no additional predictive information beyond objective values, where composite and black box methods use the same warped Gaussian process prior. Controlled numerical experiments with versions of four MOBO methods across eight synthetic and application-inspired scientific-design benchmarks examine how these conditions relate to optimization gains in the multi-objective setting, where the black box approach models the objective functions with a standard multi-output Gaussian process. Both investigations support the claim that composite modeling tends to help more when the function mapping intermediate responses to final objectives is non-bijective and non-affine.
Figures & tables
Single objective
Multi-objective
Outer mapping g
Bijective?
Final-RG reduction (%)
p
Final-HV gain (%)
p
Identity
Yes
+18.1±29.2
0.29
+0.2±0.9
0.40
Cubic h3
Yes
+17.1±21.1
0.24
+0.9±1.6
0.29
Square h2
No
+99.8±0.1
<0.01
+2.9±1.0
<0.01
Shifted square (h−3)2
No
+99.6±0.5
<0.01
+1.0±0.8
0.12
Three-root cubic
No
+45.3±16.4
0.03
+2.5±1.2
0.03
Table 1: Controlled synthetic experiments. Final regret reductions (single-objective) and final hypervolume gains (multi-objective) for Composite relative to BlackBox-GP, both in percentage (%) (mean ± SE over 20 paired runs). Positive values favor Composite. Bold marks unadjusted p<0.05 (one-sided paired t -tests for improvement).
Low-dimensional
High-dimensional
Benchmark
Method
Final-HV gain (%)
p
Benchmark
Method
Final-HV gain (%)
p
Color formulation Chaouch et al. (2022)
LogEHVI
+13.8±1.6
<0.01
Synthetic 2 Surjanovic and Bingham ()
Sp-STCH
+3.6±1.8
0.038
STCH
+8.7±1.7
<0.01
MORBO
+5.3±2.6
0.036
Synthetic 1 Deb et al. (2002)
LogEHVI
+12.7±0.6
<0.01
Lake management Ward et al. (2015)
Sp-STCH
+8.7±0.7
<0.01
STCH
+7.5±0.4
<0.01
MORBO
+4.1±1.5
0.012
Truss design ( Affine ) Tanabe and Ishibuchi (2020)
LogEHVI
+0.8±0.1
<0.01
Power flow ( Affine ) Kumar et al. (2021)
Sp-STCH
+0.2±0.2
0.17
Table 2: Scientific-design experiments. Final hypervolume gains (%, mean ± SE) for Composite relative to BlackBox-GP over 10 paired trials per benchmark (80 for Synthetic 1). Positive values favor Composite. Bold marks unadjusted p<0.05 (one-sided paired t -tests for improvement). Affine marks affine outer mappings.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Outer mapping g
BlackBox-GP
Composite
Reduction
Final-RG reduction (%)
p
Bijective on the range h attains
Exponential eh
−2.59±0.09
−2.74±0.14
+0.15±0.12
+28.91±19.01
0.11
Identity h
−1.03±0.18
−1.11±0.14
+0.09±0.16
+18.10±29.24
0.29
Cubic h3
+0.09±0.11
+0.01±0.10
+0.08±0.11
+17.10±21.07
0.23
Hyperbolic sine sinhh
+0.24±0.11
+0.16±0.09
+0.08±0.10
+17.00±19.58
0.22
Quintic h5
+0.75±0.15
+0.79±0.08
−0.05±0.16
−11.62±40.30
0.62
Appendix
Table E.1: Controlled single-objective experiments, all 16 outer mappings. Values in BlackBox-GP and Composite columns are the final log10 simple regret from each method; Reduction is their paired difference, so positive favors Composite. Final-RG reduction is that same difference as a percentage, via Eq. ( D.1 ). All values are means ± one standard error over the 20 paired runs, the percentage column by the delta method, and p is a one-sided paired t -test for positive mean reduction. Bold marks significance at the unadjusted 0.05 level.
Outer mapping g1
BlackBox-GP
Composite
Gain
Final-HV gain (%)
p
Bijective on the range h attains
Cubic h3
71.49±11.17
72.12±11.22
+0.63±1.11
+0.88±1.56
0.29
Exponential eh
71.17±11.22
71.58±11.09
+0.41±0.92
+0.58±1.29
0.33
Quintic h5
71.95±11.11
72.21±10.97
+0.26±1.44
+0.36±2.00
0.43
Identity h
70.93±11.15
71.10±11.30
+0.16±0.65
+0.23±0.92
0.40
Hyperbolic sine sinhh
71.46±11.25
71.19±11.21
−0.27±0.53
−0.38±0.75
0.69
Appendix
Table E.2: Controlled two-objective experiments, all 19 distinct settings of g1 . Values in BlackBox-GP and Composite columns are the final hypervolume; Gain is their paired difference, so positive favors Composite. Final-HV gain is that same difference as a percentage, via Eq. ( D.3 ). All values are means ± one standard error over the 20 paired runs, the percentage column by the delta method, and p is a one-sided paired t -test for positive mean gain. The BlackBox-GP and Composite columns have the largest standard errors because each mapping pools four function draws whose hypervolumes differ substantially; that spread cancels in the paired Gain. Bold marks significance at the unadjusted 0.05 level.
Controlled experiments
Scientific-design benchmarks
BoTorch
0.9.5
0.18.0
GPyTorch
1.11
1.15.2
PyTorch
2.11.0
2.12.0
Covariance
ScaleKernel(Matern 5/2) , ARD
RBFKernel , ARD, no outputscale
Lengthscale prior
Gamma(3.0,6.0)
LogNormal(2+21logd,3)
Lengthscale constraint
—
≥0.025
Appendix
Table H.1: Surrogate and software configuration for the LogEHVI and STCH output GPs in each setting. Both fit hyperparameters by fit_gpytorch_mll in double precision with a Standardize outcome transform.
Regime
Method
Initial
Adaptive
Adaptive budget spent as
Low-dimensional
LogEHVI
5
40
sequential, one candidate per step
STCH
5
40
5 weights × 8 evaluations
High-dimensional
Sp-STCH
20
100
5 weights × 20 evaluations
MORBO
20
100
20 batches × 5 candidates
Appendix
Table H.2: Initial design size and adaptive evaluation budget per trial. The high-dimensional regime covers L–A, Lake, and RCM40.
Multiobjective Bayesian optimization (MOBO) with Gaussian process (GP) surrogates is a sample efficient approach to solving multiobjective optimization problems. In MOBO, a Bayesian decision theoretic acquisition function guides the adaptive selection of new candidate inputs, on which objectives and constraints are evaluated to update the surrogate model sequentially. Existing approaches maintain independent GP models for the objectives and constraints, with new observations evaluating all objectives and constraints in a coupled fashion. However, the objectives and constraints often contain inherent correlations which, if exploited, can enable decoupled evaluations where only a subset of them are evaluated at each round. We present a new approach that leverages a multitask GP model to jointly learn all objectives and constraints, and propose a total correlation metric that enables identifying an optimal subset of objectives and constraints to be evaluated at every round, even under uniform evaluation costs. Theoretically, we show that our acquisition policy is asymptotically consistent despite decoupling and that our proposed decoupled subset selection rule maximizes the expected posterior entropy reduction about unevaluated tasks under mild conditions. Empirically, we show that our approach outperforms coupled and decoupled baselines in the state of the art.
Ashwin Renganathan, Peter Bachman
Aerospace Engineering Pennsylvania State University University Park, PA 16802 · Institute of Computational and Data Sciences (ICDS) Pennsylvania State University University Park, PA 16802
Multi-objective Bayesian optimization (MOBO) is effective in identifying the Pareto fronts for expensive black-box problems. However, most current MOBO approaches are limited to low-dimensional decision space due to its exponential sampling complexity. This paper presents decision variable interaction analysis-based MOBO, ViaMOBO, a generic framework for expensive multi-objective problems with high-dimensional decision space. The key idea of ViaMOBO is that it utilizes a variable interaction analysis model to determine whether the decision space can be completely or partially divided, and then performs local Bayesian optimization in the divided decision subspaces. Through the variable analysis model, it can be derived whether the objectives in black-box problems are separable, partially separable, or non-separable based on the potential independent or interdependent relationships among decision variables without any strong assumptions. We compare ViaMOBO with the state-of-the-art MOBO methods on both synthetic and real-world benchmarks. The experimental results demonstrate that ViaMOBO outperforms other related MOBO baselines in approximating the Pareto front of high-dimensional expensive multi-objective problems.
Hongyan Wang, Jiayu Huang, Haotian Zheng +6
School of Vehicle and Mobility Tsinghua University Beijing, China · Faculty of Engineering The Hong Kong Polytechnic University Hong Kong, China
Multi-objective Bayesian optimization (MOBO) maximizes M expensive black-box objectives with the goal of approximating the Pareto set, whose objective vectors cannot be improved in one component without sacrificing another. A common quality measure is hypervolume, the volume in objective space dominated by these vectors relative to a reference point. Expected hypervolume improvement (EHVI) directly targets this metric and is one-step Bayes-optimal, but its standard computation scales exponentially with M, making it prohibitively expensive for many-objective problems. We introduce random max-margin multi-objective Bayesian optimization (RaMMOBO), whose acquisition function lower-bounds EHVI while requiring only linear cost in M to evaluate. RaMMOBO measures the volume of a max-margin box contained in a candidate's hypervolume-improvement region and averages over random reweightings of the objective space, yielding a utility that is strictly Pareto-monotone on hypervolume-improving outcomes. It naturally handles noisy observations and different probabilistic surrogates and supports efficient gradient-based optimization. Across synthetic and real-world benchmarks, RaMMOBO matches or outperforms state-of-the-art methods, with largest gains at higher M, with acquisition optimization up to two orders of magnitude faster than qLogNEHVI. Our implementation is available at https://anonymous.4open.science/r/rammobo.
Thomas Wedenig, Paul Plantosar, Robert Peharz
Institute of Machine Learning and Neural Computation, TU Graz, Austria · Graz Center for Machine Learning (GraML), Austria