Investigating Composite Modeling in Multi-Objective Bayesian Optimization for Scientific Design
Authors: Ricky Ye, Sahas Goli, Muhammad Ali, Poompol Buathong, Peter I. Frazier
Organizations: Department of Computer Science, Cornell University · Center for Applied Mathematics, Cornell University · School of Operations Research and Information Engineering, Cornell University
Scientific design often requires balancing competing objectives that are expensive to evaluate through experiments or simulations. Multi-objective Bayesian optimization (MOBO) addresses this challenge by using probabilistic models to guide the selection of promising designs. In many applications, tunable inputs map to observable intermediate responses, which are then combined through known functions to compute the final objectives. Prior work on Bayesian optimization of composite functions shows that learning from these intermediate responses, which we call Composite modeling, can substantially improve sampling efficiency in single-objective optimization, compared to methods that condition only on black-box observations of the objective function. The benefits of Composite modeling in MOBO, however, remain understudied. In this work, we study the benefits of Composite modeling in MOBO and investigate the factors driving variation in its gains. We first establish single-objective conditions under which intermediate observations provide no additional predictive information beyond objective values, where composite and black box methods use the same warped Gaussian process prior. Controlled numerical experiments with versions of four MOBO methods across eight synthetic and application-inspired scientific-design benchmarks examine how these conditions relate to optimization gains in the multi-objective setting, where the black box approach models the objective functions with a standard multi-output Gaussian process. Both investigations support the claim that composite modeling tends to help more when the function mapping intermediate responses to final objectives is non-bijective and non-affine.
Figures & tables
Single objective
Multi-objective
Outer mapping g
Bijective?
Final-RG reduction (%)
p
Final-HV gain (%)
p
Identity
Yes
+18.1±29.2
0.29
+0.2±0.9
0.40
Cubic h3
Yes
+17.1±21.1
0.24
+0.9±1.6
0.29
Square h2
No
+99.8±0.1
<0.01
+2.9±1.0
<0.01
Shifted square (h−3)2
No
+99.6±0.5
<0.01
+1.0±0.8
0.12
Three-root cubic
No
+45.3±16.4
0.03
+2.5±1.2
0.03
Table 1: Controlled synthetic experiments. Final regret reductions (single-objective) and final hypervolume gains (multi-objective) for Composite relative to BlackBox-GP, both in percentage (%) (mean ± SE over 20 paired runs). Positive values favor Composite. Bold marks unadjusted p<0.05 (one-sided paired t -tests for improvement).
Low-dimensional
High-dimensional
Benchmark
Method
Final-HV gain (%)
p
Benchmark
Method
Final-HV gain (%)
p
Color formulation Chaouch et al. (2022)
LogEHVI
+13.8±1.6
<0.01
Synthetic 2 Surjanovic and Bingham ()
Sp-STCH
+3.6±1.8
0.038
STCH
+8.7±1.7
<0.01
MORBO
+5.3±2.6
0.036
Synthetic 1 Deb et al. (2002)
LogEHVI
+12.7±0.6
<0.01
Lake management Ward et al. (2015)
Sp-STCH
+8.7±0.7
<0.01
STCH
+7.5±0.4
<0.01
MORBO
+4.1±1.5
0.012
Truss design ( Affine ) Tanabe and Ishibuchi (2020)
LogEHVI
+0.8±0.1
<0.01
Power flow ( Affine ) Kumar et al. (2021)
Sp-STCH
+0.2±0.2
0.17
Table 2: Scientific-design experiments. Final hypervolume gains (%, mean ± SE) for Composite relative to BlackBox-GP over 10 paired trials per benchmark (80 for Synthetic 1). Positive values favor Composite. Bold marks unadjusted p<0.05 (one-sided paired t -tests for improvement). Affine marks affine outer mappings.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Outer mapping g
BlackBox-GP
Composite
Reduction
Final-RG reduction (%)
p
Bijective on the range h attains
Exponential eh
−2.59±0.09
−2.74±0.14
+0.15±0.12
+28.91±19.01
0.11
Identity h
−1.03±0.18
−1.11±0.14
+0.09±0.16
+18.10±29.24
0.29
Cubic h3
+0.09±0.11
+0.01±0.10
+0.08±0.11
+17.10±21.07
0.23
Hyperbolic sine sinhh
+0.24±0.11
+0.16±0.09
+0.08±0.10
+17.00±19.58
0.22
Quintic h5
+0.75±0.15
+0.79±0.08
−0.05±0.16
−11.62±40.30
0.62
Appendix
Table E.1: Controlled single-objective experiments, all 16 outer mappings. Values in BlackBox-GP and Composite columns are the final log10 simple regret from each method; Reduction is their paired difference, so positive favors Composite. Final-RG reduction is that same difference as a percentage, via Eq. ( D.1 ). All values are means ± one standard error over the 20 paired runs, the percentage column by the delta method, and p is a one-sided paired t -test for positive mean reduction. Bold marks significance at the unadjusted 0.05 level.
Outer mapping g1
BlackBox-GP
Composite
Gain
Final-HV gain (%)
p
Bijective on the range h attains
Cubic h3
71.49±11.17
72.12±11.22
+0.63±1.11
+0.88±1.56
0.29
Exponential eh
71.17±11.22
71.58±11.09
+0.41±0.92
+0.58±1.29
0.33
Quintic h5
71.95±11.11
72.21±10.97
+0.26±1.44
+0.36±2.00
0.43
Identity h
70.93±11.15
71.10±11.30
+0.16±0.65
+0.23±0.92
0.40
Hyperbolic sine sinhh
71.46±11.25
71.19±11.21
−0.27±0.53
−0.38±0.75
0.69
Appendix
Table E.2: Controlled two-objective experiments, all 19 distinct settings of g1 . Values in BlackBox-GP and Composite columns are the final hypervolume; Gain is their paired difference, so positive favors Composite. Final-HV gain is that same difference as a percentage, via Eq. ( D.3 ). All values are means ± one standard error over the 20 paired runs, the percentage column by the delta method, and p is a one-sided paired t -test for positive mean gain. The BlackBox-GP and Composite columns have the largest standard errors because each mapping pools four function draws whose hypervolumes differ substantially; that spread cancels in the paired Gain. Bold marks significance at the unadjusted 0.05 level.
Controlled experiments
Scientific-design benchmarks
BoTorch
0.9.5
0.18.0
GPyTorch
1.11
1.15.2
PyTorch
2.11.0
2.12.0
Covariance
ScaleKernel(Matern 5/2) , ARD
RBFKernel , ARD, no outputscale
Lengthscale prior
Gamma(3.0,6.0)
LogNormal(2+21logd,3)
Lengthscale constraint
—
≥0.025
Appendix
Table H.1: Surrogate and software configuration for the LogEHVI and STCH output GPs in each setting. Both fit hyperparameters by fit_gpytorch_mll in double precision with a Standardize outcome transform.
Regime
Method
Initial
Adaptive
Adaptive budget spent as
Low-dimensional
LogEHVI
5
40
sequential, one candidate per step
STCH
5
40
5 weights × 8 evaluations
High-dimensional
Sp-STCH
20
100
5 weights × 20 evaluations
MORBO
20
100
20 batches × 5 candidates
Appendix
Table H.2: Initial design size and adaptive evaluation budget per trial. The high-dimensional regime covers L–A, Lake, and RCM40.
Aerospace Engineering Pennsylvania State University University Park, PA 16802 · Institute of Computational and Data Sciences (ICDS) Pennsylvania State University University Park, PA 16802