Bayesian Optimization for Mixed-Variable Problems in the Natural Sciences

Yuhao Zhang; Ti John; Matthias Stosiek; Patrick Rinke

Parameter Type	Random Variable	Continuous Parameter
Binary	$Z\sim B\bigl(b(\theta)\bigr)$	$\theta\in[0,1]$
Integer	$Z=\lfloor\theta\rfloor+B\bigl(\mathcal{I}(\theta)\bigr)$	$\theta\in[0,I-1]$
Discrete	$Z=d_{i}+(d_{i+1}-d_{i})B\bigl(\mathcal{D}(\theta)\bigr)$	$\theta\in\left[d_{1},d_{2},...,d_{D}\right]$
Categorical	$Z\sim\text{CAT}\bigl(C(\boldsymbol{\theta})\bigr),\quad\boldsymbol{\theta}=(\theta^{(1)},\ldots,\theta^{(C)})$	$\theta\in\Delta^{C-1}$

Table 1: Summary of probabilistic reparameterizations for different variable types. We denote the

(C-1)

-simplex as

\Delta^{C-1}

.

Figure 1: Synthetic BS benchmark problems within a function space of dimensionality and variable type where the fractions indicate the proportions of dimensions being continuous (C), integer (I) or discrete (D).

Figure 2: PR using EI with the subtracted fitted noise hyperparameter (left, a–c) shows repeated sampling during the third iteration (c), while PR with the EI penalty term (right, d–f) selects the next maximum (f) when the penalty is applied, avoiding redundant evaluations. Panels a–c and d–f correspond to iterations 1–3 for the subtraction and penalty methods, respectively.

Figure 3: Plot showing the mean ranks in terms of composite score (as defined in Eg. 7 ) of all models across each dimension of the BS benchmark function.

Figure 4: Histogram illustrating absolute convergence performance, measured as the percentage of converged runs aggregated over all 20 BS benchmark instances (10 runs per model per instance, each with different Sobol initializations), for all models (excluding KR) using the EI AF. Convergence defined under our medium tolerance criterion counts are pooled across all 20 instances and evaluated under relative budget constraints of 25%, 50%, 75%, and 100%, where each percentage corresponds to the respective fraction of the maximum iterations for each BS variant.

Figure 5: Mean convergence plots of specific chosen models on the Chemistry a) and Actuator b) benchmark functions.

Figure 6: Convergence plots of the BOSS_on_gam models on the DUST1 and DUST2 benchmark functions using only the penalty method (Red for the EI AF & Green for the LCB AF) versus the penalty + modified AF (mAF) approach (Blue & Magenta). Sobol sampling is shown in black, and the RF model in Maroon and Orange. The bold curves are the mean model convergences from their 10 different color-coded Sobol point initiated runs.

Figure S1: Effect of kernel rounding trick method by Eduardo et al. [ 19 ] on otherwise continuous GP posterior

Figure S2: Two dimensional Butternut Squash function with both axis continuous as shown as contour plot (Left) and surface plot (Right).

Function	Init. Points	Iter. Points	Total Points
2D Butternut Squash	5	35	40
3D Butternut Squash	10	80	90
4D Butternut Squash	20	100	120
5D Butternut Squash	40	160	200
6D Butternut Squash	60	220	280
Actuator	10	80	90

Table S1: Table showing the number of initial sobol points, iterative points, and total evaluations for each benchmark function. Each model was initialised 10 times on 10 different Sobol sequenced initial points.

Authors	Daulton et al. [ 17 ]
Kernel name	meta_off
Base kernel	Matérn-5/2
Formulation	$k_{\text{scale}}\left(k_{\text{cat}}^{\text{ARD}}\times k_{\text{cont,int}}^{\text{ARD}}\times k_{\text{bin}}^{\text{ARD}}\right)+k_{\text{scale}}\left(k_{\text{cat}}^{\text{ARD}}+k_{\text{cont,int}}^{\text{ARD}}+k_{\text{bin}}^{\text{ARD}}\right)$
Scale prior	None
Lengthscale prior	None
Comments	In this study with discrete variable problems, $k_{\text{cont,int}}^{\text{ARD}}$ becomes $k_{\text{cont,int,discr}}^{\text{ARD}}$ .

Table S2: Kernel formulation used within the original PR implementation by Daulton et al.

Authors	Hvafner et al. [ 46 ]
Kernel name	hvafner_fixed
Base kernel	Matérn-5/2
Formulation	$k_{\text{scale}}\left(k_{\text{cont,int,discr}}^{\text{ARD}}\right)$
Scale prior	$\sigma^{2}=1$
Lengthscale prior	$\text{LN}\left(\sqrt{2}+\log\sqrt{D},\sqrt{3}\right)$
Comments	Scale hyperparameter fixed to 1 (when y data is standardized). $D$ is the dimensionality of the input space. Originally designed for purely continuous cases, within this study it is generalized to mix variable problems through the generalized PR method.

Table S3: Default Botorch kernel design using the dimensional aware priors by Hvafner et al.

Authors	Eduardo et al. [ 19 ]
Kernel name	KR_on_gam_Mat52
Base kernel	Matérn-5/2
Formulation	$k_{\text{scale}}\left(k_{\text{cont}}^{\text{Prod}}\times k_{\text{int}}^{\text{Prod}}\right)$
Scale prior	$\text{Gam}\left(2,\left(\frac{1}{2(\max(\boldsymbol{Y})-\min(\boldsymbol{Y}))}\right)^{2}\right)$
Lengthscale prior	$\text{Gam}(\alpha,\lambda)$
Comments	This approach rounds integer-valued variables to the nearest integer within the kernel, making it incompatible with discrete variables. Additionally, the variable transformation involved in this method prevents the use of gradient-based optimization for the acquisition function.

Table S4: BOSS-style kernel design and priors [ 36 ] utilizing the kernel rounding method by Eduardo et al. for handling continuous+Integer variable problems.

Kernel Name	BOSS_off_RBF
Base kernel	RBF
Formulation	$k_{scale}\left(k_{cont}^{Prod}\times k_{int}^{Prod}\times k_{discr}^{Prod}\times k_{cat}^{ARD}\right)$
Scale prior	-
Lengthscale prior	-
Comments	-

Table S5: Radial basis function kernel with product formulation and no priors

Kernel Name	BOSS_off_Mat52
Base kernel	Matérn-5/2
Formulation	$k_{scale}\left(k_{cont}^{Prod}\times k_{int}^{Prod}\times k_{discr}^{Prod}\times k_{cat}^{ARD}\right)$
Scale prior	-
Lengthscale prior	-
Comments	-

Table S6: Matérn-5/2 kernel with product formulation and no priors

Kernel Name	BOSS_off_Mat52_sum
Base kernel	Matérn-5/2
Formulation	$k_{scale}\left(k_{cont}^{sum}+k_{int}^{sum}+k_{discr}^{sum}+k_{cat}^{ARD}\right)$
Scale prior	-
Lengthscale prior	-
Comments	-

Table S7: Matérn-5/2 kernel with sum formulation and no priors

Kernel Name	BOSS_on_gam_Mat52
Base kernel	Matérn-5/2
Formulation	$k_{scale}\left(k_{cont}^{Prod}\times k_{int}^{Prod}\times k_{discr}^{Prod}\times k_{cat}^{ARD}\right)$
Scale prior	$\text{Gam}\left(2,\left(\frac{1}{2\left(\max(\boldsymbol{Y})-\min(\boldsymbol{Y})\right)}\right)^{2}\right)$
Lengthscale prior	$\text{Gam}(\alpha,\lambda)$
Comments	-

Table S8: Matérn-5/2 kernel with product formulation and Gamma priors

Kernel Name	BOSS_on_LN_Mat52
Base kernel	Matérn-5/2
Formulation	$k_{scale}\left(k_{cont}^{Prod}\times k_{int}^{Prod}\times k_{discr}^{Prod}\times k_{cat}^{ARD}\right)$
Scale prior	$\text{Gam}\left(2,\left(\frac{1}{2\left(\max(\boldsymbol{Y})-\min(\boldsymbol{Y})\right)}\right)^{2}\right)$
Lengthscale prior	$\text{LN}\left(\sqrt{2}+\log\sqrt{D},\sqrt{3}\right)$
Comments	-

Table S9: Matérn-5/2 kernel with log-normal prior on lengthscale and gamma prior on scale hyperparmeters

Kernel Name	BOSS_on_gam_fixed_Mat52
Base kernel	Matérn-5/2
Formulation	$k_{scale}\left(k_{cont}^{Prod}\times k_{int}^{Prod}\times k_{discr}^{Prod}\times k_{cat}^{ARD}\right)$
Scale prior	$\sigma^{2}=1$
Lengthscale prior	$\text{Gam}(\alpha,\lambda)$
Comments	-

Table S10: Matérn-5/2 kernel with fixed output variance and Gamma prior on lengthscale

Problem	Tolerance level	Y-range	Cont.	Int., Discr., Cat.
Butternut Squash	Strict	0.1%	1%	Exact
	Medium	0.5%	2%	Exact
	Loose	1%	4%	Exact
Chemistry	Strict	1%	1%	Exact
	Medium	2%	2%	Exact
	Loose	3%	3%	Exact

Table S11: Tolerance levels used to determine convergence for each benchmark problem. The “Y-range tolerance” column specifies the allowable relative difference between the model’s minimum objective value and the global minimum, expressed as a percentage of the total objective value range. The “X-range tolerance (continuous)” column indicates the maximum allowed deviation of the continuous input parameters from their true optimal values, expressed as a percentage of the input variable range. For all tolerance levels, integer and categorical parameters must exactly match the global minimum for convergence to be declared.

Figure S3: Comparison of prior variance

k_{**}

, variance reduction

k_{*}^{\top}(K+\sigma_{0}^{2}I)^{-1}k_{*}

, and posterior variance for product and sum kernels across increasing dimension

D

. Training inputs are sampled uniformly in

[0,1]^{D}

with

n_{\text{train}}=10\cdot 2^{D-2}

points (10, 20, 40, 80, 160 for

D=2\ldots 6

) with a test point at the zero-origin. The red, pink and cyan plots show the sum kernel formulation with a prior variance values of 1, 10, 100, respectively.

Figure S4: Posterior variances for the sum and product kernel formulations across eight dimensions. The training data consist of

n_{\mathrm{train}}=10\cdot 2^{D-2}

uniformly sampled points in the domain

[0,1]^{D}

, where

D

denotes the dimensionality. For the sum kernel, prior variance hyperparameters of

1

,

10

, and

100

are shown in red, pink, and cyan, respectively. The labels “near train” and “center” indicate whether the posterior variance was evaluated at a test point located near an existing training sample or at thecenter of the domain.

Figure S5: The difference in posterior mean prediction of a product formulation kernel GP model (left) and sum kernel GP model (right) where STD is the standard deviation. Both models were fitted using the same Matérn-5/2 kernel and sobol datapoints sampled from the BS function. Besides the sum kernel having lower overall uncertainty in the design space, the blue circle shows a low uncertainty region within the sum plot that is not present in the product formulation showcasing the sum kernels confidence and stronger beliefs about the objective landscape

Figure S6: Two dimensional kernel correlation maps for the product and sum kernel formulations, each constructed from one dimensional Matérn 5/2 components with unit prior variance (

\sigma_{s}^{2}

&

\sigma_{p}^{2}

= 1 ) and lengthscale equal to one. The values represent the kernel

K((x,y),(x_{0},y_{0}))

evaluated over the two dimensional input space with a fixed reference point

(x_{0},y_{0})=(0,0)

. In the product kernel, correlations decay rapidly away from the reference point, forming a localized blob centered at the origin. In contrast, the sum kernel exhibits a broader cross shaped correlation structure, remaining large along the coordinate axes, indicating that correlations persist when either coordinate is similar to the reference point.

Figure S7: The total summed standard deviation of both the sum and product Matérn-5/2 GP model predictions on fully integer cases of the BS function landscape across dimensionalities. The number of fitted sobol sampled points are [5,10,20,40,60] for each dimension respectively. Both model were fitted on the same exact points across each dimension.

Dim	Kernel	Variance / Noise	Lengthscales
2D	Product (Mat52)	var = 2.1182, noise = 0.001	$\ell=[1.0189,\ 1.1745]$
2D	Sum (Mat52)	var = 1.9897, noise = 0.001	$\ell=[1.1065,\ 0.9621]$
3D	Product (Mat52)	var = 2.2211, noise = 0.001	$\ell=[1.5205,\ 0.9352,\ 0.3170]$
3D	Sum (Mat52)	var = 2.5875, noise = 0.001	$\ell=[0.9601,\ 0.7587,\ 0.3574]$
4D	Product (Mat52)	var = 2.6357, noise = 0.001	$\ell=[0.4082,\ 2.0836,\ 1.2681,\ 0.4527]$
4D	Sum (Mat52)	var = 0.3589, noise = 0.001	$\ell=[0.1677,\ 0.0795,\ 0.4745,\ 0.5131]$

Table S12: Hyperparameters of BOSS Matérn-5/2 product and sum kernels for the 2D–6D Yuh–Styblinski–Tang test problems corresponding to Figure S7 .

Figure S8: Iterations 1-6 marked by a-f showing subtract method leading to repeated acquisitions in LCB

Figure S9: Empirical convergence of the BOSS_on_gam_Mat52 model on the DUST1 benchmark function using the acquisition functions: (a) Expected Improvement (EI) and (b) Lower Confidence Bound (LCB).

Figure S10: Empirical convergence of the BOSS_on_gam_Mat52 model on the DUST2 benchmark function using the acquisition functions: (a) Expected Improvement (EI) and (b) Lower Confidence Bound (LCB).

Figure S11: Plot showing the mean composite score ranks of all models across each dimension of the Butternut Squash benchmark function variants with continuous+integer domains under the loose tolerance

Figure S12: Plot showing the mean composite score ranks of all models across each dimension of the Butternut Squash benchmark function variants with continuous+integer domains under the loose tolerance.

Figure S13: Plot showing the mean composite score ranks of all models across each dimension of the Butternut Squash benchmark function variants with discrete domains under loose tolerance.

Model Settings	Num Ranks	Mean R.	Median R.	Min R.	Max R.
ei_BOSS_off_Mat52_sum	20	2.35	2.00	1	6
lcb_BOSS_off_Mat52_sum	20	2.85	2.00	1	9
ei_BOSS_on_gam_Mat52	20	3.40	3.50	1	6
lcb_BOSS_on_gam_Mat52	20	4.25	4.00	1	7
ei_BOSS_on_LN_Mat52	20	5.00	5.00	1	10
lcb_BOSS_on_LN_Mat52	20	5.55	5.00	1	12

Table S13: Summary of model composite score ranking statistics across all 20 variants of the BS function under the loose tolerance. Note that since the KR model only applies to the continuous+integer domains, it has only 10 computed ranks corresponding to the 10 integer+continuous domain variants of the BS functions.

Model Settings	Num Ranks	Mean R.	Median R.	Min R.	Max R.
ei_BOSS_on_gam_Mat52	10	2.70	3.00	1	4
lcb_BOSS_off_Mat52_sum	10	2.90	1.50	1	9
ei_BOSS_off_Mat52_sum	10	3.50	3.00	1	6
lcb_BOSS_on_gam_Mat52	10	3.90	3.50	1	7
ei_BOSS_on_LN_Mat52	10	5.40	5.00	3	10
ei_KR_on_gam_Mat52	10	6.20	7.00	2	12

Table S14: Summary of all model composite score ranking statistics across only the continuous+integer variable domains (10 variants) of the BS function under the loose tolerance.

Model Settings	Num Ranks	Mean R.	Median R.	Min R.	Max R.
ei_BOSS_off_Mat52_sum	10	1.40	1.00	1	3
lcb_BOSS_off_Mat52_sum	10	2.50	2.00	1	6
ei_BOSS_on_gam_Mat52	10	4.10	4.00	2	6
ei_BOSS_on_LN_Mat52	10	4.20	4.50	1	6
lcb_BOSS_on_LN_Mat52	10	4.30	5.00	1	6
lcb_BOSS_on_gam_Mat52	10	4.50	4.00	3	6

Table S15: Summary of all model composite score ranking statistics across only the discrete variable domains (10 variants) of the BS function under the loose tolerance level.

Figure S14: Histogram illustrating absolute convergence performance, measured as the percentage of converged runs aggregated over all problem instances (10 runs per model per instance, each with different Sobol initializations), for all models (excluding KR) using the LCB AF on the BS benchmark function. Convergence defined under our medium tolerance criterion counts are pooled across all 18 instances, and evaluated under relative budget constraints of 25%, 50%, 75%, and 100%, where each percentage corresponds to the respective fraction of the maximum iterations for each individual instance.

Figure S15: Box plot showing the model rank statistics in terms of composite score (as defined in Eq. 7 ) for all 20 variants of the BS benchmark function. Note that the KR_on_gam models are limited to 10 variants as it only applies to the continuous + integer cases.

Figure S16: Plot showing the mean composite score ranks of all models across each dimension of the Butternut Squash benchmark function variants with continuous+integer domains under the medium tolerance

Figure S17: Plot showing the mean composite score ranks of all models across each dimension of the Butternut Squash benchmark function variants with discrete domains under the medium tolerance.

Model Settings	Num Ranks	Mean R.	Median R.	Min R.	Max R.
ei_BOSS_off_Mat52_sum	20	2.20	2.00	1	6
lcb_BOSS_off_Mat52_sum	20	2.60	2.00	1	9
ei_BOSS_on_gam_Mat52	20	3.55	3.50	1	6
lcb_BOSS_on_gam_Mat52	20	4.25	4.00	1	6
ei_BOSS_on_LN_Mat52	20	5.30	5.00	1	11
lcb_BOSS_on_LN_Mat52	20	5.60	5.50	1	12

Table S16: Summary of model composite score ranking statistics across all 20 variants of the BS function under the medium tolerance. Note that since the KR model only applies to the continuous+integer domains, it has only 10 computed ranks corresponding to the 10 integer+continuous domain variants of the BS functions.

Model Settings	Num Ranks	Mean R.	Median R.	Min R.	Max R.
lcb_BOSS_off_Mat52_sum	10	2.70	1.00	1	9
ei_BOSS_on_gam_Mat52	10	3.00	3.00	1	5
ei_BOSS_off_Mat52_sum	10	3.00	3.00	1	6
lcb_BOSS_on_gam_Mat52	10	4.00	4.00	1	6
ei_KR_on_gam_Mat52	10	6.30	7.00	2	12
ei_BOSS_on_LN_Mat52	10	6.40	5.50	3	11

Table S17: Summary of all model composite score ranking statistics across only the continuous+integer variable domains (10 variants) of the BS function under the medium tolerance level.

Model Settings	Num Ranks	Mean R.	Median R.	Min R.	Max R.
ei_BOSS_off_Mat52_sum	10	1.40	1.00	1	3
lcb_BOSS_off_Mat52_sum	10	2.50	2.00	1	6
ei_BOSS_on_gam_Mat52	10	4.10	4.00	2	6
ei_BOSS_on_LN_Mat52	10	4.20	4.50	1	6
lcb_BOSS_on_LN_Mat52	10	4.30	5.00	1	6
lcb_BOSS_on_gam_Mat52	10	4.50	4.00	3	6

Table S18: Summary of all model composite score ranking statistics across only the discrete variable domains (10 variants) of the BS function under the medium tolerance level.

Figure S18: Plot showing the mean composite score ranks of all models across each dimension of the Butternut Squash benchmark function variants with continuous+integer domains under the strict tolerance.

Figure S19: Plot showing the mean composite score ranks of all models across each dimension of the Butternut Squash benchmark function variants with continuous+integer domains under the strict tolerance.

Figure S20: Plot showing the mean composite score ranks of all models across each dimension of the Butternut Squash benchmark function variants with discrete domains under the strict tolerance.

Model Settings	Num Ranks	Mean R.	Median R.	Min R.	Max R.
ei_BOSS_off_Mat52_sum	20	2.50	2.00	1	6
lcb_BOSS_off_Mat52_sum	20	2.65	2.00	1	9
ei_BOSS_on_gam_Mat52	20	3.90	3.00	1	11
lcb_BOSS_on_gam_Mat52	20	4.35	4.00	1	8
ei_BOSS_on_LN_Mat52	20	5.05	5.00	1	11
lcb_BOSS_on_LN_Mat52	20	5.40	5.00	1	12

Table S19: Summary of model composite score ranking statistics across all 20 variants of the BS function under the strict tolerance level. Note that since the KR model only applies to the continuous+integer domains, it has only 10 computed ranks corresponding to the 10 integer+continuous domain variants of the BS functions.

Model Settings	Num Ranks	Mean R.	Median R.	Min R.	Max R.
lcb_BOSS_off_Mat52_sum	10	2.80	1.50	1	9
ei_BOSS_off_Mat52_sum	10	3.60	3.50	1	6
ei_BOSS_on_gam_Mat52	10	3.70	2.50	1	11
lcb_BOSS_on_gam_Mat52	10	4.20	4.00	1	8
ei_BOSS_on_LN_Mat52	10	5.90	5.50	2	11
lcb_BOSS_on_LN_Mat52	10	6.50	6.00	4	12

Table S20: Summary of all model composite score ranking statistics across only the continuous+integer variable domains (10 variants) of the BS function under the strict tolerance level.

Model Settings	Num Ranks	Mean R.	Median R.	Min R.	Max R.
ei_BOSS_off_Mat52_sum	10	1.40	1.00	1	3
lcb_BOSS_off_Mat52_sum	10	2.50	2.00	1	6
ei_BOSS_on_gam_Mat52	10	4.10	4.00	2	6
ei_BOSS_on_LN_Mat52	10	4.20	4.50	1	6
lcb_BOSS_on_LN_Mat52	10	4.30	5.00	1	6
lcb_BOSS_on_gam_Mat52	10	4.50	4.00	3	6

Table S21: Summary of all model composite score ranking statistics across only the discrete variable domains (10 variants) of the BS function under the strict tolerance level.

Figure S21: Mean convergence plots over 10 runs for all benchmarked models using the EI acquisition function on the 2D BS function.

Figure S22: Mean convergence plots over 10 runs for all benchmarked models using the LCB acquisition function on the 2D BS function.

Figure S23: Mean convergence plots over 10 runs for all benchmarked models using the EI acquisition function on the 3D BS function.

Figure S24: Mean convergence plots over 10 runs for all benchmarked models using the LCB acquisition function on the 3D BS function.

Figure S25: Mean convergence plots over 10 runs for all benchmarked models using the EI acquisition function on the 4D BS function.

Figure S26: Mean convergence plots over 10 runs for all benchmarked models using the LCB acquisition function on the 4D BS function.

Figure S27: Mean convergence plots over 10 runs for all benchmarked models using the EI acquisition function on the 5D BS function.

Figure S28: Mean convergence plots over 10 runs for all benchmarked models using the LCB acquisition function on the 5D BS function.

Figure S29: Mean convergence plots over 10 runs for all benchmarked models using the EI acquisition function on the 6D BS function.

Figure S30: Mean convergence plots over 10 runs for all benchmarked models using the LCB acquisition function on the 6D BS function.

Model Settings	Converged Runs	Mean Iteration	Composite Score
ei_meta_off	6	37.67	0.015929
ei_BOSS_on_gam_Mat52	5	41.40	0.012077
lcb_meta_off	4	41.00	0.009756
lcb_BOSS_on_gam_Mat52	3	57.33	0.005233
ei_BOSS_off_Mat52_sum	0	—	0.000000
ei_hvafner_fixed	0	—	0.000000

Table S22: Composite score ranking summary of all benchmarked models on the Chemistry function under the strict tolerance level. Mean iteration is reported for successful runs only.

Model Settings	Converged Runs	Mean Iteration	Composite Score
ei_BOSS_on_gam_Mat52	6	35.83	0.016744
ei_meta_off	6	36.33	0.016514
lcb_meta_off	6	42.33	0.014173
lcb_BOSS_on_gam_Mat52	6	59.50	0.010084
ei_BOSS_off_Mat52_sum	0	—	0.000000
ei_hvafner_fixed	0	—	0.000000

Table S23: Composite score ranking summary of all benchmarked models on the Chemistry function under the medium tolerance level. Mean iteration is reported for successful runs only.

Model Settings	Converged Runs	Mean Iteration	Composite Score
ei_BOSS_on_gam_Mat52	7	35.57	0.019679
ei_meta_off	7	40.43	0.017314
lcb_BOSS_on_gam_Mat52	8	53.13	0.015059
lcb_meta_off	6	42.33	0.014173
ei_BOSS_off_Mat52_sum	1	56.00	0.001786
ei_hvafner_fixed	0	—	0.000000

Table S24: Composite score ranking summary of all benchmarked models on the Chemistry function under the loose tolerance level. Mean iteration is reported for successful runs only.

Figure S31: Convergence plots of the EI meta model on the Chemistry benchamrk function. The bold curve is the mean convergences from its 10 different sobol point initiated runs.

Figure S32: Convergence plots of the EI PMG model on the Chemistry benchamrk function. The bold curve is the mean convergences from its 10 different sobol point initiated runs.

Figure S33: Objective Landscape of the DUST function with the global minima (Glmin) indicated in Pink.

Figure S34: All acquisition points in terms of objective value from run 9 of the EI+MGP+V_BO approach (up) versus the EI+MGP+EA_BO approach (down). The acquisition function used at each iteration is shown using red and blue vertical lines corresponding to the EI and pure exploration AF respectively. The global minima objective value is -30

Model Settings	Converged Runs	Mean Iteration	Composite Score
lcb_penalty	6	25.33	0.023684
lcb_penalty_HITL_explore	10	44.10	0.022676
ei_penalty	5	24.60	0.020325
ei_penalty_HITL_explore	7	42.86	0.016333
ei_RF	4	28.00	0.014286
SOBOL_off	1	8.00	0.012500

Table S25: Composite score ranking summary of all benchmarked models on the DUST1 function under the strict tolerance level. Mean iteration is reported for successful runs only.

Model Settings	Converged Runs	Mean Iteration	Composite Score
ei_penalty	7	14.71	0.047573
ei_penalty_HITL_explore	10	29.10	0.034364
lcb_penalty_HITL_explore	10	31.00	0.032258
lcb_penalty	6	21.00	0.028571
SOBOL_off	3	23.00	0.013043
ei_RF	5	40.80	0.012255

Table S26: Composite score ranking summary of all benchmarked models on the DUST1 function under the medium tolerance level. Mean iteration is reported for successful runs only.

Model Settings	Converged Runs	Mean Iteration	Composite Score
ei_penalty	7	13.71	0.051042
lcb_penalty_HITL_explore	10	27.50	0.036364
ei_penalty_HITL_explore	10	27.90	0.035842
lcb_penalty	6	18.83	0.031858
SOBOL_off	10	35.60	0.028090
ei_RF	5	24.40	0.020492

Table S27: Composite score ranking summary of all benchmarked models on the DUST1 function under the loose tolerance level. Mean iteration is reported for successful runs only.

Figure S35: Objective Landscape of the DUST2 function with the global minima (Glmin) indicated in Pink.

Model Settings	Converged Runs	Mean Iteration	Composite Score
lcb_penalty	6	23.00	0.026087
ei_penalty_HITL_explore	7	37.29	0.018774
lcb_penalty_HITL_explore	8	51.63	0.015496
ei_penalty	5	42.20	0.011848
ei_RF	4	62.75	0.006375
SOBOL_off	0	—	0.000000

Table S28: Composite score ranking summary of all benchmarked models on the DUST2 function under the strict tolerance level. Mean iteration is reported for successful runs only.

Model Settings	Converged Runs	Mean Iteration	Composite Score
lcb_penalty	6	23.00	0.026087
ei_penalty_HITL_explore	7	34.71	0.020165
lcb_penalty_HITL_explore	8	44.63	0.017927
ei_penalty	5	41.40	0.012077
ei_RF	4	62.75	0.006375
SOBOL_off	1	68.00	0.001471

Table S29: Composite score ranking summary of all benchmarked models on the DUST2 function under the medium tolerance level. Mean iteration is reported for successful runs only.

Model Settings	Converged Runs	Mean Iteration	Composite Score
lcb_penalty	6	21.33	0.028125
ei_penalty_HITL_explore	7	33.71	0.020763
lcb_penalty_HITL_explore	9	53.78	0.016736
ei_penalty	5	40.20	0.012438
ei_RF	4	33.75	0.011852
SOBOL_off	2	80.00	0.002500

Table S30: Composite score ranking summary of all benchmarked models on the DUST2 function under the loose tolerance level. Mean iteration is reported for successful runs only.

Bayesian Optimization for Mixed-Variable Problems in the Natural Sciences

Abstract

Figures & tables

Bayesian Optimization

Exploiting Separability in Multi-Scale Grey-Box Bayesian Optimization

Frugal Bayesian Optimization: Scalable Surrogates for Data- and Resource-Limited Discovery

TagBO: LLM-Driven Task-Aware Graph Bayesian Optimization for Scientific Discovery

Abstract

Bayesian Optimization

Explore similar work

Exploiting Separability in Multi-Scale Grey-Box Bayesian Optimization

Frugal Bayesian Optimization: Scalable Surrogates for Data- and Resource-Limited Discovery

TagBO: LLM-Driven Task-Aware Graph Bayesian Optimization for Scientific Discovery