Authors: Marc Becker, Lennart Schneider, Martin Binder, Lars Kotthoff, Bernd Bischl
Organizations: Department of Statistics, LMU Munich, Munich, Germany · Munich Center for Machine Learning (MCML), Munich, Germany · University of St Andrews, St Andrews, Scotland
We present mlr3mbo, a modular toolbox for Bayesian optimization in R. mlr3mbo supports single- and multi-objective optimization, multi-point proposals, batch and asynchronous parallelization, and robust error handling. While it can be used for many standard Bayesian optimization variants in applied settings, researchers can also construct custom Bayesian optimization algorithms from its flexible building blocks. In addition to an introduction to the software, its design principles, and its building blocks, the paper presents two extensive empirical evaluations on the surrogate-based benchmark suite YAHPO Gym. To identify robust default configurations for both numeric and mixed-hierarchical optimization regimes, and to gain further insights into the respective impacts of individual settings, we run a coordinate descent search over the mlr3mbo configuration space and analyze its results. Furthermore, we benchmark mlr3mbo against a wide range of established optimizers, including HEBO, SMAC3, Ax, and Optuna, and find that it performs on par with state-of-the-art.
Figures & tables
Figure 1: Sinusoidal function used as an example.
Constructor
Description
Underlying class
p_dbl
Real valued parameter (“double”)
ParamDbl
p_int
Integer parameter
ParamInt
p_fct
Discrete valued parameter (“factor”)
ParamFct
p_lgl
Logical / Boolean parameter
ParamLgl
p_uty
Untyped parameter
ParamUty
Table 1: Parameter constructors and their underlying classes.
Figure 2: Expected Improvement (solid dark gray line) based on the mean (dashed gray line) and uncertainty (gray shaded area) prediction of the GP SM trained on an initial design of four points (black).
Key
Description
Reference
bayesopt_ego
Efficient global optimization
Snoek et al. 111 Jones et al. 70
bayesopt_emo
Multi-objective EGO
Huang et al. 64
bayesopt_mpcl
Multi-point constant liar
Ginsbourger et al. 54 Wang et al. 116
bayesopt_parego
Scalarization based multi-objective optimization
Knowles 77
bayesopt_smsego
Direct indicator based multi-objective optimization
Beume et al. 15 Ponweiser et al. 96
Table 2: Loop functions are returned by mlr_loop_functionsget(key).Callhelp() to access the help page of the loop function e.g. mlr_loop_functionsget("bayesoptego")help() .
Figure 3: Graph of our example AutoML system.
Parameter
Start value
Final value
Input trafo
None
None
Output trafo
None
Log
Init
Random
Random
Init size
0.25
0.05
Random interleave
0
0
Surrogate
GP
GP
Table 3: Start and final configuration of the CD on numeric instances. For all evaluated configurations, see Table 17 . Parameters that are only relevant for the random forest surrogate model are omitted.
Iteration
Parameter
Value
RSNS
Start configuration
–
0.76
1
Optimizer
CMA-ES
0.99
2
Output trafo
Log
1.08
3
Surrogate
GP - Matérn 3/2 - Nugget =0 - No Scaling
1.14
4
Acquisition function
LCB - λ=3
1.18
5
Surrogate
GP - Matérn 3/2 - Nugget =10−8 - No Scaling
1.18
Table 4: Optimization path of the CD on numeric instances.
Parameter
Start value
Final value
Input trafo
None
None
Output trafo
None
Log
Init
Random
Random
Init size
0.25
0.05
Random interleave
0
0
Variance estimator
ESD
LTV
Table 5: Start and final configuration of the CD on mixed-hierarchical instances. For all evaluated configurations, see Table 18 .
Iteration
Parameter
Value
RSNS
Start configuration
–
0.34
1
Output trafo
Log
0.50
2
Optimizer
LS
0.69
3
Variance estimator
LTV
0.78
4
Init size
0.1
0.78
5
Acquisition function
LCB - λ=3
0.80
Table 6: Optimization path of the CD on mixed-hierarchical instances.
Figure 4: Optimization performance of Ax , HEBO , SMAC4BB , SMAC4HPO , Optuna , and mlr3mbo over iterations. Performance is measured as the mean RSNS over all instances. Shaded ribbons show the standard error of the mean RSNS across the instances. The curves start at different budget fractions because the initial design size is different for each framework.
Figure 5: Critical difference diagrams of the average ranks of the final performance across all instances. The Friedman test is significant at α=0.05 for numeric instances. The horizontal bars indicate that frameworks connected by a bar do not differ significantly according to the Nemenyi post-hoc test at α=0.05 . The Friedman test is close to significance at α=0.05 for mixed-hierarchical instances. The critical difference plot is consistent with this result, as all frameworks are connected by a horizontal bar.
Figure 6: Runtime in minutes of Ax , HEBO , SMAC4BB , SMAC4HPO , Optuna , and mlr3mbo .
Key
Description
Reference
aei
Augmented expected improvement for noisy optimization. c controls the exploration-exploitation trade-off
Huang et al. 64
cb
Lower / upper confidence bound. λ controls the exploration-exploitation trade-off
Snoek et al. 111
ehvi
Expected hypervolume improvement for multi-criteria optimization
Emmerich et al. 41
ehvigh
Expected hypervolume improvement via GH quadrature for multi-criteria optimization
Rahat et al. 98
ei
Expected improvement
Jones et al. 70
ei_log
Expected improvement. Assumes that the surrogate models on log scale
Hutter et al. 66 Watanabe 119
Table 7: Acquisition Functions. Acquisition functions are constructed with the acqf(key) . Call help()toaccessthehelppageoftheacquisitionfunctione.g.acqf("ei")help() . The help page describes the formula of the acquisition function and the parameters that can be set.
Key
Search space type
Default settings
Reference
direct
Numeric
5⋅d restarts and 100⋅d2 function evaluations
Jones et al. 69
lbfgs
Numeric
5⋅d restarts and 100⋅d2 function evaluations
Byrd et al. 25
cmaes
Numeric
ABIPOP with unlimited restarts and 100⋅d2 function evaluations
Hansen and Ostermeier 59
random_search
Mixed and dependencies
100⋅d2 iterations
local_search
Mixed and dependencies
10 neighbors and 100⋅d2 iterations
Hutter et al. 66
Table 8: Acquisition function optimizers. Optimizers are constructed with the acqo(key) . Call help()toaccessthehelppageoftheacquisitionfunctionoptimizere.g.acqo("direct")help() . The help page describes the parameters that can be set.
Key
Description
clock_time
Clock time exceeds a given limit
combo
A combination of terminators is met
evals
Number of evaluations exceeds a given limit
none
Optimization is finished
perf_reached
Performance reaches a given level
run_time
Run time exceeds a given limit
Table 9: Terminator classes in the bbotk package. Terminators are constructed with the trm(key) function. Call help()toaccessthehelppageoftheterminatore.g.trm("stagnation")help() . The help page describes the parameters that can be set.
Parameter
Values
Dependency
Default
Input trafo
None, unitcube
–
None
Output trafo
None, standardize, log
–
None
Init
Random, LHS, Sobol
–
Random
Init size
0.05, 0.10, 0.25
–
0.25
Random interleave
0, 2, 4
–
0
Surrogate
RF, GP
–
GP
Table 10: Search space for purely numeric instances.
Parameter
Values
Dependency
Default
Input trafo
None, unitcube
–
None
Output trafo
None, standardize, log
–
None
Init
Random, LHS, Sobol
–
Random
Init size
0.05, 0.10, 0.25
–
0.25
Random interleave
0, 2, 4
–
0
Variance estimator
Jackknife, ESD, LTV
–
ESD
Table 11: Search space for mixed-hierarchical instances.
Scenario
Instance
Data set
Dimension
lcbench
189873
dionis
7
lcbench
189906
segment
7
lcbench
167168
vehicle
7
rbv2_rpart
14
mfeat-fourier
5
rbv2_rpart
40499
texture
5
rbv2_xgboost
12
mfeat-factors
14
Table 12: Summary of YAHPO-SO v1 numeric instances with data set name and search space dimension.
Scenario
Instance
Data set
Dimension
lcbench
189873
dionis
7
lcbench
189906
segment
7
lcbench
167168
vehicle
7
nb301
CIFAR10
CIFAR10
34
rbv2_ranger
16
mfeat-karhunen
8
rbv2_ranger
42
soybean
8
Table 13: Summary of YAHPO-SO v1 mixed-hierarchical instances with data set name and search space dimension.
Parameter
Value
Δ start
Δ final
Input trafo
Unitcube
0.00
-0.02
Output trafo
Log
0.06
–
None
–
-0.21
Standardize
0.00
-0.20
Init
LHS
0.01
-0.02
Sobol
0.00
-0.01
Table 14: Ablation study on the numeric search space. The ‘ Δ start’ column shows the change in RSNS when one parameter of the start configuration is changed. The ‘ Δ final’ column shows the change in RSNS when one parameter of the final configuration is changed. Dependent parameters with unsatified dependencies in the start or final configuration are omitted. When a parent parameters was changed, the dependent parameters were only changed to their default values e.g. GP to RF with 500 trees and jackknife variance estimator.
Parameter
Value
Δ start
Δ final
Input trafo
Unitcube
-0.01
-0.02
Output trafo
Log
0.16
–
None
–
-0.22
Standardize
0.02
-0.08
Init
LHS
-0.01
-0.04
Sobol
0.00
-0.06
Table 15: Ablation study on the mixed-hierarchical search space. The ‘ Δ start’ column shows the change in RSNS when one parameter of the start configuration is changed. The ‘ Δ final’ column shows the change in RSNS when one parameter of the final configuration is changed. Dependent parameters with unsatified dependencies in the start or final configuration are omitted. When a parent parameters was changed, the dependent parameters were only changed to their default values e.g. EI to LCB with a λ value of 1 and no λ decay.
Package
Version
Language
Source
mlrMBO
1.1.6
R
CRAN
mlr3
1.6.0
R
CRAN
mlr3tuning
1.6.0
R
CRAN
mlr3pipelines
0.11.0
R
CRAN
mlr3learners
0.14.0
R
CRAN
mlr3verse
0.3.1
R
CRAN
Table 16: Software packages referenced in this paper with their latest release versions.
Bayesian optimization (BO) has for sequential optimization of expensive black-box functions demonstrated practicality and effectiveness in many real-world settings. Meta-Bayesian optimization (meta-BO) focuses on improving the sample efficiency of BO by making use of information from related tasks. Although meta-BO is sample-efficient when task structure transfers, poor alignment between meta-training and test tasks can cause suboptimal queries to be suggested during online optimization. To this end, we propose a simple meta-BO algorithm that utilizes related-task information when determined useful, falling back to lookahead otherwise, within a unified framework. We demonstrate competitiveness of our method with existing approaches on function optimization tasks, while retaining strong performance in low task-relatedness regimes where test tasks share limited structure with the meta-training set.
Rahman Ejaz, Varchas Gopalaswamy, Ricardo Luna +5
Laboratory for Laser Energetics, Rochester, NY, USA · Department of Mechanical Engineering, University of Rochester, Rochester, NY, USA · Hewlett Packard Labs, Santa Clara, CA, USA +2
Modern engineering workflows increasingly rely on massive parallel simulation, driving the need for scalable, large-batch Bayesian Optimization (BO). Existing batch BO methods, however, incur large computational cost or rely on approximations that erode batch diversity. We propose B3O (Boltzmann Batch Bayesian Optimization), a framework that reframes batch generation as a pure sampling problem: drawing samples directly from the Boltzmann distribution defined by the acquisition function avoids the bottlenecks of existing large-batch methods. Theoretically, we prove that queries sampled from this distribution incur only negligible additional regret. Empirically, B3O outperforms existing batch BO methods on standard synthetic benchmarks and adapts robustly across complex applied tasks, including multi-objective electrode design and mixed-variable race car configuration.
Maximilian Bloor, Liyuan Xu, Hrvoje Stojic +1
Imperial College London London, UK · Secondmind Cambridge, UK
Multi-objective Bayesian optimization (MOBO) provides a principled framework for optimizing multiple expensive black-box functions. However, existing MOBO methods often struggle with coverage, scalability, and handling constraints and preferences. In this work we propose STAGE-BO, Sequential Targeting Adaptive Gap-Filling ε-Constraint Bayesian Optimization: by analyzing the coverage of the surrogate Pareto front, our method identifies the Pareto front point with the largest uncovered gap, and uses its coordinates to define adaptive constraints in ε-constraint method, which transforms the problem into a sequence of inequality-constrained subproblems, efficiently solved via constrained expected improvement acquisition. Our approach provides uniform Pareto coverage without hypervolume computation and naturally handles constraints and preferences. Experiments on synthetic and real-world benchmarks demonstrate superior coverage and competitive hypervolume performance against state-of-the-art baselines. Our code implementation can be found at https://github.com/YangYaohong1/STAGE-BO.
Yaohong Yang, Sammie Katt, Samuel Kaski
Department of Computer Science, Aalto University, Espoo, Finland · ELLIS Institute Finland · Department of Computer Science, University of Manchester, Manchester, United Kingdom