Optimization with SpotOptim

Thomas Bartz-Beielstein

cs.LGApr 15, 2026

Optimization with SpotOptim

Authors: Thomas Bartz-Beielstein

Organizations: Bartz & Bartz GmbH, 51643 Gummersbach, Germany

Abstract

The spotoptim package implements surrogate-model-based optimization of expensive black-box functions in Python. Building on two decades of Sequential Parameter Optimization (SPO) methodology, it provides a Kriging-based optimization loop with Expected Improvement, support for continuous, integer, and categorical variables, noise-aware evaluation via Optimal Computing Budget Allocation (OCBA), and multi-objective extensions. A success-rate-based restart mechanism detects stagnation while preserving the best solution found. The package returns scipy-compatible OptimizeResult objects and accepts any scikit-learn-compatible surrogate model. Built-in TensorBoard logging provides real-time monitoring of convergence and surrogate quality. This report describes the architecture and module structure of spotoptim, provides worked examples including neural network hyperparameter tuning, and compares the framework with BoTorch, Optuna, RayTune, BOHB, SMAC, and Hyperopt. The package is open-source (AGPL-3.0).

Figures & tables

Parameter	Role
max_iter=200	Hard evaluation budget. Stops the run if nothing else triggers first.
n_initial=5	Each LHS design (initial and every restart) uses 5 points.
window_size=3	Success rate is averaged over the last 3 infill iterations.
restart_after_n=3	3 consecutive iterations with success-rate = 0 trigger a restart.
max_restarts=2	After 2 restarts in a row that fail to improve best_y_ , stop early.

Table 1: Role of each parameter in the early-stopping example.

Figure 2: Optimization progress for the sphere function. Grey dots mark the initial Latin Hypercube design; subsequent evaluations are connected by a line. The red curve shows the best objective value found so far.

Figure 3: Surrogate model for dimensions

x_{0}

and

x_{1}

. Top row: 3-D surfaces of predictions (left) and prediction uncertainty (right). Bottom row: contour plots with evaluated points overlaid.

Figure 4: Filled contour plot of the Rosenbrock function over

[-2,2]\times[-1,3]

Figure 5: Surrogate contour plots for both Fonseca–Fleming objectives, fitted from 50 random evaluations.

Figure 6: Feature importances for the sphere optimization. Left: impurity-based (Gini) importances from a Random Forest. Right: permutation importances on the test set.

Figure 7: Surrogate prediction diagnostics. Left: actual versus predicted objective values. Right: residuals versus predicted values.

Figure 8: Hyperparameter tuning progress (demo run with reduced epochs and budget). The red curve shows the best validation MSE found so far.

Figure 9: Feature importances for the hyperparameter tuning demo. Left: impurity-based (Gini) importances. Right: permutation importances on the test set.

Black-Box Optimization

Black-box optimization finds high-quality solutions when objectives can only be queried, not directly analyzed or differentiated.

See all 176 papers

Python Tool for Visualizing Variability of Pareto Fronts over Multiple Runs
May 15, 2023Shuhei WatanabeMulti-Objective Optimization Pareto Optimization
Department of Computer Science, University of Freiburg, Germany
mlr3mbo: Bayesian Optimization in R
Mar 31, 2026Marc Becker, Lennart Schneider, Martin Binder +2Bayesian Optimization Hyperparameter Optimization
Department of Statistics, LMU Munich, Munich, Germany · Munich Center for Machine Learning (MCML), Munich, Germany · University of St Andrews, St Andrews, Scotland
Frugal Bayesian Optimization: Scalable Surrogates for Data- and Resource-Limited Discovery
Jul 31, 2026Panagiotis Krokidas, Christoforos Rekatsinas, Vassilis Sioros +3Surrogate Modeling Bayesian Optimization
Institute of Informatics and Telecommunications, National Centre for Scientific Research "Demokritos", Agia Paraskevi, Greece. · Department of Mechanical Engineering and Aeronautics, University of Patras, Patras Greece. · Department of Informatics and Telecommunications, National and Kapodistrian University of Athens, Athens, Greece. +4

May 15, 2023J/K move · Enter open · S save

Python Tool for Visualizing Variability of Pareto Fronts over Multiple Runs

Shuhei Watanabe

Department of Computer Science, University of Freiburg, Germany

Multi-Objective Optimization Pareto Optimization

Open paper AskPDF arXiv

Hyperparameter optimization is crucial to achieving high performance in deep learning. On top of the performance, other criteria such as inference time or memory requirement often need to be optimized due to some practical reasons. This motivates research on multi-objective optimization (MOO). However, Pareto fronts of MOO methods are often shown without considering the variability caused by random seeds and this makes the performance stability evaluation difficult. Although there is a concept named empirical attainment surface to enable the visualization with uncertainty over multiple runs, there is no major Python package for empirical attainment surface. We, therefore, develop a Python package for this purpose and describe the usage. The package is available at https://github.com/nabenabe0928/empirical-attainment-func. OptunaHub now provides our implementation at https://hub.optuna.org/visualization/plot_empirical_attainment_surface/.

Abstract

Black-Box Optimization

Explore similar work

Python Tool for Visualizing Variability of Pareto Fronts over Multiple Runs

mlr3mbo: Bayesian Optimization in R

Frugal Bayesian Optimization: Scalable Surrogates for Data- and Resource-Limited Discovery