Python Tool for Visualizing Variability of Pareto Fronts over Multiple Runs

Shuhei Watanabe

cs.AIMay 15, 2023

Python Tool for Visualizing Variability of Pareto Fronts over Multiple Runs

Authors: Shuhei Watanabe

Organizations: Department of Computer Science, University of Freiburg, Germany

Abstract

Hyperparameter optimization is crucial to achieving high performance in deep learning. On top of the performance, other criteria such as inference time or memory requirement often need to be optimized due to some practical reasons. This motivates research on multi-objective optimization (MOO). However, Pareto fronts of MOO methods are often shown without considering the variability caused by random seeds and this makes the performance stability evaluation difficult. Although there is a concept named empirical attainment surface to enable the visualization with uncertainty over multiple runs, there is no major Python package for empirical attainment surface. We, therefore, develop a Python package for this purpose and describe the usage. The package is available at https://github.com/nabenabe0928/empirical-attainment-func. OptunaHub now provides our implementation at https://hub.optuna.org/visualization/plot_empirical_attainment_surface/.

Figures & tables

Figure 1: Conceptual visualizations of PF ( Left ) and EASs ( Right ). Lower is better for both objectives

f_{1},f_{2}

. Left : an example of PF. The red dots in the objective space are the objective vectors that belong to PF

\mathcal{F}

(the red solid line). The blue dots are dominated by the objective vectors in

\mathcal{F}

. In other words, the blue dots do not lie on PF. Right : an example of EASs. Each dotted line is the PF of one of four independent runs. Light-colored bands show EASs of

L=2

(red) and

L=3

(blue). The red surface is attained by PFs from two independent runs, and the blue surface is attained by PFs from three independent runs.

Figure 2: Visualizations generated by Listing 2 ( Left ) and Listing 3 ( Right ). The horizontal axis shows

f_{1}

and the vertical axis shows

f_{2}

. Left : an example plot of EASs. By repeating the same process for different MOO methods, we can compare multiple methods. Right : an example plot of EAS with an uncertainty band. The light-colored band shows the variability of PF due to random seeds.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 3: The visualization obtained by Listing 4 . This example plots HV over time with a band showing the standard error.

Multi-Objective Optimization

Multi-objective optimization studies how to find and choose solutions that balance competing goals, constraints, and preferences.

See all 266 papers

Optimization with SpotOptim
Apr 15, 2026Thomas Bartz-BeielsteinBlack-Box Optimization Surrogate-Assisted Optimization
Bartz & Bartz GmbH, 51643 Gummersbach, Germany
ParetoPilot: Zero-Surrogate Offline Multi-Objective Optimization via Infer-Perturb-Guide Diffusion
Jun 3, 2026Ruiqing Sun, Sen Yang, Dawei Feng +3Diffusion Guidance Multi-Objective Optimization
College of Computer Science and Technology, National University of Defense Technology, Changsha 410073, China · Bioinformatics Center of AMMS, Beijing 100080, China
Optuna Constrained Tree-Structured Parzen Estimator Is a Joint Density Generalization of c-TPE
Jun 3, 2026Shuhei Watanabe, Kaichi IrieBayesian Optimization Constrained Optimization
Independent Researcher · Kyoto University

Apr 15, 2026J/K move · Enter open · S save

Optimization with SpotOptim

Thomas Bartz-Beielstein

Bartz & Bartz GmbH, 51643 Gummersbach, Germany

Black-Box Optimization Surrogate-Assisted Optimization Bayesian Optimization Hyperparameter Optimization

Open paper AskPDF arXiv

The spotoptim package implements surrogate-model-based optimization of expensive black-box functions in Python. Building on two decades of Sequential Parameter Optimization (SPO) methodology, it provides a Kriging-based optimization loop with Expected Improvement, support for continuous, integer, and categorical variables, noise-aware evaluation via Optimal Computing Budget Allocation (OCBA), and multi-objective extensions. A success-rate-based restart mechanism detects stagnation while preserving the best solution found. The package returns scipy-compatible OptimizeResult objects and accepts any scikit-learn-compatible surrogate model. Built-in TensorBoard logging provides real-time monitoring of convergence and surrogate quality. This report describes the architecture and module structure of spotoptim, provides worked examples including neural network hyperparameter tuning, and compares the framework with BoTorch, Optuna, RayTune, BOHB, SMAC, and Hyperopt. The package is open-source (AGPL-3.0).

Abstract

Multi-Objective Optimization

Explore similar work

Optimization with SpotOptim

ParetoPilot: Zero-Surrogate Offline Multi-Objective Optimization via Infer-Perturb-Guide Diffusion

Optuna Constrained Tree-Structured Parzen Estimator Is a Joint Density Generalization of c-TPE