Hyperparameter optimization is crucial to achieving high performance in deep learning. On top of the performance, other criteria such as inference time or memory requirement often need to be optimized due to some practical reasons. This motivates research on multi-objective optimization (MOO). However, Pareto fronts of MOO methods are often shown without considering the variability caused by random seeds and this makes the performance stability evaluation difficult. Although there is a concept named empirical attainment surface to enable the visualization with uncertainty over multiple runs, there is no major Python package for empirical attainment surface. We, therefore, develop a Python package for this purpose and describe the usage. The package is available at https://github.com/nabenabe0928/empirical-attainment-func. OptunaHub now provides our implementation at https://hub.optuna.org/visualization/plot_empirical_attainment_surface/.
Figures & tables
Figure 1: Conceptual visualizations of PF ( Left ) and EASs ( Right ). Lower is better for both objectives f1,f2 . Left : an example of PF. The red dots in the objective space are the objective vectors that belong to PF F (the red solid line). The blue dots are dominated by the objective vectors in F . In other words, the blue dots do not lie on PF. Right : an example of EASs. Each dotted line is the PF of one of four independent runs. Light-colored bands show EASs of L=2 (red) and L=3 (blue). The red surface is attained by PFs from two independent runs, and the blue surface is attained by PFs from three independent runs.
Figure 2: Visualizations generated by Listing 2 ( Left ) and Listing 3 ( Right ). The horizontal axis shows f1 and the vertical axis shows f2 . Left : an example plot of EASs. By repeating the same process for different MOO methods, we can compare multiple methods. Right : an example plot of EAS with an uncertainty band. The light-colored band shows the variability of PF due to random seeds.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 3: The visualization obtained by Listing 4 . This example plots HV over time with a band showing the standard error.
The spotoptim package implements surrogate-model-based optimization of expensive black-box functions in Python. Building on two decades of Sequential Parameter Optimization (SPO) methodology, it provides a Kriging-based optimization loop with Expected Improvement, support for continuous, integer, and categorical variables, noise-aware evaluation via Optimal Computing Budget Allocation (OCBA), and multi-objective extensions. A success-rate-based restart mechanism detects stagnation while preserving the best solution found. The package returns scipy-compatible OptimizeResult objects and accepts any scikit-learn-compatible surrogate model. Built-in TensorBoard logging provides real-time monitoring of convergence and surrogate quality. This report describes the architecture and module structure of spotoptim, provides worked examples including neural network hyperparameter tuning, and compares the framework with BoTorch, Optuna, RayTune, BOHB, SMAC, and Hyperopt. The package is open-source (AGPL-3.0).
Offline multi-objective optimization (Offline MOO) seeks Pareto-optimal designs from static datasets without additional environment interactions. Existing generative methods typically guide sampling with external surrogate or preference models, which adds training complexity and may provide unreliable guidance. We propose ParetoPilot, a plug-and-play method that guides designs to Pareto front at inference time using a pre-trained conditional diffusion model without any surrogate. ParetoPilot introduces an Infer-Perturb-Guide (IPG) engine within the reverse diffusion process. IPG first infers the individual conditional target for each sample in the batch by aligning its conditional and unconditional predictions. It then perturbs these targets collectively across the batch, balancing convergence toward the Pareto front and diversity among samples. Finally, the engine guides the generative trajectory toward the Pareto front by injecting these perturbed targets via standard Classifier-Free Guidance (CFG). Experiments on 51 tasks demonstrate that ParetoPilot achieves the best overall ranking among 16 methods and competitive hypervolume improvement.
Ruiqing Sun, Sen Yang, Dawei Feng +3
College of Computer Science and Technology, National University of Defense Technology, Changsha 410073, China · Bioinformatics Center of AMMS, Beijing 100080, China
Constrained hyperparameter optimization (HPO) is common in practice, yet Optuna's widely used constrained TPE lacks algorithmic analysis. While c-TPE proposes an expected constrained improvement (ECI) approach assuming independence between the objective and constraints, Optuna uses a single joint density over both. We show that Optuna's constrained TPE is joint c-TPE -- the same ECI acquisition function using a joint likelihood. We demonstrate joint c-TPE is invariant to constraint duplication whereas independent c-TPE degrades as the product accumulates duplicated factors. We outline practical tradeoffs between the formulations and directions for future study.