BO-Arena: An Evolving Benchmark for High-Dimensional Bayesian Optimisation
Organizations: Tübingen AI Center University of Tübingen · Lancaster University
Abstract
High-dimensional Bayesian optimisation (HDBO) is an increasingly crowded field with new methods proposed at every major conference, making it ever harder to answer a deceptively simple question: "Which method is currently state-of-the-art?" In this paper, we question the efficacy of benchmarking practices in the field, and find them to be systematically ineffective: multiple published works fail to benchmark against the strongest available baselines, or even include incorrect implementations of key methods. Without a shared reference implementation, such errors are difficult to detect during peer review. We therefore introduce BO-Arena, an actively maintained and extensible package providing canonical implementations of state-of-the-art algorithms, which allows researchers to develop new methods and benchmark them rigorously. Using BO-Arena, we find that what currently drives performance in HDBO is even simpler than previously thought. We use these insights to propose a new algorithm which is competitive with the state of the art on problems with many observations, without resorting to the complex GP approximations used in recent methods. Finally, we question whether there is currently sufficient evidence that any more complex method outperforms a simple Gaussian process baseline for HDBO.
Figures & tables
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| BO-Arena | COCO | HDBO on BBOB | poli | HPOBench | Bayesmark | |
|---|---|---|---|---|---|---|
| HDBO algorithms | 6 | 0 | 5 | 6 | 0 | 0 |
| Verified vs. original papers | ✓ | – | – | – | ||
| Native implementations | ✓ | – | – | – | ||
| Continuous problems | 48 | 24 | 24 | 0 | 40 | 28 |
| Real-world problems | ✓ | ✓ | ✓ | ✓ | ||
| Max. continuous dimensionality | 6392 | 640 | 60 | – | 2 | 3 |
Bayesian Optimization
Bayesian optimization uses probabilistic models and sequential experiments to find strong solutions when evaluating a black-box objective is costly.
See all 268 papers