cs.LGNov 2, 2026

ExaTune: Exhaustive Hyperparameter Landscape Exploration at Scale

Authors: Kevin Dick, Katie Lonsway

Organizations: BORN AI Foundations Lab, Better Outcomes Registry & Network (BORN) Ontario Children’s Hospital of Eastern Ontario Research Institute (CHEO-RI) Department of Systems & Computer Engineering, Carleton University Ottawa, ON

Abstract

Hyperparameter optimization (HPO) returns a single optimal configuration, systematically discarding the explored performance landscape that encodes how results depend on configuration choices. Fitness landscape analysis comprises a robust collection of methods from evolutionary computation offering ruggedness indices, autocorrelation functions, and fitness-distance correlation that have never been applied to machine learning hyperparameter spaces at scale, given that fine-grained ground-truth enumeration data has not existed. We present ExaTune, an open-source HPC system to produce a new artifact class: the hyperparameter landscape map. ExaTune exhaustively evaluates every point on a discretized hyperparameter grid via KK-fold cross-validation, aggregates results into reproducible parquet benchmarks, and applies a suite of five formalized landscape metrics: spatial autocorrelation (Moran's II), ruggedness, local optima density, fitness-distance correlation (FDC), and variance-based parameter importance. We release a benchmark collection of 8 complete landscape maps (totalling 285,080 exhaustively evaluated configurations) spanning five model families (SVMs, random forests, XGBoost, ridge regression, and kk-nearest neighbors) across six classification and two regression benchmarks, hosted on HuggingFace. Empirical analysis demonstrates that Bayesian optimization, Optuna, Hyperband, and Random Search at standard trial budgets systematically miss instability regions, secondary optima, and parameter interaction effects that are visible in complete landscapes; information that is structurally unavailable to any optimizer by design.

Figures & tables

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. How Many Trees in a Random Forest? A Revisited Approach with Plateau Search and Optuna Integration

    Jun 2, 2026Vadim Porvatov, Andrey Dukhovny, Andrey LangeEnsemble LearningRandom Forest

  2. Importance-Aware Scheduling for High-Dimensional Hyperparameter Optimization

    Jun 8, 2026Ruinan Wang, Ian Nabney, Mohammad GolbabaeeBayesian OptimizationHyperparameter Optimization