ExaTune: Exhaustive Hyperparameter Landscape Exploration at Scale
Authors: Kevin Dick, Katie Lonsway
Organizations: BORN AI Foundations Lab, Better Outcomes Registry & Network (BORN) Ontario Children’s Hospital of Eastern Ontario Research Institute (CHEO-RI) Department of Systems & Computer Engineering, Carleton University Ottawa, ON
Hyperparameter optimization (HPO) returns a single optimal configuration, systematically discarding the explored performance landscape that encodes how results depend on configuration choices. Fitness landscape analysis comprises a robust collection of methods from evolutionary computation offering ruggedness indices, autocorrelation functions, and fitness-distance correlation that have never been applied to machine learning hyperparameter spaces at scale, given that fine-grained ground-truth enumeration data has not existed. We present ExaTune, an open-source HPC system to produce a new artifact class: the hyperparameter landscape map. ExaTune exhaustively evaluates every point on a discretized hyperparameter grid via K-fold cross-validation, aggregates results into reproducible parquet benchmarks, and applies a suite of five formalized landscape metrics: spatial autocorrelation (Moran's I), ruggedness, local optima density, fitness-distance correlation (FDC), and variance-based parameter importance. We release a benchmark collection of 8 complete landscape maps (totalling 285,080 exhaustively evaluated configurations) spanning five model families (SVMs, random forests, XGBoost, ridge regression, and k-nearest neighbors) across six classification and two regression benchmarks, hosted on HuggingFace. Empirical analysis demonstrates that Bayesian optimization, Optuna, Hyperband, and Random Search at standard trial budgets systematically miss instability regions, secondary optima, and parameter interaction effects that are visible in complete landscapes; information that is structurally unavailable to any optimizer by design.
Figures & tables
Figure 1: Hyperparameter landscape map & the information loss of optimization-only HPO. Left: The complete hyperparameter landscape for XGBoost on the Gestational Diabetes task, with validation accuracy mapped exhaustively over learning_rate and num_estimators . The landscape reveals a global optimum on a narrow performance ridge, an adjacent instability cliff, and a secondary optimum with lower cross-run variance. Right: The same space overlaying 200 trials of Random Search and Bayesian optimization. All annotated structure is missed. ExaTune ’s landscape maps make this structure a first-class scientific artifact.
Figure 2: ExaTune system pipeline. A declarative YAML file is validated, expanded into the full Cartesian grid ΛG , and dispatched to an HPC cluster as a SLURM job array. Each worker independently estimates cross-validated performance f^(λ) and writes a JSON result. Per-job results are merged into a columnar Apache Parquet store and subsequently analyzed with the landscape metric suite (Section 4 ).
Large δO : many local attractors; local search unreliable.
Funnel index
ϕε
[0,1]
ϕε→1 : broad optimum basin; ϕε→0 : narrow peak.
Fitness-distance corr.
FDC
(−1,1)
>0.15 : big valley; local search works. <−0.15 : deceptive landscape.
Eta-squared importance
ηk2
[0,1]
Fraction of performance variance explained by parameter k alone.
Table 1: Summary of ExaTune landscape metrics. All metrics are computed over the complete grid surface f^:ΛG→R . Strategy guidance indicates the recommended HPO approach at extreme values.
Model
Dataset
Task
d
∣ΛG∣
I
R
FDC
δO
ϕ
Classification †
SVM
MNIST
C
2
0 1,000
0.966
0.035
− 0.621
—
—
Random Forest
UCI Adult
C
6
70,200
0.848
0.120
− 0.398
0.024
0.216
XGBoost
Gestational Diabetes
C
7
0 4,862
0.718
0.272
− 0.113
0.105
<0.001
XGBoost
Maternal Risk
C
7
0 4,862
0.869
0.065
− 0.120
0.293
0.086
XGBoost
Titanic
C
7
112,320
0.728
0.223
− 0.013
0.052
0.127
Table 2: ExaTune benchmark landscape collection. Landscape metrics are computed over the complete grid surface. d : number of tuned hyperparameters; ∣ΛG∣ : total evaluated configurations. I : Moran’s spatial autocorrelation ( →1 = smooth); R : ruggedness index ( →0 = smooth); FDC: fitness-distance correlation ( >0.15 = big valley, <−0.15 = deceptive); δO : optima density (fraction of grid that are strict local maxima); ϕ : funnel index (fraction of grid within ε=0.05 of the global maximum). Metrics on weighted F 1† (classification) and negative MSE ‡ (regression). ‘—’: not yet computed.
Figure 3: Landscape diversity across model families. Representative hyperparameter landscape heatmaps for (Top) California Housing:Gradient Boosting, Diabetes:Ridge Regression, Wine:KNN, (Bottom) Maternal Risk:XGBoost, Titanic:XGBoost. Color encodes cross-validated performance ( f^ ); annotations mark key topological features.
Method
Coverage
Δ score
Energy (kWh)
CO 2 (g)
Random Search
0.09–2.1%
0.0022
0.003
0.18
Bayesian Opt.
0.09–2.1%
0.0015
0.002
0.15
Hyperband
0.09–2.1%
0.0011
< 0.001
0.05
Optuna (TPE)
0.09–2.1%
0.0011
0.002
0.12
ExaTune
100%
—
3.75 †
558.8 †
† Summed over 4 of 7 datasets with complete energy records.
Table 3: HPO method comparison across 7 benchmark landscapes. Coverage: fraction of ∣ΛG∣ evaluated at a 100-trial budget (range reflects landscape sizes ∣ΛG∣∈[4,862,112,320] ). Δ score: mean absolute gap below ExaTune ’s global optimum, averaged over 7 datasets. Energy and CO 2 (CodeCarbon [ 22 ] ) summed over 4 datasets with complete ExaTune records (Diabetes, GD, Titanic, Wine); SVM/MNIST comparison data forthcoming. Full per-dataset breakdown in Appendix E .
Figure 4: Scaling behavior & carbon emission of exhaustive landscape generation. Left: Wallclock time (node-hours) as a function of grid size for each model family. Each curve traces the expected total wallclock cost as a fraction p∈0.01,0.05,0.10,…,1.00 of the full grid ∣ΛG∣ is evaluated, estimated via k=20 without-replacement bootstrap replicates at each p . The dashed grey line denotes the linear reference ∝n , i.e. total node-hours =K∑i(t^ifit+t^iscore)/3600∝n , which has unit slope in the log – log axes and serves as a baseline against which each model family’s per-configuration cost can be read as a vertical offset. The shaded region marks the feasibility frontier ( ≤100 node-hours). Right: CO 2 emissions landscape for Titanic-XGBoost ( max_depth × learning_rate ). The surface encodes per-configuration CO 2 -equivalent emissions (kg,CO 2 eq) across the exhaustively evaluated grid; the landscape is highly rugged, with sharp local peaks indicating that emissions vary non-monotonically with both hyperparameters. The configurations sampled by random search and Bayesian optimisation illustrate that neither strategy systematically avoids high-emission regions of the landscape.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Landscape
Model
Time limit
Memory
CPUs/task
MNIST
SVM (RBF)
12:00:00
8 GB
2
UCI Adult
Random Forest
02:00:00
8 GB
2
Pima Diabetes
Ridge Regression
06:10:00
2 GB
2
Gest. Diabetes
XGBoost
00:30:00
4 GB
2
Maternal Risk
XGBoost
00:30:00
4 GB
2
Titanic
XGBoost
12:30:00
4 GB
2
Appendix
Table 4: Per-task SLURM resource allocations for each benchmark landscape. Each SLURM task evaluates one grid configuration ( λ(i) ) with 5-fold CV. Total tasks per landscape =∣ΛG∣ (see Table 2 ). All jobs ran on standard Fir nodes (AMD EPYC 9655, Zen 5).
Figure 5: ExaTune dashboard for the California Housing dataset using gradient boosting.
Model (Dataset)
Hyperparameter
Type
Range / Values
mk
Spacing
SVM (MNIST)
C
F
[10−5,200]
40
Log
γ
F
[10−5,200]
25
Log
kernel
C
rbf
1
—
Grid product:
1,000
Random Forest (Adult)
n_estimators
I
[25,300] , step 25
12
L
max_depth
I
[3,15]
13
L
Appendix
Table 5: Hyperparameter grid specifications per model family. Type: I =integer, F =float, C =categorical. Spacing: L =linear (uniform step), Log =approximately logarithmic, — =unordered categorical or fixed. Fixed parameters ( italics ) are held constant; mk=1 .
Dataset
Source
n
p
Task
Metric
License
MNIST
LeCun et al. [11]
70,000
784
10-class classif.
F1 (W)
CC BY-SA 3.0
UCI Adult
Kohavi (1996), UCI ML Repo.
32,561
36 †
Binary classif.
F1 (W)
CC BY 4.0
Pima Diabetes
NIDDK / Kaggle
2,000
8
Regression ‡
Neg. MSE
CC0
Gest. Diabetes
Kaggle GDM Dataset
3,525
15
Binary classif.
F1 (W)
CC BY-NC-SA 4.0
Maternal Risk
Kaggle ( csafrit2 )
1,014
6
3-class classif.
F1 (W)
CC BY 4.0
Titanic
Kaggle competition
1,309
27 §
Binary classif.
F1 (W)
CC0
Appendix
Table 6: Benchmark dataset descriptions. n : samples as used in ExaTune (after preprocessing). p : feature dimensionality after encoding. F1 (W) = F1 weighted; Neg. MSE = negative mean squared error.
Dataset
Method
Wall-clock
Cov.
Score
Energy (kWh)
CO 2 (g)
UCI Adult (RF)
Random
56m 51s
0.14%
0.8585
0.000094
0.61
Bayesian
1h 9m 55s
0.14%
0.8587
0.010473
0.68
Hyperband
1h 19m 12s
0.14%
0.8585
0.008144
0.53
Optuna
37m 48s
0.14%
0.8585
0.006010
0.39
ExaTune
8h 26m 34s
100%
0.8592
N/A
N/A
Diabetes (Ridge)
Random
8m 45s
1.20%
0.2787
0.000062
0.004
Appendix
Table 7: Full per-dataset HPO method comparison. Energy (kWh) and CO 2 (g) measured with CodeCarbon [ 22 ] . All approximate methods use a fixed 100-trial budget. N/A : energy record not yet available. Diabetes score is R2 (negative MSE scaled); all others are F1 weighted. Bold indicates global optimum per dataset.
Hyperparameter optimization selects model and training settings to improve predictive performance, efficiency, or reliability under limited evaluation budgets.
Sberbank, Moscow 117997, Russia · Skolkovo Institute of Science and Technology (Skoltech), Moscow 121205, Russia · Federal Research Center ‘‘Computer Science and Control" of Russian Academy of Sciences (FRC CSC RAS), Moscow 119333, Russia