Early-Stopping Thresholds for ES-HyperNEAT: A Data-Driven Approach from Fitness Dynamics
Authors: Romain Claret, Arthur Gygax, Michael O'Neill, Paul Cotofrei, Pascal Felber
Organizations: University of Neuchâtel, Switzerland · Information Management Institute · Computer Science Department · Natural Computing Research & Applications Group, University College Dublin, Ireland
Most hyperparameter configurations for Evolvable-Substrate HyperNEAT (ES-HyperNEAT) produce networks that stagnate at random-guessing performance, wasting computational resources. We frame early stopping as binary classification on early fitness trajectories: for each trial, we compute the cumulative median of best-per-generation fitness and test it against a threshold derived by maximizing the F1 score on an initial 90-trial dataset. The resulting rule (generation G* = 3, threshold T* = 0.140) achieves F1 = 0.872 on 180 independent validation trials, retaining over 90% of successful trials while cutting computational cost by 41.6%. Compared to Hyperband, our domain-specific rule is 64% more efficient with higher mean fitness, though Hyperband occasionally discovers higher peak solutions. On a converged search population the rule becomes too aggressive (recall 31.1%), motivating adaptive thresholds. The specific thresholds are ES-HyperNEAT-specific, but the methodology, deriving stopping criteria from fitness dynamics classification, is applicable to other evolutionary algorithms with stagnation-prone hyperparameter spaces.
Figures & tables
Success Def.
Rule ( G∗,T∗ )
Recall
Precision
F1 - Score
Savings (%)
50th
(13, 0.145)
0.52
1.00
0.68
11.4
75th
(4, 0.143)
0.86
0.97
0.91
38.2
80th
(3, 0.140)
0.90
0.84
0.87
41.6
90th
(3, 0.140)
0.94
0.70
0.80
41.6
TABLE I: Sensitivity analysis for the success threshold percentile. Each rule was derived on the initial dataset and evaluated on the validation set.
Fig. 1: Fitness trajectories of “Successful” vs. “Unsuccessful” trials. The plot shows the mean of the median best fitness for trials labeled as “Successful” (n=19) and “Unsuccessful” (n=71).
Fig. 2: F1 score for pruning thresholds at generation G∗=3 . The plot shows the F1 score, precision, and recall for different median best fitness thresholds. The threshold that maximizes the F1 score represents the optimal trade-off between identifying unsuccessful trials and retaining successful ones.
Predicted Outcome
Actual Outcome
Successful
Unsuccessful (Pruned)
Successful
75
8
Unsuccessful
14
83
TABLE II: Confusion matrix for the pruner’s performance on the validation set. Rows represent the actual final outcome, while columns represent the pruner’s prediction.
Performance Metric
Value
Precision
0.843
Recall (Sensitivity)
0.904
Specificity
0.856
F1 Score
0.872
Accuracy
0.878
Matthews Correl. Coeff. (MCC)
0.757
TABLE III: Key performance metrics calculated from the validation confusion matrix.
Performance Metric
Our Method
Hyper- band
Baseline
Best Final Fitness Found
0.225
0.250
0.275
Avg. Top 5% Fitness
0.206
0.195
0.241
Total Cost (Gens)
10,656
29,574
40,260
Trials to 0.25
Not Reached
569
359
TABLE IV: Comparative performance between our custom pruner, the standard Hyperband pruner, and the unpruned TPE baseline.
Comparison
Mean Diff
p -value
Sig.
Cohen’s d
Our vs. HbP
0.052
<.001
Yes
1.884
Baseline vs. HbP
0.058
<.001
Yes
1.555
Our vs. Baseline
-0.006
0.069
No
-
TABLE V: Tukey’s HSD post-hoc test results comparing the three methods.
Equivalence Test Metric
Value
Equivalence Margin ( Δ )
± 0.020
Observed Mean Difference
-0.0059
TOST p -value
<.001
Conclusion
Statistically Equivalent
TABLE VI: Results of the TOST equivalence test comparing our method to the unpruned baseline.
Fig. 3: Fitness trajectories of the 8 successful trials that were incorrectly pruned (False Negatives). These trials failed to surpass the fitness threshold of 0.140 (red dashed line) by the pruning generation G∗=3 (vertical line).
Experimental Group
t -statistic
p -value
Cohen’s d
Random Search (N=90)
−0.88
0.378
−0.13
Random Unpruned (N=90)
−0.65
0.519
−0.10
Random Pruned (N=90)
−1.00
0.321
−0.15
TABLE VII: Results of paired-samples t -tests comparing final performance metrics ( Last Median Best vs. Last Best ) within each experimental group.
TABLE VIII: F1 -scores and 95% confidence intervals for the paper’s pruning rules, evaluated on the curated 180-trial validation set. Confidence intervals were calculated via bootstrapping (N=1,000 resamples).
Late-Stage Simulation Metric
Value
Simulated Computational Savings
73.2 %
F1 -Score on Late-Stage Trials
0.402
Recall on Top 20% Late-Stage Trials
31.1 %
Mean Fitness of Kept Trials
0.1870
Mean Fitness of Pruned Trials
0.1500
Statistical Separation ( p -value)
<.001
TABLE IX: Results of a counterfactual simulation applying our G∗=3 pruning rule to the final 824 trials of a prior dataset.
Fig. 4: TPE search progress with our pruner simulated on the prior dataset. Gray crosses mark trials the pruner would have dismissed; blue dots mark trials it would have kept. The dashed gray line shows the true best fitness found over all trials (unpruned baseline), while the solid orange line shows the best fitness achieved using only kept trials.
Fig. 5: Fitness trajectories of the “slow starter” trials incorrectly pruned during the baseline simulation (False Negatives).