Simple Meta-Learning for HPO and CASH Using Performance Representations
Organizations: Delft University of Technology, Delft, The Netherlands
Abstract
Bayesian optimization for Hyperparameter Optimization (HPO) and Combined Algorithm Selection and Hyperparameter Optimization (CASH) often relies on explicit hyperparameter encodings as surrogate inputs. In high-dimensional search spaces, however, such representations can make it difficult for standard surrogates to learn useful performance models. We instead study an alternative meta-learning representation that describes each configuration directly as its historical performance across prior tasks. This representation can be paired directly with standard surrogate models. Our scope is the iterative selection component of AutoML, deciding which configuration to evaluate next from a fixed candidate set, rather than a complete framework with features such as runtime estimation, cost-aware budgeting and ensembling. We further apply this representation to maximal marginal relevance (MMR), a diversity-based re-ranking criterion from recommender systems, as a posterior-free acquisition rule. When selecting and optimising across multiple models on the TabArena benchmark, the performance-based representation paired with a linear Bayesian ridge regression model outperforms dedicated meta-learning baselines, achieving the same tuned performance in roughly half the number of trials. For single-model HPO, it remains a competitive second. Across both benchmarks this method closely matches or improves on regret at a fraction of the complexity and cost, requiring no pre-training or tuning and scaling independently of search space dimensionality. Code for reproducing our results is available at https://github.com/EWitting/simple-metalearning.
Figures & tables
| Method | Line of work | Config. representation | Surrogate | Selection |
|---|---|---|---|---|
| SMAC3 ( Lindauer et al., 2022 ) | BO | Hyperparameter | Random forest | EI |
| GP-BO ( Snoek et al., 2012 ) | BO | Hyperparameter | Gaussian process | EI |
| Average ranking ( Brazdil et al., 2022 ) | AS | Performance | None (ranking) | Static |
| Active Testing ( Leite et al., 2012 ) | AS | Performance | None (matrix) | Active |
| Multi-task GP ( Swersky et al., 2013 ) | Meta-learning BO | Hyperparameter | Multi-task GP | EI |
| RGPE ( Feurer et al., 2018 ) | Meta-learning BO | Hyperparameter | GP ensemble | EI |
| CASH | HPO | ||||||
| Model | Input | Regret | Rank | Time/task | Regret | Rank | Time/task |
| Linear | Perf. | 12.19 ∗ | 4.34 ∗ | 2.7s | 9.84 ∗ | 6.14 ∗ | 1.1s |
| Hyper. | 14.59 | 5.95 | 24s | 12.79 | 7.04 | 0.7s | |
| Random Forest | Perf. | 18.53 ∗ | 5.83 ∗ | 2.0m | 11.29 | 6.51 | 1.8m |
| Hyper. | 28.59 | 8.06 | 2.7m | 13.46 | 6.59 | 53s | |
| Gaussian Process | Perf. | 23.73 | 6.99 | 7.3m | 13.40 | 6.87 | 1.5m |
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | Config key | Final hyperparameters |
| Linear-Hyper / Linear-Perf | ridge_ei / perf_vector_ridge_ei | BayesianRidge, scikit-learn defaults ( , max_iter=300 , tol=1e-3 ); Linear-Perf fits without intercept (Section C.1 ) |
| RF-Hyper | rf_ei | n_estimators=100 , min_samples_leaf=3 , random_state=42 |
| RF-Perf | perf_vector_rf_ei | n_estimators=100 , min_samples_leaf=1 (default), random_state=42 |
| GP-Hyper / GP-Perf | botorch_gp_ei / botorch_gp_perf_{cash,hpo} | BoTorch SingleTaskGP , Matérn-5/2 ARD kernel, lengthscale prior , outputscale prior , L-BFGS-B MLL optimization (Section B.6 ) |
| TwoTower-Hyper / TwoTower-Perf | two_tower_hyper_* / two_tower_perf_* | random_state=42 ; remaining settings tuned per benchmark by random search, see Table 4 |
| ABLR | ablr | feature dim 50, hidden layers , 300 epochs, lr=0.01, random_state=42 |
| Variant | Benchmark | Emb. dim | LR | Batch size | Hyper. tower |
|---|---|---|---|---|---|
| Two-Tower-Hyper | CASH | 8 | 0.007 | 1024 | |
| Two-Tower-Hyper | HPO | 8 | 0.007 | 256 | |
| Two-Tower-Perf | CASH | 8 | 0.005 | 1024 | — |
| Two-Tower-Perf | HPO | 8 | 0.020 | 512 | — |
| Model family | Wall-clock time |
|---|---|
| Two-Tower | 9h 25m |
| Gaussian Process | 7h 43m |
| Random Forest | 5h 40m |
| Meta-learning baselines | 2h 57m |
| Model-free | 0h 54m |
| Linear | 0h 41m |
| Model | Category | Configurations |
|---|---|---|
| XGBoost | Gradient boosting | 201 |
| LightGBM | Gradient boosting | 201 |
| LightGBM (AIO) | Gradient boosting | 201 |
| CatBoost | Gradient boosting | 201 |
| Extra-Trees | Tree ensemble | 201 |
| Random Forest | Tree ensemble | 201 |
| Hyperparameter | Type | Range | Scale |
|---|---|---|---|
| learning_rate | Float | Log | |
| max_depth | Integer | Log | |
| min_child_weight | Float | Log | |
| subsample | Float | Linear | |
| colsample_bylevel | Float | Linear | |
| colsample_bynode | Float | Linear |
Hyperparameter Optimization
Hyperparameter optimization selects model and training settings to improve predictive performance, efficiency, or reliability under limited evaluation budgets.
See all 178 papers