Multi-objective Hyperparameter Optimization with Expert Priors
Organizations: University of Freiburg · Prior Labs · ELLIS Institute Tübingen
Abstract
While Deep Learning (DL) experts often have prior knowledge about which hyperparameter settings yield strong performance, only few Hyperparameter Optimization (HPO) algorithms can leverage such prior knowledge and none incorporate priors over multiple objectives. As DL practitioners often need to optimize not just one but many objectives, this is a blind spot in the algorithmic landscape of HPO. To address this shortcoming, we propose the novel problem setting of multi-objective HPO with expert priors and introduce PriMO, the first HPO algorithm that can integrate user beliefs over multiple objectives. We show PriMO achieves state-of-the-art performance across 8 DL benchmarks in the multi-objective and single-objective setting. Further, we demonstrate that PriMO remains state-of-the-art at large scales by optimizing a vision-language model over three objectives.
Figures & tables
| Criterion | RS | EA | MOMF | MO-BO | PriMO |
| Utilize cheap approximations | ✕ | ✕ | ✓ | ✕ | ✓ |
| Integrate multi-objective expert priors | ✕ | ✕ | ✕ | ✕ | ✓ |
| Strong anytime performance | ✕ | ✕ | ✓ | ✕ | ✓ |
| Strong final performance | ✕ | (✓) | (✓) | ✓ | ✓ |
| Algorithm | Sampling time [s] | Model evaluation time [s] |
| PriMO | 10 | 8307 |
| BO+RW | 10 | 8741 |
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
| Benchmark | # Total HPs | # Continuous HPs | # Discrete HPs |
| cifar-100-wide_resnet-2048 | 5 | 4 | 1 |
| imagenet-resnet-512 | 5 | 4 | 1 |
| lm1b-transformer-2048 | 5 | 4 | 1 |
| translatewmt-xformer-64 | 5 | 4 | 1 |
| yahpo-lcbench 126026 | 7 | 4 | 3 |
| yahpo-lcbench 146212 | 7 | 4 | 3 |
| Benchmark Name | valid_error_rate (max) | train_cost (max) |
| cifar100 - wide_resnet - 2048 | 1.0 | 30 |
| imagenet - resnet - 512 | 1.0 | 5000 |
| lm1b - transformer - 2048 | 1.0 | 1000 |
| translatewmt - xformer - 64 | 1.0 | 20000 |
| Hyperparameter | Type | Log-scaled | Range | Space Type | Notes |
| lr_decay_factor | float | continuous | |||
| lr_initial | float | ✓ | continuous | ||
| lr_power | float | continuous | |||
| opt_momentum | float | ✓ | continuous | ||
| epoch | integer | discrete | fidelity |
| Hyperparameter | Type | Log-scaled | Range | Space Type | Notes |
| lr_decay_factor | float | continuous | |||
| lr_initial | float | ✓ | continuous | ||
| lr_power | float | continuous | |||
| opt_momentum | float | ✓ | continuous | ||
| epoch | integer | discrete | fidelity |
| Hyperparameter | Type | Log-scaled | Range | Space Type | Notes |
| lr_decay_factor | float | continuous | |||
| lr_initial | float | ✓ | continuous | ||
| lr_power | float | continuous | |||
| opt_momentum | float | ✓ | continuous | ||
| epoch | integer | discrete | fidelity |
| Hyperparameter | Type | Log-scaled | Range | Space Type | Notes |
| lr_decay_factor | float | continuous | |||
| lr_initial | float | ✓ | continuous | ||
| lr_power | float | continuous | |||
| opt_momentum | float | ✓ | continuous | ||
| epoch | integer | discrete | fidelity |
| Task ID | val_cross_entropy (max) | time (max, seconds) |
| 126026 | 1.0 | 150 |
| 146212 | 1.0 | 150 |
| 168330 | 1.0 | 5000 |
| 168868 | 1.0 | 200 |
| Hyperparameter | Type | Log-scaled | Range | Space Type | Notes |
| batch_size | integer | ✓ | discrete | ||
| learning_rate | float | ✓ | continuous | ||
| momentum | float | continuous | |||
| weight_decay | float | continuous | |||
| num_layers | integer | discrete | |||
| max_units | integer | ✓ | discrete |
| Hyperparameter | Type | Log-scaled | Range/Set | Space Type | Notes |
| lr | float | ✓ | continuous | ||
| warmup_fraction | float | continuous | |||
| vision_width | integer | discrete | |||
| text_width | integer | discrete | |||
| epochs | integer | discrete | fidelity |
| Benchmark | PriMO Best Error | BO+RW Best Error | PriMO Min Cost | BO+RW Min Cost | Superior Algorithm |
| cifar-100 | 0.185 | 0.188 | 10.46 | 11.27 | PriMO |
| imagenet | 0.178 | 0.218 | 696.0 | 707.0 | PriMO |
| lm1b-transformer | 0.586 | 0.605 | 119.5 | 117.9 | - |
| translate-wmt-xformer | 0.303 | 0.347 | 4120 | 4140 | PriMO |
| LCBench-126026 | 0.0307 | 0.0390 | 14.97 | 1.43 | - |
| LCBench-146212 | 0.0509 | 0.0841 | 7.94 | 9.49 | PriMO |
Hyperparameter Optimization
Hyperparameter optimization selects model and training settings to improve predictive performance, efficiency, or reliability under limited evaluation budgets.
See all 179 papers