While Deep Learning (DL) experts often have prior knowledge about which hyperparameter settings yield strong performance, only few Hyperparameter Optimization (HPO) algorithms can leverage such prior knowledge and none incorporate priors over multiple objectives. As DL practitioners often need to optimize not just one but many objectives, this is a blind spot in the algorithmic landscape of HPO. To address this shortcoming, we propose the novel problem setting of multi-objective HPO with expert priors and introduce PriMO, the first HPO algorithm that can integrate user beliefs over multiple objectives. We show PriMO achieves state-of-the-art performance across 8 DL benchmarks in the multi-objective and single-objective setting. Further, we demonstrate that PriMO remains state-of-the-art at large scales by optimizing a vision-language model over three objectives.
Figures & tables
Criterion
RS
EA
MOMF
MO-BO
PriMO
Utilize cheap approximations
✕
✕
✓
✕
✓
Integrate multi-objective expert priors
✕
✕
✕
✕
✓
Strong anytime performance
✕
✕
✓
✕
✓
Strong final performance
✕
(✓)
(✓)
✓
✓
Table 1: Comparison of our algorithm PriMO to prominent categories of multi-objective algorithms with respect to the identified desiderata. The algorithmic categories include Evolutionary algorithms (EA, e.g. NSGA-II , SMS-EMOA ), multi-objective multi-fidelity algorithms ( MOMF , e.g. , MOASHA , MO-HyperBand, HyperBand with Random Weights), and multi-objective Bayesian optimization ( MO-BO , e.g. , BO with random weights, BO with EHVI , ParEGO ). A ✓ indicates that the method satisfies the criterion; a ✕ indicates it does not. (✓) denotes partial fulfillment or fulfillment with additional assumptions.
Algorithm
Sampling time [s]
Model evaluation time [s]
PriMO
10 ±1
8307 ±105
BO+RW
10 ±1
8741 ±215
Table 2: Comparison of the average HPO sampling times of PriMO and BO+RW , and the evaluation cost of language modeling with a transformer on the 1B word benchmark.
Figure 1: Decay of the prior exponent γ with the increase in the number of BO samples for PriMO and the original π BO implementation. Here, nd refers to the search spaces dimensions, which we fix to 4 - the size of the search space of the PD1 benchmarks. We note that the π BO authors set β=10 .
Figure 2: Mean relative ranks ± 1 standard error of PriMO and prominent multi-objective algorithms across benchmarks and seeds under various prior conditions.
Figure 3: Mean relative ranks ± 1 standard error in the single-objective setting across benchmarks and seeds under various prior conditions.
Figure 4: Case study on the OpenCLIP pipeline across 3 objectives over 20 seeds. [Panel (a)] Mean dominated hypervolume vs. cumulative compute cost (in PFLOPs). [Panels (b), (c), and (d)] The trajectories of the best-found (incumbent) values for model parameter size, ImageNet1k error, and MS-COCO error. PriMO outperforms all other baselines, achieving both the highest hypervolume and the best individual values for each objective within the 10 PFLOP budget.
Figure 5: Mean relative ranks ± 1 standard error of various ablations of PriMO across benchmarks and seeds under various prior conditions.
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
Benchmark
# Total HPs
# Continuous HPs
# Discrete HPs
cifar-100-wide_resnet-2048
5
4
1
imagenet-resnet-512
5
4
1
lm1b-transformer-2048
5
4
1
translatewmt-xformer-64
5
4
1
yahpo-lcbench 126026
7
4
3
yahpo-lcbench 146212
7
4
3
Appendix
Table 3: Overview of benchmark search spaces.
Benchmark Name
valid_error_rate (max)
train_cost (max)
cifar100 - wide_resnet - 2048
1.0
30
imagenet - resnet - 512
1.0
5000
lm1b - transformer - 2048
1.0
1000
translatewmt - xformer - 64
1.0
20000
Appendix
Table 4: Reference values for valid_error_rate and train_cost objectives across PD1 benchmarks for HVI calculation.
Hyperparameter
Type
Log-scaled
Range
Space Type
Notes
lr_decay_factor
float
[0.010093,0.989012]
continuous
lr_initial
float
✓
[0.000010,9.779176]
continuous
lr_power
float
[0.100708,1.999376]
continuous
opt_momentum
float
✓
[0.000059,0.998993]
continuous
epoch
integer
[1,52]
discrete
fidelity
Appendix
Table 5: Hyperparameter search space table of the cifar-100-wide_resnet-2048 benchmark, including the hyperparameter ranges and fidelity bounds of epoch , as given in mf-prior-bench .
Hyperparameter
Type
Log-scaled
Range
Space Type
Notes
lr_decay_factor
float
[0.010294,0.989753]
continuous
lr_initial
float
✓
[1e−5,9.774312]
continuous
lr_power
float
[0.100225,1.999326]
continuous
opt_momentum
float
✓
[5.9e−5,0.998993]
continuous
epoch
integer
[1,99]
discrete
fidelity
Appendix
Table 6: Approximate hyperparameter search space table of the imagenet-resnet-512 benchmark, including hyperparameter ranges and fidelity bounds of epoch . Exact ranges are provided by mf-prior-bench .
Hyperparameter
Type
Log-scaled
Range
Space Type
Notes
lr_decay_factor
float
[0.010543,0.9885653]
continuous
lr_initial
float
✓
[1e−5,9.986256]
continuous
lr_power
float
[0.100811,1.999659]
continuous
opt_momentum
float
✓
[5.9e−5,0.9989986]
continuous
epoch
integer
[1,74]
discrete
fidelity
Appendix
Table 7: Hyperparameter search space of the lm1b-transformer-2048 benchmark, with the fidelity epoch as given in mf-prior-bench .
Hyperparameter
Type
Log-scaled
Range
Space Type
Notes
lr_decay_factor
float
[0.010022,0.988565]
continuous
lr_initial
float
✓
[1.00276e−5,9.842248]
continuous
lr_power
float
[0.100425,1.998593]
continuous
opt_momentum
float
✓
[5.86114e−5,0.998999]
continuous
epoch
integer
[1,19]
discrete
fidelity
Appendix
Table 8: Search space and fidelity epoch of the translatewmt-xformer-64 benchmark, as given in mf-prior-bench .
Task ID
val_cross_entropy (max)
time (max, seconds)
126026
1.0
150
146212
1.0
150
168330
1.0
5000
168868
1.0
200
Appendix
Table 9: Reference values for val_cross_entropy and time objectives across selected LCBench tasks, for HVI calculation.
Hyperparameter
Type
Log-scaled
Range
Space Type
Notes
batch_size
integer
✓
[16,512]
discrete
learning_rate
float
✓
[1e−4,0.1]
continuous
momentum
float
[0.1,0.99]
continuous
weight_decay
float
[1e−5,0.1]
continuous
num_layers
integer
[1,5]
discrete
max_units
integer
✓
[64,1024]
discrete
Appendix
Table 10: Hyperparameter search space table of the yahpo-lcbench benchmarks. This includes the hyperparameter ranges and types as typically defined in the YAHPO-Gym benchmark suite.
Hyperparameter
Type
Log-scaled
Range/Set
Space Type
Notes
lr
float
✓
[0.0001,0.01]
continuous
warmup_fraction
float
[0.0,0.75]
continuous
vision_width
integer
{32,64,128,192,256}
discrete
text_width
integer
{32,64,128,192,256}
discrete
epochs
integer
[1,10]
discrete
fidelity
Appendix
Table 11: Hyperparameter search space for the OpenCLIP benchmark. The epochs parameter serves as the fidelity for the optimization process.
Figure 6: Mean relative ranks ± 1 standard error of various ablations of the ϵ - BO of PriMO across benchmarks and seeds under various prior conditions.
Figure 7: Sensitivity study of ϵ in PriMO’s ϵ - BO using mean dominated Hypervolume ± 1 standard error across seeds for each benchmark. PriMO is under all good priors setting here.
Figure 8: Mean dominated Hypervolume ± 1 standard error of PriMO and prominent multi-objective algorithms across seeds for each benchmark. PriMO is under all good priors setting here.
Figure 9: Shown here are the Pareto fronts obtained by PriMO, compared to other non-prior MO baselines under all good prior conditions.
Benchmark
PriMO Best Error
BO+RW Best Error
PriMO Min Cost
BO+RW Min Cost
Superior Algorithm
cifar-100
0.185
0.188
10.46
11.27
PriMO
imagenet
0.178
0.218
696.0
707.0
PriMO
lm1b-transformer
0.586
0.605
119.5
117.9
-
translate-wmt-xformer
0.303
0.347
4120
4140
PriMO
LCBench-126026
0.0307
0.0390
14.97
1.43
-
LCBench-146212
0.0509
0.0841
7.94
9.49
PriMO
Appendix
Table 12: Case level comparison of performance of PriMO against BO+RW on 8 Deep Learning benchmarks based on aggregated Pareto fronts.
Figure 10: Mean relative ranks ± 1 standard error of baselines we constructed to use multi-objective priors and PriMO across benchmarks and seeds under various prior conditions.
Figure 11: Comparing the average dominated HVI over 20 evaluations and across 25 seeds, between PriMO and prior-based baselines MO-Priorband, π BO+RW and RS + Prior, under all good prior conditions.
Figure 12: Mean dominated HV across 8 Deep Learning benchmarks, showcasing remarkable recovery of PriMO from bad priors, compared against some prominent non-prior baselines.
Figure 13: Dominated HV plot comparing PriMO against prior-based baselines that were adapted to MO , under bad priors.
Figure 14: Comparing PriMO against some prominent non-prior MO baselines under the overall prior setting, with respect to the mean dominated HV across all 8 benchmarks.
Figure 15: Critical Difference diagrams at 10 evaluations comparing early performance of PriMO against the baselines – BO+RW , π BO+RW, MO-Priorband (MO-PB), MOASHA , HB+RW , MO-TPE, LogNEHVI, Random Search ( RS ), ParEGO and NSGA-II , under various prior conditions.
Figure 16: CD diagrams at 20 evaluations comparing final performance of PriMO against other non-prior and prior-guided (adapted to MO ) baselines, under various prior conditions.
Hyperparameter optimization selects model and training settings to improve predictive performance, efficiency, or reliability under limited evaluation budgets.
Large-scale hyperparameter optimization (HPO) in automated machine learning (AutoML) consumes substantial computational resources, raising growing concerns about scalability and energy efficiency. Existing methods use prior information heuristically to accelerate both black-box and multi-fidelity settings, but they lack a characterization of how prior informativeness quantitatively reduces sample complexity. In this work, we provide the first distribution-dependent sample complexity bounds for multi-fidelity HPO with priors through the formal lens of fixed-budget best-arm identification. By modeling priors directly over arm means as configuration performance, we derive explicit, distribution-dependent error bounds that quantify the relationship between priors and evaluation budget. Our analysis shows that informative priors, which concentrate probability mass on near-optimal arms, yield reductions in the number of required evaluations, whereas baseline performance is recovered with uninformative or misleading priors. We conduct proof-of-concept experiments on a synthetic benchmark and on LCBench, a common multi-fidelity HPO benchmark for deep learning, to confirm our theoretical results, achieving up to 90% budget reduction while retaining solution quality. Together, our results provide a principled foundation for prior-guided and compute-efficient green AutoML.
Leona Hennig, Jasmin Brandt, Lukas Fehring +3
Institute of Artificial Intelligence Leibniz University Hanover, Germany · University of Bielefeld Bielefeld, Germany · Institute of Artificial Intelligence, Leibniz University Hanover & L3S Research Center Hanover, Germany
Large language models (LLMs) are increasingly used as heuristic advisors for black-box optimization, yet their suggestions and self-reported confidence are not necessarily calibrated to downstream objective values. This issue becomes more pronounced in multi-objective Bayesian optimization, where different objectives may require different expert knowledge and where an LLM expert can be useful for one objective but misleading for another. We study how to use LLM-generated expert priors in discrete multi-objective Bayesian optimization without blindly trusting them. We propose an objective-wise reputation-market mechanism that treats each expert-objective pair as a falsifiable prior source. Expert weights are updated online from observed objective feedback, discounted over time, and gated by market-level trust. We then introduce a decoupled counterfactual gate that can use the LLM prior without confidence, use it with confidence, or abstain from the LLM prior entirely. Across controlled synthetic stress tests and three molecule optimization benchmarks with \qwenflash{}-generated expert priors, we find that dynamic objective-wise calibration improves robustness over fixed LLM priors. However, raw LLM confidence is not reliably beneficial: on ESOL, confidence is positively correlated with prediction error; on FreeSolv, confidence can help; and on Lipophilicity, ignoring confidence remains strongest. Our fixed three-arm counterfactual gate improves over the first counterfactual variant on ESOL and FreeSolv, while an attempted margin portfolio exposes a useful negative result: margin selection should be acquisition-aware rather than based only on one-step prior error.
Jiangyu Chen, Ban Yi
State Key Laboratory for Novel Software Technology, Nanjing University
Hyperparameter Optimization (HPO) is essential for building high-performing ML/DL models, yet conventional optimizers often struggle in high-dimensional spaces where evaluations are costly and progress is diluted across many low-impact variables. We propose Greedy Importance First (GIF), an importance-aware scheduling strategy that uses a small-sample warm start to estimate hyperparameter importance, forms importance-based groups, allocates trials proportionally, and retains a full-space fallback. We evaluate GIF under fixed evaluation budgets on five anisotropic analytic functions, Bayesmark, and NAS-Bench-301. On the higher-dimensional benchmarks, GIF reaches better incumbents with faster convergence than TPE, BOHB, Random Search, and Sequential Grouping. On Bayesmark, where the effective dimensionality is smaller, GIF remains competitive but the margins are smaller. Ablation studies show that importance estimation, proportional allocation, and the fallback step all contribute to the gains. We also verify that the HIA component recovers the intended anisotropy on the analytic benchmarks. These results suggest that GIF is a simple and plug-compatible way to improve sample efficiency in high-dimensional HPO.