Hyperparameter tuning is a challenging problem especially when the system itself involves uncertainty. Due to noisy function evaluations, optimization under uncertainty can be computationally expensive. In this paper, we present a novel Bayesian optimization framework tailored for hyperparameter tuning under uncertainty, with a focus on optimizing a scale- or precision-type parameter in stochastic models. The proposed method employs a statistical surrogate for the underlying random variable, enabling analytical evaluation of the expectation operator. Moreover, we derive a closed-form expression for the optimizer of the random acquisition function, which significantly reduces computational cost per iteration. Numerical experiments demonstrate substantial gains in optimization efficiency: about 40-fold reduction in data usage relative to a Monte Carlo-based one-dimensional optimization scheme, and an order of magnitude improvement relative to a Gaussian process-based Bayesian optimization baseline.
Figures & tables
Figure 1: Conditional model of s∣β . The left panel shows the mean of s as a power function of β for positive and negative exponents, with dashed curves indicating the 2.5th and 97.5th conditional percentiles. The right panel presents the corresponding log–log transformations ( lns vs. lnβ ), where the relationship is linear.
Figure 2: Static problem: raw data and surrogate model fit. (Left) Raw data is highly noisy with a downward linear trend. Plotted points are 1% of the data, uniformly subsampled. (Right) Bayesian GLM constructed during Bayesian optimization captures the GLM regression function accurately with quantified error, using much fewer data points.
Figure 3: Static problem: optimization results under different approaches. (Left) Posterior distribution of (β∗,f∗) well captures the true statistics. (Right) Histogram of β∗ posterior samples.
Figure 4: Static problem: progression of Bayesian optimization. (Left) Sample points during BO iterations. (Right) Evolution of β∗ posterior distribution and GLM estimate.
Table 1: Comparison of data points, Computational time, absolute accuracy, and standard deviation of posterior samples for different methods across three examples. Ratios in parentheses are relative to Bayesian GLM.
Figure 7: Static problem: progression of one-dimensional Monte-Carlo based optimization. (Left) β sample points during one-dimensional optimization iterations with 1000 MC samples. (Right) Objective function versus β , with mean trends and local regression estimates.
Figure 8: Static problem: optimization results. (Left) β∗ obtained via GP surrogate with heteroscedastic noise and Thompson sampling acquisition function. (Right) Histogram of β∗ posterior samples.
Figure 9: Static problem: optimization results. (Left) β∗ obtained via GP surrogate with qNEI acquisition function. (Right) Histogram of β∗ posterior samples.
Figure 10: Dynamics problem: (a) space structure; (b) loading.
Figure 11: Dynamics problem: raw data and surrogate model fit. (Left) Raw data is highly noisy with a downward linear trend. Plotted points are 1% of the data, uniformly subsampled. (Right) Bayesian GLM captures the GLM regression function accurately with quantified error, using much fewer data points.
Figure 12: Dynamics problem: optimization results under different approaches. (Left) Posterior distribution of (β∗,f∗) well captures the true statistics. (Right) Histogram of β∗ posterior samples.
Figure 13: Dynamics problem: progression of Bayesian optimization. (Left) Sample points during BO iterations. (Right) Evolution of β∗ posterior distribution and GLM estimate.
Figure 14: Dynamics problem: progression of one-dimensional Monte-Carlo based optimization. (Left) β sample points during one-dimensional optimization iterations with 1000 MC samples. (Right) Objective function versus β , with mean trends and local regression estimates.
Figure 15: Dynamic problem: optimization results. (Left) β∗ obtained via GP surrogate with heteroscedastic noise and Thompson sampling acquisition function. (Right) Histogram of β∗ posterior samples.
Figure 16: Dynamic problem: optimization results. (Left) β∗ obtained via GP surrogate with qNEI acquisition function. (Right) Histogram of β∗ posterior samples.
Figure 17: Diagnosis of residual distributions. (Left) Statistics of the residual distribution conditional on β . (Right) Histogram of the residual distribution for β=150 , and the best-fit PDFs using log-normal and Gamma distributions.
Figure 18: Non-power-law problem: raw data and surrogate model fit. (Left) Raw data is highly noisy with a downward linear trend. Plotted points are 1% of the data, uniformly subsampled. (Right) Bayesian GLM captures the local regression function accurately near the optima using much fewer data points.
Figure 19: Non-power-law problem: optimization results under different approaches. (Left) Posterior distribution of (β∗,f∗) well captures the true statistics. (Right) Histogram of posterior samples of β∗ , indicating residual bias due to posterior mass falling outside the optimal region.
Figure 20: Non-power-law problem: progression of Bayesian optimization. (Left) Sample points during BO iterations. (Right) Evolution of β∗ posterior distribution and GLM estimate.
Figure 21: Non-power-law problem: progression of one-dimensional Monte-Carlo based optimization. (Left) β sample points during one-dimensional optimization iterations with 1000 MC samples. (Right) Objective function versus β , with mean trends and local regression estimates.
Figure 22: Non-power-law problem: optimization results. (Left) β∗ obtained via GP surrogate with heteroscedastic noise and Thompson sampling acquisition function. (Right) Histogram of β∗ posterior samples.
Figure 23: Non-power-law problem: optimization results. (Left) β∗ obtained via GP surrogate with qNEI acquisition function. (Right) Histogram of β∗ posterior samples.
Example
Metric
Run 1
Run 2
Run 3
Static
β^
98
98
97
# evaluations
280
290
290
Dynamics
β^
40
43
43
# evaluations
280
290
290
Non power-law
β^
31
30
30
# evaluations
300
300
290
Table 2: Initialization sensitivity study across three examples. Three runs correspond to different initial datasets (all other settings fixed).
Example
Method
Mean β^∗± std
Mean σ(β∗)± std
Mean Data Points
Static ( βLR∗=103 )
Bayesian GLM
97.9±1.63
3.82±0.27
309
GP-BO (TS)
101.3±6.26
6.44±3.39
5100
GP-BO (qNEI)
97.2±5.37
6.94±4.04
5100
Dynamics ( βLR∗=40 )
Bayesian GLM
41.8±1.34
1.78±0.20
340
GP-BO (TS)
37.2±1.66
1.32±0.46
4727
GP-BO (qNEI)
38.5±2.80
1.45±0.68
4407
Table 3: Multi-seed statistical evaluation over Nseed=15 independent runs. For each method and example, we report the mean and standard deviation of the estimated optimal hyperparameter, the standard deviation of the posterior samples, σ(β∗) , and the mean number of data points.
The problem of multiobjective optimization under uncertainties is often approached by taking the expectation of each objective. In this work, we propose instead to formulate this as a Bayesian decision problem and to rely on the expected value of the hypervolume, which is to be maximized with respect to a finite set of input points. We show that this can be performed using methods based on gradients in a stochastic optimization framework, provided that care is taken with respect to dominated points. Moreover, in the absence of readily available differentiable code, we propose to use Gaussian Processes as differentiable surrogate models, in order to perform the optimization. An additional contribution in this work are some active learning strategies, through acquisition functions which helps construct a surrogate model well-designed for the multiobjective optimization problem at stake. These strategies are compared on simple analytical problems to assess their performances.
Bayesian optimization is widely used for hyperparameter optimization when model evaluations are expensive; however, noisy acquisition estimates can lead to unstable decisions. We identify acquisition estimation noise as a failure mode that was previously overlooked: even when the surrogate model and acquisition target are correctly specified, finite-sample Monte Carlo error can perturb acquisition values. This can, in turn, flip candidate rankings and lead to suboptimal BO decisions. As a remedy, we aim at variance reduction and propose an orthogonal acquisition estimator that subtracts an optimally weighted score-function control variate, which yields an acquisition residual orthogonal to posterior score directions and which thus reduces Monte Carlo variance. We further introduce OrthoBO: a Bayesian optimization framework that combines our orthogonal acquisition estimator with ensemble surrogates and an outer log transformation. We show theoretically that our estimator preserves the target, leads to variance reduction, and improves pairwise ranking stability. We further verify the theoretical properties of OrthoBO through numerical experiments where our framework reduces acquisition estimation variance, stabilizes candidate rankings, and achieves strong performance. We also demonstrate the downstream utility of OrthoBO in hyperparameter optimization for neural network training and fine-tuning.
Maresa Schröder, Pascal Janetzky, Michael Klar +1
LMU Munich · Munich Center for Machine Learning · Bosch Center for Artificial Intelligence
Bayesian optimization (BO) is widely used as a standard approach for expensive black-box optimization. However, BO algorithms often involve parameters that must be specified in advance, and their performance can strongly depend on these choices. We propose a framework for optimizing such parameters using sample paths drawn from a Gaussian process (GP) inferred from the information available at the start of BO. We use cumulative regret as the performance metric for a BO algorithm. By running the BO algorithm on the generated sample paths, we obtain an empirical estimate of its expected cumulative regret for a given parameter configuration. Optimizing this estimate allows us to identify parameter configurations that, given the currently available information, are expected to achieve low cumulative regret. Since this parameter optimization is itself a black-box optimization problem, we employ another BO procedure to solve it, which we refer to as outer BO. Through experiments, we demonstrate that the proposed framework can effectively select parameter configurations that achieve strong performance among a range of candidate configurations.