Bayesian Optimization with Fisher Information Geometry: Gradient Bounds and Trust-Region Methods

Saksham Kiroriwal; Julius Pfrommer; Jürgen Beyerer

cs.LGSep 25, 2026

Bayesian Optimization with Fisher Information Geometry: Gradient Bounds and Trust-Region Methods

Authors: Saksham Kiroriwal, Julius Pfrommer, Jürgen Beyerer

Organizations: Cognitive Industrial Systems, Fraunhofer IOSB, Karlsruhe, Germany · Karlsruhe Institute of Technology (KIT), Karlsruhe, Germany

Abstract

We study Bayesian optimization (BO) through the lens of information geometry. Pulling back the Fisher information metric through the surrogate posterior map yields a local sensitivity tensor on the input space, which leads to an upper bound on the gradient of reparameterizable acquisition functions. This view explains vanishing-gradient behavior in high-dimensional BO and provides a common interpretation of heuristics such as RAASP and dimension-scaled lengthscales. Building on this analysis, we propose FITR, a trust-region-based BO method that replaces lengthscale-based scaling by local pullback-Fisher weights. FITR is not restricted to GP kernels with explicit lengthscales. On GP benchmarks with an SE kernel, experiments show competitive performance using FITR. The proposed method also easily generalizes to non-isotropic surrogates, although the gains are more task-dependent in that setting.

Figures & tables

Figure 1: First: Conventional BO view with a GP surrogate (SE kernel) and LogEI [ 1 ] acquisition function (AF). Second: Visualization of Proposition 4.2 , where the acquisition gradient is upper-bounded by the product of the trace of the pullback Fisher tensor and the sensitivity factor of the AF. Third: Induced geometry on the input space using the pullback Fisher tensor. Fourth: Sensitivity $C_{\alpha}$ (Section 4 ) of the AF with respect to the predictive-distribution parameters on the statistical manifold. $\boldsymbol{+}$ denotes $x=0$ and $\boldsymbol{\times}$ denotes $x=1$ in input space (the same markers are used to denote the respective points in the manifold view). Gray boxes indicate regions with no observations where the trace of the pullback Fisher tensor collapses.

Figure 2: Conventional view of BO with a GP surrogate (SE kernel) and LogEI AF. The LogEI plot also shows the gradient (orange) and natural gradient (purple) vector field in the input space. The "gradient analysis" shows pullback metric trace $\sqrt{\mathrm{tr}(\mathbf{g}_{t}(\mathbf{x}))}$ and AF gradient norm $\|\nabla_{\mathbf{x}}\alpha(\mathbf{x})\|_{2}$ . Trace is non-degenerate near the observed data and collapses in regions where the GP posterior has reverted to the prior (no observations). The red ellipses are calculated using the pullback metric tensor at respective input locations.

Figure 3: Ellipses using $g(\mathbf{x})$ from ( 5 ) for different surrogates: GP with squared exponential (SE) kernel, GP with Infinite Width Bayesian Neural Network (IBNN) kernel [ 21 ] and Bayesian Neural Network (BNN).

Figure 4: BO performance (lower is better) comparison for different methods with a GP using an SE kernel. Solid curves show means over

11

trials, and shaded bands show standard error. FITR-REI and TuRBO-REI use the same GP modeling pipeline and regional trust-region center selection; they differ only in whether the trust-region shape is determined by ARD lengthscales or by the pullback Fisher tensor. FullFITR-REI was evaluated only on a subset of problems and is discussed separately in the text.

Figure 5: BO performance comparison (lower is better) for different methods with a GP using an IBNN kernel of depth

3

. Solid curves show means over

5

trials, and shaded bands show standard error.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Acquisition	Closed-form $\bar{\alpha}(\mu_{t,\mathbf{x}},\Sigma_{t,\mathbf{x}};\bm{\xi})$
EI	$\bigl(\mu_{t,\mathbf{x}}-y^{\star}\bigr)\,\Omega_{\scriptscriptstyle\mathcal{N}}(z)+\Sigma_{t,\mathbf{x}}^{1/2}\,\omega_{\scriptscriptstyle\mathcal{N}}(z)$
PI	$\Omega_{\scriptscriptstyle\mathcal{N}}(z)$
UCB	$\mu_{t,\mathbf{x}}+\sqrt{\kappa_{\mathrm{UCB}}}\,\Sigma_{t,\mathbf{x}}^{1/2}$

Appendix

Table 1: Closed-form AF

\bar{\alpha}(\bm{\theta}_{t,\mathbf{x}};\bm{\xi})

under ( 14 ) for a scalar Gaussian predictive

\bm{\theta}_{t,\mathbf{x}}=(\mu_{t,\mathbf{x}},\Sigma_{t,\mathbf{x}})

. EI and PI use incumbent

y^{\star}

; UCB uses exploration weight

\kappa_{\mathrm{UCB}}>0

Acquisition	Utility $a(y;\bm{\theta}_{t,\mathbf{x}},\bm{\xi})$	Reparameterized form $\bar{\alpha}(\bm{\theta}_{t,\mathbf{x}},\bm{\zeta};\bm{\xi})$
EI	$\max\!\left(\mathrm{ReLU}\!\left(\bm{y}-y^{\star}\right)\right)$	$\max\!\left(\mathrm{ReLU}\!\left(\bm{\mu}_{t,\bm{X}}+\bm{L}_{t,\bm{X}}\bm{\zeta}-y^{\star}\right)\right)$
PI	$\max\!\left(\mathbf{1}^{-}\!\left(\bm{y}-y^{\star}\right)\right)$	$\max\!\left(\mathrm{sigmoid}\!\left(\frac{\bm{\mu}_{t,\bm{X}}+\bm{L}_{t,\bm{X}}\bm{\zeta}-y^{\star}}{\tau}\right)\right)$
UCB	$\max\!\left(\bm{\mu}_{t,\bm{X}}+\sqrt{\kappa_{\mathrm{UCB}}\pi/2}\,\bigl\|\bm{y}-\bm{\mu}_{t,\bm{X}}\bigr\|\right)$	$\max\!\left(\bm{\mu}_{t,\bm{X}}+\sqrt{\kappa_{\mathrm{UCB}}\pi/2}\,\bigl\|\bm{L}_{t,\bm{X}}\bm{\zeta}\bigr\|\right)$
SR	$\max(\bm{y})$	$\max\!\left(\bm{\mu}_{t,\bm{X}}+\bm{L}_{t,\bm{X}}\bm{\zeta}\right)$
ES	$-\mathrm{H}\!\left(\mathbb{E}_{\bm{y}_{b}\mid\bm{y}_{a}}\!\left[\mathbf{1}^{+}\!\left(\bm{y}_{b}-\max(\bm{y}_{b})\right)\right]\right)$	$-\mathrm{H}\!\left(\mathbb{E}_{\bm{\zeta}_{b}}\!\left[\mathrm{softmax}\!\left(\frac{\bm{\mu}_{b\mid a}+\bm{L}_{b\mid a}\bm{\zeta}_{b}}{\tau}\right)\right]\right)$

Appendix

Table 2: Utility

a

and integrand

\bar{\alpha}(\cdot,\bm{\zeta};\cdot)

inside

\mathbb{E}_{\bm{\zeta}}

for reparameterized acquisitions under ( 2 ) for a Gaussian predictive [ 23 , 40 ] .

\mathbf{1}^{+/-}

denote the right/left-continuous Heaviside step functions;

\mathrm{ReLU}(u)=\max(u,0)

;

\mathrm{sigmoid}(u)=(1+e^{-u})^{-1}

;

\mathrm{H}

is Shannon entropy. EI (Expected Improvement) and PI (Probability of Improvement) use incumbent threshold

y^{\star}

; PI uses temperature

\tau>0

; the PI row is a sigmoid-smoothed surrogate for Monte Carlo gradients and is not the analytic PI functional

\Omega_{\scriptscriptstyle\mathcal{N}}(z)

. UCB uses exploration weight

\kappa_{\mathrm{UCB}}>0

. For ES (Entropy Search), subscripts

a

and

b

mark the query set and discretization points;

\bm{\mu}_{b\mid a}

and

\bm{L}_{b\mid a}

are the conditional posterior mean and Cholesky factor at

b

given an outcome at

a

. SR (Simple Regret) does not use extra parameters.

Figure 6: Per-problem diagnostics for TuRBO and FITR on the GP-SE benchmark suite. The first column reports the average time for getting the next candidate in BO iterations. The second column shows the trust-region length

\kappa_{t}

. The last two columns show the FITR-specific Tikhonov regularization and anisotropy diagnostics. Solid lines show means over the

11

seeded runs and shaded bands show standard error.

Acquisition	Analytic bound on $C_{\alpha}$	Monte Carlo bound on $C_{\alpha}$
EI	$\Sigma_{t,\mathbf{x}}\bigl(1+1/(4\pi)\bigr)$	$\tfrac{3}{2}\Sigma_{t,\mathbf{x}}$
UCB	$\Sigma_{t,\mathbf{x}}\bigl(1+\kappa/2\bigr)$	$\Sigma_{t,\mathbf{x}}\bigl(1+\kappa\pi/4\bigr)$
PI	$1/(2\pi)$	$\tfrac{3}{32\tau^{2}}\Sigma_{t,\mathbf{x}}$

Appendix

Table 3: Calculated bounds on

C_{\alpha}(\bm{\theta}_{t,\mathbf{x}})

for the scalar Gaussian predictive. If

\Sigma_{t,\mathbf{x}}

is bounded, which holds for any GP with finite prior variance, the shown AF sensitivities are also bounded. Analytic PI is bounded globally on

\mathcal{S}

. Since all entries depend only on

(\mu_{t,\mathbf{x}},\Sigma_{t,\mathbf{x}})

, they apply to any surrogate with a Gaussian predictive.

Bayesian Optimization

Bayesian optimization uses probabilistic models and sequential experiments to find strong solutions when evaluating a black-box objective is costly.

See all 243 papers

Rethinking Trust Region Bayesian Optimization in High Dimensions
Apr 24, 2026Wei-Ting Tang, Joel A. PaulsonBayesian Optimization Black-Box Optimization
University of Wisconsin–Madison
Information Theoretic Bayesian Optimization over the Probability Simplex
Mar 10, 2026Federico Pavesi, Antonio Candelieri, Noémie JaquierBayesian Optimization Constrained Optimization
Department of Robotics, Perception and Learning, KTH Royal Institute of Technology, Stockholm, Sweden · Department of Economics, Management and Statistics, University of Milan, Bicocca, Milan, Italy
Adaptive Candidate Point Thompson Sampling for High-Dimensional Bayesian Optimization
Apr 10, 2026Donney Fan, Geoff PleissBayesian Optimization Thompson Sampling
University of British Columbia Vector Institute

Apr 24, 2026J/K move · Enter open · S save

Rethinking Trust Region Bayesian Optimization in High Dimensions

Wei-Ting Tang, Joel A. Paulson

University of Wisconsin–Madison

Bayesian Optimization Black-Box Optimization Trust-Region Optimization Gaussian Process Regression

Open paper AskPDF arXiv

Trust Region Bayesian Optimization (TuRBO) is an effective strategy for alleviating the curse of dimensionality in high-dimensional black-box optimization. However, inappropriate lengthscale design can cause the local Gaussian process (GP) model within the trust region to degenerate, leading to suboptimal performance in high dimensions. In this work, we show that TuRBO's local GP may remain either excessively complex or overly simple as the dimension

D

and trust region side length

L

vary. To address this issue, we propose a straightforward variant, AdaScale-TuRBO, which scales the GP lengthscale with both the problem dimension and trust region size, thereby preserving kernel geometry and maintaining consistent prior complexity. Empirically, we show that AdaScale-TuRBO can robustly outperform standard TuRBO and other popular high-dimensional BO methods on synthetic benchmarks and real-world trajectory planning tasks.

Abstract

Bayesian Optimization

Explore similar work

Rethinking Trust Region Bayesian Optimization in High Dimensions

Information Theoretic Bayesian Optimization over the Probability Simplex

Adaptive Candidate Point Thompson Sampling for High-Dimensional Bayesian Optimization