We study Bayesian optimization (BO) through the lens of information geometry. Pulling back the Fisher information metric through the surrogate posterior map yields a local sensitivity tensor on the input space, which leads to an upper bound on the gradient of reparameterizable acquisition functions. This view explains vanishing-gradient behavior in high-dimensional BO and provides a common interpretation of heuristics such as RAASP and dimension-scaled lengthscales. Building on this analysis, we propose FITR, a trust-region-based BO method that replaces lengthscale-based scaling by local pullback-Fisher weights. FITR is not restricted to GP kernels with explicit lengthscales. On GP benchmarks with an SE kernel, experiments show competitive performance using FITR. The proposed method also easily generalizes to non-isotropic surrogates, although the gains are more task-dependent in that setting.
Figures & tables
Figure 1: First: Conventional BO view with a GP surrogate (SE kernel) and LogEI [ 1 ] acquisition function (AF). Second: Visualization of Proposition 4.2 , where the acquisition gradient is upper-bounded by the product of the trace of the pullback Fisher tensor and the sensitivity factor of the AF. Third: Induced geometry on the input space using the pullback Fisher tensor. Fourth: Sensitivity Cα (Section 4 ) of the AF with respect to the predictive-distribution parameters on the statistical manifold. + denotes x=0 and × denotes x=1 in input space (the same markers are used to denote the respective points in the manifold view). Gray boxes indicate regions with no observations where the trace of the pullback Fisher tensor collapses.
Figure 2: Conventional view of BO with a GP surrogate (SE kernel) and LogEI AF. The LogEI plot also shows the gradient (orange) and natural gradient (purple) vector field in the input space. The "gradient analysis" shows pullback metric trace tr(gt(x)) and AF gradient norm ∥∇xα(x)∥2 . Trace is non-degenerate near the observed data and collapses in regions where the GP posterior has reverted to the prior (no observations). The red ellipses are calculated using the pullback metric tensor at respective input locations.
Figure 3: Ellipses using g(x) from ( 5 ) for different surrogates: GP with squared exponential (SE) kernel, GP with Infinite Width Bayesian Neural Network (IBNN) kernel [ 21 ] and Bayesian Neural Network (BNN).
Figure 4: BO performance (lower is better) comparison for different methods with a GP using an SE kernel. Solid curves show means over 11 trials, and shaded bands show standard error. FITR-REI and TuRBO-REI use the same GP modeling pipeline and regional trust-region center selection; they differ only in whether the trust-region shape is determined by ARD lengthscales or by the pullback Fisher tensor. FullFITR-REI was evaluated only on a subset of problems and is discussed separately in the text.
Figure 5: BO performance comparison (lower is better) for different methods with a GP using an IBNN kernel of depth 3 . Solid curves show means over 5 trials, and shaded bands show standard error.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Acquisition
Closed-form αˉ(μt,x,Σt,x;ξ)
EI
(μt,x−y⋆)ΩN(z)+Σt,x1/2ωN(z)
PI
ΩN(z)
UCB
μt,x+κUCBΣt,x1/2
Appendix
Table 1: Closed-form AF αˉ(θt,x;ξ) under ( 14 ) for a scalar Gaussian predictive θt,x=(μt,x,Σt,x) . EI and PI use incumbent y⋆ ; UCB uses exploration weight κUCB>0 .
Acquisition
Utility a(y;θt,x,ξ)
Reparameterized form αˉ(θt,x,ζ;ξ)
EI
max(ReLU(y−y⋆))
max(ReLU(μt,X+Lt,Xζ−y⋆))
PI
max(1−(y−y⋆))
max(sigmoid(τμt,X+Lt,Xζ−y⋆))
UCB
max(μt,X+κUCBπ/2y−μt,X)
max(μt,X+κUCBπ/2Lt,Xζ)
SR
max(y)
max(μt,X+Lt,Xζ)
ES
−H(Eyb∣ya[1+(yb−max(yb))])
−H(Eζb[softmax(τμb∣a+Lb∣aζb)])
Appendix
Table 2: Utility a and integrand αˉ(⋅,ζ;⋅) inside Eζ for reparameterized acquisitions under ( 2 ) for a Gaussian predictive [ 23 , 40 ] . 1+/− denote the right/left-continuous Heaviside step functions; ReLU(u)=max(u,0) ; sigmoid(u)=(1+e−u)−1 ; H is Shannon entropy. EI (Expected Improvement) and PI (Probability of Improvement) use incumbent threshold y⋆ ; PI uses temperature τ>0 ; the PI row is a sigmoid-smoothed surrogate for Monte Carlo gradients and is not the analytic PI functional ΩN(z) . UCB uses exploration weight κUCB>0 . For ES (Entropy Search), subscripts a and b mark the query set and discretization points; μb∣a and Lb∣a are the conditional posterior mean and Cholesky factor at b given an outcome at a . SR (Simple Regret) does not use extra parameters.
Figure 6: Per-problem diagnostics for TuRBO and FITR on the GP-SE benchmark suite. The first column reports the average time for getting the next candidate in BO iterations. The second column shows the trust-region length κt . The last two columns show the FITR-specific Tikhonov regularization and anisotropy diagnostics. Solid lines show means over the 11 seeded runs and shaded bands show standard error.
Acquisition
Analytic bound on Cα
Monte Carlo bound on Cα
EI
Σt,x(1+1/(4π))
23Σt,x
UCB
Σt,x(1+κ/2)
Σt,x(1+κπ/4)
PI
1/(2π)
32τ23Σt,x
Appendix
Table 3: Calculated bounds on Cα(θt,x) for the scalar Gaussian predictive. If Σt,x is bounded, which holds for any GP with finite prior variance, the shown AF sensitivities are also bounded. Analytic PI is bounded globally on S . Since all entries depend only on (μt,x,Σt,x) , they apply to any surrogate with a Gaussian predictive.
Trust Region Bayesian Optimization (TuRBO) is an effective strategy for alleviating the curse of dimensionality in high-dimensional black-box optimization. However, inappropriate lengthscale design can cause the local Gaussian process (GP) model within the trust region to degenerate, leading to suboptimal performance in high dimensions. In this work, we show that TuRBO's local GP may remain either excessively complex or overly simple as the dimension D and trust region side length L vary. To address this issue, we propose a straightforward variant, AdaScale-TuRBO, which scales the GP lengthscale with both the problem dimension and trust region size, thereby preserving kernel geometry and maintaining consistent prior complexity. Empirically, we show that AdaScale-TuRBO can robustly outperform standard TuRBO and other popular high-dimensional BO methods on synthetic benchmarks and real-world trajectory planning tasks.
Bayesian optimization is a data-efficient technique that has been shown to be extremely powerful to optimize expensive, black-box, and possibly noisy objective functions. Many applications involve optimizing probabilities and mixtures which naturally belong to the probability simplex, a constrained non-Euclidean domain defined by non-negative entries summing to one. This paper introduces α-GaBO, a novel family of Bayesian optimization algorithms over the probability simplex. Our approach is grounded in information geometry, a branch of Riemannian geometry which endows the simplex with a Riemannian metric and a class of connections. Based on information geometry theory, we construct Matérn kernels that reflect the geometry of the probability simplex, as well as a one-parameter family of geometric optimizers for the acquisition function. We validate our method on benchmark functions and on a variety of real-world applications including mixtures of components, mixtures of classifiers, and a robotic control task, showing its increased performance compared to constrained Euclidean approaches.
Federico Pavesi, Antonio Candelieri, Noémie Jaquier
Department of Robotics, Perception and Learning, KTH Royal Institute of Technology, Stockholm, Sweden · Department of Economics, Management and Statistics, University of Milan, Bicocca, Milan, Italy
In Bayesian optimization, Thompson sampling selects the evaluation point by sampling from the posterior distribution over the objective function maximizer. Because this sampling problem is intractable for Gaussian process (GP) surrogates, the posterior distribution is typically restricted to fixed discretizations (i.e., candidate points) that become exponentially sparse as dimensionality increases. While previous works aim to increase candidate point density through scalable GP approximations, our orthogonal approach increases density by adaptively reducing the search space during sampling. Specifically, we introduce Adaptive Candidate Thompson Sampling (ACTS), which generates candidate points in subspaces guided by the gradient of a surrogate model sample. ACTS is a simple drop-in replacement for existing TS methods -- including those that use trust regions or other local approximations -- producing better samples of maxima and improved optimization across synthetic and real-world benchmarks.