Bayesian Optimization on Function Spaces via Sparse RKHS Manifolds

Davide Sartor; Meghan E. Huber; Donghyun Kim; Nathan Wycoff

stat.MLOct 5, 2026

Bayesian Optimization on Function Spaces via Sparse RKHS Manifolds

Authors: Davide Sartor, Meghan E. Huber, Donghyun Kim, Nathan Wycoff

Organizations: Department of Mathematics and Statistics University of Massachusetts Amherst · Department of Mechanical and Industrial Engineering University of Massachusetts Amherst · Manning College of Information and Computer Sciences University of Massachusetts Amherst

Abstract

Bayesian Optimization (BO) has become an established methodology for minimizing black-box functions of a vector input. Often, however, this parameter vector arises from the discretization of an inherently functional relationship. Several recent articles have considered the Functional Bayesian Optimization (FBO) setting, in which the variable to be optimized is not a member of a finite dimensional vector space, but rather an infinite dimensional function space. In this work, we propose $L^0$ Manifold Optimization (L0MO), a simple approach to FBO which searches the subset of a Reproducing Kernel Hilbert Space (RKHS) consisting of functions with a sparse representation in the kernel functions, optimizing both the kernel locations and their coefficients. We discuss in detail the relationship between our method and existing ones, providing a unifying lens through which to view prior works. To assess our method against the state of the art, we conduct an extensive computational study, and along the way develop a novel set of benchmark test functions which port standard finite-dimensional ones to the infinite dimensional domain. Our experiments demonstrate that, on balance, the proposed method achieves superior performance across a wide range of test benchmarks.

Figures & tables

Figure 1 : A schematic view inside

\mathcal{H}

, illustrating our approximation strategy. The sparse manifolds form a stratified set of increasing dimension, depicted as a curve, a surface patch, and a ball. For a target

u\in\mathcal{H}

that does not admit a finite sparse representation, the best

m

-sparse approximation

u_{m}\in\bigcup_{i\leq m}\mathcal{M}^{i}

moves closer to

u

m

grows.

Figure 2 : Method comparison across benchmark functions: projections (top), functional ridges (middle) and real-world tasks (bottom). Each plot shows the running best target function value (median and 95% CI ) as a function of the number of acquisitions.

Figure 3 : Best functions found by L0MO : brachistochrone paths (left) and MNIST activation functions, i.e. ReLU plus the learned offset (right). Transparent lines show individual runs; bold lines show the average across runs.

Method	Sinc-1D [ $10^{-2}$ ]	Sinc-2D [ $10^{-2}$ ]	Sinc-3D [ $10^{-2}$ ]	Sinc-4D [ $10^{-2}$ ]	Sinc-5D [ $10^{-2}$ ]
Random	$8.02\ [6.26,8.89]$	$6.80\ [6.52,7.20]$	$5.22\ [5.04,5.41]$	$3.62\ [3.58,3.75]$	$4.59\ [4.49,4.72]$
Kundu [ 24 ]	$8.48\ [7.79,8.69]$	$6.45\ [6.37,6.55]$	$5.09\ [5.00,5.19]$	$4.39\ [4.19,4.54]$	$4.85\ [4.75,4.92]$
Vien [ 40 ]	$2.58\ [2.31,2.82]$	$4.74\ [4.37,5.06]$	$3.96\ [3.78,4.05]$	$2.73\ [2.68,2.76]$	$3.68\ [3.64,3.71]$
Shilton [ 36 ]	$6.86\ [6.16,7.72]$	$6.66\ [6.42,7.04]$	$5.46\ [5.12,5.68]$	$5.37\ [4.79,6.00]$	$6.55\ [6.05,6.96]$
Vellanki [ 39 ]	$9.65\ [9.62,9.73]$	✗	✗	✗	✗
L0MO	$\mathbf{1.93\ [1.85,2.04]}$	$\mathbf{2.70\ [2.65,2.81]}$	$\mathbf{3.40\ [3.36,3.44]}$	$\mathbf{2.73\ [2.70,2.79]}$	$\mathbf{3.67\ [3.63,3.70]}$

Table 1 : Average regret (median and 95% CI) across methods and benchmark functions, one row per function class. ✗ indicates the method does not support the corresponding problem class.

Method	Sinc-1D [ $10^{-2}$ ]	Sinc-2D [ $10^{-2}$ ]	Sinc-3D [ $10^{-2}$ ]	Sinc-4D [ $10^{-2}$ ]	Sinc-5D [ $10^{-2}$ ]
Random	$6.67\ [5.01,7.43]$	$6.38\ [6.01,6.99]$	$4.99\ [4.76,5.23]$	$3.46\ [3.33,3.60]$	$4.45\ [4.26,4.59]$
Kundu [ 24 ]	$2.38\ [2.06,2.76]$	$3.85\ [3.64,4.01]$	$3.59\ [3.53,3.63]$	$2.64\ [2.62,2.67]$	$3.62\ [3.60,3.66]$
Vien [ 40 ]	$0.31\ [0.26,0.33]$	$2.02\ [1.82,2.21]$	$3.16\ [3.10,3.25]$	$\mathbf{2.43\ [2.42,2.44]}$	$\mathbf{3.36\ [3.36,3.37]}$
Shilton [ 36 ]	$0.19\ [0.18,0.26]$	$3.56\ [3.45,3.81]$	$3.43\ [3.40,3.48]$	$2.60\ [2.58,2.63]$	$3.42\ [3.40,3.43]$
Vellanki [ 39 ]	$6.24\ [6.10,6.46]$	✗	✗	✗	✗
L0MO	$\mathbf{0.06\ [0.05,0.07]}$	$\mathbf{0.76\ [0.70,0.80]}$	$\mathbf{2.74\ [2.69,2.77]}$	$2.47\ [2.46,2.48]$	$3.40\ [3.38,3.43]$

Table 2 : Final regret (median and 95% CI) across methods and benchmark functions, one row per function class. ✗ indicates the method does not support the corresponding problem class.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4 : Ratio between the Monte Carlo estimate of the ridge functionals on

n

uniform points and their closed form value (median and 95% CI over

32

random candidates).

Time (s)	Random	Kundu [ 24 ]	Vien [ 40 ]	Shilton [ 36 ]	Vellanki [ 39 ]	L0MO
Surrogate fit	✗	$85.5$	$88.5$	$81.1$	$154.1$	$86.9$
Acquisition	$3.9$	$254.9$	$586.1$	$891.7$	$79.7$	$202.2$

Appendix

Table 3 : Total wall-clock time per run (seconds) spent fitting the surrogate and optimizing the acquisition, over the

110

-evaluation budget, on a single CPU core and averaged across benchmarks.

Benchmark	Kundu [ 24 ]		Vien [ 40 ]		Shilton [ 36 ]		Vellanki [ 39 ]		L0MO
	Fit	Acq.	Fit	Acq.	Fit	Acq.	Fit	Acq.	Fit	Acq.
GramacyLee	$66$	$191$	$58$	$496$	$49$	$71$	$108$	$72$	$86$	$78$
Ackley	$87$	$171$	$63$	$455$	$45$	$30$	✗	✗	$99$	$102$
Hartmann	$69$	$120$	$69$	$437$	$67$	$42$	✗	✗	$82$	$89$
Rosenbrock	$96$	$174$	$74$	$503$	$72$	$36$	✗	✗	$98$	$129$
Michalewicz	$67$	$173$	$88$	$756$	$80$	$46$	✗	✗	$92$	$234$

Appendix

Table 4 : Median total wall-clock time per run (seconds) spent fitting the surrogate and optimizing the acquisition, over the

110

-evaluation budget, as recorded in the main sweep on a heterogeneous cluster. Times include jax compilation and are only indicative across rows.

Figure 5 : Effect of the GP kernel profile on L0MO across benchmark functions. Each line corresponds to a different smoothness assumption on the surrogate.

Benchmark	Random		Kundu [ 24 ]		Vien [ 40 ]		Shilton [ 36 ]		Vellanki [ 39 ]		L0MO
	Avg	Final	Avg	Final	Avg	Final	Avg	Final	Avg	Final	Avg	Final
GramacyLee [ $10^{-1}$ ]	$\mathbf{5.53}$	$5.19$	$6.58$	$6.37$	$\mathbf{5.45}$	$5.13$	$\mathbf{5.50}$	$\mathbf{5.13}$	$5.46$	$\mathbf{5.13}$	$\mathbf{5.39}$	$\mathbf{5.13}$
Ackley [ $10^{1}$ ]	$1.58$	$1.46$	$1.60$	$1.51$	$1.61$	$1.58$	$1.39$	$1.16$	✗	✗	$\mathbf{0.66}$	$\mathbf{0.09}$
Hartmann [ $10^{0}$ ]	$3.14$	$3.04$	$2.94$	$2.65$	$2.55$	$2.38$	$2.58$	$2.43$	✗	✗	$\mathbf{1.36}$	$\mathbf{0.78}$
Rosenbrock [ $10^{4}$ ]	$6.57$	$6.10$	$5.91$	$5.33$	$5.51$	$5.10$	$5.07$	$4.43$	✗	✗	$\mathbf{3.17}$	$\mathbf{0.66}$
Michalewicz [ $10^{0}$ ]	$4.20$	$4.11$	$3.96$	$3.72$	$3.74$	$3.38$	$3.81$	$3.65$	✗	✗	$\mathbf{3.24}$	$\mathbf{2.93}$

Appendix

Table 5 : Median final and average regret with the Matérn-5/2 GP profile for every method, at the selected RKHS lengthscale. Methods tied for first in bold.

Benchmark	RKHS scale	Surrogate kernel
		Kundu [ 24 ]	Vien [ 40 ]	Shilton [ 36 ]	L0MO
GramacyLee	$0.4$	$\mathcal{C}(\infty)$	$\mathcal{C}(1)$	$\mathcal{C}(2)$	$\mathcal{C}(0)$
Ackley	$0.4$	$\mathcal{C}(2)$	$\mathcal{C}(0)$	$\mathcal{C}(1)$	$\mathcal{C}(2)$
Hartmann	$0.4$	$\mathcal{C}(1)$	$\mathcal{C}(\infty)$	$\mathcal{C}(\infty)$	$\mathcal{C}(2)$
Rosenbrock	$0.4$	$\mathcal{C}(2)$	$\mathcal{C}(\infty)$	$\mathcal{C}(\infty)$	$\mathcal{C}(1)$
Michalewicz	$0.4$	$\mathcal{C}(\infty)$	$\mathcal{C}(2)$	$\mathcal{C}(2)$	$\mathcal{C}(\infty)$

Appendix

Table 6 : Selected hyperparameters per method and benchmark. The GP kernel determines the order of differentiability

\mathcal{C}(\nu)

of the surrogate model (

\mathcal{C}(\infty)

= RBF). The RKHS lengthscale

\ell

is shared by all methods on a benchmark.

$\ell$	Random		Kundu [ 24 ]		Vien [ 40 ]		Shilton [ 36 ]		Vellanki [ 39 ]		L0MO
	Avg	Final	Avg	Final	Avg	Final	Avg	Final	Avg	Final	Avg	Final
$0.05$	$4.40$	$4.70$	$3.70$	$3.20$	$2.80$	$2.30$	$2.20$	$2.90$	$3.20$	$3.20$	$\mathbf{2.10}$	$\mathbf{1.70}$
$0.1$	$4.20$	$4.40$	$3.40$	$3.30$	$3.10$	$2.90$	$2.60$	$2.60$	$3.20$	$3.20$	$\mathbf{1.70}$	$\mathbf{1.60}$
$0.2$	$4.30$	$4.40$	$3.80$	$3.60$	$3.10$	$3.40$	$2.20$	$2.50$	$3.40$	$3.40$	$\mathbf{1.10}$	$\mathbf{1.10}$
$0.4$	$3.50$	$3.80$	$4.10$	$4.20$	$3.10$	$3.50$	$1.90$	$2.40$	$4.60$	$4.00$	$\mathbf{1.10}$	$\mathbf{1.10}$

Appendix

Table 7 : Mean rank over the ten main benchmarks at each RKHS lengthscale

\ell

, each method using its own best GP profile at that

\ell

. A method ties with any better one whose median lies within its own 95% CI. Lower is better; best per row in bold.

RKHS lengthscale $\ell=0.05$
Benchmark	Random		Kundu [ 24 ]		Vien [ 40 ]		Shilton [ 36 ]		Vellanki [ 39 ]		L0MO
	Avg	Final	Avg	Final	Avg	Final	Avg	Final	Avg	Final	Avg	Final
GramacyLee [ $10^{-1}$ ]	$7.30$	$6.37$	$6.53$	$6.37$	$6.90$	$6.37$	$6.46$	$6.37$	$\mathbf{5.52}$	$\mathbf{5.13}$	$6.89$	$6.37$
Ackley [ $10^{1}$ ]	$1.69$	$1.68$	$1.69$	$1.68$	$\mathbf{1.69}$	$1.68$	$\mathbf{1.69}$	$\mathbf{1.66}$	✗	✗	$\mathbf{1.69}$	$1.68$
Hartmann [ $10^{0}$ ]	$3.48$	$3.48$	$3.48$	$3.47$	$3.47$	$\mathbf{3.46}$	$\mathbf{3.47}$	$3.47$	✗	✗	$\mathbf{3.47}$	$\mathbf{3.46}$
Rosenbrock [ $10^{4}$ ]	$7.87$	$7.86$	$7.86$	$7.86$	$\mathbf{7.86}$	$\mathbf{7.84}$	$\mathbf{7.86}$	$7.85$	✗	✗	$7.86$	$7.84$

Appendix

Table 8 : Median final and average regret at every RKHS lengthscale, each method at its own best GP profile. Each benchmark carries a common power of ten. Methods tied for first (see table 7 ) in bold.

Figure 6 : Effect of natural gradient preconditioning for Vien [ 40 ] across benchmark functions.

Figure 7 : Effect of natural gradient preconditioning for L0MO across benchmark functions.

Figure 8 : Effect of truncating the small singular values of the pullback metric in the natural gradient preconditioner of L0MO , compared with the pseudo-inverse and the Euclidean gradient.

Figure 9 : Effect of restricting L0MO to a single sheet of the parameterization, with Euclidean and natural gradients, on the benchmarks where the single-sheet variant was run.

Figure 10 : Effect of the initial candidate distribution across benchmark functions.

Figure 11 : Growing complexity schedule compared with a fixed number of support points

k

for L0MO .

Explore similar work

May 10, 2026cs.LG

Bayesian Optimization with Structured Measurements: A Vector-Valued RKHS Framework

Bayesian optimization (BO) is an efficient framework for optimizing expensive black-box functions. However, it is typically formulated as learning an end-to-end mapping from inputs to scalar objectives, thereby discarding the potentially rich information whenever a structured system output is available. In this work, we study Bayesian optimization over a vector-valued operator with structured measurements, where each measurement observes multidimensional or functional outputs, e.g., trajectories or spatial fields, rather than a single scalar value. The objective is then defined as a linear functional of these measurements. This allows each observation to reveal substantially richer information about the underlying system compared to scalar observations. Assuming the unknown operator lies in a vector-valued reproducing kernel Hilbert space (RKHS), we derive high-probability concentration bounds for the kernel ridge regression (KRR) estimator directly in the measurement space, characterizing uncertainty in a general Hilbert space. Building on these results, we propose an algorithm based on the upper confidence bound (UCB) acquisition function with regret guarantees under mild assumptions, recovering sublinear rates for common kernels. Empirically, we demonstrate that leveraging structured measurements leads to improved sample efficiency by enabling efficient transfer of information across objectives and adaptation to time-varying settings.

Wenbin Wang, Colin N. Jones

Automatic Control Laboratory, EPFL

Bayesian Optimization Reproducing Kernel Hilbert Spaces Kernel Hilbert Spaces Concentration Bounds

Ask PDF

Date pendingmath.OC

Nonlinear Dimensionality Reduction Techniques for Bayesian Optimization

Bayesian optimisation (BO) enables sample-efficient global optimisation of expensive black-box functions but remains challenging in high dimensions. We investigate nonlinear dimensionality reduction to a sequence of low-dimensional latent-space BO (LSBO) problems. Early LSBO used linear random and supervised embeddings; building on Grosnit et al., we employ variational autoencoders (VAEs), deep metric loss for structured latent manifolds, and retraining to adapt the encoder-decoder pair to newly sampled regions. We couple LSBO with sequential domain reduction (SDR) directly in latent space (SDR-LSBO), narrowing search domains as evidence accumulates. Implemented in GPU-accelerated BoTorch with Mat'ern-5/2 Gaussian-process surrogates, our methods improve benchmark optimisation quality, and retraining can enhance BO performance. Comparisons with adaptive supervised linear random embeddings demonstrate the effectiveness of VAE-based BO for nonlinear low-dimensional structures. We analyse BO-VAE with a fixed pretrained representation, decomposing ambient-space simple regret into latent BO error and a fixed VAE-induced representation gap. Under a PAC-Bayes-certified reconstruction condition and standard fixed-prior assumptions for expected improvement with a Mat'ern-5/2 kernel, latent BO error vanishes as the evaluation budget increases, whereas the representation gap remains fixed and may impose a non-vanishing error floor. Visualisations empirically assess accessibility of the ambient optimum through the learned decoder. To our knowledge, this is the first study combining SDR with VAE-based LSBO. Our analysis clarifies metric shaping and retraining choices critical for scalable latent-space BO. For reproducibility, source code is available at https://github.com/L-Lok/Nonlinear-Dimensionality-Reduction-Techniques-for-Bayesian-Optimization.git.

Luo Long, Coralia Cartis, Paz Fink Shustin

Mathematical Institute, University of Oxford, Radcliffe Observatory, Andrew Wiles Building, Woodstock Rd, Oxford, OX2 6GG, United Kingdom.

Bayesian Optimization Dimensionality Reduction Variance Reduction Latent Space Representation Learning

Ask PDF

Apr 30, 2026cs.LG

Bayesian Optimization in Linear Time

Bayesian optimization is a sequential method for minimizing objective functions that are expensive to evaluate and about which few assumptions can be made. By using all gathered data to train a Gaussian process model for the function and adaptively employing a mixture of global exploration and local exploitation, this method has been used for optimization in many fields including machine learning, automotive engineering and reinforcement learning. However, the standard method suffers from two problems: 1) with cubic computational complexity in the training-set size it eventually becomes computationally infeasible to train the model, and 2) globally modeling the objective function is not necessarily optimal given the local nature of minimization. Using flexible and recursive binary partitioning of the search space, we adapt both the modeling and acquisitive aspects of standard Bayesian optimization to work harmoniously with the partitioning scheme, thereby ameliorating both standard shortcomings. We compare our method against a commonly used Bayesian optimization library on seven challenging test functions, ranging in dimensionality from

6

124

, and show that our method achieves superior optimization performance in all tests. In addition our method has linear computational complexity.

Jesse Schneider, William J. Welch

Department of Statistics, University of British Columbia, Vancouver, BC V6T 1Z4, Canada

Bayesian Optimization Gaussian Process Prompt Optimization

Ask PDF