Bayesian Optimization on Function Spaces via Sparse RKHS Manifolds

Davide Sartor; Meghan E. Huber; Donghyun Kim; Nathan Wycoff

stat.MLOct 5, 2026

Bayesian Optimization on Function Spaces via Sparse RKHS Manifolds

Authors: Davide Sartor, Meghan E. Huber, Donghyun Kim, Nathan Wycoff

Organizations: Department of Mathematics and Statistics University of Massachusetts Amherst · Department of Mechanical and Industrial Engineering University of Massachusetts Amherst · Manning College of Information and Computer Sciences University of Massachusetts Amherst

Abstract

Bayesian Optimization (BO) has become an established methodology for minimizing black-box functions of a vector input. Often, however, this parameter vector arises from the discretization of an inherently functional relationship. Several recent articles have considered the Functional Bayesian Optimization (FBO) setting, in which the variable to be optimized is not a member of a finite dimensional vector space, but rather an infinite dimensional function space. In this work, we propose $L^0$ Manifold Optimization (L0MO), a simple approach to FBO which searches the subset of a Reproducing Kernel Hilbert Space (RKHS) consisting of functions with a sparse representation in the kernel functions, optimizing both the kernel locations and their coefficients. We discuss in detail the relationship between our method and existing ones, providing a unifying lens through which to view prior works. To assess our method against the state of the art, we conduct an extensive computational study, and along the way develop a novel set of benchmark test functions which port standard finite-dimensional ones to the infinite dimensional domain. Our experiments demonstrate that, on balance, the proposed method achieves superior performance across a wide range of test benchmarks.

Figures & tables

Figure 1 : A schematic view inside

\mathcal{H}

, illustrating our approximation strategy. The sparse manifolds form a stratified set of increasing dimension, depicted as a curve, a surface patch, and a ball. For a target

u\in\mathcal{H}

that does not admit a finite sparse representation, the best

m

-sparse approximation

u_{m}\in\bigcup_{i\leq m}\mathcal{M}^{i}

moves closer to

u

m

grows.

Figure 2 : Method comparison across benchmark functions: projections (top), functional ridges (middle) and real-world tasks (bottom). Each plot shows the running best target function value (median and 95% CI ) as a function of the number of acquisitions.

Figure 3 : Best functions found by L0MO : brachistochrone paths (left) and MNIST activation functions, i.e. ReLU plus the learned offset (right). Transparent lines show individual runs; bold lines show the average across runs.

Method	Sinc-1D [ $10^{-2}$ ]	Sinc-2D [ $10^{-2}$ ]	Sinc-3D [ $10^{-2}$ ]	Sinc-4D [ $10^{-2}$ ]	Sinc-5D [ $10^{-2}$ ]
Random	$8.02\ [6.26,8.89]$	$6.80\ [6.52,7.20]$	$5.22\ [5.04,5.41]$	$3.62\ [3.58,3.75]$	$4.59\ [4.49,4.72]$
Kundu [ 24 ]	$8.48\ [7.79,8.69]$	$6.45\ [6.37,6.55]$	$5.09\ [5.00,5.19]$	$4.39\ [4.19,4.54]$	$4.85\ [4.75,4.92]$
Vien [ 40 ]	$2.58\ [2.31,2.82]$	$4.74\ [4.37,5.06]$	$3.96\ [3.78,4.05]$	$2.73\ [2.68,2.76]$	$3.68\ [3.64,3.71]$
Shilton [ 36 ]	$6.86\ [6.16,7.72]$	$6.66\ [6.42,7.04]$	$5.46\ [5.12,5.68]$	$5.37\ [4.79,6.00]$	$6.55\ [6.05,6.96]$
Vellanki [ 39 ]	$9.65\ [9.62,9.73]$	✗	✗	✗	✗
L0MO	$\mathbf{1.93\ [1.85,2.04]}$	$\mathbf{2.70\ [2.65,2.81]}$	$\mathbf{3.40\ [3.36,3.44]}$	$\mathbf{2.73\ [2.70,2.79]}$	$\mathbf{3.67\ [3.63,3.70]}$

Table 1 : Average regret (median and 95% CI) across methods and benchmark functions, one row per function class. ✗ indicates the method does not support the corresponding problem class.

Method	Sinc-1D [ $10^{-2}$ ]	Sinc-2D [ $10^{-2}$ ]	Sinc-3D [ $10^{-2}$ ]	Sinc-4D [ $10^{-2}$ ]	Sinc-5D [ $10^{-2}$ ]
Random	$6.67\ [5.01,7.43]$	$6.38\ [6.01,6.99]$	$4.99\ [4.76,5.23]$	$3.46\ [3.33,3.60]$	$4.45\ [4.26,4.59]$
Kundu [ 24 ]	$2.38\ [2.06,2.76]$	$3.85\ [3.64,4.01]$	$3.59\ [3.53,3.63]$	$2.64\ [2.62,2.67]$	$3.62\ [3.60,3.66]$
Vien [ 40 ]	$0.31\ [0.26,0.33]$	$2.02\ [1.82,2.21]$	$3.16\ [3.10,3.25]$	$\mathbf{2.43\ [2.42,2.44]}$	$\mathbf{3.36\ [3.36,3.37]}$
Shilton [ 36 ]	$0.19\ [0.18,0.26]$	$3.56\ [3.45,3.81]$	$3.43\ [3.40,3.48]$	$2.60\ [2.58,2.63]$	$3.42\ [3.40,3.43]$
Vellanki [ 39 ]	$6.24\ [6.10,6.46]$	✗	✗	✗	✗
L0MO	$\mathbf{0.06\ [0.05,0.07]}$	$\mathbf{0.76\ [0.70,0.80]}$	$\mathbf{2.74\ [2.69,2.77]}$	$2.47\ [2.46,2.48]$	$3.40\ [3.38,3.43]$

Table 2 : Final regret (median and 95% CI) across methods and benchmark functions, one row per function class. ✗ indicates the method does not support the corresponding problem class.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4 : Ratio between the Monte Carlo estimate of the ridge functionals on

n

uniform points and their closed form value (median and 95% CI over

32

random candidates).

Time (s)	Random	Kundu [ 24 ]	Vien [ 40 ]	Shilton [ 36 ]	Vellanki [ 39 ]	L0MO
Surrogate fit	✗	$85.5$	$88.5$	$81.1$	$154.1$	$86.9$
Acquisition	$3.9$	$254.9$	$586.1$	$891.7$	$79.7$	$202.2$

Appendix

Table 3 : Total wall-clock time per run (seconds) spent fitting the surrogate and optimizing the acquisition, over the

110

-evaluation budget, on a single CPU core and averaged across benchmarks.

Benchmark	Kundu [ 24 ]		Vien [ 40 ]		Shilton [ 36 ]		Vellanki [ 39 ]		L0MO
	Fit	Acq.	Fit	Acq.	Fit	Acq.	Fit	Acq.	Fit	Acq.
GramacyLee	$66$	$191$	$58$	$496$	$49$	$71$	$108$	$72$	$86$	$78$
Ackley	$87$	$171$	$63$	$455$	$45$	$30$	✗	✗	$99$	$102$
Hartmann	$69$	$120$	$69$	$437$	$67$	$42$	✗	✗	$82$	$89$
Rosenbrock	$96$	$174$	$74$	$503$	$72$	$36$	✗	✗	$98$	$129$
Michalewicz	$67$	$173$	$88$	$756$	$80$	$46$	✗	✗	$92$	$234$

Appendix

Table 4 : Median total wall-clock time per run (seconds) spent fitting the surrogate and optimizing the acquisition, over the

110

-evaluation budget, as recorded in the main sweep on a heterogeneous cluster. Times include jax compilation and are only indicative across rows.

Figure 5 : Effect of the GP kernel profile on L0MO across benchmark functions. Each line corresponds to a different smoothness assumption on the surrogate.

Benchmark	Random		Kundu [ 24 ]		Vien [ 40 ]		Shilton [ 36 ]		Vellanki [ 39 ]		L0MO
	Avg	Final	Avg	Final	Avg	Final	Avg	Final	Avg	Final	Avg	Final
GramacyLee [ $10^{-1}$ ]	$\mathbf{5.53}$	$5.19$	$6.58$	$6.37$	$\mathbf{5.45}$	$5.13$	$\mathbf{5.50}$	$\mathbf{5.13}$	$5.46$	$\mathbf{5.13}$	$\mathbf{5.39}$	$\mathbf{5.13}$
Ackley [ $10^{1}$ ]	$1.58$	$1.46$	$1.60$	$1.51$	$1.61$	$1.58$	$1.39$	$1.16$	✗	✗	$\mathbf{0.66}$	$\mathbf{0.09}$
Hartmann [ $10^{0}$ ]	$3.14$	$3.04$	$2.94$	$2.65$	$2.55$	$2.38$	$2.58$	$2.43$	✗	✗	$\mathbf{1.36}$	$\mathbf{0.78}$
Rosenbrock [ $10^{4}$ ]	$6.57$	$6.10$	$5.91$	$5.33$	$5.51$	$5.10$	$5.07$	$4.43$	✗	✗	$\mathbf{3.17}$	$\mathbf{0.66}$
Michalewicz [ $10^{0}$ ]	$4.20$	$4.11$	$3.96$	$3.72$	$3.74$	$3.38$	$3.81$	$3.65$	✗	✗	$\mathbf{3.24}$	$\mathbf{2.93}$

Appendix

Table 5 : Median final and average regret with the Matérn-5/2 GP profile for every method, at the selected RKHS lengthscale. Methods tied for first in bold.

Benchmark	RKHS scale	Surrogate kernel
		Kundu [ 24 ]	Vien [ 40 ]	Shilton [ 36 ]	L0MO
GramacyLee	$0.4$	$\mathcal{C}(\infty)$	$\mathcal{C}(1)$	$\mathcal{C}(2)$	$\mathcal{C}(0)$
Ackley	$0.4$	$\mathcal{C}(2)$	$\mathcal{C}(0)$	$\mathcal{C}(1)$	$\mathcal{C}(2)$
Hartmann	$0.4$	$\mathcal{C}(1)$	$\mathcal{C}(\infty)$	$\mathcal{C}(\infty)$	$\mathcal{C}(2)$
Rosenbrock	$0.4$	$\mathcal{C}(2)$	$\mathcal{C}(\infty)$	$\mathcal{C}(\infty)$	$\mathcal{C}(1)$
Michalewicz	$0.4$	$\mathcal{C}(\infty)$	$\mathcal{C}(2)$	$\mathcal{C}(2)$	$\mathcal{C}(\infty)$

Appendix

Table 6 : Selected hyperparameters per method and benchmark. The GP kernel determines the order of differentiability

\mathcal{C}(\nu)

of the surrogate model (

\mathcal{C}(\infty)

= RBF). The RKHS lengthscale

\ell

is shared by all methods on a benchmark.

$\ell$	Random		Kundu [ 24 ]		Vien [ 40 ]		Shilton [ 36 ]		Vellanki [ 39 ]		L0MO
	Avg	Final	Avg	Final	Avg	Final	Avg	Final	Avg	Final	Avg	Final
$0.05$	$4.40$	$4.70$	$3.70$	$3.20$	$2.80$	$2.30$	$2.20$	$2.90$	$3.20$	$3.20$	$\mathbf{2.10}$	$\mathbf{1.70}$
$0.1$	$4.20$	$4.40$	$3.40$	$3.30$	$3.10$	$2.90$	$2.60$	$2.60$	$3.20$	$3.20$	$\mathbf{1.70}$	$\mathbf{1.60}$
$0.2$	$4.30$	$4.40$	$3.80$	$3.60$	$3.10$	$3.40$	$2.20$	$2.50$	$3.40$	$3.40$	$\mathbf{1.10}$	$\mathbf{1.10}$
$0.4$	$3.50$	$3.80$	$4.10$	$4.20$	$3.10$	$3.50$	$1.90$	$2.40$	$4.60$	$4.00$	$\mathbf{1.10}$	$\mathbf{1.10}$

Appendix

Table 7 : Mean rank over the ten main benchmarks at each RKHS lengthscale

\ell

, each method using its own best GP profile at that

\ell

. A method ties with any better one whose median lies within its own 95% CI. Lower is better; best per row in bold.

RKHS lengthscale $\ell=0.05$
Benchmark	Random		Kundu [ 24 ]		Vien [ 40 ]		Shilton [ 36 ]		Vellanki [ 39 ]		L0MO
	Avg	Final	Avg	Final	Avg	Final	Avg	Final	Avg	Final	Avg	Final
GramacyLee [ $10^{-1}$ ]	$7.30$	$6.37$	$6.53$	$6.37$	$6.90$	$6.37$	$6.46$	$6.37$	$\mathbf{5.52}$	$\mathbf{5.13}$	$6.89$	$6.37$
Ackley [ $10^{1}$ ]	$1.69$	$1.68$	$1.69$	$1.68$	$\mathbf{1.69}$	$1.68$	$\mathbf{1.69}$	$\mathbf{1.66}$	✗	✗	$\mathbf{1.69}$	$1.68$
Hartmann [ $10^{0}$ ]	$3.48$	$3.48$	$3.48$	$3.47$	$3.47$	$\mathbf{3.46}$	$\mathbf{3.47}$	$3.47$	✗	✗	$\mathbf{3.47}$	$\mathbf{3.46}$
Rosenbrock [ $10^{4}$ ]	$7.87$	$7.86$	$7.86$	$7.86$	$\mathbf{7.86}$	$\mathbf{7.84}$	$\mathbf{7.86}$	$7.85$	✗	✗	$7.86$	$7.84$

Appendix

Table 8 : Median final and average regret at every RKHS lengthscale, each method at its own best GP profile. Each benchmark carries a common power of ten. Methods tied for first (see table 7 ) in bold.

Figure 6 : Effect of natural gradient preconditioning for Vien [ 40 ] across benchmark functions.

Figure 7 : Effect of natural gradient preconditioning for L0MO across benchmark functions.

Figure 8 : Effect of truncating the small singular values of the pullback metric in the natural gradient preconditioner of L0MO , compared with the pseudo-inverse and the Euclidean gradient.

Figure 9 : Effect of restricting L0MO to a single sheet of the parameterization, with Euclidean and natural gradients, on the benchmarks where the single-sheet variant was run.

Figure 10 : Effect of the initial candidate distribution across benchmark functions.

Figure 11 : Growing complexity schedule compared with a fixed number of support points

k

for L0MO .

Explore similar work

Bayesian Optimization with Structured Measurements: A Vector-Valued RKHS Framework
May 10, 2026Wenbin Wang, Colin N. JonesBayesian Optimization Reproducing Kernel Hilbert Spaces
Automatic Control Laboratory, EPFL
Nonlinear Dimensionality Reduction Techniques for Bayesian Optimization
Date pendingLuo Long, Coralia Cartis, Paz Fink ShustinBayesian Optimization Dimensionality Reduction
Mathematical Institute, University of Oxford, Radcliffe Observatory, Andrew Wiles Building, Woodstock Rd, Oxford, OX2 6GG, United Kingdom.
Bayesian Optimization in Linear Time
Apr 30, 2026Jesse Schneider, William J. WelchBayesian Optimization Gaussian Process
Department of Statistics, University of British Columbia, Vancouver, BC V6T 1Z4, Canada

May 10, 2026, cs.LGJ/K move · Enter open · S save

Bayesian Optimization with Structured Measurements: A Vector-Valued RKHS Framework

Wenbin Wang, Colin N. Jones

Automatic Control Laboratory, EPFL

Bayesian Optimization Reproducing Kernel Hilbert Spaces Kernel Hilbert Spaces Concentration Bounds

Open paper AskPDF arXiv

Bayesian optimization (BO) is an efficient framework for optimizing expensive black-box functions. However, it is typically formulated as learning an end-to-end mapping from inputs to scalar objectives, thereby discarding the potentially rich information whenever a structured system output is available. In this work, we study Bayesian optimization over a vector-valued operator with structured measurements, where each measurement observes multidimensional or functional outputs, e.g., trajectories or spatial fields, rather than a single scalar value. The objective is then defined as a linear functional of these measurements. This allows each observation to reveal substantially richer information about the underlying system compared to scalar observations. Assuming the unknown operator lies in a vector-valued reproducing kernel Hilbert space (RKHS), we derive high-probability concentration bounds for the kernel ridge regression (KRR) estimator directly in the measurement space, characterizing uncertainty in a general Hilbert space. Building on these results, we propose an algorithm based on the upper confidence bound (UCB) acquisition function with regret guarantees under mild assumptions, recovering sublinear rates for common kernels. Empirically, we demonstrate that leveraging structured measurements leads to improved sample efficiency by enabling efficient transfer of information across objectives and adaptation to time-varying settings.

Abstract

Explore similar work

Bayesian Optimization with Structured Measurements: A Vector-Valued RKHS Framework

Nonlinear Dimensionality Reduction Techniques for Bayesian Optimization

Bayesian Optimization in Linear Time