SABLE: Scalable Affine Bayesian Latent-space Exploration

Erik Bodin; Lisa Maria Kreusser; Donney Fan; Henry Moss; Geoff Pleiss; Nicolas Durrande

cs.LGOct 22, 2026

SABLE: Scalable Affine Bayesian Latent-space Exploration

Authors: Erik Bodin, Lisa Maria Kreusser, Donney Fan, Henry Moss, Geoff Pleiss, Nicolas Durrande

Organizations: Monumo · University of British Columbia · Lancaster University · Vector Institute

Abstract

Complex objects such as engineering designs and proteins are often difficult to optimise because their design spaces exhibit high-dimensional structure and objective evaluations are expensive. Generative models provide continuous latent representations, but their latent spaces can have thousands to tens of thousands of dimensions, creating a regime where the latent dimension $D$ greatly exceeds the evaluation budget. We introduce SABLE, a Bayesian optimisation method for high-dimensional latent spaces with an isotropic distribution and limited budget. SABLE uses a Bayesian affine surrogate for prediction and acquisition, treating higher-order structure as surrogate discrepancy. After $N$ observations, the slope update has dimension at most $N-1$ , yielding a decomposition of each query into an in-span component optimised by expected improvement (EI) in at most $N$ dimensions and an off-span component governed by a scalar norm. We show that the off-span norm implicitly selected by affine EI can lead to underexploration, and SABLE instead prescribes it using a finite-horizon schedule, without restricting search to a learned subspace. A recompression scheme reduces information to $\mathcal O(M)$ pseudo-observations, bounding refitting cost by $\mathcal O(M^2D+M^3)$ while keeping parameter inference, prediction, and acquisition decisions to tens of milliseconds. Across synthetic benchmarks and latent-space optimisation tasks, SABLE achieves high sample efficiency and reduces decision time by several orders of magnitude at large budgets.

Figures & tables

Panel of Figure 1 — Figure 1: Overview of SABLE. (a) Each query is decomposed into an in-span component and an orthogonal off-span component. EI chooses the in-span direction, while the exploration schedule prescribes the off-span norm. Because the design span $\mathcal{V}_{N}$ has dimension at most $N$ , EI optimisation requires at most $N$ coordinates, independently of $D$ . (b) As observations accumulate, their representation is recompressed to $\mathcal{O}(M)$ pseudo-observations. Strong information is retained and weaker information is summarised, making model-refitting cost independent of $N$ for fixed $M$ .

Panel of Figure 2 — Figure 2: Effect of state compression. Left: best observed value for compression budgets $M\in\{10,100,300,500\}$ at $D=10{,}000$ , compared with the uncapped representation (dashed) and the mean global maximum of the ten objective draws. Lines show means over ten runs (and objective draws) and bands show $5$ th– $95$ th percentiles. Right: measured wall-clock time per decision.

		$\cos(\hat{\bm{s}},\bm{v}_{\star})$				attained value
		affine	mild	mod.	severe	affine	mild	mod.	severe
Standard EI (no schedule)	MLE	1.00	-0.00	-0.01	-0.01	1.00	0.48	0.32	0.16
	MAP	0.78	0.44	0.30	0.16	0.78	0.60	0.54	0.50
	Marg.	0.77	0.44	0.29	0.19	0.77	0.59	0.54	0.49
naive model-derived schedule: $\lambda_{k}=\nicefrac{{\sigma_{1,k}^{2}}}{{\sigma_{\varepsilon,k}^{2}}}$	MLE	0.13	0.00	0.00	0.01	0.49	0.49	0.36	0.23
	MAP	0.98	0.59	0.41	0.23	0.98	0.67	0.55	0.45

Table 1: Model fitting and off-span exploration at

N=D=1000

with the uncapped representation. Left: recovery of the known degree-one component,

\cos(\hat{\bm{s}},\bm{v}_{\star})

. Right: attained value: maximum of the best observed point and the posterior-mean recommendation. Entries are means over ten seeds. Bold: the column best and every entry within two standard deviations.

Figure 3: Controlled long-horizon evaluation, $D=N=10{,}000$ . Lines show mean best observed value over ten runs, with $5$ th– $95$ th percentile bands. Markers at $N=10{,}000$ show attained value, the better of the best observed point and the posterior-mean recommendation.

Figure 4: Scientific latent-space optimisation. Table: best observed value, reported as mean and $5$ th– $95$ th percentile range over ten seeds; bold: within two standard errors of the column best. † : truncated at 48 h, reaching $N=303$ – $1{,}132$ . Bottom: best observed value against wall-clock time.

Figure 5: High-budget DS3 optimisation . Best observed value against evaluations; thick lines are medians and thin lines the individual runs.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Problem	$D$	Generative model	Generated object	Objective	Prior median
Turbulence	$6{,}000$	diffusion-lagr	velocity trajectory	DNS distribution match	$0.57$
Peptides	$13{,}440$	AMP-Diffusion	peptide sequence	hydrophobic moment	$0.34$
Calorimeter DS2	$6{,}480$	CaloDiffusion	shower voxel grid	energy-deposition match	$0.07$
Calorimeter DS3	$40{,}500$	CaloDiffusion	shower voxel grid	energy-deposition match	$0.05$

Appendix

Table 2: Generative-model optimisation problems. “Prior median” is the median objective value obtained from independent draws of the generative model’s latent prior

\mathcal{N}(\bm{0},I_{D})

. Datasets 2 and 3 form a matched calorimeter pair at different detector discretisations.

		$\cos(\hat{\bm{s}},\bm{v}_{\star})$				attained value
		affine	mild	mod.	severe	affine	mild	mod.	severe
no schedule	MLE	1.00	-0.00	-0.01	-0.01	1.00	0.48	0.32	0.16
	MAP	0.78	0.44	0.30	0.16	0.78	0.60	0.54	0.50
	Marg.	0.77	0.44	0.29	0.19	0.77	0.59	0.54	0.49
$\lambda_{\mathrm{ref}}$ from model	MLE	0.13	0.00	0.00	0.01	0.49	0.49	0.36	0.23
	MAP	0.98	0.59	0.41	0.23	0.98	0.67	0.55	0.45

Appendix

Table 3: Complete schedule-by-treatment grid, noise-free , at

N=D=1000

with the uncapped posterior. Left: recovery of the known degree-one component,

\cos(\hat{\bm{s}},\bm{v}_{\star})

. Right: attained value, the better of the best observed point and the posterior-mean recommendation. Means over ten paired seeds. Bold: the column best and every entry within two standard deviations of it. Table 1 shows the no-schedule, model-derived, and

\lambda_{\mathrm{ref}}=10^{5}

rows.

		$\cos(\hat{\bm{s}},\bm{v}_{\star})$				attained value
		affine	mild	mod.	severe	affine	mild	mod.	severe
no schedule	MLE	0.01	0.02	0.01	0.01	0.37	0.26	0.19	0.13
	MAP	0.14	0.10	0.07	0.06	0.15	0.15	0.15	0.16
	Marg.	0.16	0.11	0.08	0.05	0.16	0.16	0.16	0.15
$\lambda_{\mathrm{ref}}$ from model	MLE	-0.00	-0.00	-0.00	-0.01	0.48	0.33	0.24	0.17
	MAP	0.36	0.27	0.19	0.11	0.36	0.30	0.25	0.21

Appendix

Table 4: Complete schedule-by-treatment grid, noisy observations (

\sigma_{\varepsilon}=0.05

), at

N=D=1000

with the uncapped posterior; layout and bold rule as in Table 3 . Best-so-far and recommendations are scored on the noiseless objective.

Figure 6: Per-step diagnostics, marginalised variance scales, noise-free. $N=D=1000$ , uncapped posterior, ten paired seeds (mean and $5$ th– $95$ th percentiles). Rows: best-so-far, value of the posterior-mean recommendation $F(\hat{\bm{s}})$ , degree-one recovery $\cos(\hat{\bm{s}},\bm{v}_{\star})$ , off-span norm $t$ of the selected query, fitted amplitude $\hat{a}=\sqrt{D}\,\hat{\sigma}_{1}$ , and fitted discrepancy scale $\hat{\sigma}_{\varepsilon}$ . Columns: the affine control and the mild, moderate, and severe non-affine objectives. Curves: no schedule (dashed), the model-derived reference (dash-dotted), and the fixed ladder $\lambda_{\mathrm{ref}}\in\{10^{0},10^{1},10^{3},10^{5}\}$ .

Figure 7: Per-step diagnostics, MAP variance scales, noise-free. Layout as in Figure 6 .

Figure 8: Per-step diagnostics, maximum-likelihood variance scales, noise-free. Layout as in Figure 6 . Without a schedule the fitted discrepancy scale collapses and the off-span norm stays near one for the whole run.

Figure 9: Per-step diagnostics, marginalised variance scales, noisy observations ( $\sigma_{\varepsilon}=0.05$ ). Layout as in Figure 6 ; best-so-far and the recommendation are scored on the noiseless objective.

Figure 10: Per-step diagnostics, MAP variance scales, noisy observations ( $\sigma_{\varepsilon}=0.05$ ). Layout as in Figure 6 .

Figure 11: Per-step diagnostics, maximum-likelihood variance scales, noisy observations ( $\sigma_{\varepsilon}=0.05$ ). Layout as in Figure 6 .

	affine	mild	mod.	severe
Comparisons
Random design + posterior argmax	0.93 [0.92, 0.93]	0.46 [0.45, 0.47]	0.31 [0.29, 0.33]	0.17 [0.16, 0.19]
Orthogonal design + posterior argmax	1.00 [1.00, 1.00]	0.45 [0.42, 0.47]	0.28 [0.25, 0.31]	0.13 [0.12, 0.14]
sep-CMA-ES	0.60 [0.59, 0.61]	0.50 [0.49, 0.51]	0.48 [0.46, 0.49]	0.45 [0.44, 0.46]
Thompson sampling	0.79 [0.78, 0.79]	0.49 [0.48, 0.51]	0.35 [0.30, 0.38]	0.20 [0.11, 0.27]
Ours (marginalised, $M=300$ )

Appendix

Table 5: Attained value at

D=N=10{,}000

M=300

: the better of the best observed point and the posterior-mean recommendation, mean and

5

–

95\%

over ten paired seeds. Bold: the column best and all within two standard deviations. The model-free baselines expose no recommendation and report their best observed point. Prescribing the off-span schedule beats every comparison on every non-affine objective.

Figure 12: Scientific latent-space optimisation against evaluation count. Best observed value against evaluation count for the runs of Figure 4 ; lines show means and bands $5$ th– $95$ th percentiles over ten seeds. Standard BO and Linear BO timed out at $48$ h of wall clock.

Bayesian Optimization

Bayesian optimization uses probabilistic models and sequential experiments to find strong solutions when evaluating a black-box objective is costly.

See all 240 papers

Sep 16, 2026

Search at the Cost of Sampling: Nearly-Instant Latent Space Bayesian Optimization

Generative models are increasingly central to many de novo discovery pipelines, in which designs are generated at scale and filtered through virtual screens to determine a set of candidates to experimentally validate. While Bayesian optimization (BO) is a natural fit for this setting, as it uses past evaluations to guide future proposals, the computational overhead required for its sequential decision-making becomes a bottleneck when virtual screens are relatively cheap. We make BO practical in this regime by exploiting the unique combination of a linear model constrained to a spherical domain where high-dimensional latents concentrate. We build off recent work justifying the use of linear surrogates, while deriving nearly closed-form solutions to the surrogate modelling and acquisition problems that exploit spherical symmetry. The result is at least a 100x speedup over state-of-the art baselines, with matching or improved performance across molecular and image generation benchmarks. Altogether, our method makes BO a practical drop-in for de novo pipelines where it was previously too slow to consider.

Donney Fan, Colin Doumont, Aleksandra Kalisz +4

University of British Columbia · Vector Institute · Tübingen AI Center +3

Surrogate Modeling Bayesian Optimization Latent Space Optimization

Ask PDF

Oct 17, 2025

Nonlinear Dimensionality Reduction Techniques for Bayesian Optimization

Bayesian optimisation (BO) enables sample-efficient global optimisation of expensive black-box functions but remains challenging in high dimensions. We investigate nonlinear dimensionality reduction to a sequence of low-dimensional latent-space BO (LSBO) problems. Early LSBO used linear random and supervised embeddings; building on Grosnit et al., we employ variational autoencoders (VAEs), deep metric loss for structured latent manifolds, and retraining to adapt the encoder-decoder pair to newly sampled regions. We couple LSBO with sequential domain reduction (SDR) directly in latent space (SDR-LSBO), narrowing search domains as evidence accumulates. Implemented in GPU-accelerated BoTorch with Mat'ern-5/2 Gaussian-process surrogates, our methods improve benchmark optimisation quality, and retraining can enhance BO performance. Comparisons with adaptive supervised linear random embeddings demonstrate the effectiveness of VAE-based BO for nonlinear low-dimensional structures. We analyse BO-VAE with a fixed pretrained representation, decomposing ambient-space simple regret into latent BO error and a fixed VAE-induced representation gap. Under a PAC-Bayes-certified reconstruction condition and standard fixed-prior assumptions for expected improvement with a Mat'ern-5/2 kernel, latent BO error vanishes as the evaluation budget increases, whereas the representation gap remains fixed and may impose a non-vanishing error floor. Visualisations empirically assess accessibility of the ambient optimum through the learned decoder. To our knowledge, this is the first study combining SDR with VAE-based LSBO. Our analysis clarifies metric shaping and retraining choices critical for scalable latent-space BO. For reproducibility, source code is available at https://github.com/L-Lok/Nonlinear-Dimensionality-Reduction-Techniques-for-Bayesian-Optimization.git.

Luo Long, Coralia Cartis, Paz Fink Shustin

Mathematical Institute, University of Oxford, Radcliffe Observatory, Andrew Wiles Building, Woodstock Rd, Oxford, OX2 6GG, United Kingdom.

Variational Autoencoders Bayesian Optimization Dimensionality Reduction Latent Space Optimization

Ask PDF

Oct 6, 2026

Work While They Sleep: Exploiting Evaluation Latency for Fully Bayesian Optimization

Black-box optimization problems are ubiquitous across science and engineering, often dealing with expensive objective functions. This objective latency has two consequences during optimization: (i) the objective evaluation dominates execution time, and (ii) sample-efficient algorithms are crucial to accelerate development and avoid wasting resources. Bayesian optimization (BO) methods are the de facto choice of planners for suggesting the next point to try. Standard BO fits the surrogate model's hyperparameters with a point estimate. Alternatively, a fully Bayesian approach uses model averaging to account for uncertainty over the hyperparameters, leading to better uncertainty estimates---useful in the low-data regime that is pervasive in BO. However, it is often prohibitively expensive and thus rarely used. In this work, we propose ELF-BO, an algorithm that uses the objective evaluation latency to headstart the computation of the next suggestion, allowing for fully Bayesian optimization without incurring substantial decision-time costs. This is done by sampling from the hyperparameter posterior while the objective is being evaluated, only requiring reweighting of the samples once the objective value is observed. Across synthetic functions and real-world applications, we show that ELF-BO matches the performance of fully Bayesian methods while only incurring decision latency on par with or better than standard BO. Thus, ELF-BO makes fully Bayesian optimization practical in real-world use cases.

Gustavo Sutter, Alejandro Comas-Leon, David Holzmüller +4

Cheriton School of Computer Science, University of Waterloo, Waterloo, ON, Canada · Vector Institute, Toronto, ON, Canada · SODA Team, INRIA Saclay, Palaiseau, France +3

Bayesian Optimization Surrogate-Assisted Optimization

Ask PDF

Abstract

Bayesian Optimization

Explore similar work

Search at the Cost of Sampling: Nearly-Instant Latent Space Bayesian Optimization

Nonlinear Dimensionality Reduction Techniques for Bayesian Optimization

Work While They Sleep: Exploiting Evaluation Latency for Fully Bayesian Optimization