IQS-BO: In-Context Query Selection for Bayesian Optimisation

Luca Geminiani; Nadja Klein

cs.LGOct 1, 2026

IQS-BO: In-Context Query Selection for Bayesian Optimisation

Authors: Luca Geminiani, Nadja Klein

Organizations: Scientific Computing Center (SCC), Karlsruhe Institute of Technology (KIT)

Abstract

Bayesian Optimisation (BO) is a powerful framework for the optimisation of expensive black-box functions, but typically requires refitting a surrogate and maximising an acquisition function at every evaluation step. In-context approaches based on Prior-data Fitted Networks (PFNs) amortise part of this cost by pre-training transformers on functions drawn from synthetic priors. PFNs4BO amortises the surrogate but still relies on a numerically maximised acquisition function, while FIBO performs BO fully in-context by sampling optimiser locations from a learned density, which fixes the decision rule and admits no surrogate. Learned acquisition functions score a finite candidate set with a trained network, but, lacking a label for the query, learn the score by reinforcement learning on previously solved tasks. We propose IQS-BO, a PFN that learns the query decision by supervised learning on synthetic priors. In a single forward pass, IQS-BO predicts the probability that each candidate maximises the objective over the set, and we show that the minimiser of its objective is the posterior probability of this event. The model can be pre-trained without a surrogate for fully in-context BO, or take the predictions of a fixed probabilistic surrogate as additional input, amortising only the decision step. Our method proposes queries at a fraction of the cost of acquisition-based methods, while either matching or outperforming standard BO with Gaussian processes (GPs) and available in-context methods on synthetic and real-world benchmarks. Finally, we propose a mixture prior for pre-training PFNs which combines samples from GPs with functions exhibiting warped inputs, isolated narrow optima, or plateaus that are poorly modeled by stationary kernels common in GP surrogates. We show that pre-training on this prior can lead to improved optimisation performance.

Figures & tables

Figure 1: One optimisation round of traditional BO, PFNs4BO ( Müller et al., 2023 ) , FIBO ( de Carvalho et al., 2025 ) and IQS-BO with and without a surrogate.

\mathcal{D}_{k}

: observations after

k

evaluations,

x_{k+1}

: next query,

\mathcal{C}_{k}

: candidates drawn from

\mathcal{D}_{k}

\alpha

: acquisition function,

J^{*}

: index of the best candidate. Red: fitted or optimised every round. Green: one forward pass of a pre-trained network. Red-to-green: fixed surrogate of either kind. Blue: sampling without learning. Grey: quantities passed between steps.

Figure 2: Average rank (left) and average regret (right) per iteration, ten 3D problems,

n_{\text{init}}=40

, 90 seeds,

q=2

, with

95\%

confidence intervals. Top, methods pre-trained on the mixture prior ( † ). Bottom, methods pre-trained on the HEBO prior or on GP draws ( ⋆ ). Ranks are computed within each row, and regret is normalised by the mean initial regret of each problem.

	$R^{\text{AUC}}$						$R^{\text{final}}$
	$d=3$			$d=4$			$d=3$			$d=4$
$n_{\text{init}}$	10	20	40	10	20	40	10	20	40	10	20	40
IQS-BO †	3.1 $\pm$ .1	2.8 $\pm$ .1	2.8 $\pm$ .1	3.3 $\pm$ .1	3.5 $\pm$ .1	3.4 $\pm$ .1	2.8 $\pm$ .1	2.6 $\pm$ .1	2.7 $\pm$ .1	3.4 $\pm$ .1	3.5 $\pm$ .1	3.3 $\pm$ .1
IQS-BO-PFN †	3.1 $\pm$ .1	2.7 $\pm$ .1	2.6 $\pm$ .1	3.6 $\pm$ .1	3.5 $\pm$ .1	3.3 $\pm$ .1	2.8 $\pm$ .1	2.5 $\pm$ .1	2.5 $\pm$ .1	3.5 $\pm$ .1	3.5 $\pm$ .1	3.3 $\pm$ .1
IQS-BO-GP †	3.6 $\pm$ .1	3.3 $\pm$ .1	3.3 $\pm$ .1	3.6 $\pm$ .1	3.2 $\pm$ .1	3.1 $\pm$ .1	2.8 $\pm$ .1	2.8 $\pm$ .1	2.7 $\pm$ .1	3.2 $\pm$ .1	3.0 $\pm$ .1	2.9 $\pm$ .1
PFNs4BO †	3.8 $\pm$ .1	4.2 $\pm$ .1	4.2 $\pm$ .1	4.1 $\pm$ .1	4.4 $\pm$ .1	5.1 $\pm$ .1	4.9 $\pm$ .1	5.1 $\pm$ .1	5.1 $\pm$ .1	5.2 $\pm$ .1	5.3 $\pm$ .1	5.4 $\pm$ .1

Table 1: Mean rank on

R^{\text{AUC}}

(left) and

R^{\text{final}}

(right), over 90 seeds and the ten 3D or five 4D problems,

\pm

95\%

bootstrap CI half-width. † mixture prior, ⋆ HEBO or GP prior. Ranks are computed within each block, and bold marks the best rank per column.

Figure 3: Paired win rates (Section 4.1 ), ten 3D problems, 900 paired runs, mean with

95\%

CI. Left, on

\tilde{r}(t)

per iteration,

n_{\text{init}}=40

. Right, on

R^{\text{final}}

, each model pre-trained on the mixture prior against its original prior.

Figure 4: GAP after 200 evaluations against the time spent proposing queries over a complete run, one marker per problem and method, seven 3D problems,

n_{\text{init}}=40

, mean over 90 seeds. Upper left is better.

Variant	$n_{\text{init}}{=}10$	$20$	$40$
space-filling	$84.2{\scriptstyle\pm 2.2}$	$87.9{\scriptstyle\pm 2.0}$	$88.4{\scriptstyle\pm 1.8}$
$M{=}128$	$57.9{\scriptstyle\pm 3.1}$	$56.7{\scriptstyle\pm 3.2}$	$56.6{\scriptstyle\pm 3.1}$
$M{=}2048$	$48.3{\scriptstyle\pm 3.2}$	$46.4{\scriptstyle\pm 3.2}$	$50.3{\scriptstyle\pm 3.2}$
top- $q$ batch	$59.2{\scriptstyle\pm 3.2}$	$57.9{\scriptstyle\pm 3.2}$	$55.6{\scriptstyle\pm 3.2}$

Table 2: Win rate (%) on

R^{\text{final}}

of IQS-BO against each variant, means

\pm

95\%

CI over ten 3D problems and 900 paired runs.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Learned object	Training signal	Surrogate	Next query
PFNs4BO	posterior predictive	function values, prior	is one	maximise acquisition
FIBO	density over optimum	global optimiser, prior	none	sample
MetaBO, FSAF	candidate score	RL reward, solved tasks	GP, refit	argmax
NAP	candidate score	RL reward, solved tasks	learned jointly	argmax
POLAR	candidate score	RL reward, solved tasks	pretrained	argmax
IQS-BO	posterior over set	best candidate, prior	none or frozen	argmax

Appendix

Table 3: What each method learns, from which signal, and how it proposes the next query.

channel	width	carried by	definition
coordinates	$d$	all tokens	$x\in[0,1]^{d}$
distances	2	all tokens	$\min_{i}\lVert x-x_{i}\rVert_{2}$ and $\lVert x-x^{+}\rVert_{2}$
role	4	all tokens	one-hot token type
state	2	all tokens	$1/\sqrt{n}$ , $\log_{10}\sigma_{\mathrm{noise}}$
value	1	observations, filled pending	$(y_{i}-\bar{y})/s_{y}$ with $s_{y}\geq 10^{-3}$ , or fantasy $\hat{y}$
rank	1	observations	$\Phi^{-1}\big((r_{i}+\tfrac{1}{2})/n\big)$

Appendix

Table 4: Token channels. Statistics are over the

n

observed values, and

x^{+}

is the incumbent. Channels are zero on tokens that do not carry them. The noise channel holds the task’s noise level in pre-training and a fixed

\sigma_{0}

at deployment (Appendix B.2 ).

model	feature	width	carried by	definition
IQS-BO-PFN	layer states	512	all tokens	$\sum_{\ell}\operatorname{softmax}(\alpha)_{\ell}\,\mathrm{state}_{\ell,s}$ , learned $\alpha\in\mathbb{R}^{12}$
IQS-BO-PFN	summaries	5	candidates	mean, $68\%$ half-width, $10/50/90\%$ quantiles
IQS-BO-GP	mean	1	all tokens	posterior mean $\mu(x)$
	std. dev.	1	all tokens	latent posterior $\sigma(x)$
	correlation	1	all tokens	$\operatorname{corr}\big(f(x),f(x^{+})\big)$
	hyperparameters	$d+2$	all tokens	log-lengthscales, log-noise, constant mean

Appendix

Table 5: Surrogate features

m_{s}

. The surrogate is conditioned on the observations and on the filled pending points at their fantasy values. Empty pending tokens receive zeros.

lengthscales	$\ell_{i}\sim\mathrm{Gamma}(1.2107,1.5212)$
output scale	$s^{2}\sim\mathrm{Gamma}(0.8452,0.3994)$
noise	$\sigma^{2}\sim\mathrm{LogNormal}(-4.63,0.5)$ , with relative noise $\sqrt{\sigma^{2}/s^{2}}$ clamped to $[10^{-3},10]$
active dimensions	each dimension dropped with probability $0.3$ , keeping at least one

Appendix

Table 6: HEBO prior. Functions are drawn as

f\sim\mathcal{GP}(0,s^{2}k_{3/2})

, with

k_{3/2}

the Matérn-3/2 ARD kernel with lengthscales

\ell

. Gamma distributions are given by shape and rate.

family	weight	definition
stationary	$0.3$	$\mathrm{rff}$
warped	$\approx 0.233$	$\mathrm{rff}\circ\kappa$ , $\kappa_{i}(x)=1-(1-x^{\beta_{i}})^{\alpha_{i}}$ $\alpha_{i},\beta_{i}\sim\mathrm{LogNormal}(0,0.75)$
bumps	$\approx 0.233$	$\sum_{k\leq K}h_{k}\exp\big(-\tfrac{1}{2}\lVert(x-c_{k})/w_{k}\rVert^{2}\big)$ $K\sim\mathcal{U}\{2,\dots,10\}$ , $c_{k}\sim\mathcal{U}[0,1]^{d}$ , $w_{ki}\sim\mathrm{LogU}(0.03,2)$
plateau	$\approx 0.233$	$\sum_{j\leq L}\mathrm{sigm}\big((\mathrm{rff}-t_{j})/\tau\big)+10^{-3}\,\mathrm{rff}$ $L\sim\mathcal{U}\{2,\dots,7\}$ , $\tau\sim\mathrm{LogU}(0.01,0.1)$

Appendix

Table 7: Definition of the mixture prior. The warped family uses Kumaraswamy CDFs

\kappa_{i}

( Kumaraswamy, 1980 ; Snoek et al., 2014 ) , with

\alpha_{i},\beta_{i}

clamped to

[0.2,5]

. In the bumps family, the division by

w_{k}

is elementwise, and

\mathrm{sigm}

denotes the logistic function.

Table 12

problem	$d$	native domain	source	problem	$d$	native domain	source
Ackley	3, 4	$[-32.768,32.768]^{d}$	BoTorch	Six-Hump Camel	3	$[-3,3]\times[-2,2]$	BoTorch
Levy	3, 4	$[-10,10]^{d}$	BoTorch	bioprocess	3	$[0,1]^{3}$	this work
Rosenbrock	3, 4	$[-2.048,2.048]^{d}$	BoTorch	fullerenes	3	emulator	Olympus
Rastrigin	3	$[-5.12,5.12]^{3}$	BoTorch	colours	3	emulator	Olympus
Michalewicz	3	$[0,\pi]^{3}$	BoTorch	benzylation	4	emulator	Olympus
Hartmann	3	$[0,1]^{3}$	BoTorch	GP-prior draws	4	$[0,1]^{4}$	FIBO prior

Appendix

Table 10: Benchmark problems. Minimisation problems are negated.

Table 14

Figure 5: As Figure 2 , at

n_{\text{init}}=10

	model	GP-MAP	GP-FB	PFNs4BO-HEBO	PFNs4BO-BNN	FIBO	Sobol
$n_{\text{init}}=10$	IQS-BO †	$77.4{\scriptstyle\pm 2.4}$	$77.3{\scriptstyle\pm 2.4}$	$86.4{\scriptstyle\pm 2.1}$	$85.1{\scriptstyle\pm 2.2}$	$92.7{\scriptstyle\pm 1.7}$	$96.4{\scriptstyle\pm 1.2}$
	IQS-BO-PFN †	$76.1{\scriptstyle\pm 2.6}$	$77.3{\scriptstyle\pm 2.4}$	$86.3{\scriptstyle\pm 2.2}$	$85.7{\scriptstyle\pm 2.2}$	$90.9{\scriptstyle\pm 1.9}$	$95.4{\scriptstyle\pm 1.4}$
	IQS-BO-GP †	$77.6{\scriptstyle\pm 2.6}$	$78.8{\scriptstyle\pm 2.4}$	$85.7{\scriptstyle\pm 2.1}$	$83.9{\scriptstyle\pm 2.2}$	$91.0{\scriptstyle\pm 1.8}$	$96.3{\scriptstyle\pm 1.2}$
	IQS-BO ⋆	$65.2{\scriptstyle\pm 2.7}$	$67.4{\scriptstyle\pm 2.7}$	$75.3{\scriptstyle\pm 2.7}$	$75.4{\scriptstyle\pm 2.8}$	$83.6{\scriptstyle\pm 2.2}$	$94.2{\scriptstyle\pm 1.5}$
	IQS-BO-PFN ⋆	$49.7{\scriptstyle\pm 3.1}$	$51.0{\scriptstyle\pm 3.0}$	$60.4{\scriptstyle\pm 2.9}$	$61.8{\scriptstyle\pm 3.1}$	$72.0{\scriptstyle\pm 2.8}$	$90.9{\scriptstyle\pm 1.8}$
	IQS-BO-GP ⋆	$71.0{\scriptstyle\pm 2.8}$	$71.6{\scriptstyle\pm 2.7}$	$77.0{\scriptstyle\pm 2.6}$	$76.7{\scriptstyle\pm 2.6}$	$85.3{\scriptstyle\pm 2.1}$	$93.7{\scriptstyle\pm 1.6}$

Appendix

Table 13: Paired win rate (%) on

R^{\text{final}}

of each IQS-BO variant against each baseline, ten 3D problems, 900 paired runs per cell,

\pm

95\%

bootstrap CI half-width. † mixture prior, ⋆ HEBO prior.

Figure 7: Average regret per iteration on each 3D problem,

n_{\text{init}}=10

(top),

20

(middle) and

40

(bottom), 90 seeds, with

95\%

confidence intervals.

Figure 8: As Figure 2 , on the five 4D problems,

n_{\text{init}}=40

. Curves for

n_{\text{init}}=10

and

20

show the same pattern.

Method	Ackley	Levy	Rosenbrock	Benzylation	GP prior
IQS-BO †	$\mathbf{0.111}{\scriptstyle\pm 0.006}$	$0.016{\scriptstyle\pm 0.008}$	$0.028{\scriptstyle\pm 0.002}$	$0.141{\scriptstyle\pm 0.016}$	$0.079{\scriptstyle\pm 0.011}$
IQS-BO-PFN †	$0.118{\scriptstyle\pm 0.006}$	$0.031{\scriptstyle\pm 0.013}$	$0.026{\scriptstyle\pm 0.002}$	$0.135{\scriptstyle\pm 0.016}$	$0.043{\scriptstyle\pm 0.007}$
IQS-BO-GP †	$0.123{\scriptstyle\pm 0.006}$	$0.024{\scriptstyle\pm 0.009}$	$\mathbf{0.025}{\scriptstyle\pm 0.002}$	$0.113{\scriptstyle\pm 0.018}$	$0.018{\scriptstyle\pm 0.004}$
PFNs4BO †	$0.432{\scriptstyle\pm 0.020}$	$0.116{\scriptstyle\pm 0.016}$	$0.080{\scriptstyle\pm 0.008}$	$\mathbf{0.000}{\scriptstyle\pm 0.000}$	$0.142{\scriptstyle\pm 0.008}$
FIBO †	$0.295{\scriptstyle\pm 0.014}$	$0.138{\scriptstyle\pm 0.025}$	$0.059{\scriptstyle\pm 0.006}$	$0.049{\scriptstyle\pm 0.005}$	$0.152{\scriptstyle\pm 0.009}$
IQS-BO ⋆	$0.228{\scriptstyle\pm 0.048}$	$\mathbf{0.006}{\scriptstyle\pm 0.003}$	$0.038{\scriptstyle\pm 0.003}$	$0.102{\scriptstyle\pm 0.018}$	$0.076{\scriptstyle\pm 0.010}$

Appendix

Table 14: Mean normalised final regret per 4D problem,

n_{\text{init}}=40

, 90 seeds,

\pm

95\%

CI. The GP-prior column averages ten instances. Bold marks the best value per problem.

Figure 9: Average regret per iteration on each 4D problem (columns),

n_{\text{init}}=10

20

and

40

(rows), 90 seeds (900 runs for the ten GP-prior draws), with

95\%

confidence intervals.

	$R^{\text{final}}$			$R^{\text{AUC}}$
$n_{\text{init}}$	10	20	40	10	20	40
IQS-BO-GP † vs EI-CAND	$66.1{\scriptstyle\pm 3.1}$	$68.2{\scriptstyle\pm 3.1}$	$70.3{\scriptstyle\pm 2.9}$	$53.3{\scriptstyle\pm 3.2}$	$61.6{\scriptstyle\pm 3.2}$	$69.6{\scriptstyle\pm 3.0}$
EI-CAND vs GP-MAP	$70.7{\scriptstyle\pm 3.1}$	$71.2{\scriptstyle\pm 2.9}$	$67.3{\scriptstyle\pm 3.1}$	$61.7{\scriptstyle\pm 3.2}$	$61.9{\scriptstyle\pm 3.2}$	$62.5{\scriptstyle\pm 3.1}$
IQS-BO-GP † vs GP-MAP	$77.6{\scriptstyle\pm 2.7}$	$78.3{\scriptstyle\pm 2.7}$	$81.1{\scriptstyle\pm 2.6}$	$62.0{\scriptstyle\pm 3.2}$	$70.4{\scriptstyle\pm 3.0}$	$75.1{\scriptstyle\pm 2.8}$
IQS-BO-GP ⋆ vs EI-CAND	$54.3{\scriptstyle\pm 3.3}$	$57.7{\scriptstyle\pm 3.3}$	$56.2{\scriptstyle\pm 3.3}$	$30.1{\scriptstyle\pm 3.0}$	$37.2{\scriptstyle\pm 3.2}$	$46.8{\scriptstyle\pm 3.2}$
IQS-BO-GP ⋆ vs GP-MAP	$71.0{\scriptstyle\pm 2.9}$	$75.0{\scriptstyle\pm 2.8}$	$70.1{\scriptstyle\pm 3.0}$	$38.8{\scriptstyle\pm 3.2}$	$49.0{\scriptstyle\pm 3.3}$	$58.7{\scriptstyle\pm 3.2}$

Appendix

Table 15: Paired win rate (%) of the first method against the second, ten 3D problems, 900 paired runs per cell,

\pm

95\%

bootstrap CI half-width. † mixture prior, ⋆ HEBO prior. EI-CAND requires no pre-training, so its comparison with GP-MAP is the same under both priors and is shown once.

	$R^{\text{final}}$			$R^{\text{AUC}}$
$n_{\text{init}}$	10	20	40	10	20	40
IQS-BO	$66.7{\scriptstyle\pm 3.1}$	$68.3{\scriptstyle\pm 3.1}$	$68.8{\scriptstyle\pm 3.1}$	$76.1{\scriptstyle\pm 2.8}$	$75.7{\scriptstyle\pm 2.8}$	$76.7{\scriptstyle\pm 2.7}$
IQS-BO-PFN	$77.5{\scriptstyle\pm 2.8}$	$76.0{\scriptstyle\pm 2.8}$	$76.2{\scriptstyle\pm 2.8}$	$80.7{\scriptstyle\pm 2.6}$	$82.6{\scriptstyle\pm 2.5}$	$83.4{\scriptstyle\pm 2.3}$
IQS-BO-GP	$62.2{\scriptstyle\pm 3.2}$	$61.8{\scriptstyle\pm 3.1}$	$65.8{\scriptstyle\pm 3.1}$	$70.3{\scriptstyle\pm 3.0}$	$70.9{\scriptstyle\pm 2.9}$	$68.7{\scriptstyle\pm 3.1}$
PFNs4BO	$53.0{\scriptstyle\pm 3.3}$	$55.5{\scriptstyle\pm 3.2}$	$62.9{\scriptstyle\pm 3.2}$	$56.7{\scriptstyle\pm 3.2}$	$54.3{\scriptstyle\pm 3.2}$	$63.0{\scriptstyle\pm 3.2}$
FIBO	$51.1{\scriptstyle\pm 3.3}$	$47.9{\scriptstyle\pm 3.3}$	$50.9{\scriptstyle\pm 3.3}$	$50.9{\scriptstyle\pm 3.2}$	$53.2{\scriptstyle\pm 3.3}$	$54.2{\scriptstyle\pm 3.2}$

Appendix

Table 16: Paired win rate (%) of each model pre-trained on the mixture prior against the same model pre-trained on its original prior, ten 3D problems, 900 paired runs per cell,

\pm

95\%

bootstrap CI half-width.

	Rosenbrock (3D)			Rosenbrock (4D)			GP draws (4D)			Benzylation (4D)
$n_{\text{init}}$	10	20	40	10	20	40	10	20	40	10	20	40
IQS-BO	$62{\scriptstyle\pm 10}$	$58{\scriptstyle\pm 10}$	$64{\scriptstyle\pm 10}$	$58{\scriptstyle\pm 10}$	$67{\scriptstyle\pm 9}$	$78{\scriptstyle\pm 8}$	$55{\scriptstyle\pm 3}$	$53{\scriptstyle\pm 3}$	$46{\scriptstyle\pm 3}$	$56{\scriptstyle\pm 6}$	$48{\scriptstyle\pm 6}$	$39{\scriptstyle\pm 7}$
IQS-BO-PFN	$72{\scriptstyle\pm 9}$	$64{\scriptstyle\pm 10}$	$61{\scriptstyle\pm 10}$	$80{\scriptstyle\pm 8}$	$92{\scriptstyle\pm 5}$	$91{\scriptstyle\pm 6}$	$63{\scriptstyle\pm 3}$	$65{\scriptstyle\pm 3}$	$65{\scriptstyle\pm 3}$	$45{\scriptstyle\pm 6}$	$45{\scriptstyle\pm 6}$	$45{\scriptstyle\pm 7}$
IQS-BO-GP	$44{\scriptstyle\pm 10}$	$26{\scriptstyle\pm 9}$	$51{\scriptstyle\pm 10}$	$69{\scriptstyle\pm 9}$	$66{\scriptstyle\pm 10}$	$61{\scriptstyle\pm 10}$	$46{\scriptstyle\pm 3}$	$44{\scriptstyle\pm 3}$	$48{\scriptstyle\pm 3}$	$58{\scriptstyle\pm 6}$	$45{\scriptstyle\pm 7}$	$32{\scriptstyle\pm 6}$
PFNs4BO	$49{\scriptstyle\pm 10}$	$56{\scriptstyle\pm 10}$	$74{\scriptstyle\pm 9}$	$26{\scriptstyle\pm 9}$	$28{\scriptstyle\pm 9}$	$40{\scriptstyle\pm 10}$	$49{\scriptstyle\pm 3}$	$58{\scriptstyle\pm 3}$	$59{\scriptstyle\pm 3}$	$78{\scriptstyle\pm 6}$	$95{\scriptstyle\pm 3}$	$97{\scriptstyle\pm 2}$
FIBO	$50{\scriptstyle\pm 10}$	$44{\scriptstyle\pm 10}$	$46{\scriptstyle\pm 10}$	$48{\scriptstyle\pm 10}$	$60{\scriptstyle\pm 10}$	$51{\scriptstyle\pm 11}$	$20{\scriptstyle\pm 2}$	$16{\scriptstyle\pm 2}$	$13{\scriptstyle\pm 2}$	$100{\scriptstyle\pm 0}$	$100{\scriptstyle\pm 0}$	$100{\scriptstyle\pm 0}$

Appendix

Table 17: Paired win rate (%) on

R^{\text{final}}

of each model pre-trained on the mixture prior against the same model pre-trained on its original prior, on the problems for which GP surrogates are well specified,

\pm

95\%

CI. 90 paired runs per cell (900 for the GP draws).

Figure 10: Average regret per iteration of IQS-BO , IQS-BO-PFN , PFNs4BO and FIBO pre-trained on the mixture prior (solid) and on their original prior (dashed), with GP-MAP as reference, ten 3D problems,

n_{\text{init}}=40

, 90 seeds, with

95\%

confidence intervals.

	method	Ackley	Hartm.	Bioproc.	Levy	Michal.	Rastr.	Rosenb.	mean	s/run
$q=10$	IQS-BO	$.940{\scriptscriptstyle\pm 6}$	$\mathbf{.999{\scriptscriptstyle\pm 0}}$	$\mathbf{.999{\scriptscriptstyle\pm 0}}$	$\mathbf{.998{\scriptscriptstyle\pm 1}}$	$.853{\scriptscriptstyle\pm 51}$	$\mathbf{.750{\scriptscriptstyle\pm 39}}$	$.909{\scriptscriptstyle\pm 20}$	$\mathbf{.921{\scriptscriptstyle\pm 10}}$	$0.9$
	IQS-BO-PFN	$\mathbf{.949{\scriptscriptstyle\pm 6}}$	$.998{\scriptscriptstyle\pm 1}$	$.998{\scriptscriptstyle\pm 1}$	$.954{\scriptscriptstyle\pm 13}$	$\mathbf{.874{\scriptscriptstyle\pm 47}}$	$.741{\scriptscriptstyle\pm 38}$	$.933{\scriptscriptstyle\pm 30}$	$\mathbf{.921{\scriptscriptstyle\pm 10}}$	$3.2$
	GP-MAP	$.839{\scriptscriptstyle\pm 13}$	$\mathbf{.999{\scriptscriptstyle\pm 0}}$	$.995{\scriptscriptstyle\pm 2}$	$.857{\scriptscriptstyle\pm 35}$	$.760{\scriptscriptstyle\pm 57}$	$.682{\scriptscriptstyle\pm 40}$	$\mathbf{.973{\scriptscriptstyle\pm 6}}$	$.872{\scriptscriptstyle\pm 11}$	$54.5$
	GP-FB	$.834{\scriptscriptstyle\pm 14}$	$\mathbf{.999{\scriptscriptstyle\pm 0}}$	$.994{\scriptscriptstyle\pm 2}$	$.885{\scriptscriptstyle\pm 29}$	$.719{\scriptscriptstyle\pm 61}$	$.679{\scriptscriptstyle\pm 39}$	$.971{\scriptscriptstyle\pm 11}$	$.869{\scriptscriptstyle\pm 11}$	$350.3$
	PFNs4BO-HEBO	$.773{\scriptscriptstyle\pm 20}$	$.968{\scriptscriptstyle\pm 10}$	$.879{\scriptscriptstyle\pm 41}$	$.946{\scriptscriptstyle\pm 16}$	$.797{\scriptscriptstyle\pm 39}$	$.714{\scriptscriptstyle\pm 31}$	$.855{\scriptscriptstyle\pm 37}$	$.848{\scriptscriptstyle\pm 11}$	$182.6$
	PFNs4BO-BNN	$.719{\scriptscriptstyle\pm 18}$	$.975{\scriptscriptstyle\pm 11}$	$.987{\scriptscriptstyle\pm 8}$	$.949{\scriptscriptstyle\pm 15}$	$.547{\scriptscriptstyle\pm 59}$	$.678{\scriptscriptstyle\pm 35}$	$.850{\scriptscriptstyle\pm 39}$	$.815{\scriptscriptstyle\pm 12}$	$131.8$

Appendix

Table 18: GAP after 200 evaluations per problem and mean over the seven problems, mean over 90 seeds, with the

95\%

CI half-width in units of

10^{-3}

as subscript, and seconds spent proposing queries over a complete run. Bold marks the best value per problem and batch size.

$q$	1	2	10	20
IQS-BO †	0.011 [0.011, 0.011]	0.020 [0.020, 0.022]	0.085 [0.085, 0.085]	0.170 [0.170, 0.171]
IQS-BO-PFN †	0.030 [0.030, 0.030]	0.060 [0.060, 0.060]	0.302 [0.301, 0.303]	0.610 [0.604, 0.612]
IQS-BO-GP †	0.096 [0.096, 0.096]	0.105 [0.105, 0.106]	0.185 [0.183, 0.186]	0.282 [0.280, 0.284]
FIBO	0.021 [0.021, 0.021]	0.022 [0.022, 0.022]	0.022 [0.022, 0.022]	0.022 [0.022, 0.023]
GP-MAP	0.233 [0.233, 0.233]	0.232 [0.232, 0.233]	3.47 [3.47, 3.47]	10.7 [10.7, 10.8]
PFNs4BO-HEBO	3.18 [3.17, 3.19]	6.78 [6.76, 6.79]	32.6 [32.6, 32.6]	64.9 [64.9, 65.0]

Appendix

Table 19: Median seconds per proposed batch on a GPU workstation (Quadro P2200, Xeon W-1350P), Ackley in 3D,

n=30

observations, ten repeats (three for PFNs4BO-HEBO ), with the interquartile range in brackets. † mixture prior.

Figure 11: GAP per evaluation on each problem of the batch study (rows), for

q=10

20

and

50

(columns), networks pre-trained on the HEBO prior,

n_{\text{init}}=40

, 90 seeds, with

95\%

confidence intervals. The y-axis of every panel shows the GAP. Solid lines are checkpoints pre-trained at the deployed batch size, dashed lines the

q=2

checkpoints used at that batch size.

Explore similar work

Jun 8, 2026cs.LG

In-Context Learning for Latent Space Bayesian Optimization

Bayesian optimization (BO) is a central tool for sample-efficient design, and latent-space Bayesian optimization (LSBO) extends it to structured objects such as molecules and proteins. In parallel, tabular foundation models such as TabPFN and TabICL now achieve state-of-the-art regression performance and are increasingly used as BO surrogates. Because their Bayesian behavior is induced by large synthetic pretraining collections, the composition of this pretraining distribution is crucial. LSBO creates a distinctive mismatch: the induced map from latent code to objective value differs markedly from the regression tasks used to train current in-context models. We address this mismatch by complementing the pretraining stage of tabular foundation model surrogates with synthetic optimization tasks defined on the latent space of a molecular VAE. The continued-pretraining objective features a regularizer that anchors the model to the original checkpoint, preserving its broad regression prior while avoiding overspecialization to the adaptation tasks. On held-out molecular optimization benchmarks, the resulting model achieves strong performance, supporting the relevance of LSBO-specific adaptation for in-context surrogates.

Tuan A. Vu, Harri Lähdesmäki, Julien Martinelli

Aalto University

Bayesian Optimization Molecular Optimization In-Context Learning Latent Space Rapid Adaptation

Ask PDF

Jun 5, 2026cs.LG

$α$ -PFN: Fast Entropy Search via In-Context Learning

Information-theoretic acquisition functions such as Entropy Search (ES) offer a principled exploration-exploitation framework for Bayesian optimization (BO). However, their practical implementation relies on complicated and slow approximations, i.e., a Monte Carlo estimation of the information gain. This complexity can introduce numerical errors and requires specialized, hand-crafted implementations. We propose a two-stage amortization strategy that learns to approximate entropy search-based acquisition functions using Prior-data Fitted Networks (PFNs) in a single forward pass. A first PFN is trained to be conditioned on information about the optima; second, the

α

-PFN is trained to predict the expected information gain by training on information gains measured with the first PFN. The

α

-PFN offers a flexible learned approximation, which replaces the complex heuristic approximations with a single forward pass per candidate, enabling rapid and extensible acquisition evaluation. Empirically, our approach is competitive with state-of-the-art entropy search implementations on synthetic and real-world benchmarks, while accelerating the different entropy search variants across all our experiments, with speed ups over 50x. Source code: https://github.com/automl/AlphaPFN.

Herilalaina Rakotoarison, Steven Adriaensen, Tom Viering +4

University of Helsinki · University of Freiburg · Delft University of Technology +2

Bayesian Optimization Search Algorithms Entropy In-Context Learning

Ask PDF

Jun 1, 2026cs.LG

Local Preferential Bayesian Optimization

Bayesian optimization (BO) is a popular and effective approach for tuning expensive, noisy experiments, but requires the formulation of an explicit objective function. Preferential BO (PBO) removes this requirement by learning from pairwise human feedback, yet existing methods struggle to efficiently optimize beyond low- and medium-dimensional problems due to their global search approaches. We address this limitation by developing a family of local PBO methods that transfer key ideas from high-dimensional BO to the preferential setting. In particular, we introduce local PBO methods which adapt trust-region and derivative-informed local search to pairwise preference feedback, where the latter exploits first- and second-order derivatives of the Laplace-approximated GP posterior. Our benchmark on GP sample paths, standard optimization benchmark functions, and policy-search tasks shows that local PBO methods are especially effective in high-dimensional and complex landscapes with steep optima. Compared with global preference-based baselines, they can substantially reduce cumulative regret, making them particularly useful for real-world preference-based optimization tasks such as policy search.

Johanna Menn, Miriam Kober, Paul Brunzema +2

Institute for Data Science in Mechanical Engineering, RWTH Aachen University, Aachen, Germany · Department of Clinical Research, University of Bern, Bern, Switzerland · Center for Reproducible Science and Research Synthesis, University of Zurich, Zurich, Switzerland +1

Multi-Objective Bayesian Optimization Algorithms Bayesian Optimization Large Neighborhood Search Sequential Preference Optimization

Ask PDF

Abstract

Explore similar work

In-Context Learning for Latent Space Bayesian Optimization

ααα-PFN: Fast Entropy Search via In-Context Learning

Local Preferential Bayesian Optimization

$α$ -PFN: Fast Entropy Search via In-Context Learning