Time-Aware Latent Space Bayesian Optimization

Tuan A. Vu; Julien Martinelli; Harri Lähdesmäki

stat.MLMar 1, 2026

Time-Aware Latent Space Bayesian Optimization

Authors: Tuan A. Vu, Julien Martinelli, Harri Lähdesmäki

Organizations: Aalto University

Abstract

Latent-space Bayesian optimization (LSBO) extends Bayesian optimization to structured domains, such as molecular design, by searching in the continuous latent space of a generative model. However, most LSBO methods assume a fixed objective, whereas real design campaigns often face temporal drift (e.g., evolving preferences or shifting targets). Bringing time-varying BO into LSBO is nontrivial: drift can affect not only the surrogate, but also the latent search space geometry induced by the representation. We propose Time-Aware Latent-space Bayesian Optimization (TALBO), which incorporates time in both the surrogate and the learned generative representation via a GP-prior variational autoencoder, yielding a latent space aligned as objectives evolve. To evaluate timevarying LSBO systematically, we adapt widely used molecular design tasks to drifting multi-property objectives and introduce metrics tailored to changing targets. Across these benchmarks, TALBO consistently outperforms strong LSBO baselines and remains robust across drift speeds and design choices, while remaining competitive under actually time-invariant objectives.

Figures & tables

Figure 1: Overview of our setting. At each iteration

t

, a black-box objective

f_{t}

is defined as a weighted combination of two molecular properties, with a predefined time-varying weight schedule (details in Section 6.1 ). Left: The best score under the current objective

f_{t}

among all molecules evaluated up to iteration

t

, (mean

\pm

std over 10 seeds with different initial datasets). Top-right: mean property components. Bottom-right: objective weights (identical across seeds). TALBO , our time-aware method adapts more rapidly to changes in objective importance, resulting in consistently higher attained values

\tilde{f}_{t}

Task	Scoring functions
Median molecules 1	$\mathrm{sim}(\text{camphor},\mathrm{ECFC4})$ $\mathrm{sim}(\text{menthol},\mathrm{ECFC4})$
Median molecules 2	$\mathrm{sim}(\text{tadalafil},\mathrm{ECFC6})$ $\mathrm{sim}(\text{sildenafil},\mathrm{ECFC6})$
Composite Similarity 1	$\mathrm{sim}(\text{Sitagliptin},\mathrm{ECFC4})$ $\mathrm{sim}(\text{Ranolazine},\mathrm{AP})$ $\mathrm{sim}(\text{Osimertinib},\mathrm{FCFC4})$
Composite Similarity 2	$\mathrm{sim}(\text{Celecoxib},\mathrm{ECFC4})$ $\mathrm{sim}(\text{Troglitazone},\mathrm{ECFC4})$ $\mathrm{sim}(\text{Thiothixene},\mathrm{ECFC4})$

Table 1: Tasks and their associated scoring functions.

Baseline	$\ell_{w}=0.1$	$\ell_{w}=0.2$	$\ell_{w}=0.3$
TALBO	1.98 $\pm$ 1.23	1.75 $\pm$ 1.08	1.90 $\pm$ 1.15
InvBO	3.77 $\pm$ 1.39	3.90 $\pm$ 1.24	3.52 $\pm$ 1.38
CoBO	2.48 $\pm$ 1.04	2.45 $\pm$ 1.30	2.30 $\pm$ 1.16
LOL-BO	2.42 $\pm$ 1.01	2.73 $\pm$ 1.11	2.92 $\pm$ 1.10
LS-BO	6.41 $\pm$ 0.65	6.62 $\pm$ 0.60	6.59 $\pm$ 0.65
TurBO	4.76 $\pm$ 1.06	4.76 $\pm$ 1.19	4.80 $\pm$ 1.07

Table 2: Ablation study. Final baseline rank (mean

\pm\tfrac{1}{2}

std over all tasks and 10 random seeds) for different GP lengthscales

\ell_{w}\in\{0.1,0.2,0.3\}

(Equation 22 ). Ranks are computed per task and seed from cumulative regret (lower is better) and then averaged across tasks and seeds. Bold entries denote the best (lowest) average rank for each

\ell_{w}

, and underlined entries the second-best. TALBO achieves the best average rank across all tested lengthscales, demonstrating robustness to varying rates of objective drift.

Figure 2: Performance comparison across dynamic multi-objective tasks. First row: current-objective best-so-far value. Second row: cumulative regret, expressed as percentage improvement relative to InvBO . Third row: baseline rank at each iteration based on cumulative regret (lower is better). Fourth row: time-varying objective weights governing the dynamic multi-objective trade-off, using $\ell_{w}=0.2$ for $k_{w}$ (Equation 22 ). Mean $\pm\tfrac{1}{2}$ std over 10 random seeds. All trajectories are smoothed using a moving average for improved readability. Across all tasks, TALBO consistently outperforms other baselines, attaining higher best-found values, lower regret, and better average rank.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure S1: Performance comparison with pre-training dataset baselines. Current-objective best-so-far value for each dynamic multi-objective task, augmented with statistics derived from the pre-training dataset. Colored curves correspond to optimization baselines (mean

\pm\tfrac{1}{2}

std over 10 random seeds). The solid black curve denotes the best achievable value within the pre-training dataset at each time step, i.e.,

\max_{j}f_{t}(x_{j})

evaluated over all molecules in the dataset. Dashed black curves indicate top-percentile subsets of the dataset (5%, 1%, 0.1%, and 0.01% best), providing reference performance levels attainable without adaptive optimization. All trajectories are smoothed using a centered moving average for improved readability. The comparison highlights both how challenging it is to outperform strong pre-training set reference levels with only a few thousand queries, and the extent to which dynamic BO methods can nonetheless surpass best-in-dataset metrics.

Figure S2: Per-component dataset optima under drifting priorities. For each task, we evaluate each objective component on the full GuacaMol pre-training corpus (1.27M molecules) using the same time-dependent weighting schedule as in the benchmark, and plot the best attainable component value at each iteration. The resulting trajectories exhibit clear regime changes and transitions across components, confirming that the weight schedule induces meaningful time variation in the underlying objective landscape.

Figure S3: Ablation on the dual temporal modeling mechanism. For each task and iteration, baselines are ranked within each seed using cumulative regret, and then averaged across 10 random seeds. Curves show the mean

\pm\tfrac{1}{2}

standard deviation over seeds. The setting is similar as Figure 2 :

\ell_{w}=0.2

. Explicitly modeling time leads to overall performance gains, though the relative contribution of temporal modeling in the generative model versus the surrogate varies across tasks.

Panel of Figure 8 — Figure S4: Performance comparison across dynamic multi-objective tasks for different GP lengthscales $\ell_{w}={0.1}$ (top figure) and $\ell_{w}=0.3$ (bottom figure). First row: current-objective best-so-far value. Second row: cumulative regret, expressed as percentage improvement relative to InvBO. Third row: baseline rank at each iteration based on cumulative regret (lower is better). Fourth row: time-varying objective weights governing the dynamic multi-objective trade-off. Mean $\pm\tfrac{1}{2}$ std over 10 random seeds, All trajectories are smoothed using a centered moving average for improved readability.

Figure S5: Time-invariant objective. Best-found value as a function of oracle calls for three representative tasks when the objective remains fixed over time. Mean

\pm\tfrac{1}{2}

std over 10 random seeds. In the absence of temporal drift, differences between methods are task-dependent, and time-awareness does not provide a systematic advantage, as expected. All trajectories are smoothed using a centered moving average for readability. TALBO remains competitive without exhibiting systematic degradation in the static setting.

Figure S6: Performance comparison across dynamic multi-objective tasks (instantaneous evaluation). First row: instantaneous objective value $f_{t}(\mathbf{x}_{t})$ evaluated at each iteration. Second row: cumulative regret, expressed as percentage improvement relative to InvBO . Third row: baseline rank at each iteration based on cumulative regret (lower is better). Mean $\pm\tfrac{1}{2}$ std over 10 random seeds. All trajectories are smoothed using a moving average for improved readability. TALBO consistently achieves competitive instantaneous performance and favorable regret-based ranking across tasks.

Optimization setup
Optimization horizon $T$	600
Initial dataset size $N_{\text{init}}$	100
Batch size $B$	10
Number of seeds	10
Time-varying objective generation
Temporal kernel (Equation 22 )	Squared-exponential $k_{\text{w}}$

Appendix

Table S1: Main optimization, objective-generation, and modeling settings used in the experiments.

DGBFGP
Latent dimension	$d=256$
Max token length	$L_{\max}=128$
Dataset	GuacaMol
Basis / features	Hilbert-space features ( basis_func=hs ), basis size $M=128$
GP KL scale	gpB =0.001 ( gp_kl_scale_B )
Hyperparameters	batch size $1024$ , learning rate $5\times 10^{-4}$

Appendix

Table S2: DGBFGP configuration and training hyperparameters used in the experiments.

Index	Covariate name
1	molecular weight
2	number of H-bond donors
3	number of H-bond acceptors
4	number of rotatable bonds
5	Quantitative Estimation of Drug-likeness (QED)
6	Synthetic Accessibility Score (SAS)

Appendix

Table S3: Covariates used by TALBO for Multi-property optimization tasks.

Figure S7: Histograms of molecular covariates used during pretraining (Table S3 ).

InfoTransformerVAE
Latent dimension	$d=256$ ( bottleneck_size =2, d_model =128)
Max decode length	$L_{\max}=128$
KL weight	$\beta_{\mathrm{KL}}=0.1$ ( kl_factor )

Appendix

Table S4: InfoTransformerVAE architecture and training hyperparameters used in the experiments

Molecule Design

See all 43 papers

In-Context Learning for Latent Space Bayesian Optimization
Jun 8, 2026Tuan A. Vu, Harri Lähdesmäki, Julien MartinelliSynthetic Data Pretraining Molecular Optimization
Aalto University
Search at the Cost of Sampling: Nearly-Instant Latent Space Bayesian Optimization
Sep 16, 2026Donney Fan, Colin Doumont, Aleksandra Kalisz +4Surrogate Modeling Bayesian Optimization
University of British Columbia · Vector Institute · Tübingen AI Center +3
Work While They Sleep: Exploiting Evaluation Latency for Fully Bayesian Optimization
Oct 6, 2026Gustavo Sutter, Alejandro Comas-Leon, David Holzmüller +4Bayesian Optimization Surrogate-Assisted Optimization
Cheriton School of Computer Science, University of Waterloo, Waterloo, ON, Canada · Vector Institute, Toronto, ON, Canada · SODA Team, INRIA Saclay, Palaiseau, France +3

Jun 8, 2026J/K move · Enter open · S save

In-Context Learning for Latent Space Bayesian Optimization

Tuan A. Vu, Harri Lähdesmäki, Julien Martinelli

Aalto University

Synthetic Data Pretraining Molecular Optimization Surrogate Modeling Bayesian Optimization Latent Space Optimization Tabular ICL

Open paper AskPDF arXiv

Bayesian optimization (BO) is a central tool for sample-efficient design, and latent-space Bayesian optimization (LSBO) extends it to structured objects such as molecules and proteins. In parallel, tabular foundation models such as TabPFN and TabICL now achieve state-of-the-art regression performance and are increasingly used as BO surrogates. Because their Bayesian behavior is induced by large synthetic pretraining collections, the composition of this pretraining distribution is crucial. LSBO creates a distinctive mismatch: the induced map from latent code to objective value differs markedly from the regression tasks used to train current in-context models. We address this mismatch by complementing the pretraining stage of tabular foundation model surrogates with synthetic optimization tasks defined on the latent space of a molecular VAE. The continued-pretraining objective features a regularizer that anchors the model to the original checkpoint, preserving its broad regression prior while avoiding overspecialization to the adaptation tasks. On held-out molecular optimization benchmarks, the resulting model achieves strong performance, supporting the relevance of LSBO-specific adaptation for in-context surrogates.

Abstract

Molecule Design

Explore similar work

In-Context Learning for Latent Space Bayesian Optimization

Search at the Cost of Sampling: Nearly-Instant Latent Space Bayesian Optimization

Work While They Sleep: Exploiting Evaluation Latency for Fully Bayesian Optimization