Bayesian Optimization

Offline model-based reinforcement learning (MBRL) serves as a competitive framework that can learn well-performing policies solely from pre-collected data with the help of learned dynamics models. To fully unleash the power of offline MBRL, model selection plays a pivotal role in determining the dynamics model utilized for downstream policy learning. However, offline MBRL conventionally relies on validation or off-policy evaluation, which are rather inaccurate due to the inherent distribution shift in offline RL. To tackle this, we propose BOMS, an active model selection framework that enhances model selection in offline MBRL with only a small online interaction budget, through the lens of Bayesian optimization (BO). Specifically, we recast model selection as BO and enable probabilistic inference in BOMS by proposing a novel model-induced kernel, which is theoretically grounded and computationally efficient. Through extensive experiments, we show that BOMS improves over the baseline methods with a small amount of online interaction comparable to only $1\%$-$2.5\%$ of offline training data on various RL tasks.

Time-Varying Bayesian Optimization (TVBO) is the go-to framework for optimizing a time-varying, expensive, noisy black-box function $f$. However, most of the asymptotic guarantees offered by TVBO algorithms rely on the assumption that observations are acquired at a constant frequency. As the GP inference complexity scales with the cube of its dataset size, this assumption is unrealistic in the long run. In this paper, we relax this assumption and derive the first upper regret bound that explicitly accounts for changes in the observations sampling frequency. Based on this analysis, we formulate practical recommendations about dataset sizes and stale data policies of TVBO algorithms. We illustrate how an algorithm (BOLT) that follows these recommendations performs better than the state-of-the-art of TVBO through experiments on synthetic and real-world problems.

Complex dynamical systems, such as particle accelerators, require tuning over high-dimensional, nonlinear state and parameter spaces while experimental measurements and high-fidelity simulations can be expensive. Learned latent representations provide a compact domain for such optimization, but poorly supported regions of the learned manifold may decode into unrealistic physical states and yield deceptively favorable objectives. To address this challenge, we propose the Classifier-pruned Bayesian Optimization-based Latent-space Tuner (CBOL-Tuner), which performs Bayesian optimization over a temporally structured latent representation of 6D beam phase-space dynamics. The CBOL-Tuner integrates a conditional variational autoencoder for latent space representation, a long short-term memory network for temporal dynamics, a lightweight neural network for parameter estimation, and a classifier-pruned Bayesian optimizer to adaptively search and filter the latent space for optimal solutions. This framework enables feasibility-aware exploration of learned scientific representations for accelerator tuning.