Enhancing Offline Model-Based RL via Active Model Selection: A Bayesian Optimization Perspective

Yu-Wei Yang; Wei Hung; Yun-Ming Chan; Xi Liu; Ping-Chun Hsieh

cs.LGFeb 17, 2025

Enhancing Offline Model-Based RL via Active Model Selection: A Bayesian Optimization Perspective

Authors: Yu-Wei Yang, Wei Hung, Yun-Ming Chan, Xi Liu, Ping-Chun Hsieh

Organizations: Department of Computer Science, National Yang Ming Chiao Tung University, Hsinchu, Taiwan · Applied Machine Learning, Meta AI, Menlo Park, CA, USA

Abstract

Offline model-based reinforcement learning (MBRL) serves as a competitive framework that can learn well-performing policies solely from pre-collected data with the help of learned dynamics models. To fully unleash the power of offline MBRL, model selection plays a pivotal role in determining the dynamics model utilized for downstream policy learning. However, offline MBRL conventionally relies on validation or off-policy evaluation, which are rather inaccurate due to the inherent distribution shift in offline RL. To tackle this, we propose BOMS, an active model selection framework that enhances model selection in offline MBRL with only a small online interaction budget, through the lens of Bayesian optimization (BO). Specifically, we recast model selection as BO and enable probabilistic inference in BOMS by proposing a novel model-induced kernel, which is theoretically grounded and computationally efficient. Through extensive experiments, we show that BOMS improves over the baseline methods with a small amount of online interaction comparable to only $1\%$ - $2.5\%$ of offline training data on various RL tasks.

Figures & tables

	hopper (medium)	walker2d (medium)	halfcheetah (medium)
Best model	2073 $\pm 822$	4002 $\pm 238$	6607 $\pm 295$
Validation	821 $\pm 440$	39 $\pm 61$	5978 $\pm 140$
OPE	495 $\pm 265$	3618 $\pm 241$	4355 $\pm 581$

Table 1: Total rewards achieved by MOPO ( Yu et al., 2020b ) under validation-based and OPE-based model selection on D4RL. The performance of the best model is also reported to highlight the sub-optimality of the existing schemes.

Figure 1: An illustration of the BOMS framework.

Update the posterior

\mathcal{N}(\mu_{t}(M),\sigma^{2}_{t}(M))

of each dynamics model

M\in\cM

Algorithm 1 BOMS

Tasks		BOMS			MOPO	OPE
Tasks		$T=5$	$T=10$	$T=20$	MOPO	OPE
	med	2.6 $\pm$ 0.7	2.2 $\pm$ 0.2	1.9 $\pm$ 0.0	99.9	7.5
walker2d	med-r	69.9 $\pm$ 17.7	44.0 $\pm$ 26.7	16.9 $\pm$ 17.5	77.1	94.3
	med-e	91.8 $\pm$ 4.8	86.2 $\pm$ 13.0	74.3 $\pm$ 24.9	99.9	19.4
	med	27.0 $\pm$ 11.3	20.6 $\pm$ 2.6	20.4 $\pm$ 2.8 0	66.0	78.4
hopper	med-r	5.5 $\pm$ 1.4	5.1 $\pm$ 2.6	5.2 $\pm$ 2.6	3.1	59.3

Table 2: Normalized regrets of BOMS and the baselines. The best performance of each row is highlighted in bold. We use ”med”, ”med-r”, ”mid-e” as the shorthands for medium, medium-replay, and medium-expert.

Door Open	BOMS			MOPO	OPE
Door Open	$T=5$	$T=10$	$T=20$	MOPO	OPE
Success Rate	1.3 $\pm$ 4.0	6.5 $\pm$ 15.7	16.3 $\pm$ 23.5	0.0	0.0
Regrets	57.7 $\pm$ 2.6	51.5 $\pm$ 17.3	39.6 $\pm$ 25.9	66.7	86.4

Table 3: Success rates and normalized regrets of BOMS and the baselines on the Meta-World door-open task.

Figure 2: Comparison of BOMS and the baselines in inference regret. BOMS achieves lower regrets than Validation and OPE in all the tasks after 5 iterations, which correspond to only $1\%$ - $2.5\%$ of the offline training data. The x-axis denotes the number of BO selection iterations $t$ in Algorithm 1 .

Figure 3: Comparison of various designs of model distance for BOMS in inference regret. These results corroborate the proposed model-induced kernel. The x-axis denotes the number of BO selection iterations $t$ in Algorithm 1 .

Figure 4: The inference regrets of BOMS with distance measure calculated under different rollout lengths. The x-axis denotes the number of BO selection iterations $t$ in Algorithm 1 .

Tasks		BOMS-Augmented RAMBO			RAMBO
Tasks		$T=5$	$T=10$	$T=20$	RAMBO
walker2d	med	79.3 $\pm$ 10.9	70.2 $\pm$ 24.7	37.7 $\pm$ 38.1	73.6
	med-r	93.0 $\pm$ 2.6	93.0 $\pm$ 2.5	92.7 $\pm$ 2.5	99.3
	med-e	53.3 $\pm$ 10.6	43.0 $\pm$ 15.6	38.9 $\pm$ 17.2	65.4

Table 4: Normalized regrets under vanilla RAMBO and BOMS-augmented RAMBO.

Tasks		$\alpha$	BOMS			MOPO	OPE
Tasks		$\alpha$	$T=5$	$T=10$	$T=20$	MOPO	OPE
	med	10.0	36.7 $\pm$ 39.7	12.5 $\pm$ 29.2	2.0 $\pm$ 1.5	100.0	7.5
	med	1.0	2.6 $\pm$ 0.7	2.2 $\pm$ 0.2	1.9 $\pm$ 0.0	100.0	7.5
	med	0.1	61.6 $\pm$ 47.0	17.9 $\pm$ 25.3	8.6 $\pm$ 7.8	100.0	7.5
	med-r	10.0	62.3 $\pm$ 28.6	53.5 $\pm$ 23.3	38.4 $\pm$ 21.8	77.1	94.3
walker2d	med-r	1.0	69.9 $\pm$ 17.7	44.0 $\pm$ 26.7	17.0 $\pm$ 17.5	77.1	94.3

Table 5: The normalized regrets of BOMS with different

\alpha

and the baselines.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Comparison of BOMS and the baselines in inference regret. BOMS achieves lower regrets than Validation and OPE in almost all tasks after 5-10 iterations.

Panel of Figure 6 — Figure 6: Comparison of various designs of model distance for BOMS in inference regret. These results corroborate the proposed model-induced kernel.

Model Selection

Model selection chooses models, features, or configurations that balance predictive fit, generalization, complexity, and deployment constraints.

See all 96 papers

Fully Offline Reinforcement Learning
May 28, 2025Mattie Fellows, Clarisse Wibault, Uljad Berdica +3Bayesian RL Offline RL
Foerster Lab for AI Research (FLAIR), Department of Engineering Science, University of Oxford · Machine Learning Research Group, Department of Engineering Science, University of Oxford
Model-based Offline RL via Robust Value-Aware Model Learning with Implicitly Differentiable Adaptive Weighting
Mar 9, 2026Zhongjian Qiao, Jiafei Lyu, Boxiang Lyu +3Offline RL Model-Based RL
CityUHK · Tencent · UChicago +1
Active Offline-to-Online Reinforcement Learning
Jul 13, 2026Alper Kamil Bozkurt, Shangtong Zhang, Yuichi MotaiOffline RL RL Fine-Tuning
Virginia Commonwealth University, USA · University of Virginia, USA

May 28, 2025J/K move · Enter open · S save

Fully Offline Reinforcement Learning

Mattie Fellows, Clarisse Wibault, Uljad Berdica, Johannes Forkel+2

Foerster Lab for AI Research (FLAIR), Department of Engineering Science, University of Oxford · Machine Learning Research Group, Department of Engineering Science, University of Oxford

Bayesian RL Offline RL Policy Learning Hyperparameter Optimization Model-Based RL

Open paper AskPDF arXiv

Offline RL (ORL) promises safe and sample-efficient deployment but existing methods rely on undocumented online interactions for hyperparameter tuning and lack reliable fully offline estimates of initial online performance. We introduce SOReL, a fully offline Bayesian model-based RL method that learns a posterior over dynamics, estimates policy value via predictive uncertainty, and enables complete offline hyperparameter selection. We further propose TOReL, which extends this tuning framework to arbitrary model-free and model-based ORL algorithms. We provide a regret analysis showing that Bayesian offline RL achieves the minimax-optimal parametric rate under standard regularity conditions. Together, our methods establish a practical and theoretically grounded framework for fully offline RL.

Abstract

Model Selection

Explore similar work

Fully Offline Reinforcement Learning

Model-based Offline RL via Robust Value-Aware Model Learning with Implicitly Differentiable Adaptive Weighting

Active Offline-to-Online Reinforcement Learning