Organizations: Department of Computer Science, National Yang Ming Chiao Tung University, Hsinchu, Taiwan · Applied Machine Learning, Meta AI, Menlo Park, CA, USA
Offline model-based reinforcement learning (MBRL) serves as a competitive framework that can learn well-performing policies solely from pre-collected data with the help of learned dynamics models. To fully unleash the power of offline MBRL, model selection plays a pivotal role in determining the dynamics model utilized for downstream policy learning. However, offline MBRL conventionally relies on validation or off-policy evaluation, which are rather inaccurate due to the inherent distribution shift in offline RL. To tackle this, we propose BOMS, an active model selection framework that enhances model selection in offline MBRL with only a small online interaction budget, through the lens of Bayesian optimization (BO). Specifically, we recast model selection as BO and enable probabilistic inference in BOMS by proposing a novel model-induced kernel, which is theoretically grounded and computationally efficient. Through extensive experiments, we show that BOMS improves over the baseline methods with a small amount of online interaction comparable to only 1%-2.5% of offline training data on various RL tasks.
Figures & tables
hopper (medium)
walker2d (medium)
halfcheetah (medium)
Best model
2073 ±822
4002 ±238
6607 ±295
Validation
821 ±440
39 ±61
5978 ±140
OPE
495 ±265
3618 ±241
4355 ±581
Table 1: Total rewards achieved by MOPO ( Yu et al., 2020b ) under validation-based and OPE-based model selection on D4RL. The performance of the best model is also reported to highlight the sub-optimality of the existing schemes.
Figure 1: An illustration of the BOMS framework.
Update the posterior N(μt(M),σt2(M)) of each dynamics model M∈\cM .
Algorithm 1 BOMS
Tasks
BOMS
MOPO
OPE
T=5
T=10
T=20
med
2.6 ± 0.7
2.2 ± 0.2
1.9 ± 0.0
99.9
7.5
walker2d
med-r
69.9 ± 17.7
44.0 ± 26.7
16.9 ± 17.5
77.1
94.3
med-e
91.8 ± 4.8
86.2 ± 13.0
74.3 ± 24.9
99.9
19.4
med
27.0 ± 11.3
20.6 ± 2.6
20.4 ± 2.8 0
66.0
78.4
hopper
med-r
5.5 ± 1.4
5.1 ± 2.6
5.2 ± 2.6
3.1
59.3
Table 2: Normalized regrets of BOMS and the baselines. The best performance of each row is highlighted in bold. We use ”med”, ”med-r”, ”mid-e” as the shorthands for medium, medium-replay, and medium-expert.
Door Open
BOMS
MOPO
OPE
T=5
T=10
T=20
Success Rate
1.3 ± 4.0
6.5 ± 15.7
16.3 ± 23.5
0.0
0.0
Regrets
57.7 ± 2.6
51.5 ± 17.3
39.6 ± 25.9
66.7
86.4
Table 3: Success rates and normalized regrets of BOMS and the baselines on the Meta-World door-open task.
Figure 2: Comparison of BOMS and the baselines in inference regret. BOMS achieves lower regrets than Validation and OPE in all the tasks after 5 iterations, which correspond to only 1% - 2.5% of the offline training data. The x-axis denotes the number of BO selection iterations t in Algorithm 1 .
Figure 3: Comparison of various designs of model distance for BOMS in inference regret. These results corroborate the proposed model-induced kernel. The x-axis denotes the number of BO selection iterations t in Algorithm 1 .
Figure 4: The inference regrets of BOMS with distance measure calculated under different rollout lengths. The x-axis denotes the number of BO selection iterations t in Algorithm 1 .
Tasks
BOMS-Augmented RAMBO
RAMBO
T=5
T=10
T=20
walker2d
med
79.3 ± 10.9
70.2 ± 24.7
37.7 ± 38.1
73.6
med-r
93.0 ± 2.6
93.0 ± 2.5
92.7 ± 2.5
99.3
med-e
53.3 ± 10.6
43.0 ± 15.6
38.9 ± 17.2
65.4
Table 4: Normalized regrets under vanilla RAMBO and BOMS-augmented RAMBO.
Tasks
α
BOMS
MOPO
OPE
T=5
T=10
T=20
med
10.0
36.7 ± 39.7
12.5 ± 29.2
2.0 ± 1.5
100.0
7.5
med
1.0
2.6 ± 0.7
2.2 ± 0.2
1.9 ± 0.0
100.0
7.5
med
0.1
61.6 ± 47.0
17.9 ± 25.3
8.6 ± 7.8
100.0
7.5
med-r
10.0
62.3 ± 28.6
53.5 ± 23.3
38.4 ± 21.8
77.1
94.3
walker2d
med-r
1.0
69.9 ± 17.7
44.0 ± 26.7
17.0 ± 17.5
77.1
94.3
Table 5: The normalized regrets of BOMS with different α and the baselines.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Comparison of BOMS and the baselines in inference regret. BOMS achieves lower regrets than Validation and OPE in almost all tasks after 5-10 iterations.
Figure 6: Comparison of various designs of model distance for BOMS in inference regret. These results corroborate the proposed model-induced kernel.
Offline RL (ORL) promises safe and sample-efficient deployment but existing methods rely on undocumented online interactions for hyperparameter tuning and lack reliable fully offline estimates of initial online performance. We introduce SOReL, a fully offline Bayesian model-based RL method that learns a posterior over dynamics, estimates policy value via predictive uncertainty, and enables complete offline hyperparameter selection. We further propose TOReL, which extends this tuning framework to arbitrary model-free and model-based ORL algorithms. We provide a regret analysis showing that Bayesian offline RL achieves the minimax-optimal parametric rate under standard regularity conditions. Together, our methods establish a practical and theoretically grounded framework for fully offline RL.
Foerster Lab for AI Research (FLAIR), Department of Engineering Science, University of Oxford · Machine Learning Research Group, Department of Engineering Science, University of Oxford
Model-based offline reinforcement learning (RL) aims to enhance offline RL with a dynamics model that facilitates policy exploration. However, model exploitation could occur due to inevitable model errors, degrading algorithm performance. Adversarial model learning offers a theoretical framework to mitigate model exploitation by solving a maximin formulation. Within such a paradigm, RAMBO~\citep{rigter2022rambo} has emerged as a representative and most popular method that provides a practical implementation with model gradient. However, we empirically reveal that severe Q-value underestimation and gradient explosion can occur in RAMBO with only slight hyperparameter tuning, suggesting that it tends to be overly conservative and suffers from unstable model updates. To address these issues, we propose RObust value-aware Model learning with Implicitly differentiable adaptive weighting (ROMI). Instead of updating the dynamics model with model gradient, ROMI introduces a novel robust value-aware model learning approach. This approach requires the dynamics model to predict future states with values close to the minimum Q-value within a scale-adjustable state uncertainty set, enabling controllable conservatism and stable model updates. To further improve out-of-distribution (OOD) generalization during multi-step rollouts, we propose implicitly differentiable adaptive weighting, a bi-level optimization scheme that adaptively achieves dynamics- and value-aware model learning. Empirical results on D4RL and NeoRL datasets show that ROMI significantly outperforms RAMBO and achieves competitive or superior performance compared to other state-of-the-art methods on datasets where RAMBO typically underperforms. Code is available at https://github.com/zq2r/ROMI.git.
Background: Offline reinforcement learning (RL) enables effective policies to be trained from large, previously collected datasets and subsequently improved through limited online interaction. This offline-to-online RL (O2O-RL) paradigm is particularly promising in nonstationary domains where interaction is costly or potentially hazardous. Standard O2O-RL pipelines train multiple candidate policies offline, evaluate them using off-policy or online evaluation, and then deploy and fine-tune the policy with the highest estimated value. However, as in offline pretraining, fine-tuning performance is highly sensitive to the choice of algorithm and hyperparameters, making it risky to commit to a single policy. Objectives: We study active policy selection for fine-tuning under a limited interaction budget in O2O-RL settings. To our knowledge, this is the first work to address this problem. Methods: We formulate the problem by identifying a fundamental trade-off between allocating online interactions to policy evaluation, which helps identify high-performing policies, and allocating them to fine-tuning, which improves policy performance. We then propose an approach that balances this trade-off by actively selecting policies for fine-tuning based on upper-confidence bounds on their future performance. These bounds are derived from locally linear performance forecasts fitted to observations obtained through online evaluation. Results: Across a diverse range of experiments, the proposed approach consistently outperforms existing O2O-RL baselines. Conclusions: Actively selecting and fine-tuning policies uses limited online interaction budgets more effectively than either committing to a single policy or dividing the budget equally among all policies. Our framework also advances offline RL toward practical deployment in real-world systems where online interaction is costly or risky.