Fully Offline Reinforcement Learning

Mattie Fellows; Clarisse Wibault; Uljad Berdica; Johannes Forkel; Maike Osborne; Jakob N. Foerster

cs.LGMay 28, 2025

Fully Offline Reinforcement Learning

Authors: Mattie Fellows, Clarisse Wibault, Uljad Berdica, Johannes Forkel, Maike Osborne, Jakob N. Foerster

Organizations: Foerster Lab for AI Research (FLAIR), Department of Engineering Science, University of Oxford · Machine Learning Research Group, Department of Engineering Science, University of Oxford

Abstract

Offline RL (ORL) promises safe and sample-efficient deployment but existing methods rely on undocumented online interactions for hyperparameter tuning and lack reliable fully offline estimates of initial online performance. We introduce SOReL, a fully offline Bayesian model-based RL method that learns a posterior over dynamics, estimates policy value via predictive uncertainty, and enables complete offline hyperparameter selection. We further propose TOReL, which extends this tuning framework to arbitrary model-free and model-based ORL algorithms. We provide a regret analysis showing that Bayesian offline RL achieves the minimax-optimal parametric rate under standard regularity conditions. Together, our methods establish a practical and theoretically grounded framework for fully offline RL.

Explore similar work

Regularized Offline Policy Optimization with Posterior Hybrid Bayesian Belief
May 30, 2026Hongqiang Lin, Pengfei Wang, Nenggan ZhengOffline Reinforcement Learning Posterior
Active Offline-to-Online Reinforcement Learning
Jul 13, 2026Alper Kamil Bozkurt, Shangtong Zhang, Yuichi MotaiOffline Reinforcement Learning Limited Data
The Three Regimes of Offline-to-Online Reinforcement Learning
Oct 1, 2025Lu Li, Tianwei Ni, Yihao Sun +1Offline Reinforcement Learning Reinforcement Fine-Tuning

May 30, 2026 · cs.AIJ/K move · Enter open · S save

Regularized Offline Policy Optimization with Posterior Hybrid Bayesian Belief

Hongqiang Lin, Pengfei Wang, Nenggan Zheng

Zhejiang University, Hangzhou, China.

Offline reinforcement learning (RL) aims to optimize policies from pre-collected datasets. A bottleneck of this paradigm is managing epistemic uncertainty, which arises from limited data coverage (sample-level) and the ambiguity in identifying transition dynamics from finite data (model-level). To provide a unified quantification of these uncertainties, Bayesian RL has been proposed by treating the dynamics model as a random variable and maintaining a corresponding belief. Despite its theoretical appeal, policy optimization in Bayesian RL remains computationally challenging as it requires solving composite objectives with expectations. Prior methods either employ search-based techniques with poor computational scalability or impose restrictive posterior assumptions that sacrifice the adaptability of Bayesian RL. To address these limitations, we propose Posterior Hybrid Bayesian Belief (PhyB), which reformulates the expectation as a convex combination over a subset of dynamics models. Theoretical analysis demonstrates that the objective discrepancy induced by this approximation remains bounded. Based on PhyB, we develop an iterative regularized policy optimization algorithm that provides metric-agnostic guarantees for monotonic improvement until convergence. Empirical results demonstrate that PhyB achieves state-of-the-art performance on various benchmarks.

Offline Reinforcement Learning Posterior

Open paper Ask about it PDF

Abstract

Explore similar work

Regularized Offline Policy Optimization with Posterior Hybrid Bayesian Belief

Active Offline-to-Online Reinforcement Learning

The Three Regimes of Offline-to-Online Reinforcement Learning