cs.LGNov 17, 2026

Towards Optimism-Pessimism Trade-off in Model-based Offline-to-Online Reinforcement Learning

Authors: Guochen Zhou, Yijun Yang, Qiqi Duan, Qing Su, Weiming Ou, Li Shen, Shihao Ji, Yuhui Shi, 

Organizations: Department of Computer Science, City University of Hong Kong · Tsinghua University · Jiangxi University of Finance and Economics · University of Connecticut · 

Abstract

Model-based offline-to-online reinforcement learning (RL) enables sample-efficient adaptation by leveraging offline pre-training for online fine-tuning. However, the distribution shifts between offline and online stages often hinder fine-tuning performance. Many existing methods encode the optimism-pessimism trade-off through a single-objective formulation. Selecting its trade-off coefficient through online fine-tuning can require costly bi-level optimization. We identify the balance established during offline training as a key challenge: in our motivating experiments, optimistic policies adapt better to novel tasks, whereas pessimistic policies perform better on the same online tasks. To address this challenge, we propose a bi-objective formulation that captures this trade-off, yielding a Pareto policy pool during offline training. This policy pool enables adaptive selection during online fine-tuning across diverse online tasks. To generate the pool, we introduce Multi-Objective Soft Actor-critIC (MOSAIC), which solves bi-objective problems and constructs diverse Pareto policies. After offline training, a contextual bandit algorithm hierarchically selects the most suitable policy for fine-tuning at each online interaction step. Empirically, our pipeline, Hierarchical Pareto Policy Pool (HiP3), achieves the strongest aggregate performance across both same and novel online tasks and maintains its advantage under stronger distribution shifts. Comprehensive ablations further clarify the roles of the policy pool, MOSAIC-based pool construction, and the online selection mechanism.

Figures & tables

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Active Offline-to-Online Reinforcement Learning

    Jul 13, 2026Alper Kamil Bozkurt, Shangtong Zhang, Yuichi MotaiOffline RLRL Fine-Tuning

  2. The Three Regimes of Offline-to-Online Reinforcement Learning

    Oct 1, 2025Lu Li, Tianwei Ni, Yihao Sun +1Offline RLRL Fine-Tuning

  3. Policy Plasticity Matters in Offline-to-Online Reinforcement Learning: Refitting Offline Policies for Online Adaptation

    Sep 27, 2026Yuheng Huang, Yunpeng Qing, Yixiao Chi +2Offline-To-Online RL