Towards Optimism-Pessimism Trade-off in Model-based Offline-to-Online Reinforcement Learning
Authors: Guochen Zhou, Yijun Yang, Qiqi Duan, Qing Su, Weiming Ou, Li Shen, Shihao Ji, Yuhui Shi, Zhisong Zhang
Organizations: Department of Computer Science, City University of Hong Kong · Tsinghua University · Jiangxi University of Finance and Economics · University of Connecticut · Shanghai University of Finance and Economics · Sun Yat-sen University · Southern University of Science and Technology
Model-based offline-to-online reinforcement learning (RL) enables sample-efficient adaptation by leveraging offline pre-training for online fine-tuning. However, the distribution shifts between offline and online stages often hinder fine-tuning performance. Many existing methods encode the optimism-pessimism trade-off through a single-objective formulation. Selecting its trade-off coefficient through online fine-tuning can require costly bi-level optimization. We identify the balance established during offline training as a key challenge: in our motivating experiments, optimistic policies adapt better to novel tasks, whereas pessimistic policies perform better on the same online tasks. To address this challenge, we propose a bi-objective formulation that captures this trade-off, yielding a Pareto policy pool during offline training. This policy pool enables adaptive selection during online fine-tuning across diverse online tasks. To generate the pool, we introduce Multi-Objective Soft Actor-critIC (MOSAIC), which solves bi-objective problems and constructs diverse Pareto policies. After offline training, a contextual bandit algorithm hierarchically selects the most suitable policy for fine-tuning at each online interaction step. Empirically, our pipeline, Hierarchical Pareto Policy Pool (HiP3), achieves the strongest aggregate performance across both same and novel online tasks and maintains its advantage under stronger distribution shifts. Comprehensive ablations further clarify the roles of the policy pool, MOSAIC-based pool construction, and the online selection mechanism.
Figures & tables
On the Same Online Tasks
On Novel Online Tasks
AWAC
PEX
Cal-QL
MBPO
HiP3 (Ours)
AWAC
PEX
Cal-QL
MBPO
HiP3 (Ours)
Random
HalfCheetah
35.6
± 0.5
56.9
± 1.2
15.4
± 5.6
66.0
± 15.4
82.6
± 2.3
33.2
± 0.4
42.2
± 2.6
11.1
± 2.5
47.1
± 22.1
63.5
± 2.2
Hopper
11.7
± 0.2
22.2
± 4.1
12.7
± 8.5
98.2
± 6.1
105.7
± 4.1
9.2
± 0.1
23.5
± 17.2
11.7
± 4.7
86.4
± 4.6
95.2
± 5.5
Walker2d
12.2
± 6.7
9.8
± 2.2
1.6
± 1.3
51.4
± 35.9
35.7
± 26.0
7.0
± 1.3
8.5
± 0.7
-0.6
± 1.1
44.6
± 11.4
49.2
± 26.7
Medium
HalfCheetah
43.6
± 0.8
44.1
± 0.3
49.7
± 0.3
66.0
± 15.4
90.1
± 4.1
33.9
± 0.4
30.4
± 0.9
35.4
± 0.1
47.1
± 22.1
69.3
± 2.1
Hopper
101.5
± 0.4
99.4
± 1.2
83.3
± 12.3
98.2
± 6.1
106.7
± 3.0
88.5
± 0.1
74.8
± 15.9
37.6
± 4.6
86.4
± 4.6
95.4
± 3.8
Table 1: Performance comparison on the same and novel online tasks. Novel online tasks are created by modifying the environments’ gravity, friction, noise, and reward terms. Scores for each dataset are normalized according to the D4RL benchmark [ 9 ] : Normalized Score=100×expert return−random returnreturn−random return . Full results can be found in Table 21 in the Appendix.
Figure 3: Offline-to-online learning curves on the same and novel online tasks. The blue and yellow shaded regions denote offline pre-training and online fine-tuning, respectively. Curves are averaged over 5 random seeds with 10 evaluations per seed, and shaded areas show standard deviation. Full results across all datasets are available in Figure 8 and Figure 9 in the Appendix.
Method
HiP3
SO2
WSRL
FQL
IQL-EDIS
Cal-QL-EDIS
hc-r-v2
42.9±3.9
29.8±0.7
38.7±1.5
22.2±1.8
15.7±0.6
48.9±4.6
hc-mr-v2
55.8±4.6
35.9±0.5
52.4±0.5
44.1±0.3
27.2±0.3
43.0±2.1
hop-r-v2
65.4±6.3
14.0±11.3
9.5±0.6
12.9±0.4
6.0±0.1
8.7±1.5
hop-mr-v2
102.1±4.1
17.1±1.1
87.3±0.3
50.8±14.8
12.8±0.7
75.2±0.9
wk-r-v2
55.2±7.2
39.5±2.8
53.6±7.6
5.7±1.8
3.5±0.3
27.0±11.9
wk-mr-v2
74.9±5.2
45.3±10.6
75.9±5.9
29.3±12.0
54.5±0.8
67.6±10.1
Table 2: Performance under more challenging distribution shifts. We modify environment dynamics and reward functions to create harder adaptation scenarios. (Notations: hc = halfcheetah, hop = hopper, wk = walker2d; r = random, mr = medium-replay.)
Figure 4: Ablations on novel online tasks: (a) removing the policy pool; (b) policy-selection algorithms. Full curves are provided in Figure 10 and Figure 11 in the Appendix, respectively.
Table 3: Ablation and controlled comparison studies. (a) Comparison with a matched linear-scalarization baseline. (b) Comparison of candidate-set initialization strategies, including target-interaction-free candidate ranking via FQE.
Figure 5: Heat maps visualize the policy preferences Pt , obtained by softmax-normalizing the LinUCB scores. HiP3 executes kt=argmaxkPt(k) at each interaction step. A sharply peaked preference profile indicates a clear preference for one candidate, whereas a diffuse profile indicates smaller preference margins among candidates. The accompanying curves compare HiP3 with the five candidate policies on two representative novel online tasks. Additional quantitative and qualitative analyses are provided in Appendix D.5.2 .
Domain
Mean Dominant-policy Selection Frequency
Range Across Settings
HalfCheetah
37.6%
28.6%-62.0%
Hopper
35.1%
29.0%-44.4%
Walker2d
54.4%
46.1%-58.8%
Overall
42.3%
28.6%-62.0%
Table 4: Quantitative analysis of LinUCB policy selection behavior on 12 novel online task settings. The dominant policy is defined as the candidate selected most frequently over the full online run. Its selection frequency is the fraction of online steps assigned to that candidate.
Appendix figures & tables24 assets
Supplementary material from the paper’s appendix.
Appendix
Hyper-parameter
Value
Ensemble size
7
Number of elite models
5
Model architecture
FC(200, 200, 200, 200)
Batch size
256
Optimizer
Adam
Learning rate
1×10−3
Appendix
Table 5: The hyper-parameters used for environment model M
Method
HiP3
HiP3-LS
DiME+AWAC
AWAC
halfcheetah-random-v2
63.5±2.2
47.5±17.1
28.4±1.6
33.2±0.4
halfcheetah-medium-replay-v2
69.4±3.6
57.8±15.4
36.0±4.1
29.9±3.8
hopper-random-v2
95.2±5.5
62.3±21.9
7.0±4.9
9.2±0.1
hopper-medium-replay-v2
100.1±11.0
82.9±6.0
51.2±8.0
29.6±2.8
mean
82.1±5.5
62.6±15.1
30.7±4.7
25.5±1.8
Appendix
Table 6: Comparison with Linear Scalarization Method
Variant
Offline Time (h)
Online Time (h)
End-to-End Time (h)
Single-policy
5.8h
8.4h
14.2h
HiP3-LS
6.3h
8.4h
14.7h
HiP3
7.1h
8.5h
15.6h
Appendix
Table 7: End-to-end computational cost of matched HiP3 variants. All variants use the same total 1M-step offline policy-optimization budget and 100K-step online fine-tuning budget. Wall-clock times are averaged over 3 seeds on a common NVIDIA A6000 GPU.
Method
HiP3 (offline)
Cal-QL (offline)
PEX (offline)
MOPO (offline)
Hopper
7.5h
9.2h
2.9h
5.9h
HalfCheetah
8.3h
8.7h
4.0h
7.0h
Walker2d
5.6h
7.5h
2.7h
5.2h
Appendix
Table 8: Computation cost comparison for offline stage
Method
HiP3 (online)
MBPO (online)
Hopper
7.7h
7.8h
HalfCheetah
8.5h
8.9h
Walker2d
9.4h
9.7h
Appendix
Table 9: Computation cost comparison for online stage
Table 10: The hyper-parameters of HiP3 used for offline and online learning on the MuJoCo tasks.
Offline Dataset
Same Online Task
Novel Online Task
Optimistic
Pessimistic
Optimistic
Pessimistic
halfcheetah-r
48.4
56.1
57.6
12.8
halfcheetah-mr
56.2
85.9
54.5
53.0
hopper-r
45.7
100.6
74.2
31.6
hopper-mr
78.6
102.2
86.5
37.7
mean
57.2
86.2
68.2
33.8
Appendix
Table 11: Generality of the optimism-pessimism crossover behavior. Optimistic and pessimistic policies are selected before observing the online task and independently fine-tuned for 100K online steps without Top- K ranking, LinUCB selection, or policy switching. The results show a consistent crossover between adaptation on the same and novel online tasks.
Domain
Task
HiP3
Cal-QL
AWAC
CQL
Adroit
pen-human-v1
35.8±10.2
27.6±24.8
36.6±10.6
1.2±5.0
pen-cloned-v1
54.6±4.7
-3.2±0.1
47.3±11.7
13.2±6.4
mean
45.2±7.5
12.2±12.5
42.0±11.2
7.2±5.7
Maze2D
medium-dense-v1
142.4±21.2
77.3±21.2
48.7±26.4
59.2±24.1
Appendix
Table 12: Generalization to additional domains. We evaluate HiP3 on high-dimensional manipulation tasks (Adroit) and navigation tasks (Maze2D), and compare with widely-used O2O RL baselines (Cal-QL, AWAC, CQL). HiP3 achieves competitive or superior performance across domains.
Table 13: Relative performance drop comparison on same and novel online tasks. The drop is measured between the pre-fine-tuning performance and the minimum performance during online adaptation. (Notations: r = random, mr = medium-replay.)
Task
Minimum Normalized Score
Recovery Step, Median [Range]
Same-HC-r
33.2±1.3
8K [ 7K , 9K ]
Same-HC-mr
46.6±15.7
11K [ 7K , 18K ]
Same-Hop-r
14.5±1.0
24K [ 15K , 31K ]
Same-Hop-mr
43.0±5.9
18K [ 12K , 28K ]
Novel-HC-r
27.0±2.8
7K [ 6K , 10K ]
Novel-HC-mr
39.3±21.2
9K [ 7K , 12K ]
Appendix
Table 14: Absolute transient degradation and recovery analysis during online fine-tuning. Minimum normalized scores are averaged over three seeds. Recovery steps are reported as median [minimum, maximum] across seeds.
Sweep
Value
HC-r
HC-mr
Hop-r
Hop-mr
Mean
κ
0.5
66.8 ± 3.0
66.4 ± 3.9
92.2 ± 5.1
94.6 ± 8.3
80.0
κ
1.5
63.5 ± 2.2
69.4 ± 3.6
95.2 ± 5.5
100.1 ± 11.0
82.1
κ
2.5
61.9 ± 0.4
68.7 ± 1.6
96.7 ± 7.3
97.4 ± 10.6
81.2
(τa,τb)
(0.05,0.95)
64.1 ± 2.6
69.3 ± 2.0
88.0 ± 8.2
97.6 ± 17.5
79.8
(τa,τb)
(0.1,0.9)
63.5 ± 2.2
69.4 ± 3.6
95.2 ± 5.5
100.1 ± 11.0
82.1
(τa,τb)
(0.2,0.8)
63.9 ± 2.3
70.2 ± 3.1
89.5 ± 3.5
95.8 ± 16.7
79.9
Appendix
Table 15: Sensitivity analysis of the uncertainty-squashing constant κ and reference-vector range (τa,τb) . Default configurations are bolded. Each non-default setting changes only the corresponding hyperparameter while keeping all other configurations fixed.
Sweep
Value
HC-r
HC-mr
Hop-r
Hop-mr
Mean
ϵ
0.025
62.7 ± 1.5
67.3 ± 3.6
91.4 ± 5.1
96.0 ± 4.3
79.4
ϵ
0.05
63.5 ± 2.2
69.4 ± 3.6
95.2 ± 5.5
100.1 ± 11.0
82.1
ϵ
0.075
58.7 ± 3.6
72.3 ± 2.5
98.5 ± 5.0
94.2 ± 7.9
80.9
ψ
0.995
64.1 ± 4.8
72.5 ± 3.3
94.3 ± 7.2
94.7 ± 12.1
81.4
ψ
0.999
63.5 ± 2.2
69.4 ± 3.6
95.2 ± 5.5
100.1 ± 11.0
82.1
ψ
0.9995
63.0 ± 2.7
68.7 ± 4.8
90.6 ± 4.6
104.5 ± 6.4
81.7
Appendix
Table 16: Sensitivity analysis of the perturbation size ϵ and similarity threshold ψ . Default configurations are bolded. The same experimental protocol is used across all settings.
Method
HiP3 (default, mmepoch=5)
HiP3 (mmepoch=1)
HiP3 (mmepoch=10)
halfcheetah-random-v2
63.5±2.2
48.9±3.7
63.1±2.6
halfcheetah-medium-replay-v2
69.4±3.6
56.7±3.1
66.3±2.3
hopper-random-v2
95.2±5.5
25.0±9.8
92.3±9.0
hopper-medium-replay-v2
100.1±11.0
26.1±8.2
100.8±6.9
mean
82.1±5.5
39.2±6.2
80.6±5.2
Appendix
Table 17: Ablation study on world model training impact
Figure 6: Ablation study on the number of policies K when fine-tuning the policy pool on six novel online tasks. The normalized score is averaged over 5 random seeds (each with 10 evaluations). The curves exclusively illustrate the performance during the online stage, where LinUCB is used for policy selection from the pool.
Method
HiP3 (LinUCB)
HiP3 (KernelUCB)
halfcheetah-random-v2
63.5±2.2
62.0±2.8
halfcheetah-medium-replay-v2
69.4±3.6
63.5±4.9
hopper-random-v2
95.2±5.5
86.8±11.7
hopper-medium-replay-v2
100.1±11.0
99.0±2.8
mean
82.1±5.5
77.8±5.6
Appendix
Table 18: Impact of bandit model capacity on online adaptation performance
Figure 7: Online policy-selection dynamics across the novel online tasks. The heat maps visualize the policy preferences Pt obtained by softmax-normalizing the LinUCB scores at each online interaction step. The visualizations show whether the selector develops a concentrated preference for one candidate or maintains comparable preferences across multiple candidates during adaptation.
Figure 8: Offline-to-online learning curves on the same online tasks . Curves show mean over five seeds; shaded regions denote standard deviation.
Figure 9: Offline-to-online learning curves on the novel online tasks . These tasks are generated via modifying the gravity, noise scales, and reward functions of the original MuJoCo tasks. Curves show mean over five seeds; shaded regions denote standard deviation.
Figure 10: Complete comparison between HiP3 and the single-policy variant on four representative novel online tasks. Curves show mean and standard deviation over five seeds.
Figure 11: Ablation study on the policy selection algorithms for six novel online tasks. The normalized score is averaged over 5 random seeds (each with 10 evaluations). The curves exclusively illustrate the performance during the online stage, where various bandit algorithms are evaluated.
Reference number
ref_vec=5 (HiP3)
ref_vec=4
ref_vec=2
halfcheetah-r-v2
63.5±2.2
63.3±1.2
63.5±1.5
halfcheetah-mr-v2
69.4±3.6
67.7±4.5
68.4±3.8
hopper-r-v2
95.2±5.5
86.1±8.2
87.5±10.4
hopper-mr-v2
100.1±11.0
90.4±10.7
93.8±12.3
mean
82.1±5.5
76.8±6.2
78.3±7.0
Appendix
Table 19: Ablation study for different number of reference vectors. For fairness, we fix the total training budget to 1M steps and allocate it evenly across reference vectors. When too many reference vectors are used, each policy receives insufficient training, leading to performance deterioration. Conversely, too few reference vectors reduce the diversity of the policy pool. The ablation study shows that using five reference vectors achieves the best balance between policy diversity and overall performance.
Novel Online Task
HiP3
Single-Policy HiP3
halfcheetah-r-v2
63.5±2.2
57.1±11.8
halfcheetah-medium-v2
69.3±2.1
54.2±8.3
halfcheetah-medium-replay-v2
69.4±3.6
67.9±15.4
halfcheetah-medium-expert-v2
74.7±2.0
70.3±4.9
hopper-r-v2
95.2±5.5
83.2±0.3
hopper-medium-v2
95.4±3.8
88.5±1.0
Appendix
Table 20: Complete comparison between HiP3 and its single-policy variant on all 12 novel online tasks. The single-policy variant removes the offline Pareto policy pool while retaining the same model-based online fine-tuning procedure. HiP3 uses the five seeds reported in the main results, while the single-policy variant uses three seeds per task. Both methods use the same offline data, environment model, SAC-based backbone, and online fine-tuning budget.
Table 21: Performance comparison on the same and novel online tasks. The novel online tasks are generated by adjusting the gravity, noise scales, and reward functions of the original MuJoCo environments. ∗ indicates the results reported by the paper. Note that FamO2O reports results with a 1M-step online fine-tuning budget and is therefore not directly budget-matched to the 100K-step setting used for the other methods. We fine-tune all other methods for 100K steps, using the same hyper-parameter settings provided by the original papers. Scores for each dataset are normalized according to the D4RL benchmark [ 9 ] : Normalized Score=100×expert return−random returnreturn−random return .
Table 22: Comparisons with model-based reinforcement learning baselines. * indicates results reported by the paper. FOWM results are reported from its paper; TD-MPC2 is evaluated for 100K online training steps.
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
Background: Offline reinforcement learning (RL) enables effective policies to be trained from large, previously collected datasets and subsequently improved through limited online interaction. This offline-to-online RL (O2O-RL) paradigm is particularly promising in nonstationary domains where interaction is costly or potentially hazardous. Standard O2O-RL pipelines train multiple candidate policies offline, evaluate them using off-policy or online evaluation, and then deploy and fine-tune the policy with the highest estimated value. However, as in offline pretraining, fine-tuning performance is highly sensitive to the choice of algorithm and hyperparameters, making it risky to commit to a single policy. Objectives: We study active policy selection for fine-tuning under a limited interaction budget in O2O-RL settings. To our knowledge, this is the first work to address this problem. Methods: We formulate the problem by identifying a fundamental trade-off between allocating online interactions to policy evaluation, which helps identify high-performing policies, and allocating them to fine-tuning, which improves policy performance. We then propose an approach that balances this trade-off by actively selecting policies for fine-tuning based on upper-confidence bounds on their future performance. These bounds are derived from locally linear performance forecasts fitted to observations obtained through online evaluation. Results: Across a diverse range of experiments, the proposed approach consistently outperforms existing O2O-RL baselines. Conclusions: Actively selecting and fine-tuning policies uses limited online interaction budgets more effectively than either committing to a single policy or dividing the budget equally among all policies. Our framework also advances offline RL toward practical deployment in real-world systems where online interaction is costly or risky.
Offline-to-online reinforcement learning (RL) has emerged as a practical paradigm that leverages offline datasets for pretraining and online interactions for fine-tuning. However, its empirical behavior is highly inconsistent: design choices of online fine-tuning that work well in one setting can fail completely in another. Guided by the stability--plasticity principle, we propose a framework that can explain this inconsistency: We argue that efficient fine-tuning must preserve the utility of the stronger offline prior, whether that is the pretrained policy or the offline dataset, while maintaining sufficient plasticity. This perspective identifies three regimes of online fine-tuning, each requiring distinct stability properties. We validate this framework through a large-scale empirical study, finding that the results strongly align with its predictions in 45 out of 63 cases, with only 3 opposite mismatches. This work provides a framework for guiding design choices in offline-to-online RL based on the relative performance of the offline dataset and the pretrained policy.
Lu Li, Tianwei Ni, Yihao Sun +1
Mila – Québec AI Institute · Université de Montréal · CIFAR AI Chair
Offline-to-Online Reinforcement Learning (O2O RL) has emerged as a practical paradigm that pre-trains the policy using static offline datasets and subsequently adapts the policy through online interactions. Existing O2O methods primarily address the transition through value calibration, while generally treating the offline-trained policy as a given initialization. We instead study O2O adaptation from the perspective of network plasticity, asking whether the offline-trained policy remains sufficiently adaptable for online learning. Controlled experiments show that prolonged optimization on static offline data progressively reduces network plasticity even after offline performance has largely saturated, and that lower plasticity is associated with weaker subsequent online improvement. Motivated by these observations, we propose REstoring plasticity via Fresh Initialization and policy Transfer (REFIT), a lightweight model-level method for the O2O transition. Before online fine-tuning, REFIT distills the offline policy into a freshly initialized student while temporarily freezing a random subset of student units, transferring the learned offline behavior to a more plastic policy initialization. Extensive experiments on D4RL and OGBench demonstrate that REFIT consistently achieves higher aggregate performance than existing O2O plug-in methods across both Cal-QL and IQL backbones, while plasticity diagnostics and ablations provide further evidence of restored network plasticity.
Yuheng Huang, Yunpeng Qing, Yixiao Chi +2
Zhejiang University · Carnegie Mellon University · Nanyang Technological University +1