Policy Plasticity Matters in Offline-to-Online Reinforcement Learning: Refitting Offline Policies for Online Adaptation

Yuheng Huang; Yunpeng Qing; Yixiao Chi; Yilun Kong; Changqing Zou

cs.LGSep 27, 2026

Policy Plasticity Matters in Offline-to-Online Reinforcement Learning: Refitting Offline Policies for Online Adaptation

Authors: Yuheng Huang, Yunpeng Qing, Yixiao Chi, Yilun Kong, Changqing Zou

Organizations: Zhejiang University · Carnegie Mellon University · Nanyang Technological University · Zhejiang University & Zhejiang Lab

Abstract

Offline-to-Online Reinforcement Learning (O2O RL) has emerged as a practical paradigm that pre-trains the policy using static offline datasets and subsequently adapts the policy through online interactions. Existing O2O methods primarily address the transition through value calibration, while generally treating the offline-trained policy as a given initialization. We instead study O2O adaptation from the perspective of network plasticity, asking whether the offline-trained policy remains sufficiently adaptable for online learning. Controlled experiments show that prolonged optimization on static offline data progressively reduces network plasticity even after offline performance has largely saturated, and that lower plasticity is associated with weaker subsequent online improvement. Motivated by these observations, we propose REstoring plasticity via Fresh Initialization and policy Transfer (REFIT), a lightweight model-level method for the O2O transition. Before online fine-tuning, REFIT distills the offline policy into a freshly initialized student while temporarily freezing a random subset of student units, transferring the learned offline behavior to a more plastic policy initialization. Extensive experiments on D4RL and OGBench demonstrate that REFIT consistently achieves higher aggregate performance than existing O2O plug-in methods across both Cal-QL and IQL backbones, while plasticity diagnostics and ablations provide further evidence of restored network plasticity.

Figures & tables

Panel of Figure 1 — Figure 1: Network plasticity and online adaptation in a controlled O2O setting. The dashed line marks the transition from offline pre-training to online fine-tuning. (a) The bimodal reward landscape, offline data distribution, and policy trajectories. (b) Actor FAU throughout offline training and subsequent online fine-tuning. (c) Learning curve throughout training.

Panel of Figure 2 — Figure 2: Offline over-optimization significantly degrades model plasticity, and the plasticity metric FAU is shown to be positively correlated with online performance improvements. (a) Overall performance and FAU curves across varying offline pre-training budgets. (b) Correlation between online initial policy FAU and performance improvement.

Environment	IQL ( Kostrikov et al., 2021 )			Cal-QL ( Nakamoto et al., 2023 )
Environment	Baseline	OPT	Ours (REFIT)	Baseline	PARS	Ours (REFIT)
antmaze-m-p	76.2 $\pm$ 1.5	88.0 $\pm$ 3.0	86.6 $\pm$ 4.8	92.2 $\pm$ 2.1	92.4 $\pm$ 3.0	95.4 $\pm$ 1.7
antmaze-m-d	71.6 $\pm$ 2.7	88.8 $\pm$ 1.2	89.2 $\pm$ 2.7	93.2 $\pm$ 0.8	89.6 $\pm$ 2.9	96.2 $\pm$ 0.8
antmaze-l-p	46.0 $\pm$ 2.8	61.6 $\pm$ 5.1	63.8 $\pm$ 8.3	70.6 $\pm$ 7.5	78.6 $\pm$ 3.6	78.2 $\pm$ 5.0
antmaze-l-d	50.6 $\pm$ 5.7	63.6 $\pm$ 3.6	65.2 $\pm$ 4.0	53.6 $\pm$ 17.4	79.2 $\pm$ 12.4	80.2 $\pm$ 5.0
AntMaze total	244.4	302.0	304.8	309.6	339.8	350.0

Table 1: Performance of our REFIT and baselines on AntMaze, MuJoCo, and OGBench, averaged over five random seeds. Bold and underlined values denote the best and second-best results within each backbone, respectively.

Figure 4: O2O training curves on D4RL and OGBench. The dashed line marks the transition from offline pre-training to online fine-tuning, at which REFIT is applied.

Figure 5: Evolution of actor FAU across the offline-to-online transition.

Method	hc-m	hc-r	wal-m-r	hop-m-r	ant-m-p	ant-m-d	Score
Baseline	48.1 $\pm$ 0.1	13.8 $\pm$ 3.9	74.8 $\pm$ 2.8	77.5 $\pm$ 18.1	76.2 $\pm$ 1.5	71.6 $\pm$ 2.7	362.0 (-23.3%)
Reset	47.3 $\pm$ 0.4	38.6 $\pm$ 2.4	84.0 $\pm$ 10.6	86.5 $\pm$ 29.9	71.2 $\pm$ 6.8	74.0 $\pm$ 5.5	401.6 (-14.9%)
Reset + Distillation	49.8 $\pm$ 0.2	45.7 $\pm$ 1.9	89.3 $\pm$ 11.2	97.9 $\pm$ 6.2	83.5 $\pm$ 1.8	79.2 $\pm$ 23.6	445.4 (-5.6%)
REFIT	50.0 $\pm$ 0.3	46.3 $\pm$ 1.4	96.9 $\pm$ 2.2	102.7 $\pm$ 0.4	86.6 $\pm$ 4.8	89.2 $\pm$ 2.7	471.7

Table 2: Ablation of REFIT design components. All results are averaged over 5 random seeds.

Env	0.0	0.25	0.5	0.75	1.0
halfcheetah-m	49.8 $\pm$ 0.2	50.0 $\pm$ 0.2	50.0 $\pm$ 0.3	50.1 $\pm$ 0.4	47.3 $\pm$ 0.4
halfcheetah-r	45.7 $\pm$ 1.9	46.2 $\pm$ 1.3	46.3 $\pm$ 1.4	46.5 $\pm$ 1.8	38.6 $\pm$ 2.4
antmaze-m-p	83.5 $\pm$ 1.8	86.0 $\pm$ 3.7	86.6 $\pm$ 4.8	85.8 $\pm$ 4.1	71.2 $\pm$ 6.8
antmaze-m-d	79.2 $\pm$ 23.6	86.4 $\pm$ 3.0	89.2 $\pm$ 2.7	84.4 $\pm$ 1.5	74.0 $\pm$ 5.5
Total	258.2	268.6	272.1	266.8	231.1

Table 3: Performance of REFIT under various freeze ratio settings. The two lowest scores in each row are marked in red .

Figure 6: Learning curves of REFIT with different distillation targets.

Figure 7: Performance of REFIT under different distillation budget settings.

Panel of Figure 8 — Figure 8: Robustness of REFIT across different offline training budgets. (a) Normalized online performance of REFIT and the corresponding IQL baseline after online fine-tuning under different offline training budgets. (b) Online learning curves of REFIT and the baseline across different offline training budgets. (c) FAU during online fine-tuning.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Suite	Token	Meaning	Example
Environment	hc	halfcheetah	hc-m
	hop	hopper	hop-m-r
	wal	walker2d	wal-r
	ant	antmaze	ant-l-d
MuJoCo Postfix	r	random	hc-r
	m	medium	hop-m

Appendix

Table 4: Environment naming convention. Tokens are composed from left to right, as illustrated in the examples.

Figure 9: Wall-clock time of IQL+REFIT and the IQL baseline, averaged over five random seeds.

Method	hc-m	hc-r	wal-m-r	hop-m-r	ant-m-p	ant-m-d	Total
IQL	48.1 $\pm$ 0.1	13.8 $\pm$ 3.9	74.8 $\pm$ 2.8	77.5 $\pm$ 18.1	76.2 $\pm$ 1.5	71.6 $\pm$ 2.7	362.0
IQL+CPR	51.7 $\pm$ 0.5	26.0 $\pm$ 4.0	86.0 $\pm$ 11.7	101.8 $\pm$ 3.9	66.0 $\pm$ 3.3	71.0 $\pm$ 6.8	402.5
IQL+REFIT	50.0 $\pm$ 0.3	46.3 $\pm$ 1.4	96.9 $\pm$ 2.2	102.7 $\pm$ 0.4	86.6 $\pm$ 4.8	89.2 $\pm$ 2.7	471.7
Cal-QL	65.3 $\pm$ 3.0	33.4 $\pm$ 3.8	86.7 $\pm$ 8.7	68.9 $\pm$ 7.8	92.2 $\pm$ 2.1	93.2 $\pm$ 0.8	439.7
Cal-QL+CPR	43.3 $\pm$ 0.6	2.2 $\pm$ 0.1	17.6 $\pm$ 2.8	30.3 $\pm$ 7.9	27.0 $\pm$ 10.3	26.0 $\pm$ 13.9	146.3
Cal-QL+REFIT	51.1 $\pm$ 0.9	39.2 $\pm$ 3.5	97.9 $\pm$ 5.8	87.6 $\pm$ 7.5	95.4 $\pm$ 1.7	96.2 $\pm$ 0.8	467.4

Appendix

Table 5: Normalized scores on six representative tasks.

Figure 10: Performance comparison between IQL, IQL From Scratch and IQL+REFIT

Figure 11: Impact of network reset during distillation.

Environment	$\text{KL}_{\text{fwd}}$	$\text{KL}_{\text{rev}}$	MSE
antmaze-m-d	86.9 $\pm$ 6.7	87.0 $\pm$ 5.2	89.2 $\pm$ 2.7
antmaze-m-p	87.8 $\pm$ 2.7	84.8 $\pm$ 5.2	86.6 $\pm$ 4.8
halfcheetah-m	49.9 $\pm$ 0.2	50.0 $\pm$ 0.6	50.0 $\pm$ 0.3
halfcheetah-r	46.3 $\pm$ 1.5	47.5 $\pm$ 1.7	46.3 $\pm$ 1.4
Total Score	270.9 $\pm$ 2.9	269.2 $\pm$ 2.9	272.1 $\pm$ 2.8

Appendix

Table 6: Ablation of distillation loss. Results report mean

\pm

standard deviation.

Method	antmaze-m-d	antmaze-m-p	halfcheetah-m	halfcheetah-r	Total Score
Default	89.2 $\pm$ 2.7	86.6 $\pm$ 4.8	50.0 $\pm$ 0.3	46.3 $\pm$ 1.4	272.1 $\pm$ 2.8
Copy Std	86.4 $\pm$ 4.5	84.3 $\pm$ 4.0	50.1 $\pm$ 0.3	46.9 $\pm$ 2.9	267.7 $\pm$ 2.3

Appendix

Table 7: Ablation of standard-deviation transfer. “Default” denotes the standard REFIT configuration where the log-standard-deviation parameters are reset along with the rest of the student network; “Copy Std” denotes the variant that inherits the standard deviation from the offline teacher. Results are reported as mean

\pm

standard deviation across random seeds.

Figure 12: Online performance with and without unfreezing the parameter subset after distillation.

Figure 13: Weight Norm of REFIT and baseline throughout the training process.

Figure 14: SRank of REFIT and baseline throughout the training process.

Offline-To-Online RL

Offline-to-online reinforcement learning adapts policies learned from fixed datasets through further interaction with the environment.

See all 69 papers

Adaptive Policy Selection and Fine-Tuning under Interaction Budgets for Offline-to-Online Reinforcement Learning
May 6, 2026Alper Kamil Bozkurt, Xiaoan Xu, Shangtong Zhang +2Sequential Decision Making RL Fine-Tuning
Virginia Commonwealth University, Richmond, VA · Duke University, Durham, NC · University of Virginia, Charlottesville, VA
Active Offline-to-Online Reinforcement Learning
Jul 13, 2026Alper Kamil Bozkurt, Shangtong Zhang, Yuichi MotaiOffline RL RL Fine-Tuning
Virginia Commonwealth University, USA · University of Virginia, USA
The Three Regimes of Offline-to-Online Reinforcement Learning
Oct 1, 2025Lu Li, Tianwei Ni, Yihao Sun +1Offline RL RL Fine-Tuning
Mila – Québec AI Institute · Université de Montréal · CIFAR AI Chair

May 6, 2026J/K move · Enter open · S save

Adaptive Policy Selection and Fine-Tuning under Interaction Budgets for Offline-to-Online Reinforcement Learning

Alper Kamil Bozkurt, Xiaoan Xu, Shangtong Zhang, Miroslav Pajic+1

Virginia Commonwealth University, Richmond, VA · Duke University, Durham, NC · University of Virginia, Charlottesville, VA

Sequential Decision Making RL Fine-Tuning Sample-Efficient RL Offline-To-Online RL

Open paper AskPDF arXiv

In offline-to-online reinforcement learning (O2O-RL), policies are first safely trained offline using previously collected datasets and then further fine-tuned for tasks via limited online interactions. In a typical O2O-RL pipeline, candidate policies trained with offline RL are evaluated via either off-policy evaluation (OPE) or online evaluation (OE). The policy with the highest estimated value is then deployed and continually fine-tuned. However, this setup has two main issues. First, OPE can be unreliable, making it risky to deploy a policy based solely on those estimates, whereas OE may identify a viable policy with substantial online interaction, which could have been used for fine-tuning. Second--and more importantly--it is also often not possible to determine a priori whether a pretrained policy will improve with post-deployment fine-tuning, especially in non-stationary environments. As a result, procedures committing to a single deployed policy are impractical in many real-world settings. Moreover, a naive remedy that exhaustively fine-tunes all candidates would violate interaction budget constraints and is likewise infeasible. In this paper, we propose a novel adaptive approach for policy selection and fine-tuning under online interaction budgets in O2O-RL. Following the standard pipeline, we first train a set of candidate policies with different offline RL algorithms and hyperparameters; we then perform OPE to obtain initial performance estimates. We next adaptively select and fine-tune the policies based on their predicted performance via an upper-confidence-bound approach thereby making efficient use of online interactions. We demonstrate that our approach improves upon O2O-RL baselines with various benchmarks.

Abstract

Offline-To-Online RL

Explore similar work

Adaptive Policy Selection and Fine-Tuning under Interaction Budgets for Offline-to-Online Reinforcement Learning

Active Offline-to-Online Reinforcement Learning

The Three Regimes of Offline-to-Online Reinforcement Learning