Critic-Free Pretraining for Efficient Online Reinforcement Learning Fine-Tuning

Daoyi Li; Yixian Zhang; Wenbo Ding; Yu Wang; Chao Yu

cs.LGAug 11, 2026

Critic-Free Pretraining for Efficient Online Reinforcement Learning Fine-Tuning

Authors: Daoyi Li, Yixian Zhang, Wenbo Ding, Yu Wang, Chao Yu

Organizations: Tsinghua University

Abstract

Offline-to-online (O2O) reinforcement learning aims to leverage policies pretrained on static datasets while improving them through online interaction. However, directly reusing an offline-trained critic can hinder online fine-tuning: as the policy and data distribution change rapidly, value estimates inherited from offline training may become misaligned with the online environment, leading to inaccurate policy improvement and inefficient exploration. To address this problem, we introduce Critic-Free Pretraining: an efficient paradigm that completely abandons the approach of offline critic training, allowing a freshly initialized critic to adapt without inheriting biased estimates. CFP is compatible with various mainstream O2O algorithms and consistently matches or improves upon conventional O2O algorithms across a diverse set of tasks, with particularly pronounced gains on several challenging tasks.

Explore similar work

Active Offline-to-Online Reinforcement Learning
Jul 13, 2026Alper Kamil Bozkurt, Shangtong Zhang, Yuichi MotaiOffline Reinforcement Learning Limited Data
The Three Regimes of Offline-to-Online Reinforcement Learning
Oct 1, 2025Lu Li, Tianwei Ni, Yihao Sun +1Offline Reinforcement Learning Reinforcement Fine-Tuning
Adaptive Policy Selection and Fine-Tuning under Interaction Budgets for Offline-to-Online Reinforcement Learning
May 6, 2026Alper Kamil Bozkurt, Xiaoan Xu, Shangtong Zhang +2Offline Reinforcement Learning Off-Policy Evaluation

Jul 13, 2026 · cs.LGJ/K move · Enter open · S save

Active Offline-to-Online Reinforcement Learning

Alper Kamil Bozkurt, Shangtong Zhang, Yuichi Motai

Virginia Commonwealth University, USA · University of Virginia, USA

Background: Offline reinforcement learning (RL) enables effective policies to be trained from large, previously collected datasets and subsequently improved through limited online interaction. This offline-to-online RL (O2O-RL) paradigm is particularly promising in nonstationary domains where interaction is costly or potentially hazardous. Standard O2O-RL pipelines train multiple candidate policies offline, evaluate them using off-policy or online evaluation, and then deploy and fine-tune the policy with the highest estimated value. However, as in offline pretraining, fine-tuning performance is highly sensitive to the choice of algorithm and hyperparameters, making it risky to commit to a single policy. Objectives: We study active policy selection for fine-tuning under a limited interaction budget in O2O-RL settings. To our knowledge, this is the first work to address this problem. Methods: We formulate the problem by identifying a fundamental trade-off between allocating online interactions to policy evaluation, which helps identify high-performing policies, and allocating them to fine-tuning, which improves policy performance. We then propose an approach that balances this trade-off by actively selecting policies for fine-tuning based on upper-confidence bounds on their future performance. These bounds are derived from locally linear performance forecasts fitted to observations obtained through online evaluation. Results: Across a diverse range of experiments, the proposed approach consistently outperforms existing O2O-RL baselines. Conclusions: Actively selecting and fine-tuning policies uses limited online interaction budgets more effectively than either committing to a single policy or dividing the budget equally among all policies. Our framework also advances offline RL toward practical deployment in real-world systems where online interaction is costly or risky.

Offline Reinforcement Learning Limited Data

Open paper Ask about it PDF

Abstract

Explore similar work

Active Offline-to-Online Reinforcement Learning

The Three Regimes of Offline-to-Online Reinforcement Learning

Adaptive Policy Selection and Fine-Tuning under Interaction Budgets for Offline-to-Online Reinforcement Learning