Freeze, Don't Clone: Fine-Tuning with Expressive Policies Without Retaining Offline Data

Sami Nur Islam; Johan Obando-Ceron; Doina Precup; Aaron Courville; Pablo Samuel Castro

cs.LGOct 26, 2026

Freeze, Don't Clone: Fine-Tuning with Expressive Policies Without Retaining Offline Data

Authors: Sami Nur Islam, Johan Obando-Ceron, Doina Precup, Aaron Courville, Pablo Samuel Castro

Organizations: Mila – Québec AI Institute · McGill University · Université de Montréal · CIFAR AI Chair

Abstract

Offline-to-online reinforcement learning uses offline pretraining to accelerate subsequent online learning. While many methods assume the offline dataset remains available throughout fine-tuning, there are many settings where this is not the case. Prior work attributes failures in this setting primarily to critic recalibration under distribution shift. We identify a complementary actor-side failure: once the offline data disappears, continued behavioral cloning (BC) no longer preserves the offline behavior distribution and instead refits the behavioral model to the agent's limited, and nonstationary, online experience. We demonstrate that stopping online behavioral fitting while preserving the pretrained expressive prior substantially improves no-retention fine-tuning across implicit flow Q-learning (IFQL), flow Q-learning (FQL), and action-chunked RL (QC) on seven OGBench domains. We trace this failure to a behavioral feedback loop: continuing online behavioral cloning rapidly erases the pretrained prior, inducing representational collapse and action diversity loss that starves the critic of useful exploration. Freezing the prior breaks this cycle, preserving offline coverage while allowing value learning to drive continuous improvement.

Figures & tables

Panel of Figure 1 — Figure 1: Stopping BC enables no-retention fine-tuning. Left: After offline pretraining, we freeze the behavioral prior and continue updating the critic using online data. Right: Aggregate performance across seven OGBench tasks after $1$ M offline gradient steps (gray) and $1$ M online environment steps (white). Stopping BC consistently improves methods with expressive policies.

Figure 2: Sensitivity to the BC loss weight. Online fine-tuning performance across varying BC loss weights (

0.1\times

10\times

default) without offline data retention. Scaling the BC weight up or down (yellow to dark brown) fails to prevent collapse in both IFQL and FQL. Only removing the BC loss entirely ( BC-Off , green) avoids collapse and enables sustained online learning.

Figure 3: Stopping BC improves no-retention fine-tuning across expressive policy classes. Performance of IFQL, FQL, and QC (Action-Chunked RL) across five OGBench domains (25 tasks). The vertical dashed line marks the transition from offline pretraining to online fine-tuning. Continuing BC without offline data ( BC-On , orange dashed) causes catastrophic performance collapse. BC-Off (red dashed) rescues online fine-tuning and often approaches the performance of the standard retained-data baseline (blue solid). The full plot with the seven domains can be found in the Appendix B.4 .

Figure 4: When does freezing the behavioral prior enable no-retention fine-tuning? Col 1: In IFQL, fine-tuning succeeds with a standard Bellman double critic (navy) but collapses entirely when its expressive flow prior is ablated in favor of a Gaussian actor (blue). Col 2: Removing ReBRAC’s BC penalty ( BC-Off ) does not prevent collapse (scaled

0.0-0.2

to highlight the immediate drop), demonstrating that omitting BC without a decoupled prior is insufficient. Col 3: A frozen flow prior supports continued online improvement without retention even after pretraining on low-quality offline data. Cols 4 & 5: Seeding replay with 25k online transitions prevents BC-On from flatlining across both IFQL and FQL, yet BC-Off consistently achieves superior asymptotic success.

$\tau$	cube- double	antsoccer- arena
$0$ (immediate)	0.85	0.71
$200$ k	0.71	0.49
$500$ k	0.53	0.50
$800$ k	0.61	0.51
$1$ M (never)	0.40	0.19

Table 1: Earlier BC termination performs best. Final success after

1

M online steps.

Figure 5: Online fine-tuning on Robomimic under no data retention. Removing the BC objective ( BC-Off ) prevents catastrophic degradation and matches offline retention performance across human demonstrations.

Figure 6: Effect of turning BC off with diffusion priors (IDQL). Performance across Cube Double and Ant Soccer .

Panel of Figure 7 — Figure 7: Continued online-only BC forgets the pretrained behavior. Left and middle: Held-out offline action MSE for IFQL and FQL, respectively. Continuing BC without retaining offline data sharply increases error, whereas BC-Off preserves the pretrained behavior. Right: BC Loss on the offline data (log scale).

Figure 8: Online BC induces actor collapse and critic data starvation. ( Middle) When BC remains active without offline data, the behavior prior’s internal representations degenerate (feature rank collapses), which destroys the policy’s cross-state and within-state diversity across both IFQL and FQL. (Right) Confined to narrow data collection, the critic is starved of diverse, high-value trajectories. This causes the critic’s own feature rank to collapse, destroying its ability to accurately separate high-return states from low-return states (shrinking quantile gap). BC-Off rescues this entire chain. The details of how these metrics are computed are covered in Appendix C .

Panel of Figure 9 — Figure 9: Spatial collapse and degraded exploration under online Behavioral Cloning. Left: PCA projections of online rollouts onto the offline manifold at 1.2M steps. Stripped of its exploratory capacity by BC, the agent is forced into repetitive failure modes, leading to severely restricted state-space coverage. Right: Short-horizon state displacement on cube-double . Continued BC restricts the agent to increasingly localized behavior, while BC-Off preserves purposeful, wide-area navigation.

Figure 10: Online BC drives progressive parameter drift and candidate collapse. Left: Per-layer relative drift (log

x

-axis). Under BC-On (solid), drift penetrates chronologically, beginning with the output-adjacent layer (

10^{3}

–

10^{4}

steps). BC-Off (dashed) exactly freezes IFQL and anchors FQL via distillation. Middle: Macroscopic drift and downstream impact for IFQL on cube-double . Under BC-On , unbounded parameter drift occurs from the pretrained initialization. Right: Adam’s second-moment estimate (

\hat{v}

, log scale). Under BC-On (solid), gradient magnitude plummets immediately at the onset of online training and remains orders of magnitude below BC-Off (dashed), indicating a weakened optimization signal.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
Default MLP hidden dimensions	$(512,512,512,512)$
Activation	GELU
Optimizer	Adam
Learning rate	$3\times 10^{-4}$
Batch size	$256$
Target-network update rate $\tau$	$0.005$

Appendix

Table 2: Base architecture and optimization hyperparameters.

Task	IFQL $N$	FQL $\alpha$	QC $N$	ReBRAC $(\alpha_{1},\alpha_{2})$
cube-double	$32$	$300$	$32$	$(0.1,0)$
puzzle-4x4	$32$	$1000$	$32$	$(0.3,0.01)$
antsoccer-arena	$64$	$30$	$64$	$(0.01,0.01)$
humanoidmaze-medium	$32$	$100$	$32$	$(0.01,0.01)$
humanoidmaze-large	$32$	$100$	$32$	$(0.01,0.01)$

Appendix

Table 3: Task-specific hyperparameters adopted from the FQL and QC papers. IFQL and QC report the number of candidate actions or sequences

N

, FQL reports its behavioral-regularization coefficient

\alpha

, and ReBRAC reports the actor and critic BC coefficients

(\alpha_{1},\alpha_{2})

Setting	Value
Offline pretraining steps	$1{,}000{,}000$
Online fine-tuning steps	$1{,}000{,}000$
Replay buffer capacity	$2{,}000{,}000$ transitions
Evaluation episodes per checkpoint	$50$
Evaluation interval	Every $100{,}000$ steps
Seeds per configuration	3

Appendix

Table 4: Training and evaluation protocol used in our experiments.

Task	BC-On	BC-Off	BC-Off , no distillation
cube-double	$0.000$	$\mathbf{0.977\pm 0.014}$	$0.000$
antsoccer-arena	$0.000$	$0.303\pm 0.206$	$0.000$

Appendix

Table 5: FQL requires a connection to the frozen flow. Final success under no-retention fine-tuning. Freezing the behavioral flow succeeds only when the one-step actor remains distilled toward it.

Figure 11: Online fine-tuning across all seven OGBench domains. Performance across 35 tasks (5 per domain) for IFQL, FQL, and QC.

Figure 12: Resetting the optimizer’s Adam state at the online switch never rescues the discard+BC-on collapse (left, all six cells at exactly zero change) and has only small, inconsistent effects once BC is already off (right). The failure mode is not stale optimizer statistics.

RL Fine-Tuning

Reinforcement learning fine-tuning adapts pretrained models and policies through task rewards, preference feedback, or environment interaction.

See all 231 papers

Critic-Free Pretraining for Efficient Online Reinforcement Learning Fine-Tuning
Aug 11, 2026Daoyi Li, Yixian Zhang, Wenbo Ding +2Reinforcement Learning Critic-Free RL
Tsinghua University
The Three Regimes of Offline-to-Online Reinforcement Learning
Oct 1, 2025Lu Li, Tianwei Ni, Yihao Sun +1Offline RL RL Fine-Tuning
Mila – Québec AI Institute · Université de Montréal · CIFAR AI Chair
When Policies Cannot Be Retrained: A Unified Closed-Form View of Post-Training Steering in Offline Reinforcement Learning
Apr 23, 2026Elias Hossain, Mohammad Jahid Ibna Basher, Ivan Garibay +2Offline RL RL Post-Training
College of Engineering and Computer Science University of Central Florida, Orlando, FL, USA

Aug 11, 2026J/K move · Enter open · S save

Critic-Free Pretraining for Efficient Online Reinforcement Learning Fine-Tuning

Daoyi Li, Yixian Zhang, Wenbo Ding, Yu Wang+1

Tsinghua University

Reinforcement Learning Critic-Free RL RL Fine-Tuning Policy Learning Offline-To-Online RL

Open paper AskPDF arXiv

Offline-to-online (O2O) reinforcement learning aims to leverage policies pretrained on static datasets while improving them through online interaction. However, directly reusing an offline-trained critic can hinder online fine-tuning: as the policy and data distribution change rapidly, value estimates inherited from offline training may become misaligned with the online environment, leading to inaccurate policy improvement and inefficient exploration. To address this problem, we introduce Critic-Free Pretraining: an efficient paradigm that completely abandons the approach of offline critic training, allowing a freshly initialized critic to adapt without inheriting biased estimates. CFP is compatible with various mainstream O2O algorithms and consistently matches or improves upon conventional O2O algorithms across a diverse set of tasks, with particularly pronounced gains on several challenging tasks.

Abstract

RL Fine-Tuning

Explore similar work

Critic-Free Pretraining for Efficient Online Reinforcement Learning Fine-Tuning

The Three Regimes of Offline-to-Online Reinforcement Learning

When Policies Cannot Be Retrained: A Unified Closed-Form View of Post-Training Steering in Offline Reinforcement Learning