Offline-to-online reinforcement learning uses offline pretraining to accelerate subsequent online learning. While many methods assume the offline dataset remains available throughout fine-tuning, there are many settings where this is not the case. Prior work attributes failures in this setting primarily to critic recalibration under distribution shift. We identify a complementary actor-side failure: once the offline data disappears, continued behavioral cloning (BC) no longer preserves the offline behavior distribution and instead refits the behavioral model to the agent's limited, and nonstationary, online experience. We demonstrate that stopping online behavioral fitting while preserving the pretrained expressive prior substantially improves no-retention fine-tuning across implicit flow Q-learning (IFQL), flow Q-learning (FQL), and action-chunked RL (QC) on seven OGBench domains. We trace this failure to a behavioral feedback loop: continuing online behavioral cloning rapidly erases the pretrained prior, inducing representational collapse and action diversity loss that starves the critic of useful exploration. Freezing the prior breaks this cycle, preserving offline coverage while allowing value learning to drive continuous improvement.
Figures & tables
Figure 1: Stopping BC enables no-retention fine-tuning. Left: After offline pretraining, we freeze the behavioral prior and continue updating the critic using online data. Right: Aggregate performance across seven OGBench tasks after 1 M offline gradient steps (gray) and 1 M online environment steps (white). Stopping BC consistently improves methods with expressive policies.
Figure 2: Sensitivity to the BC loss weight. Online fine-tuning performance across varying BC loss weights ( 0.1× to 10× default) without offline data retention. Scaling the BC weight up or down (yellow to dark brown) fails to prevent collapse in both IFQL and FQL. Only removing the BC loss entirely ( BC-Off , green) avoids collapse and enables sustained online learning.
Figure 3: Stopping BC improves no-retention fine-tuning across expressive policy classes. Performance of IFQL, FQL, and QC (Action-Chunked RL) across five OGBench domains (25 tasks). The vertical dashed line marks the transition from offline pretraining to online fine-tuning. Continuing BC without offline data ( BC-On , orange dashed) causes catastrophic performance collapse. BC-Off (red dashed) rescues online fine-tuning and often approaches the performance of the standard retained-data baseline (blue solid). The full plot with the seven domains can be found in the Appendix B.4 .
Figure 4: When does freezing the behavioral prior enable no-retention fine-tuning? Col 1: In IFQL, fine-tuning succeeds with a standard Bellman double critic (navy) but collapses entirely when its expressive flow prior is ablated in favor of a Gaussian actor (blue). Col 2: Removing ReBRAC’s BC penalty ( BC-Off ) does not prevent collapse (scaled 0.0−0.2 to highlight the immediate drop), demonstrating that omitting BC without a decoupled prior is insufficient. Col 3: A frozen flow prior supports continued online improvement without retention even after pretraining on low-quality offline data. Cols 4 & 5: Seeding replay with 25k online transitions prevents BC-On from flatlining across both IFQL and FQL, yet BC-Off consistently achieves superior asymptotic success.
τ
cube- double
antsoccer- arena
0 (immediate)
0.85
0.71
200 k
0.71
0.49
500 k
0.53
0.50
800 k
0.61
0.51
1 M (never)
0.40
0.19
Table 1: Earlier BC termination performs best. Final success after 1 M online steps.
Figure 5: Online fine-tuning on Robomimic under no data retention. Removing the BC objective ( BC-Off ) prevents catastrophic degradation and matches offline retention performance across human demonstrations.
Figure 6: Effect of turning BC off with diffusion priors (IDQL). Performance across Cube Double and Ant Soccer .
Figure 7: Continued online-only BC forgets the pretrained behavior. Left and middle: Held-out offline action MSE for IFQL and FQL, respectively. Continuing BC without retaining offline data sharply increases error, whereas BC-Off preserves the pretrained behavior. Right: BC Loss on the offline data (log scale).
Figure 8: Online BC induces actor collapse and critic data starvation. ( Middle) When BC remains active without offline data, the behavior prior’s internal representations degenerate (feature rank collapses), which destroys the policy’s cross-state and within-state diversity across both IFQL and FQL. (Right) Confined to narrow data collection, the critic is starved of diverse, high-value trajectories. This causes the critic’s own feature rank to collapse, destroying its ability to accurately separate high-return states from low-return states (shrinking quantile gap). BC-Off rescues this entire chain. The details of how these metrics are computed are covered in Appendix C .
Figure 9: Spatial collapse and degraded exploration under online Behavioral Cloning. Left: PCA projections of online rollouts onto the offline manifold at 1.2M steps. Stripped of its exploratory capacity by BC, the agent is forced into repetitive failure modes, leading to severely restricted state-space coverage. Right: Short-horizon state displacement on cube-double . Continued BC restricts the agent to increasingly localized behavior, while BC-Off preserves purposeful, wide-area navigation.
Figure 10: Online BC drives progressive parameter drift and candidate collapse. Left: Per-layer relative drift (log x -axis). Under BC-On (solid), drift penetrates chronologically, beginning with the output-adjacent layer ( 103 – 104 steps). BC-Off (dashed) exactly freezes IFQL and anchors FQL via distillation. Middle: Macroscopic drift and downstream impact for IFQL on cube-double . Under BC-On , unbounded parameter drift occurs from the pretrained initialization. Right: Adam’s second-moment estimate ( v^ , log scale). Under BC-On (solid), gradient magnitude plummets immediately at the onset of online training and remains orders of magnitude below BC-Off (dashed), indicating a weakened optimization signal.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Value
Default MLP hidden dimensions
(512,512,512,512)
Activation
GELU
Optimizer
Adam
Learning rate
3×10−4
Batch size
256
Target-network update rate τ
0.005
Appendix
Table 2: Base architecture and optimization hyperparameters.
Task
IFQL N
FQL α
QC N
ReBRAC (α1,α2)
cube-double
32
300
32
(0.1,0)
puzzle-4x4
32
1000
32
(0.3,0.01)
antsoccer-arena
64
30
64
(0.01,0.01)
humanoidmaze-medium
32
100
32
(0.01,0.01)
humanoidmaze-large
32
100
32
(0.01,0.01)
Appendix
Table 3: Task-specific hyperparameters adopted from the FQL and QC papers. IFQL and QC report the number of candidate actions or sequences N , FQL reports its behavioral-regularization coefficient α , and ReBRAC reports the actor and critic BC coefficients (α1,α2) .
Setting
Value
Offline pretraining steps
1,000,000
Online fine-tuning steps
1,000,000
Replay buffer capacity
2,000,000 transitions
Evaluation episodes per checkpoint
50
Evaluation interval
Every 100,000 steps
Seeds per configuration
3
Appendix
Table 4: Training and evaluation protocol used in our experiments.
Task
BC-On
BC-Off
BC-Off , no distillation
cube-double
0.000
0.977±0.014
0.000
antsoccer-arena
0.000
0.303±0.206
0.000
Appendix
Table 5: FQL requires a connection to the frozen flow. Final success under no-retention fine-tuning. Freezing the behavioral flow succeeds only when the one-step actor remains distilled toward it.
Figure 11: Online fine-tuning across all seven OGBench domains. Performance across 35 tasks (5 per domain) for IFQL, FQL, and QC.
Figure 12: Resetting the optimizer’s Adam state at the online switch never rescues the discard+BC-on collapse (left, all six cells at exactly zero change) and has only small, inconsistent effects once BC is already off (right). The failure mode is not stale optimizer statistics.
Offline-to-online (O2O) reinforcement learning aims to leverage policies pretrained on static datasets while improving them through online interaction. However, directly reusing an offline-trained critic can hinder online fine-tuning: as the policy and data distribution change rapidly, value estimates inherited from offline training may become misaligned with the online environment, leading to inaccurate policy improvement and inefficient exploration. To address this problem, we introduce Critic-Free Pretraining: an efficient paradigm that completely abandons the approach of offline critic training, allowing a freshly initialized critic to adapt without inheriting biased estimates. CFP is compatible with various mainstream O2O algorithms and consistently matches or improves upon conventional O2O algorithms across a diverse set of tasks, with particularly pronounced gains on several challenging tasks.
Offline-to-online reinforcement learning (RL) has emerged as a practical paradigm that leverages offline datasets for pretraining and online interactions for fine-tuning. However, its empirical behavior is highly inconsistent: design choices of online fine-tuning that work well in one setting can fail completely in another. Guided by the stability--plasticity principle, we propose a framework that can explain this inconsistency: We argue that efficient fine-tuning must preserve the utility of the stronger offline prior, whether that is the pretrained policy or the offline dataset, while maintaining sufficient plasticity. This perspective identifies three regimes of online fine-tuning, each requiring distinct stability properties. We validate this framework through a large-scale empirical study, finding that the results strongly align with its predictions in 45 out of 63 cases, with only 3 opposite mismatches. This work provides a framework for guiding design choices in offline-to-online RL based on the relative performance of the offline dataset and the pretrained policy.
Lu Li, Tianwei Ni, Yihao Sun +1
Mila – Québec AI Institute · Université de Montréal · CIFAR AI Chair
Offline reinforcement learning (RL) can learn effective policies from fixed datasets, but deployment objectives may change after training, and in many applications the trained actor cannot be retrained because of data, cost, or governance constraints. We study deployment-time adaptation for frozen offline actors using Product-of-Experts (PoE) composition with a goal-conditioned prior. Our main practical finding is graceful degradation rather than universal performance gain: under degraded or random priors, precision-weighted composition remains anchored to the frozen actor, while additive and prior-only adaptation collapse, and a KL-budget selector often recovers a near-oracle operating point. We also make explicit a closed-form identity in the frozen-actor setting: for diagonal-Gaussian actors and priors, PoE with coefficient alpha yields the same deterministic policy as KL-regularized adaptation with beta = alpha / (1 - alpha), with posterior covariances differing only by a global scalar factor. Empirically, across four D4RL environments (3,900 MuJoCo episodes), we observe a 4/5/3 HELP/FROZEN/HURT split. Extending the analysis to six harder cells and two AntMaze diagnostics reveals an actor-competence ceiling: medium-expert remains HURT in all 9 cells at every tested alpha, while AntMaze with a behavior-cloned frozen actor yields zero success for all composition rules. Overall, PoE and KL-regularized adaptation are best viewed as a single actor-anchored safety mechanism for deployment-time steering.
Elias Hossain, Mohammad Jahid Ibna Basher, Ivan Garibay +2
College of Engineering and Computer Science University of Central Florida, Orlando, FL, USA