ANCHOR: KL-Anchored Adaptation for Stable Offline-to-Online RL
Authors: Geonwoo Cho, Jaegyun Im, Doyoon Kim, Lexin Li
Organizations: University of Michigan, Ann Arbor · Gwangju Institute of Science and Technology · Pohang University of Science and Technology · University of California, Berkeley
Offline-to-online reinforcement learning (RL) follows the pretrain-finetune paradigm: an agent is first pretrained on historical data and then adapted through online interaction. We study offline-to-online adaptation without offline replay, where the agent must improve using only the pretrained model and newly collected experience. A key challenge is to avoid catastrophic failure, a sharp early performance collapse during online fine-tuning, while still achieving strong asymptotic performance. Through controlled experiments, we show that removing offline replay can induce catastrophic failure, while retaining strong conservative regularization from offline RL can suppress final online performance. We also find that common stabilization strategies do not reliably prevent early collapse, and that the critic-side mechanisms we test are insufficient on their own. Motivated by these findings, we propose ANCHOR, a simple adaptation method that anchors the actor to the offline-pretrained policy early in online fine-tuning, anneals this constraint over time, and retains only a small residual conservative critic regularizer. We provide a theoretical interpretation showing that KL anchoring helps preserve critic reliability across nearby occupancy distributions, with a two-phase regret bound explaining when anchoring should be maintained or relaxed. Across FrankaKitchen, AntMaze, and Adroit tasks, ANCHOR substantially reduces catastrophic failure while maintaining strong asymptotic performance. Ablations show that KL annealing and the residual conservative weight are important for balancing early stability and online adaptation, while additional analysis shows that ANCHOR is most beneficial when the pretrained policy contains useful behavior to preserve. Code is available at https://github.com/Cho-Geonwoo/ANCHOR.
Figures & tables
Figure 1: Controlled analysis of offline data and conservative regularization. Success rates of CQL during online fine-tuning across four tasks. (a) Keeping the conservative regularizer α during online fine-tuning can reduce asymptotic performance. (b) Removing offline replay can induce early performance degradation.
Figure 2: ANCHOR improves both early stability and final performance. Existing methods do not consistently achieve both low catastrophic failure and strong asymptotic performance across tasks. In contrast, ANCHOR reduces early performance collapse while attaining strong final performance across tasks. The aggregate curve reports the mean success rate averaged over the six tasks.
Figure 3: KL annealing improves early stability while preserving final performance. ANCHOR is compared with the fixed-KL and no-KL variants. The fixed-KL variant improves early stability but can limit final improvement, while the no-KL variant adapts aggressively but causes early collapse.
Figure 4: Benefit of ANCHOR depends on pretrained policy quality. ANCHOR reduces catastrophic failure more clearly when the offline-pretrained policy starts from a higher initial success rate. Curves with the same color correspond to the same pretrained policy. Results are averaged over three seeds.
Appendix figures & tables27 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Hyperparameter and architectural tweaks do not consistently mitigate catastrophic failure. From top to bottom, the plots correspond to: batch size, actor learning rate, critic learning rate, critic hidden dimension, actor hidden dimension, LayerNorm in the critic, LayerNorm in the actor, and the duration of the offline pretraining phase. Across all settings and tasks, none of these factors consistently mitigates catastrophic failure.
Figure 6: SQOR and SQOG do not reliably predict catastrophic failure. We compare success rate, the spurious Q-optimism ratio (SQOR), and the spurious Q-optimism gap (SQOG) across four tasks. Both metrics show task-dependent and inconsistent alignment with early performance collapse. The vertical dashed line marks the timestep at which the largest performance drop occurs.
Figure 7: Q-update volatility does not reliably predict catastrophic failure. We compare success rate and critic-update volatility across four tasks. Volatility shows weak and inconsistent correspondence with early performance collapse.
Figure 8: Sensitivity to batch size and UTD ratio. We vary the batch size (top) and UTD ratio (bottom) across three tasks. ANCHOR is generally robust to these choices, although larger batch size and larger UTD can increase learning instability.
Figure 9: Effect of the initial KL weight. A larger initial KL weight ( β0=2.0 ) produces lower KL loss early in online fine-tuning, indicating that the KL penalty more strongly constrains deviation from the offline policy. As the KL coefficient is annealed, the gap between β0=0.5 and β0=2.0 gradually narrows.
Figure 10: Applying KL anchoring to IQL. Adding KL anchoring with annealing to IQL reduces the initial performance drop, but does not improve asymptotic performance over the IQL baseline.
Figure 11: Additional environments reveal limitations of KL-anchored adaptation. In kitchen-complete, ANCHOR remains competitive despite near-zero initial success. In relocate-binary, all methods fail to make meaningful progress. In antmaze-ultra-diverse, most methods do not improve over the offline-pretrained policy, suggesting that harder sparse-reward exploration requires mechanisms beyond KL anchoring alone.
Figure 12: Stronger preservation is insufficient in antmaze-ultra-diverse. Keeping both the conservative coefficient and KL coefficient fixed yields a higher mean success rate than ANCHOR during online fine-tuning, but does not significantly improve asymptotic performance.
AntMaze
Adroit
Kitchen
large diverse
large play
door
pen
partial
mixed
ANCHOR
αon
0.0
0.0
0.0
0.0
1.0
1.0
TKL
80,000
80,000
40,000
80,000
160,000
160,000
β0
0.4
0.2
0.2
1.0
1.0
1.0
Optimization
Appendix
Table 1: Hyperparameters of ANCHOR.
Task
ANCHOR (Ours)
WSRL
PORL
Cal-QL
CQL
IQL
SAC
antmaze-large-diverse-v2
2.75 ± 0.02
2.73 ± 0.01
3.35 ± 0.07
3.44 ± 0.01
2.82 ± 0.02
2.08 ± 0.02
1.53 ± 0.0
antmaze-large-play-v2
2.75 ± 0.01
3.48 ± 0.08
3.54 ± 0.06
3.26 ± 0.03
2.34 ± 0.03
2.09 ± 0.03
1.52 ± 0.0
kitchen-partial-v0
4.31 ± 0.02
3.37 ± 0.0
3.41 ± 0.03
4.35 ± 0.01
3.28 ± 0.05
1.41 ± 0.01
1.38 ± 0.0
kitchen-mixed-v0
3.88 ± 0.03
3.61 ± 0.01
3.51 ± 0.13
4.36 ± 0.0
3.33 ± 0.02
1.46 ± 0.02
1.36 ± 0.0
door-binary-v0
3.07 ± 0.01
2.9 ± 0.01
2.93 ± 0.01
3.55 ± 0.01
2.43 ± 0.01
1.28 ± 0.02
0.78 ± 0.01
pen-binary-v0
2.33 ± 0.02
2.77 ± 0.01
2.79 ± 0.01
3.51 ± 0.01
2.29 ± 0.01
1.13 ± 0.1
0.67 ± 0.0
Appendix
Table 2: Online training time (in decimal hours) comparison across baselines.
Figure 13: Representative episodes across nine tasks. Each row shows temporally ordered frames from a trajectory, demonstrating navigation (antmaze-ultra-diverse-v2, antmaze-large-diverse-v2, antmaze-large-play-v2), kitchen manipulation (kitchen-partial-v0, kitchen-mixed-v0, kitchen-complete-v0), and dexterous control (pen-binary-v0, door-binary-v0, relocate-binary-v0).
Task
ANCHOR (Ours)
WSRL
PORL
Cal-QL
CQL
IQL
SAC
antmaze-large-diverse-v2
0.97 ± 0.01
0.68 ± 0.17
0.9 ± 0.07
0.97 ± 0.01
0.92 ± 0.05
0.06 ± 0.05
0.0 ± 0.0
antmaze-large-play-v2
0.96 ± 0.02
0.95 ± 0.02
0.47 ± 0.2
0.9 ± 0.07
0.95 ± 0.03
0.01 ± 0.01
0.0 ± 0.0
kitchen-partial-v0
1.0 ± 0.0
0.93 ± 0.06
0.91 ± 0.06
0.7 ± 0.09
0.44 ± 0.12
0.4 ± 0.05
0.49 ± 0.07
kitchen-mixed-v0
0.81 ± 0.09
0.72 ± 0.11
0.88 ± 0.06
0.45 ± 0.12
0.4 ± 0.06
0.31 ± 0.09
0.3 ± 0.05
door-binary-v0
1.0 ± 0.0
0.8 ± 0.2
0.6 ± 0.24
0.19 ± 0.19
0.18 ± 0.18
0.0 ± 0.0
0.2 ± 0.2
pen-binary-v0
1.0 ± 0.0
0.97 ± 0.01
0.99 ± 0.01
0.94 ± 0.02
0.81 ± 0.09
0.33 ± 0.1
0.78 ± 0.04
Appendix
Table 3: Performance at 400k online steps. Bold indicates the best performance per task, and underline indicates the second best.
Task
ANCHOR (Ours)
WSRL
PORL
Cal-QL
CQL
IQL
SAC
antmaze-large-diverse-v2
0.97 ± 0.01
0.73 ± 0.18
0.84 ± 0.07
0.95 ± 0.03
0.96 ± 0.02
0.03 ± 0.03
0.0 ± 0.0
antmaze-large-play-v2
0.96 ± 0.02
0.98 ± 0.01
0.37 ± 0.23
0.89 ± 0.07
0.95 ± 0.03
0.01 ± 0.01
0.0 ± 0.0
kitchen-partial-v0
1.0 ± 0.0
0.95 ± 0.05
0.9 ± 0.06
0.7 ± 0.09
0.44 ± 0.12
0.39 ± 0.05
0.47 ± 0.06
kitchen-mixed-v0
0.83 ± 0.1
0.7 ± 0.12
0.88 ± 0.06
0.45 ± 0.12
0.4 ± 0.06
0.3 ± 0.08
0.25 ± 0.03
door-binary-v0
0.97 ± 0.01
0.8 ± 0.2
0.6 ± 0.24
0.17 ± 0.17
0.19 ± 0.19
0.0 ± 0.0
0.2 ± 0.2
pen-binary-v0
0.94 ± 0.02
0.96 ± 0.02
0.95 ± 0.02
0.96 ± 0.01
0.72 ± 0.05
0.4 ± 0.05
0.84 ± 0.02
Appendix
Table 4: Performance at 350k online steps. Bold indicates the best performance per task, and underline indicates the second best.
Task
ANCHOR (Ours)
WSRL
PORL
Cal-QL
CQL
IQL
antmaze-large-diverse-v2
0.33 ± 0.09
0.24 ± 0.05
0.39 ± 0.04
0.26 ± 0.04
0.26 ± 0.05
0.03 ± 0.02
antmaze-large-play-v2
0.29 ± 0.05
0.29 ± 0.02
0.37 ± 0.02
0.24 ± 0.04
0.28 ± 0.05
0.0 ± 0.0
kitchen-partial-v0
0.7 ± 0.03
0.73 ± 0.06
0.7 ± 0.04
0.68 ± 0.05
0.7 ± 0.06
0.3 ± 0.06
kitchen-mixed-v0
0.47 ± 0.06
0.47 ± 0.06
0.49 ± 0.07
0.52 ± 0.06
0.58 ± 0.03
0.4 ± 0.04
door-binary-v0
0.26 ± 0.04
0.31 ± 0.19
0.24 ± 0.03
0.21 ± 0.07
0.08 ± 0.08
0.05 ± 0.04
pen-binary-v0
0.69 ± 0.03
0.66 ± 0.05
0.74 ± 0.03
0.75 ± 0.05
0.58 ± 0.03
0.89 ± 0.04
Appendix
Table 5: Performance after the offline phase.
Table 6: Catastrophic failure (0-400K window).
Figure 14: Benefit of ANCHOR depends on pretrained policy quality. ANCHOR reduces catastrophic failure more clearly when the offline-pretrained policy starts from a higher initial success rate. Curves with the same color correspond to the same pretrained policy. Results are averaged over three seeds.
Figure 15: Common stabilization strategies do not prevent catastrophic failure. Effects of varying standard online fine-tuning choices across four tasks. Top : UTD ratio; Middle : warmup length; Bottom : Cal-QL conservative-regularizer weight, αCal-QL . Increasing UTD, extending warmup, or tuning αCal-QL does not reliably prevent early performance collapse.
Figure 16: Critic-shift regularization does not prevent catastrophic failure. Effects of varying the critic-shift penalty strength αKL across four tasks. Top : critic distribution shift during online fine-tuning, measured by DKL(softmax(Qoff)∥softmax(Qθ)) . Bottom : corresponding success rates. Increasing αKL slows the critic shift, but does not reliably prevent early performance collapse.
Figure 17: Hyperparameter sensitivity of ANCHOR. We vary the initial KL weight β0 (top), the KL annealing interval TKL (middle), and the residual online conservative weight αon (bottom). Here, α∗ denotes an intermediate value between 0 and αoff , where αoff is the conservative regularization weight used during offline pretraining. ANCHOR remains stable under moderate changes to KL-related hyperparameters, while αon controls the balance between early stability and final improvement.
Task
Metric
PEX
PROTO
ITD3-BC
ANCHOR
antmaze-large-play
Final
0.91±.02
0.97±.02
0.52±.11
0.96±.02
CF
0.13±.03
0.07±.04
0.08±.04
0.11±.09
kitchen-partial
Final
0.58±.11
0.97±.02
0.89±.06
1.00±.00
CF
0.71±.07
0.47±.11
0.69±.07
0.06±.02
pen-binary
Final
0.94±.04
0.95±.03
0.86±.06
1.00±.00
CF
0.49±.01
0.46±.06
0.15±.03
0.14±.04
Appendix
Table 8: Policy-side adaptations under CQL initialization. Five-seed mean ± SE; final success is higher-is-better and 0–100K CF is lower-is-better. ITD3-BC abbreviates the Improving TD3-BC-style adaptation.
Task
Annealed BC 1→0
Annealed BC 10→0
ANCHOR
antmaze-large-play
.94±.02/.23±.04
.85±.05/.31±.05
.96±.02/.11±.09
kitchen-partial
.99±.01/.69±.07
.99±.01/.11±.06
1.00±.00/.06±.02
pen-binary
.95±.02/.62±.07
.98±.01/.15±.06
1.00±.00/.14±.04
Task
Fixed BC 1
Fixed BC 10
ANCHOR
kitchen-partial
.96±.02/.68±.10
.57±.08/.22±.08
1.00±.00/.06±.02
pen-binary
.97±.02/.57±.03
.93±.02/.07±.04
1.00±.00/.14±.04
Appendix
Table 9: Five-seed BC controls. Each cell gives final success / 0–100K CF, both as mean ± SE. Data access and critic schedules are detailed in Appendix N.3 .
Metric
ANCHOR
Strongest baseline
Final success at 400K
0.993 [0.981, 1.000]
WSRL: 0.944 [0.879, 0.972]
CF over 0–100K
0.081 [0.048, 0.123]
IQL: 0.135 [0.104, 0.171]
Appendix
Table 10: Aggregate IQM and 95% bootstrap confidence intervals. The baseline in each row is the strongest by that aggregate metric.
Baseline
Final success at 400K
CF over 0–100K
WSRL
0.710 [0.607, 0.813]
0.767 [0.653, 0.880]
PORL
0.657 [0.553, 0.763]
0.857 [0.763, 0.940]
Cal-QL
0.790 [0.693, 0.880]
0.747 [0.627, 0.853]
CQL
0.853 [0.763, 0.937]
0.757 [0.653, 0.860]
IQL
1.000 [1.000, 1.000]
0.590 [0.513, 0.667]
SAC
0.983 [0.950, 1.000]
—
Appendix
Table 11: Probability of improvement of ANCHOR, with 95% bootstrap confidence intervals. These compare aggregate distributions across the six tasks.
Task
Metric
WSRL
α -only
Annealed BC
ANCHOR
kitchen-mixed
Novelty
.85±.01
.79±.02
.77±.01
.44±.01
CF
.61±.02
.57±.03
.58±.03
.03±.01
door-binary
Novelty
.98±.02
.96±.03
.96±.03
.95±.04
CF
.06±.06
.05±.05
.08±.08
.07±.07
Appendix
Table 12: Five-seed novelty and failure diagnostics. Novelty is measured over 0–30K; CF over 0–100K for kitchen-mixed and 0–60K for door-binary. Novelty is a visitation statistic, not an objective to minimize.
Backbone
Task
Base final
+A final
Base CF
+A CF
FQL
scene
1.00±.00
1.00±.00
0.26±.06
0.16±.07
FQL
puzzle-4x4
0.40±.22
0.40±.22
0.07±.01
0.09±.01
IFQL
humanoidmaze-medium
0.25±.07
0.92±.01
0.71±.02
0.19±.08
IFQL
antsoccer-arena
0.08±.05
0.40±.04
0.18±.03
0.21±.06
Appendix
Table 13: OGBench extension, five seeds (mean ± SE). “+A” uses the implicit-policy proxy in Equation 10 . Short task names are expanded in Appendix N.6 . CF is measured over 0–100K.
Task
Replay, no α
No replay, no α
Replay, with α
kitchen-partial
.69±.07/.76±.08
.68±.06/.60±.06
.58±.10/.62±.13
kitchen-mixed
.49±.06/.70±.02
.49±.07/.41±.13
.48±.07/.70±.05
door-binary
.15±.04/1.00±.00
.25±.03/.79±.20
.28±.03/.74±.19
pen-binary
.65±.06/.93±.03
.61±.04/.89±.03
.28±.09/.94±.03
Average
.50±.06/.85±.03
.51±.04/.67±.07
.40±.05/.75±.06
Appendix
Table 14: Replay/conservatism control, five seeds. Cells give CF / final success (mean ± SE). The average row is the reported aggregate.
Task
TKL=40K
TKL=160K
antmaze-large-play
.90±.05/.05±.00
1.00±.00/.12±.07
kitchen-partial
1.00±.00/.03±.01
.99±.01/.04±.03
pen-binary
1.00±.00/.35±.10
1.00±.00/.18±.12
Appendix
Table 15: Two-seed shared-weight duration check. Cells give final success / 0–100K CF, mean ± SE. The initial-weight value is unresolved in the retained experiment record.
Offline-to-online (O2O) reinforcement learning aims to leverage policies pretrained on static datasets while improving them through online interaction. However, directly reusing an offline-trained critic can hinder online fine-tuning: as the policy and data distribution change rapidly, value estimates inherited from offline training may become misaligned with the online environment, leading to inaccurate policy improvement and inefficient exploration. To address this problem, we introduce Critic-Free Pretraining: an efficient paradigm that completely abandons the approach of offline critic training, allowing a freshly initialized critic to adapt without inheriting biased estimates. CFP is compatible with various mainstream O2O algorithms and consistently matches or improves upon conventional O2O algorithms across a diverse set of tasks, with particularly pronounced gains on several challenging tasks.
Offline reinforcement learning (RL) can learn effective policies from fixed datasets, but deployment objectives may change after training, and in many applications the trained actor cannot be retrained because of data, cost, or governance constraints. We study deployment-time adaptation for frozen offline actors using Product-of-Experts (PoE) composition with a goal-conditioned prior. Our main practical finding is graceful degradation rather than universal performance gain: under degraded or random priors, precision-weighted composition remains anchored to the frozen actor, while additive and prior-only adaptation collapse, and a KL-budget selector often recovers a near-oracle operating point. We also make explicit a closed-form identity in the frozen-actor setting: for diagonal-Gaussian actors and priors, PoE with coefficient alpha yields the same deterministic policy as KL-regularized adaptation with beta = alpha / (1 - alpha), with posterior covariances differing only by a global scalar factor. Empirically, across four D4RL environments (3,900 MuJoCo episodes), we observe a 4/5/3 HELP/FROZEN/HURT split. Extending the analysis to six harder cells and two AntMaze diagnostics reveals an actor-competence ceiling: medium-expert remains HURT in all 9 cells at every tested alpha, while AntMaze with a behavior-cloned frozen actor yields zero success for all composition rules. Overall, PoE and KL-regularized adaptation are best viewed as a single actor-anchored safety mechanism for deployment-time steering.
Elias Hossain, Mohammad Jahid Ibna Basher, Ivan Garibay +2
College of Engineering and Computer Science University of Central Florida, Orlando, FL, USA
Offline-to-online reinforcement learning uses offline pretraining to accelerate subsequent online learning. While many methods assume the offline dataset remains available throughout fine-tuning, there are many settings where this is not the case. Prior work attributes failures in this setting primarily to critic recalibration under distribution shift. We identify a complementary actor-side failure: once the offline data disappears, continued behavioral cloning (BC) no longer preserves the offline behavior distribution and instead refits the behavioral model to the agent's limited, and nonstationary, online experience. We demonstrate that stopping online behavioral fitting while preserving the pretrained expressive prior substantially improves no-retention fine-tuning across implicit flow Q-learning (IFQL), flow Q-learning (FQL), and action-chunked RL (QC) on seven OGBench domains. We trace this failure to a behavioral feedback loop: continuing online behavioral cloning rapidly erases the pretrained prior, inducing representational collapse and action diversity loss that starves the critic of useful exploration. Freezing the prior breaks this cycle, preserving offline coverage while allowing value learning to drive continuous improvement.
Sami Nur Islam, Johan Obando-Ceron, Doina Precup +2
Mila – Québec AI Institute · McGill University · Université de Montréal +1