ANCHOR: KL-Anchored Adaptation for Stable Offline-to-Online RL

Geonwoo Cho; Jaegyun Im; Doyoon Kim; Lexin Li

Figure 5: Hyperparameter and architectural tweaks do not consistently mitigate catastrophic failure. From top to bottom, the plots correspond to: batch size, actor learning rate, critic learning rate, critic hidden dimension, actor hidden dimension, LayerNorm in the critic, LayerNorm in the actor, and the duration of the offline pretraining phase. Across all settings and tasks, none of these factors consistently mitigates catastrophic failure.

Figure 6: SQOR and SQOG do not reliably predict catastrophic failure. We compare success rate, the spurious Q-optimism ratio (SQOR), and the spurious Q-optimism gap (SQOG) across four tasks. Both metrics show task-dependent and inconsistent alignment with early performance collapse. The vertical dashed line marks the timestep at which the largest performance drop occurs.

Figure 7: Q-update volatility does not reliably predict catastrophic failure. We compare success rate and critic-update volatility across four tasks. Volatility shows weak and inconsistent correspondence with early performance collapse.

Figure 8: Sensitivity to batch size and UTD ratio. We vary the batch size (top) and UTD ratio (bottom) across three tasks. ANCHOR is generally robust to these choices, although larger batch size and larger UTD can increase learning instability.

Figure 9: Effect of the initial KL weight. A larger initial KL weight (

\beta_{0}=2.0

) produces lower KL loss early in online fine-tuning, indicating that the KL penalty more strongly constrains deviation from the offline policy. As the KL coefficient is annealed, the gap between

\beta_{0}=0.5

and

\beta_{0}=2.0

gradually narrows.

Figure 10: Applying KL anchoring to IQL. Adding KL anchoring with annealing to IQL reduces the initial performance drop, but does not improve asymptotic performance over the IQL baseline.

Figure 11: Additional environments reveal limitations of KL-anchored adaptation. In kitchen-complete, ANCHOR remains competitive despite near-zero initial success. In relocate-binary, all methods fail to make meaningful progress. In antmaze-ultra-diverse, most methods do not improve over the offline-pretrained policy, suggesting that harder sparse-reward exploration requires mechanisms beyond KL anchoring alone.

Figure 12: Stronger preservation is insufficient in antmaze-ultra-diverse. Keeping both the conservative coefficient and KL coefficient fixed yields a higher mean success rate than ANCHOR during online fine-tuning, but does not significantly improve asymptotic performance.

ANCHOR
	AntMaze		Adroit		Kitchen
	large diverse	large play	door	pen	partial	mixed
$\alpha_{\mathrm{on}}$	0.0	0.0	0.0	0.0	1.0	1.0
$T_{\mathrm{KL}}$	80,000	80,000	40,000	80,000	160,000	160,000
$\beta_{0}$	0.4	0.2	0.2	1.0	1.0	1.0
Optimization

Appendix

Table 1: Hyperparameters of ANCHOR.

Task	ANCHOR (Ours)	WSRL	PORL	Cal-QL	CQL	IQL	SAC
antmaze-large-diverse-v2	2.75 $\pm$ 0.02	2.73 $\pm$ 0.01	3.35 $\pm$ 0.07	3.44 $\pm$ 0.01	2.82 $\pm$ 0.02	2.08 $\pm$ 0.02	1.53 $\pm$ 0.0
antmaze-large-play-v2	2.75 $\pm$ 0.01	3.48 $\pm$ 0.08	3.54 $\pm$ 0.06	3.26 $\pm$ 0.03	2.34 $\pm$ 0.03	2.09 $\pm$ 0.03	1.52 $\pm$ 0.0
kitchen-partial-v0	4.31 $\pm$ 0.02	3.37 $\pm$ 0.0	3.41 $\pm$ 0.03	4.35 $\pm$ 0.01	3.28 $\pm$ 0.05	1.41 $\pm$ 0.01	1.38 $\pm$ 0.0
kitchen-mixed-v0	3.88 $\pm$ 0.03	3.61 $\pm$ 0.01	3.51 $\pm$ 0.13	4.36 $\pm$ 0.0	3.33 $\pm$ 0.02	1.46 $\pm$ 0.02	1.36 $\pm$ 0.0
door-binary-v0	3.07 $\pm$ 0.01	2.9 $\pm$ 0.01	2.93 $\pm$ 0.01	3.55 $\pm$ 0.01	2.43 $\pm$ 0.01	1.28 $\pm$ 0.02	0.78 $\pm$ 0.01
pen-binary-v0	2.33 $\pm$ 0.02	2.77 $\pm$ 0.01	2.79 $\pm$ 0.01	3.51 $\pm$ 0.01	2.29 $\pm$ 0.01	1.13 $\pm$ 0.1	0.67 $\pm$ 0.0

Appendix

Table 2: Online training time (in decimal hours) comparison across baselines.

Figure 13: Representative episodes across nine tasks. Each row shows temporally ordered frames from a trajectory, demonstrating navigation (antmaze-ultra-diverse-v2, antmaze-large-diverse-v2, antmaze-large-play-v2), kitchen manipulation (kitchen-partial-v0, kitchen-mixed-v0, kitchen-complete-v0), and dexterous control (pen-binary-v0, door-binary-v0, relocate-binary-v0).

Task	ANCHOR (Ours)	WSRL	PORL	Cal-QL	CQL	IQL	SAC
antmaze-large-diverse-v2	0.97 $\pm$ 0.01	0.68 $\pm$ 0.17	0.9 $\pm$ 0.07	0.97 $\pm$ 0.01	0.92 $\pm$ 0.05	0.06 $\pm$ 0.05	0.0 $\pm$ 0.0
antmaze-large-play-v2	0.96 $\pm$ 0.02	0.95 $\pm$ 0.02	0.47 $\pm$ 0.2	0.9 $\pm$ 0.07	0.95 $\pm$ 0.03	0.01 $\pm$ 0.01	0.0 $\pm$ 0.0
kitchen-partial-v0	1.0 $\pm$ 0.0	0.93 $\pm$ 0.06	0.91 $\pm$ 0.06	0.7 $\pm$ 0.09	0.44 $\pm$ 0.12	0.4 $\pm$ 0.05	0.49 $\pm$ 0.07
kitchen-mixed-v0	0.81 $\pm$ 0.09	0.72 $\pm$ 0.11	0.88 $\pm$ 0.06	0.45 $\pm$ 0.12	0.4 $\pm$ 0.06	0.31 $\pm$ 0.09	0.3 $\pm$ 0.05
door-binary-v0	1.0 $\pm$ 0.0	0.8 $\pm$ 0.2	0.6 $\pm$ 0.24	0.19 $\pm$ 0.19	0.18 $\pm$ 0.18	0.0 $\pm$ 0.0	0.2 $\pm$ 0.2
pen-binary-v0	1.0 $\pm$ 0.0	0.97 $\pm$ 0.01	0.99 $\pm$ 0.01	0.94 $\pm$ 0.02	0.81 $\pm$ 0.09	0.33 $\pm$ 0.1	0.78 $\pm$ 0.04

Appendix

Table 3: Performance at 400k online steps. Bold indicates the best performance per task, and underline indicates the second best.

Task	ANCHOR (Ours)	WSRL	PORL	Cal-QL	CQL	IQL	SAC
antmaze-large-diverse-v2	0.97 $\pm$ 0.01	0.73 $\pm$ 0.18	0.84 $\pm$ 0.07	0.95 $\pm$ 0.03	0.96 $\pm$ 0.02	0.03 $\pm$ 0.03	0.0 $\pm$ 0.0
antmaze-large-play-v2	0.96 $\pm$ 0.02	0.98 $\pm$ 0.01	0.37 $\pm$ 0.23	0.89 $\pm$ 0.07	0.95 $\pm$ 0.03	0.01 $\pm$ 0.01	0.0 $\pm$ 0.0
kitchen-partial-v0	1.0 $\pm$ 0.0	0.95 $\pm$ 0.05	0.9 $\pm$ 0.06	0.7 $\pm$ 0.09	0.44 $\pm$ 0.12	0.39 $\pm$ 0.05	0.47 $\pm$ 0.06
kitchen-mixed-v0	0.83 $\pm$ 0.1	0.7 $\pm$ 0.12	0.88 $\pm$ 0.06	0.45 $\pm$ 0.12	0.4 $\pm$ 0.06	0.3 $\pm$ 0.08	0.25 $\pm$ 0.03
door-binary-v0	0.97 $\pm$ 0.01	0.8 $\pm$ 0.2	0.6 $\pm$ 0.24	0.17 $\pm$ 0.17	0.19 $\pm$ 0.19	0.0 $\pm$ 0.0	0.2 $\pm$ 0.2
pen-binary-v0	0.94 $\pm$ 0.02	0.96 $\pm$ 0.02	0.95 $\pm$ 0.02	0.96 $\pm$ 0.01	0.72 $\pm$ 0.05	0.4 $\pm$ 0.05	0.84 $\pm$ 0.02

Appendix

Table 4: Performance at 350k online steps. Bold indicates the best performance per task, and underline indicates the second best.

Task	ANCHOR (Ours)	WSRL	PORL	Cal-QL	CQL	IQL
antmaze-large-diverse-v2	0.33 $\pm$ 0.09	0.24 $\pm$ 0.05	0.39 $\pm$ 0.04	0.26 $\pm$ 0.04	0.26 $\pm$ 0.05	0.03 $\pm$ 0.02
antmaze-large-play-v2	0.29 $\pm$ 0.05	0.29 $\pm$ 0.02	0.37 $\pm$ 0.02	0.24 $\pm$ 0.04	0.28 $\pm$ 0.05	0.0 $\pm$ 0.0
kitchen-partial-v0	0.7 $\pm$ 0.03	0.73 $\pm$ 0.06	0.7 $\pm$ 0.04	0.68 $\pm$ 0.05	0.7 $\pm$ 0.06	0.3 $\pm$ 0.06
kitchen-mixed-v0	0.47 $\pm$ 0.06	0.47 $\pm$ 0.06	0.49 $\pm$ 0.07	0.52 $\pm$ 0.06	0.58 $\pm$ 0.03	0.4 $\pm$ 0.04
door-binary-v0	0.26 $\pm$ 0.04	0.31 $\pm$ 0.19	0.24 $\pm$ 0.03	0.21 $\pm$ 0.07	0.08 $\pm$ 0.08	0.05 $\pm$ 0.04
pen-binary-v0	0.69 $\pm$ 0.03	0.66 $\pm$ 0.05	0.74 $\pm$ 0.03	0.75 $\pm$ 0.05	0.58 $\pm$ 0.03	0.89 $\pm$ 0.04

Appendix

Table 5: Performance after the offline phase.

Table 6: Catastrophic failure (0-400K window).

Figure 14: Benefit of ANCHOR depends on pretrained policy quality. ANCHOR reduces catastrophic failure more clearly when the offline-pretrained policy starts from a higher initial success rate. Curves with the same color correspond to the same pretrained policy. Results are averaged over three seeds.

Figure 15: Common stabilization strategies do not prevent catastrophic failure. Effects of varying standard online fine-tuning choices across four tasks. Top : UTD ratio; Middle : warmup length; Bottom : Cal-QL conservative-regularizer weight,

\alpha_{\mathrm{Cal\text{-}QL}}

. Increasing UTD, extending warmup, or tuning

\alpha_{\mathrm{Cal\text{-}QL}}

does not reliably prevent early performance collapse.

Figure 16: Critic-shift regularization does not prevent catastrophic failure. Effects of varying the critic-shift penalty strength

\alpha_{\mathrm{KL}}

across four tasks. Top : critic distribution shift during online fine-tuning, measured by

D_{\mathrm{KL}}\big(\mathrm{softmax}(Q_{\mathrm{off}})\,\|\,\mathrm{softmax}(Q_{\theta})\big)

. Bottom : corresponding success rates. Increasing

\alpha_{\mathrm{KL}}

slows the critic shift, but does not reliably prevent early performance collapse.

Figure 17: Hyperparameter sensitivity of ANCHOR. We vary the initial KL weight

\beta_{0}

(top), the KL annealing interval

T_{\mathrm{KL}}

(middle), and the residual online conservative weight

\alpha_{\mathrm{on}}

(bottom). Here,

\alpha^{*}

denotes an intermediate value between

0

and

\alpha_{\mathrm{off}}

, where

\alpha_{\mathrm{off}}

is the conservative regularization weight used during offline pretraining. ANCHOR remains stable under moderate changes to KL-related hyperparameters, while

\alpha_{\mathrm{on}}

controls the balance between early stability and final improvement.

Task	Metric	PEX	PROTO	ITD3-BC	ANCHOR
antmaze-large-play	Final	$0.91\pm.02$	$0.97\pm.02$	$0.52\pm.11$	$0.96\pm.02$
	CF	$0.13\pm.03$	$0.07\pm.04$	$0.08\pm.04$	$0.11\pm.09$
kitchen-partial	Final	$0.58\pm.11$	$0.97\pm.02$	$0.89\pm.06$	$1.00\pm.00$
	CF	$0.71\pm.07$	$0.47\pm.11$	$0.69\pm.07$	$0.06\pm.02$
pen-binary	Final	$0.94\pm.04$	$0.95\pm.03$	$0.86\pm.06$	$1.00\pm.00$
	CF	$0.49\pm.01$	$0.46\pm.06$	$0.15\pm.03$	$0.14\pm.04$

Appendix

Table 8: Policy-side adaptations under CQL initialization. Five-seed mean

\pm

SE; final success is higher-is-better and 0–100K CF is lower-is-better. ITD3-BC abbreviates the Improving TD3-BC-style adaptation.

Task	Annealed BC $1\to 0$	Annealed BC $10\to 0$	ANCHOR
antmaze-large-play	$.94\pm.02\,/\,.23\pm.04$	$.85\pm.05\,/\,.31\pm.05$	$.96\pm.02\,/\,.11\pm.09$
kitchen-partial	$.99\pm.01\,/\,.69\pm.07$	$.99\pm.01\,/\,.11\pm.06$	$1.00\pm.00\,/\,.06\pm.02$
pen-binary	$.95\pm.02\,/\,.62\pm.07$	$.98\pm.01\,/\,.15\pm.06$	$1.00\pm.00\,/\,.14\pm.04$
Task	Fixed BC 1	Fixed BC 10	ANCHOR
kitchen-partial	$.96\pm.02\,/\,.68\pm.10$	$.57\pm.08\,/\,.22\pm.08$	$1.00\pm.00\,/\,.06\pm.02$
pen-binary	$.97\pm.02\,/\,.57\pm.03$	$.93\pm.02\,/\,.07\pm.04$	$1.00\pm.00\,/\,.14\pm.04$

Appendix

Table 9: Five-seed BC controls. Each cell gives final success / 0–100K CF, both as mean

\pm

SE. Data access and critic schedules are detailed in Appendix N.3 .

Metric	ANCHOR	Strongest baseline
Final success at 400K	0.993 [0.981, 1.000]	WSRL: 0.944 [0.879, 0.972]
CF over 0–100K	0.081 [0.048, 0.123]	IQL: 0.135 [0.104, 0.171]

Appendix

Table 10: Aggregate IQM and 95% bootstrap confidence intervals. The baseline in each row is the strongest by that aggregate metric.

Baseline	Final success at 400K	CF over 0–100K
WSRL	0.710 [0.607, 0.813]	0.767 [0.653, 0.880]
PORL	0.657 [0.553, 0.763]	0.857 [0.763, 0.940]
Cal-QL	0.790 [0.693, 0.880]	0.747 [0.627, 0.853]
CQL	0.853 [0.763, 0.937]	0.757 [0.653, 0.860]
IQL	1.000 [1.000, 1.000]	0.590 [0.513, 0.667]
SAC	0.983 [0.950, 1.000]	—

Appendix

Table 11: Probability of improvement of ANCHOR, with 95% bootstrap confidence intervals. These compare aggregate distributions across the six tasks.

Task	Metric	WSRL	$\alpha$ -only	Annealed BC	ANCHOR
kitchen-mixed	Novelty	$.85\pm.01$	$.79\pm.02$	$.77\pm.01$	$.44\pm.01$
	CF	$.61\pm.02$	$.57\pm.03$	$.58\pm.03$	$.03\pm.01$
door-binary	Novelty	$.98\pm.02$	$.96\pm.03$	$.96\pm.03$	$.95\pm.04$
	CF	$.06\pm.06$	$.05\pm.05$	$.08\pm.08$	$.07\pm.07$

Appendix

Table 12: Five-seed novelty and failure diagnostics. Novelty is measured over 0–30K; CF over 0–100K for kitchen-mixed and 0–60K for door-binary. Novelty is a visitation statistic, not an objective to minimize.

Backbone	Task	Base final	+A final	Base CF	+A CF
FQL	scene	$1.00\pm.00$	$1.00\pm.00$	$0.26\pm.06$	$0.16\pm.07$
FQL	puzzle-4x4	$0.40\pm.22$	$0.40\pm.22$	$0.07\pm.01$	$0.09\pm.01$
IFQL	humanoidmaze-medium	$0.25\pm.07$	$0.92\pm.01$	$0.71\pm.02$	$0.19\pm.08$
IFQL	antsoccer-arena	$0.08\pm.05$	$0.40\pm.04$	$0.18\pm.03$	$0.21\pm.06$

Appendix

Table 13: OGBench extension, five seeds (mean

\pm

SE). “+A” uses the implicit-policy proxy in Equation 10 . Short task names are expanded in Appendix N.6 . CF is measured over 0–100K.

Task	Replay, no $\alpha$	No replay, no $\alpha$	Replay, with $\alpha$
kitchen-partial	$.69\pm.07\,/\,.76\pm.08$	$.68\pm.06\,/\,.60\pm.06$	$.58\pm.10\,/\,.62\pm.13$
kitchen-mixed	$.49\pm.06\,/\,.70\pm.02$	$.49\pm.07\,/\,.41\pm.13$	$.48\pm.07\,/\,.70\pm.05$
door-binary	$.15\pm.04\,/\,1.00\pm.00$	$.25\pm.03\,/\,.79\pm.20$	$.28\pm.03\,/\,.74\pm.19$
pen-binary	$.65\pm.06\,/\,.93\pm.03$	$.61\pm.04\,/\,.89\pm.03$	$.28\pm.09\,/\,.94\pm.03$
Average	$.50\pm.06\,/\,.85\pm.03$	$.51\pm.04\,/\,.67\pm.07$	$.40\pm.05\,/\,.75\pm.06$

Appendix

Table 14: Replay/conservatism control, five seeds. Cells give CF / final success (mean

\pm

SE). The average row is the reported aggregate.

Task	$T_{\mathrm{KL}}=40\mathrm{K}$	$T_{\mathrm{KL}}=160\mathrm{K}$
antmaze-large-play	$.90\pm.05\,/\,.05\pm.00$	$1.00\pm.00\,/\,.12\pm.07$
kitchen-partial	$1.00\pm.00\,/\,.03\pm.01$	$.99\pm.01\,/\,.04\pm.03$
pen-binary	$1.00\pm.00\,/\,.35\pm.10$	$1.00\pm.00\,/\,.18\pm.12$

Appendix

Table 15: Two-seed shared-weight duration check. Cells give final success / 0–100K CF, mean

\pm

SE. The initial-weight value is unresolved in the retained experiment record.

ANCHOR: KL-Anchored Adaptation for Stable Offline-to-Online RL

Abstract

Figures & tables

Appendix

Offline-To-Online RL

Critic-Free Pretraining for Efficient Online Reinforcement Learning Fine-Tuning

When Policies Cannot Be Retrained: A Unified Closed-Form View of Post-Training Steering in Offline Reinforcement Learning

Freeze, Don't Clone: Fine-Tuning with Expressive Policies Without Retaining Offline Data

Abstract

Offline-To-Online RL

Explore similar work

Critic-Free Pretraining for Efficient Online Reinforcement Learning Fine-Tuning

When Policies Cannot Be Retrained: A Unified Closed-Form View of Post-Training Steering in Offline Reinforcement Learning

Freeze, Don't Clone: Fine-Tuning with Expressive Policies Without Retaining Offline Data