Provably Efficient Offline-to-Online Value Adaptation with General Function Approximation

Shangzhe Li; Weitong Zhang

cs.LGApr 15, 2026

Provably Efficient Offline-to-Online Value Adaptation with General Function Approximation

Authors: Shangzhe Li, Weitong Zhang

Organizations: UNC Chapel Hill

Abstract

We study value adaptation in offline-to-online reinforcement learning under general function approximation. Starting from an imperfect offline pretrained $Q$ -function, the learner aims to adapt it to the target environment using only a limited amount of online interaction. We first characterize the difficulty of this setting by establishing a minimax lower bound, showing that even when the pretrained $Q$ -function is close to optimal $Q^\star$ , online adaptation can be no more efficient than pure online RL on certain hard instances. On the positive side, under a novel structural condition on the offline-pretrained value functions, we propose O2O-LSVI, an adaptation algorithm with problem-dependent sample complexity that provably improves over pure online RL. Finally, we complement our theory with neural-network experiments that demonstrate the practical effectiveness of the proposed method.

Figures & tables

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 1: Results on Linear MDP. O2O-LSVI and LSVI-UCB with a fixed true action gap of

0.10

, reference tolerance

\beta_{\mathrm{ref}}=0.40

, and reference error

0.44

3

60

entries. Both learners use

d=5

features. (a) Mean cumulative pseudo-regret across five seeds, with a linear episode axis. (b) Mean paired regret difference; negative values favor O2O-LSVI. Shaded regions in (a) and (b) are pointwise

95\%

Student-

t

intervals with four degrees of freedom, not simultaneous confidence bands. (c) Mean fraction of entries within each reference class satisfying the containment test at a checkpoint; these are table-entry fractions, not visitation-weighted or cumulative rates. Panels (b) and (c) use logarithmic episode axes. Lines connect saved checkpoints; panel (a) also includes the known origin

R_{0}=0

Settings	Offline RL		Offline-to-Online Adaptation
Settings	CQL	IQL	Cal-QL	Ours
Umaze	94.0 $\pm$ 1.6	77.0 $\pm$ 0.7	76.8 $\pm$ 7.5 $\rightarrow$ 99.8 $\pm$ 0.4	85.8 $\pm$ 3.3 $\rightarrow$ 99.8 $\pm$ 0.4
Medium-Play	59.0 $\pm$ 11.2	71.8 $\pm$ 3.0	71.8 $\pm$ 3.3 $\rightarrow$ 98.8 $\pm$ 1.6	70.3 $\pm$ 2.0 $\rightarrow$ 99.3 $\pm$ 1.3
Large-Play	28.8 $\pm$ 7.8	38.5 $\pm$ 8.7	31.8 $\pm$ 8.9 $\rightarrow$ 97.3 $\pm$ 1.8	35.3 $\pm$ 4.0 $\rightarrow$ 98.5 $\pm$ 1.6

Appendix

Table 1: Empirical Results on AntMaze. We compare our method with existing offline-to-online adaptation and offline RL baselines on AntMaze. Our method achieves performance comparable to or better than existing baselines. Results are reported as D4RL scores averaged over four random seeds.

Hyperparameter	Value
Ensemble Size	5
$\beta$	50
CQL Conservative Coefficient $\alpha$ (Offline & Online)	5.0
Discount factor	0.99
Q Network Learning Rate	3e-4
Policy Network Learning Rate	1e-4

Appendix

Table 2: Hyperparameters. We demonstrate the hyperparameters used in the empirical experiment with the practical implementation of O2O-LSVI.

Explore similar work

Do You Really Need to Pretrain Q-Functions for Online RL Fine-Tuning?
Jul 29, 2026Perry Dong, Ron Polonsky, Dorsa Sadigh +1Continuous Control RL Fine-Tuning
Stanford University
Adaptive Policy Selection and Fine-Tuning under Interaction Budgets for Offline-to-Online Reinforcement Learning
May 6, 2026Alper Kamil Bozkurt, Xiaoan Xu, Shangtong Zhang +2Sequential Decision Making RL Fine-Tuning
Virginia Commonwealth University, Richmond, VA · Duke University, Durham, NC · University of Virginia, Charlottesville, VA
Critic-Free Pretraining for Efficient Online Reinforcement Learning Fine-Tuning
Aug 11, 2026Daoyi Li, Yixian Zhang, Wenbo Ding +2Reinforcement Learning Critic-Free RL
Tsinghua University

Jul 29, 2026, cs.LGJ/K move · Enter open · S save

Do You Really Need to Pretrain Q-Functions for Online RL Fine-Tuning?

Perry Dong, Ron Polonsky, Dorsa Sadigh, Chelsea Finn

Stanford University

Continuous Control RL Fine-Tuning Policy Learning Offline-To-Online RL

Open paper AskPDF arXiv

Pre-training followed by fine-tuning has become the dominant recipe for learning performant policies, and in value-based reinforcement learning (RL) this raises a natural question: given a pretrained policy, should the Q-function be pretrained on offline data too? Conventional wisdom suggests it should, but recent results show that online RL with a randomly-initialized Q-function can result in highly performant and reliable policies without needing to pretrain the Q-function. In this paper, we systematically study whether pretraining the Q-function actually helps when fine-tuning on top of a pretrained base policy. We find, surprisingly, that naive Q-function pretraining often provides little benefit over random initialization. We show this stems from a fundamental mismatch: the Q-function learned during pretraining targets the pretrained policy's Q-function, not the Q-function that online fine-tuning converges to, and this gap persists even after offline value maximization. Motivated by this finding, we propose Initialization via Policy Ensemble (IPE), a simple method that trains multiple diverse policies and uses their pooled rollouts to bootstrap the Q-function learning in online RL. Across a suite of challenging continuous control benchmarks, IPE yields an average 1.26x improvement in fine-tuning performance over naive Q-function pre-training.

Abstract

Explore similar work

Do You Really Need to Pretrain Q-Functions for Online RL Fine-Tuning?

Adaptive Policy Selection and Fine-Tuning under Interaction Budgets for Offline-to-Online Reinforcement Learning

Critic-Free Pretraining for Efficient Online Reinforcement Learning Fine-Tuning