Offline-Online Reinforcement Learning for Linear Mixture MDPs

Zhongjun Zhang; Sean R. Sinclair

cs.LGApr 13, 2026

Offline-Online Reinforcement Learning for Linear Mixture MDPs

Authors: Zhongjun Zhang, Sean R. Sinclair

Organizations: Department of Industrial Engineering and Management Sciences, Northwestern University

Abstract

We study offline-online reinforcement learning in linear mixture Markov decision processes (MDPs) under environment shift. In the offline phase, data are collected by an unknown behavior policy and may come from a mismatched environment, while in the online phase the learner interacts with the target environment. We propose an algorithm that adaptively leverages offline data. When the offline data are informative, either due to sufficient coverage or small environment shift, the algorithm provably improves over purely online learning. When the offline data are uninformative, it safely ignores them and matches the online-only performance. We establish regret upper bounds that explicitly characterize when offline data are beneficial, together with nearly matching lower bounds. Numerical experiments further corroborate our theoretical findings.

Figures & tables

\widehat{V}_{k,h}(s)\leftarrow\max_{a\in\mathcal{A}}\widehat{Q}_{k,h}(s,a),\qquad\widehat{Q}_{k,h}(s,a)\leftarrow r(s,a)+\max_{\theta\in\mathcal{B}_{k}\cap\mathcal{C}_{k}}\sum_{j=1}^{d}\theta_{j}\phi_{j}(\cdot\mid s,a)^{\top}\widehat{V}_{k,h+1}.

Algorithm 1 O - O UCRL - VTR

Figure 1: Graphical illustration of the sufficient conditions over the informativeness of offline data for

K=10^{3}

M^{\mathrm{off}}=1.5\times 10^{11}

\epsilon=10^{-3}

H=3

d=5

, and

B=10

. The admissible region is computed directly from Eq. 13 . When

\tau\lesssim 10^{-7}

, no value of

\Delta

yields informative offline data. Once

\tau

exceeds this threshold, larger

\tau

admits a larger environment shift

\Delta

, with an approximately linear dependence. The deviation from linearity near

\tau\approx 10^{-7}

is mainly caused by the logarithmic confidence term in Eq. 13 .

Panel of Figure 2 — Figure 2: Comparison of UCRL , COMPLETE , O - O UCRL - VTR ( Optimal ), and O - O UCRL - VTR ( Pessimistic ) under different environment shifts $\Delta$ , coverage $\tau$ , $K$ , and $M^{\mathrm{off}}$ .

Figure 3: Cumulative regret versus

\tau

for different

\Delta(\tau)

, with

K=4\times 10^{3}

and

M^{\mathrm{off}}=10^{5}

Problem setting specifications
Symbol	Definition
$\mathcal{S},\mathcal{A}$	State space and action space
$H,K,d$	Episode horizon, number of online episodes, and feature dimension of the linear mixture MDP
$\phi(s^{\prime}\mid s,a)$	Feature map $\phi:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\to\mathbb{R}^{d}$
$\phi_{j}(s^{\prime}\mid s,a)$	The $j$ th coordinate of the feature vector $\phi(s^{\prime}\mid s,a)$
$P^{\mathrm{off}},P$	Offline and online transition probability

Table 1: Common notation.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

\widehat{V}_{k,h}(s)\leftarrow\max_{a\in\mathcal{A}}\widehat{Q}_{k,h}(s,a),\qquad\widehat{Q}_{k,h}(s,a)\leftarrow r(s,a)+\max_{\theta\in\mathcal{B}_{k}^{\prime}}\sum_{j=1}^{d}\theta_{j}\phi_{j}(\cdot\mid s,a)^{\top}\widehat{V}_{k,h+1}.

Appendix

Algorithm 2 COMPLETE

Panel of Figure 4 — Figure 4: Supplementary results to Figure 2 , including DP-LSVI and UCBVI . The setting in Figure 4(a) matches that of Figure 2(a) , and the setting in Figure 4(b) matches that of Figure 2(b) .

Offline-To-Online RL

Offline-to-online reinforcement learning adapts policies learned from fixed datasets through further interaction with the environment.

See all 68 papers

A Unified Algorithmic Framework for Hybrid Reinforcement Learning in Tabular MDPs with Shifted Transition Dynamics
Jul 28, 2026Zheshun Wu, Renjie Zheng, Jinhang Zuo +2Non-Stationary RL Regret Minimization in RL
Southern University of Science and Technology · Harbin Institute of Technology, Shenzhen Shenzhen, Guangdong, China · Southern University of Science and Technology Shenzhen, Guangdong, China +4
Offline-to-Online Learning in Linear Bandits
Jun 3, 2026Kushagra Chandak, Toshinori Kitamura, Xiaoqi TanMulti-Armed Bandits Contextual Bandits
Department of Computing Science, University of Alberta, Canada
Information-Directed Offline-to-Online Reinforcement Learning
May 28, 2026Keru ChenRegret Minimization in RL Bayesian RL
School of Electrical, Computer and Energy Engineering, Arizona State University

Jul 28, 2026J/K move · Enter open · S save

A Unified Algorithmic Framework for Hybrid Reinforcement Learning in Tabular MDPs with Shifted Transition Dynamics

Zheshun Wu, Renjie Zheng, Jinhang Zuo, Zenglin Xu+1

Southern University of Science and Technology · Harbin Institute of Technology, Shenzhen Shenzhen, Guangdong, China · Southern University of Science and Technology Shenzhen, Guangdong, China+4

Non-Stationary RL Regret Minimization in RL Reinforcement Learning Offline-To-Online RL

Open paper AskPDF arXiv

This paper investigates a hybrid reinforcement learning setting in tabular Markov Decision Processes (MDPs), where an agent aims to learn an optimal policy by combining online interactions with a target environment and offline data from a source environment. A central challenge is that offline data may be collected from outdated environments with shifted transition dynamics, making naive integration of historical data ineffective. To address this, we propose a unified algorithmic framework featuring two algorithms: MIN-UCB-VI for regret minimization and MAX-LCB-VI for best policy identification. Both algorithms leverage fine-grained bias information to more effectively exploit offline data under general transition shifts. We provide theoretical guarantees for our framework, including both instance-dependent and independent upper bounds on regret and sub-optimality gap. Furthermore, we establish matching lower bounds to demonstrate the optimality of our approach and validate our theoretical findings through extensive experiments.

Abstract

Offline-To-Online RL

Explore similar work

A Unified Algorithmic Framework for Hybrid Reinforcement Learning in Tabular MDPs with Shifted Transition Dynamics

Offline-to-Online Learning in Linear Bandits

Information-Directed Offline-to-Online Reinforcement Learning