stat.MLNov 1, 2026

Direct Knowledge Transfer for Offline Q-Learning

Authors: Zhanye Luo, Guangyan Gan, Lin William Cong

Organizations: Faculty of Business and Economics, The University of Hong Kong · Nanyang Business School, Nanyang Technological University, Singapore

Abstract

Data-driven sequential decision making in marketing and healthcare can benefit from observations collected in related environments, particularly when data in the environment of interest are limited. Transferring such information requires accounting for differences in rewards and transition dynamics across environments. We study knowledge transfer for offline reinforcement learning through a joint estimation framework that directly integrates source information into target value-function learning. We propose Direct Transfer Fitted Q-Iteration (DFQI), which updates the target Q-function while simultaneously borrowing the source information through an L1 penalty that links the target coefficients to a weighted aggregate of source coefficients, with both determined jointly within each Bellman update. Using sieve approximation, we establish statistical guarantees for target estimation and policy learning in stationary discounted environments with common or heterogeneous transition kernels. Empirical studies further demonstrate the effectiveness of the proposed DFQI framework in transferring source information to improve policy learning with limited target data.

Figures & tables

Explore similar work

Nov 7, 2026

Bayesian transfer for offline Q-learning with multi-task Gaussian processes

Offline reinforcement learning can benefit from data collected in related tasks, yet indiscriminate knowledge transfer may lead to negative transfer under task mismatch and limited target-data coverage. We propose a target-aware Bayesian transfer framework for offline Q-learning based on multi-task Gaussian processes that jointly model task-specific Q-functions. Source tasks are assessed using target predictive evidence and maximum mean discrepancy (MMD), while posterior uncertainty and behavior-supported action sets guide conservative Bellman updates and policy extraction. To account for the dependence induced by Bellman bootstrapping, we distinguish Bellman innovations from bootstrap errors and establish stability and predictive source-selection results under explicit regularity conditions. We further derive selection-adaptive posterior contraction and offline policy-performance bounds, identifying the structural conditions under which source information can be pooled to achieve minimax-optimal rates in a shared-function Gaussian sequence experiment. Simulation studies with heterogeneous source tasks demonstrate improved target Q-function estimation and reduced material negative transfer compared with indiscriminate source pooling. An application to human decision-making data further shows higher average expected immediate payoff than both target-only learning and matched-size random source selection. These results highlight the benefits of combining target-aware source selection with uncertainty-aware conservative learning for reliable and sample-efficient offline reinforcement learning.
May 27, 2026

Reward Transfer from Inverse Reinforcement Learning: A Coupled Minimax Approach

We study the transfer of rewards learned using inverse reinforcement learning from expert demonstrations in one environment to reinforcement learning in a new, different environment. This arises naturally when demonstrations are collected in a controlled environment. We formulate the problem as a joint system of Bellman equations across the source and target environments and develop minimax estimators for the target soft-qq-function. Whereas a sequential solution approach first estimates the source reward and then plugs it into the target control problem, a coupled approach solves the source and target system of equations jointly. We show that, in contrast to the sequential approach, the coupled approach removes the first-order influence of source Bellman residual error. We characterize the local behavior of each approach, develop finite-sample soft-qq-function error bounds, and prove regret guarantees for the resulting soft-control policy. An empirical investigation using a sepsis simulator validates the theoretical comparison.
Aug 14, 2026

Offline Deep Q* Estimation with Diffusion Models

In offline RL, estimating the optimal action-value function Q∗Q^* can be formulated as solving the optimal Bellman equation based solely on offline observations. A fundamental challenge is that the reward function and transition kernel are unknown, so the optimal Bellman operator is not directly observable from data. To address this issue, we propose a novel framework that decouples operator estimation from value function learning. In this approach, we first formulate conditional diffusion models to estimate the reward law and transition kernel, which induces a data-driven approximation of the optimal Bellman operator. We then plug these estimators into the Bellman equation and obtain a deep estimator of Q∗Q^* by minimizing the empirical Bellman residual over a neural network function class. Theoretically, we first establish sharp nonasymptotic convergence rates for learning the optimal Bellman operator. We establish the smoothness of Q∗Q^* through the Bellman equation using the the Hölder assumptions on the conditional reward and transition densities. With suitable sample sizes, we obtain the oracle value-stage rate O~(n−2αdx+da+2α)\widetilde{\mathcal O}\bigl(n^{-\frac{2α}{d_x+d_a+2α}}\bigr) for the excess Bellman residual risk. We then translate this risk bound into an expected L2L^2 estimation error of order O~(n−αdx+da+2α)\widetilde{\mathcal O}\bigl(n^{-\fracα{d_x+d_a+2α}}\bigr) under a concentrability condition. For the total offline budget n~=n+nr+nx\widetilde n=n+n_r+n_x, we obtain a value estimation rate of O~(n~−α2dx+da+2α)\widetilde{\mathcal O}\bigl(\widetilde n^{-\fracα{2d_x+d_a+2α}}\bigr) and establish an information-theoretic lower bound of order n~−αdx+da+2α\widetilde n^{-\fracα{d_x+d_a+2α}}. Importantly, our theoretical analysis does not rely on Bellman completeness assumptions commonly used in deep RL theory.