Robust Transfer Learning with Side Information

Akram S. Awad; Shihab Ahmed; Yue Wang; George K. Atia

stat.MLMar 9, 2026

Robust Transfer Learning with Side Information

Authors: Akram S. Awad, Shihab Ahmed, Yue Wang, George K. Atia

Organizations: Department of Electrical and Computer Engineering, University of Central Florida, USA · Department of Computer Science, University of Central Florida, USA

Abstract

Robust Markov Decision Processes (MDPs) address environmental shift through distributionally robust optimization (DRO) by finding an optimal worst-case policy within an uncertainty set of transition kernels. However, standard DRO approaches require enlarging the uncertainty set under large shifts, which leads to overly conservative and pessimistic policies. In this paper, we propose a framework for transfer under environment shift that derives a robust target-domain policy via estimate-centered uncertainty sets, constructed through constrained estimation that integrates limited target samples with side information about the source-target dynamics. The side information includes bounds on feature moments, distributional distances, and density ratios, yielding improved kernel estimates and tighter uncertainty sets. The side information includes bounds on feature moments, distributional distances, and density ratios, yielding improved kernel estimates and tighter uncertainty sets. Error bounds and convergence results are established for both robust and non-robust value functions. Moreover, we provide a finite-sample guarantee on the learned robust policy and analyze the robust sub-optimality gap. Under mild low-dimensional structure on the transition model, the side information reduces this gap and improves sample efficiency. We assess the performance of our approach across OpenAI Gym environments and classic control problems, consistently demonstrating superior target-domain performance over state-of-the-art robust and non-robust baselines.

Figures & tables

Panel of Figure 1 — Figure 1: (a) Environment shift: The source and target domain environments are relatively distant. (b) Over-conservative case: the source uncertainty set’s radius is enlarged to include the target domain, which leads to an overly conservative policy. (c) Our approach : We construct the uncertainty set around the estimated target dynamics, which are closer to the true target dynamics and therefore the set requires a smaller radius.

Distance IBE	Density IBE	Moment IBE	LDS IBE
$\max_{q\in\Delta(\mathcal{S})}\sum_{s^{\prime}\in\mathcal{S}}N_{s,a}(s^{\prime})\log q(s^{\prime})$ s.t. $\quad\text{dist}(q,\mathsf{P}^{s,a}_{\text{s}})\leq d_{s,a}$	$\max_{q\in\Delta(\mathcal{S})}\sum_{s^{\prime}\in\mathcal{S}}N_{s,a}(s^{\prime})\log q(s^{\prime})$ s.t. $q/\mathsf{P}^{s,a}_{\text{s}}\leq B_{s,a}$	$\max_{q\in\Delta(\mathcal{S})}\sum_{s^{\prime}\in\mathcal{S}}N_{s,a}(s^{\prime})\log q(s^{\prime})$ s.t. $\|\mu(q)-\mu(\mathsf{P}^{s,a}_{\mathrm{s}})\|\leq\beta_{s,a}$	$\max_{\theta\in\mathbb{R}^{\mathsf{d_{0}}}}\sum_{s^{\prime}\in\mathcal{S}}N_{s,a}(s^{\prime})\log\mathsf{P}^{s,a}_{\theta}(s^{\prime})$

Table 1: Information-Based Estimation (IBE) under different side information

\Phi

Panel of Figure 2 — Figure 2: Target domain performance for the non-robust setting as a function of sample size for CartPole . The legend is shared between Figures 2 and 3 .

Panel of Figure 3 — Figure 3: Target domain performance for the robust setting as a function of sample size for CartPole .

Panel of Figure 4 — Figure 4: Suboptimality gap as a function of sample size $(N)$ (log-log scale) in the CartPole environment for LDS-IBE, for non-robust (left) and robust (right) scenarios.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Transition kernel	Description
$\mathsf{P}^{s,a}_{\text{s}}(s_{intended})$	The agent moves to the state in the intended direction with probability $(1-\alpha)(r_{s}+2\frac{(1-r_{s})}{\mathcal{\|S\|}})$
$\mathsf{P}^{s,a}_{\text{s}}(s_{opposite})$	The agent may move to the state opposite the intended direction with probability $\alpha(r_{s}+2\frac{(1-r_{s})}{\mathcal{\|S\|}})$
$\mathsf{P}^{s,a}_{\text{s}}(s_{other})$	The agent may move to all other states with probability $\frac{(1-r_{s})}{\mathcal{\|S\|}}$
$\mathsf{P}^{s,a}_{\text{t}}(s_{intended})$	The agent moves to the state in the intended direction with probability $\alpha(r_{t}+2\frac{(1-r_{t})}{\mathcal{\|S\|}})$
$\mathsf{P}^{s,a}_{\text{t}}(s_{opposite})$	The agent may move to the state opposite the intended direction with probability $(1-\alpha)(r_{t}+2\frac{(1-r_{t})}{\mathcal{\|S\|}})$
$\mathsf{P}^{s,a}_{\text{t}}(s_{other})$	The agent may move to all other states with probability $\frac{(1-r_{t})}{\mathcal{\|S\|}}$

Appendix

Table 2: A Description of the source and target domain transition kernels for toy-text environments.

Panel of Figure 5 — Figure 5: A depiction OpenAI Gym toy text environments: (a) Frozen Lake environment (b) Taxi environment (c) Cliff Walking environment

Transition kernel	Description
$\mathsf{P}^{s,a}_{\text{s}}(s_{random_{1}})$	The agent moves to a random state $s_{random_{1}}$ with probability $\alpha r_{s}$
$\mathsf{P}^{s,a}_{\text{s}}(s_{random_{2}})$	The agent moves to a random state $s_{random_{1}}$ with probability $(1-\alpha)r_{s}$
$\mathsf{P}^{s,a}_{\text{s}}(s_{other})$	The agent may move to all other states with probability with probability $\frac{r_{s}}{\|\mathcal{S}\|}$ .
$\mathsf{P}^{s,a}_{\text{t}}(s_{random_{1}})$	The agent moves to a random state $s_{random_{1}}$ with probability $(1-\alpha)r_{t}$
$\mathsf{P}^{s,a}_{\text{t}}(s_{random_{2}})$	The agent moves to a random state $s_{random_{1}}$ with probability $\alpha r_{t}$
$\mathsf{P}^{s,a}_{\text{t}}(s_{other})$	The agent may move to all other states with probability with probability $\frac{r_{t}}{\|\mathcal{S}\|}$ .

Appendix

Table 3: A Description of the source and target domain transition kernels for classic control environments.

Panel of Figure 6 — Figure 6: A depiction OpenAI Gym classic control problems: (a) Cart Pole. (b) Acrobot. (c) Pendulum.

Panel of Figure 7 — Figure 7: Average total variation distance between the estimated and the target domain transition kernels over $20$ runs as a function of sample size for the toy text environments.

Panel of Figure 8 — Figure 8: Target domain performance for the non-robust setting averaged over $20$ runs as a function of sample size for Cliff Walking environment, and the corresponding $95\%$ confidence intervals.

Panel of Figure 9 — Figure 9: Target domain performance for the non-robust setting averaged over $20$ runs as a function of sample size for Taxi environment, and the corresponding $95\%$ confidence intervals.

Panel of Figure 10 — Figure 10: Target domain performance for the non-robust setting averaged over $20$ runs as a function of sample size for Frozen Lake environment, and the corresponding $95\%$ confidence intervals.

Panel of Figure 11 — Figure 11: Target domain performance for the non-robust setting averaged over $20$ runs as a function of sample size for Acrobot environment, and the corresponding $95\%$ confidence intervals.

Panel of Figure 12 — Figure 12: Target domain performance for the non-robust setting averaged over $20$ runs as a function of sample size for Pendulum environment, and the corresponding $95\%$ confidence intervals.

Panel of Figure 13 — Figure 13: Target domain performance for the robust setting averaged over $20$ runs as a function of sample size for Cliff Walking environment, and the corresponding $95\%$ confidence intervals.

Panel of Figure 14 — Figure 14: Target domain performance for the robust setting averaged over $20$ runs as a function of sample size for Taxi environment, and the corresponding $95\%$ confidence intervals.

Panel of Figure 15 — Figure 15: Target domain performance for the robust setting averaged over $20$ runs as a function of sample size for Frozen Lake environment, and the corresponding $95\%$ confidence intervals.

Panel of Figure 16 — Figure 16: Target domain performance for the robust setting averaged over $20$ runs as a function of sample size for Acrobot environment, and the corresponding $95\%$ confidence intervals.

Panel of Figure 17 — Figure 17: Target domain performance for the robust setting averaged over $20$ runs as a function of sample size for Pendulum environment, and the corresponding $95\%$ confidence intervals.

Figure 18: The Evaluation (EV) Error and the bound for each MLE method as a function of the sample size for the non-robust scenario

Figure 19: The Evaluation (EV) Error and the bound for each IBE method as a function of the sample size for the robust setting.

Figure 20: The trace of the FIM as a function of the sample size $n$ (left) and its inverse (right).

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,717 papers

Robust Regularized Policy Iteration under Transition Uncertainty
Mar 10, 2026Hongqiang Lin, Zhenghui Fu, Weihao Tang +4Offline RL Robust RL
College of Computer Science and Technology, Zhejiang University, Hangzhou, China · School of Artificial Intelligence, Optics and Electronics (iOPEN), Northwestern Polytechnical University, Xi’an, China · School of Software Technology, Zhejiang University, Hangzhou, China +2
Statistical Properties of Robust Learning under Distributional Shifts
Aug 13, 2026Zhiyi Li, Xiaojie Mao, Yunbei Xu +1Distribution Shift Robustness Distributionally Robust Optimization
National University of Singapore · Tsinghua University · University College London
Robust Average-Reward Markov Decision Processes: Minimax-Optimal Learning via Plug-in Reductions
Aug 6, 2026Yuepeng Yang, Yuxin Chen, Yuejie ChiRobust Markov Decision Processes Average-Reward RL
*Department of Statistics and Data Science, Yale University. · †Department of Statistics and Data Science, the Wharton School, University of Pennsylvania.

Mar 10, 2026J/K move · Enter open · S save

Robust Regularized Policy Iteration under Transition Uncertainty

Hongqiang Lin, Zhenghui Fu, Weihao Tang, Pengfei Wang+3

College of Computer Science and Technology, Zhejiang University, Hangzhou, China · School of Artificial Intelligence, Optics and Electronics (iOPEN), Northwestern Polytechnical University, Xi’an, China · School of Software Technology, Zhejiang University, Hangzhou, China+2

Offline RL Robust RL Distributionally Robust RL

Open paper AskPDF arXiv

Offline reinforcement learning (RL) enables data-efficient and safe policy learning without online exploration, but its performance often degrades under distribution shift. The learned policy may visit out-of-distribution state-action pairs where value estimates and learned dynamics are unreliable. To address policy-induced extrapolation and transition uncertainty in a unified framework, we formulate offline RL as robust policy optimization, treating the transition kernel as a decision variable within an uncertainty set and optimizing the policy against the worst-case dynamics. We propose Robust Regularized Policy Iteration (RRPI), which replaces the intractable max-min bilevel objective with a tractable KL-regularized surrogate and derives an efficient policy iteration procedure based on a robust regularized Bellman operator. We provide theoretical guarantees by showing that the proposed operator is a

γ

-contraction and that iteratively updating the surrogate yields monotonic improvement of the original robust objective with convergence. Experiments on D4RL benchmarks demonstrate that RRPI achieves strong average performance, outperforming recent baselines including percentile-based methods on the majority of environments while remaining competitive on the rest. Moreover, RRPI exhibits robust performance by aligning lower

Q

-values with high epistemic uncertainty, which prevents the policy from executing unreliable out-of-distribution actions.

Abstract

Reinforcement Learning

Explore similar work

Robust Regularized Policy Iteration under Transition Uncertainty

Statistical Properties of Robust Learning under Distributional Shifts

Robust Average-Reward Markov Decision Processes: Minimax-Optimal Learning via Plug-in Reductions