Cross-domain reinforcement learning (CDRL) is meant to improve the data efficiency of RL by leveraging the data samples collected from a source domain to facilitate the learning in a similar target domain. Despite its potential, cross-domain transfer in RL is known to have two fundamental and intertwined challenges: (i) The source and target domains can have distinct state space or action space, and this makes direct transfer infeasible and thereby requires more sophisticated inter-domain mappings; (ii) The transferability of a source-domain model in RL is not easily identifiable a priori, and hence CDRL can be prone to negative effect during transfer. In this paper, we propose to jointly tackle these two challenges through the lens of cross-domain Bellman consistency and hybrid critic. Specifically, we first introduce the notion of cross-domain Bellman consistency as a way to measure transferability of a source-domain model. Then, we propose QAvatar, which combines the Q functions from both the source and target domains with an adaptive hyperparameter-free weight function. Through this design, we characterize the convergence behavior of QAvatar and show that QAvatar achieves reliable transfer in the sense that it effectively leverages a source-domain Q function for knowledge transfer to the target domain. Through experiments, we demonstrate that QAvatar achieves favorable transferability across various RL benchmark tasks, including locomotion and robot arm manipulation. Our code is available at https://rl-bandits-lab.github.io/Cross-Domain-RL/.
Figures & tables
Figure 1: Training curves of Q Avatar and benchmark methods: (a)-(b) Locomotion tasks; (d)-(e) Robot arm manipulation tasks in Robosuite; (f) Navigation task from CarGoal0 to DoggoGoal0.
Figure 2Figure 3
Figure 5: Performance comparison on DMC tasks.
Figure 5
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Source and target domains of the grid navigation example.
Algorithm
HalfCheetah
Ant
Door Opening
Table Wiping
Navigation
Q Avatar
11586.0 ± 1224.4
2858.8 ± 848.0
216.6 ± 131.3
76.6 ± 13.5
38.5 ± 13.2
SAC
10986.0 ± 1821.8
1620.0 ± 527.2
94.8 ± 23.9
47.6 ± 11.0
19.7 ± 13.6
FT
10756.8 ± 1070.8
1644.3 ± 748.2
129.9 ± 34.6
42.1 ± 15.4
12.5 ± 9.0
PAR
8097.4 ± 3962.0
737.6 ± 45.3
33.7 ± 18.6
17.9 ± 11.8
0.0 ± 0.0
CAT-SAC
8756.5 ± 1264.3
1628.9 ± 200.6
63.2 ± 33.3
23.7 ± 10.7
2.7 ± 2.4
CAT
46.1 ± 149.9
17.1 ± 27.3
34.7 ± 8.4
55.5 ± 29.7
-0.1 ± 0.2
Appendix
Table 2: Final rewards of Q Avatarand all baselines in the experiments.
Figure 9: Training curves of ablation experiments. (a) Unrelated transfer scenario where the target domain is Table-Wiping with UR5e. (b) Training curves in non-stationary Ant-v3. (c) Transfer from two source domains to a target domain.
Figure 11: Cross-domain Bellman error and TD error of the target Q-function during training across all main experiments.
Environment
State
Action
Src
Tar
Src
Tar
HalfCheetah
17
23
6
9
Ant
111
133
8
10
Door Opening
46
51
8
7
Table Wiping
37
34
7
6
Goal Navigation
40
72
2
12
Appendix
Table 3: Dimensions of the source and target domains (“Src” and “Tar” represent the source domain and the target domain.)
Figure 12: The environments of the source domains and the target domains. (a),(c): Source domains – Original MuJoCo environments. (b),(d): Target domains – Modified MuJoCo environments.
Figure 13: The environments of the source and target domains. (a)–(c): Source domains. (d)–(f): Target domains.
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
Cross-domain offline reinforcement learning (CDRL) aims to improve policy learning in a target domain by leveraging data collected from a source domain. Existing works typically assess the transferability of source-domain data by measuring its similarity to target-domain transitions, and implicitly perform transition-level selection. Transitions that are considered similar are assigned higher weights or rewards, while dissimilar ones are down-weighted. However, transition-level similarity does not necessarily imply consistency in long-term returns. Even visually or dynamically similar transitions may lead to significantly different outcomes in the target domain, which can mislead policy learning and degrade performance. To address this issue, we revisit the fundamental objective of policy learning. Since policy optimization ultimately relies on Bellman targets to evaluate the quality of decisions, we propose to assess the transferability of source-domain transitions based on their alignment with target-domain Bellman targets, rather than superficial transition similarity. Based on this insight, we propose a method termed Target-Aligned Bellman Backup (TABB), which selectively leverages source-domain data by measuring their contribution to accurate Bellman target estimation in the target domain. We evaluate TABB across a broad range of cross-domain offline RL settings with highly limited target-domain data. Experimental results show that TABB consistently achieves strong performance.
Wei Liu, Ting Long
School of Artificial Intelligence Jilin University China
Cross-domain reinforcement learning (RL) aims to learn transferable policies under dynamics shifts between source and target domains. A key challenge lies in the lack of target-domain environment interaction and reward supervision, which prevents direct policy learning. To address this challenge, we propose Bridging Dynamics Gaps for Cross-Domain Reinforcement Learning (BDGxRL), a novel framework that leverages Diffusion Schrödinger Bridge (DSB) to align source transitions with target-domain dynamics encoded in offline demonstrations. Moreover, we introduce a reward modulation mechanism that estimates rewards based on state transitions, applying to DSB-aligned samples to ensure consistency between rewards and target-domain dynamics. BDGxRL performs target-oriented policy learning entirely within the source domain, without access to the target environment or its rewards. Experiments on MuJoCo cross-domain benchmarks demonstrate that BDGxRL outperforms state-of-the-art baselines and shows strong adaptability under transition dynamics shifts.
Hanping Zhang, Yuhong Guo
School of Computer Science, Carleton University, Ottawa, Canada
Transferring policies across domains poses a vital challenge in reinforcement learning, due to the dynamics mismatch between the source and target domains. In this paper, we consider the setting of online dynamics adaptation, where policies are trained in the source domain with sufficient data, while only limited interactions with the target domain are allowed. There are a few existing works that address the dynamics mismatch by employing domain classifiers, value-guided data filtering, or representation learning. Instead, we study the domain adaptation problem from a generative modeling perspective. Specifically, we introduce DADiff, a diffusion-based framework that leverages the discrepancy between source and target domain generative trajectories in the generation process of the next state to estimate the dynamics mismatch. Both reward modification and data selection variants are developed to adapt the policy to the target domain. We also provide a theoretical analysis to show that the performance difference of a given policy between the two domains is bounded by the generative trajectory deviation. More discussions on the applicability of the variants and the connection between our theoretical analysis and the prior work are further provided. We conduct extensive experiments in environments with various shifts to validate the effectiveness of our method. The results demonstrate that our method provides superior performance compared to existing approaches, effectively addressing the dynamics mismatch. We provide the code of our method at https://github.com/hanyang-chen/DADiff-release
Hanyang Chen, Anirudh Satheesh, Longchao Da +1
Arizona State University · University of Maryland, College Park