Cross-Domain Policy Optimization via Bellman Consistency and Hybrid Critics

Ming-Hong Chen; Kuan-Chen Pan; You-De Huang; Xi Liu; Ping-Chun Hsieh

cs.LGMar 12, 2026

Cross-Domain Policy Optimization via Bellman Consistency and Hybrid Critics

Authors: Ming-Hong Chen, Kuan-Chen Pan, You-De Huang, Xi Liu, Ping-Chun Hsieh

Organizations: National Yang Ming Chiao Tung University, Hsinchu, Taiwan · Applied Machine Learning, Meta AI, Menlo Park, CA, USA

Abstract

Cross-domain reinforcement learning (CDRL) is meant to improve the data efficiency of RL by leveraging the data samples collected from a source domain to facilitate the learning in a similar target domain. Despite its potential, cross-domain transfer in RL is known to have two fundamental and intertwined challenges: (i) The source and target domains can have distinct state space or action space, and this makes direct transfer infeasible and thereby requires more sophisticated inter-domain mappings; (ii) The transferability of a source-domain model in RL is not easily identifiable a priori, and hence CDRL can be prone to negative effect during transfer. In this paper, we propose to jointly tackle these two challenges through the lens of cross-domain Bellman consistency and hybrid critic. Specifically, we first introduce the notion of cross-domain Bellman consistency as a way to measure transferability of a source-domain model. Then, we propose $Q$ Avatar, which combines the Q functions from both the source and target domains with an adaptive hyperparameter-free weight function. Through this design, we characterize the convergence behavior of $Q$ Avatar and show that $Q$ Avatar achieves reliable transfer in the sense that it effectively leverages a source-domain Q function for knowledge transfer to the target domain. Through experiments, we demonstrate that $Q$ Avatar achieves favorable transferability across various RL benchmark tasks, including locomotion and robot arm manipulation. Our code is available at https://rl-bandits-lab.github.io/Cross-Domain-RL/.

Figures & tables

Panel of Figure 1 — Figure 1: Training curves of $Q$ Avatar and benchmark methods: (a)-(b) Locomotion tasks; (d)-(e) Robot arm manipulation tasks in Robosuite; (f) Navigation task from CarGoal0 to DoggoGoal0.

Panel of Figure 5 — Figure 5: Performance comparison on DMC tasks.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 8 — Figure 8: Source and target domains of the grid navigation example.

Algorithm	HalfCheetah	Ant	Door Opening	Table Wiping	Navigation
$Q$ Avatar	11586.0 $\pm$ 1224.4	2858.8 $\pm$ 848.0	216.6 $\pm$ 131.3	76.6 $\pm$ 13.5	38.5 $\pm$ 13.2
SAC	10986.0 $\pm$ 1821.8	1620.0 $\pm$ 527.2	94.8 $\pm$ 23.9	47.6 $\pm$ 11.0	19.7 $\pm$ 13.6
FT	10756.8 $\pm$ 1070.8	1644.3 $\pm$ 748.2	129.9 $\pm$ 34.6	42.1 $\pm$ 15.4	12.5 $\pm$ 9.0
PAR	8097.4 $\pm$ 3962.0	737.6 $\pm$ 45.3	33.7 $\pm$ 18.6	17.9 $\pm$ 11.8	0.0 $\pm$ 0.0
CAT-SAC	8756.5 $\pm$ 1264.3	1628.9 $\pm$ 200.6	63.2 $\pm$ 33.3	23.7 $\pm$ 10.7	2.7 $\pm$ 2.4
CAT	46.1 $\pm$ 149.9	17.1 $\pm$ 27.3	34.7 $\pm$ 8.4	55.5 $\pm$ 29.7	-0.1 $\pm$ 0.2

Appendix

Table 2: Final rewards of

Q

Avatarand all baselines in the experiments.

Panel of Figure 9 — Figure 9: Training curves of ablation experiments. (a) Unrelated transfer scenario where the target domain is Table-Wiping with UR5e. (b) Training curves in non-stationary Ant-v3. (c) Transfer from two source domains to a target domain.

Panel of Figure 10 — Figure 10: Performance evaluation of ECC.

\displaystyle Q^{(t)}_{\tar,j}=\arg\min_{Q_{\tar}}\hat{\mathbb{E}}_{(s,a,r,s^{\prime})\in B_{\text{SAC}}}\Big[\big\rvert r+\gamma\mathbb{E}_{a^{\prime}\sim\pi^{(t)}(\cdot\rvert s^{\prime})}\big[Q_{\tar}(s^{\prime},a^{\prime})-\beta\log(\pi(a^{\prime}|s^{\prime}))\big]-Q_{\tar}(s,a)\big\rvert^{2}\Big].

(88)

Appendix

Algorithm 3 Practical Implementation of

Q

Avatar

Panel of Figure 11 — Figure 11: Cross-domain Bellman error and TD error of the target Q-function during training across all main experiments.

Environment	State		Action
Environment	Src	Tar	Src	Tar
HalfCheetah	17	23	6	9
Ant	111	133	8	10
Door Opening	46	51	8	7
Table Wiping	37	34	7	6
Goal Navigation	40	72	2	12

Appendix

Table 3: Dimensions of the source and target domains (“Src” and “Tar” represent the source domain and the target domain.)

Panel of Figure 12 — Figure 12: The environments of the source domains and the target domains. (a),(c): Source domains – Original MuJoCo environments. (b),(d): Target domains – Modified MuJoCo environments.

Panel of Figure 13 — Figure 13: The environments of the source and target domains. (a)–(c): Source domains. (d)–(f): Target domains.

Parameter	Value
critic/actor learning rate	0.0003
mapping function learning rate	0.0001
batch size	256
replay buffer size	$10^{6}$
optimizer for Actor/critic	Adam
optimizer for mapping function	AdamW

Appendix

Table 4: A list of hyperparameters of

Q

Avatar.

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,717 papers

Target-Aligned Bellman Backup for Cross-domain Offline Reinforcement Learning
May 21, 2026Wei Liu, Ting LongOffline RL Cross-Domain Transfer Learning
School of Artificial Intelligence Jilin University China
Bridging Dynamics Gaps via Diffusion Schrödinger Bridge for Cross-Domain Reinforcement Learning
Feb 27, 2026Hanping Zhang, Yuhong GuoReinforcement Learning Cross-Domain Transfer Learning
School of Computer Science, Carleton University, Ottawa, Canada
DADiff: Diffusion-Driven Cross-Domain Policy Adaptation for Reinforcement Learning
Jul 17, 2026Hanyang Chen, Anirudh Satheesh, Longchao Da +1Transfer Learning Generalization in RL
Arizona State University · University of Maryland, College Park

May 21, 2026J/K move · Enter open · S save

Target-Aligned Bellman Backup for Cross-domain Offline Reinforcement Learning

Wei Liu, Ting Long

School of Artificial Intelligence Jilin University China

Offline RL Cross-Domain Transfer Learning

Open paper AskPDF arXiv

Cross-domain offline reinforcement learning (CDRL) aims to improve policy learning in a target domain by leveraging data collected from a source domain. Existing works typically assess the transferability of source-domain data by measuring its similarity to target-domain transitions, and implicitly perform transition-level selection. Transitions that are considered similar are assigned higher weights or rewards, while dissimilar ones are down-weighted. However, transition-level similarity does not necessarily imply consistency in long-term returns. Even visually or dynamically similar transitions may lead to significantly different outcomes in the target domain, which can mislead policy learning and degrade performance. To address this issue, we revisit the fundamental objective of policy learning. Since policy optimization ultimately relies on Bellman targets to evaluate the quality of decisions, we propose to assess the transferability of source-domain transitions based on their alignment with target-domain Bellman targets, rather than superficial transition similarity. Based on this insight, we propose a method termed Target-Aligned Bellman Backup (TABB), which selectively leverages source-domain data by measuring their contribution to accurate Bellman target estimation in the target domain. We evaluate TABB across a broad range of cross-domain offline RL settings with highly limited target-domain data. Experimental results show that TABB consistently achieves strong performance.

Abstract

Reinforcement Learning

Explore similar work

Target-Aligned Bellman Backup for Cross-domain Offline Reinforcement Learning

Bridging Dynamics Gaps via Diffusion Schrödinger Bridge for Cross-Domain Reinforcement Learning

DADiff: Diffusion-Driven Cross-Domain Policy Adaptation for Reinforcement Learning