Bridging Dynamics Gaps via Diffusion Schrödinger Bridge for Cross-Domain Reinforcement Learning

Hanping Zhang; Yuhong Guo

cs.LGFeb 27, 2026

Bridging Dynamics Gaps via Diffusion Schrödinger Bridge for Cross-Domain Reinforcement Learning

Authors: Hanping Zhang, Yuhong Guo

Organizations: School of Computer Science, Carleton University, Ottawa, Canada

Abstract

Cross-domain reinforcement learning (RL) aims to learn transferable policies under dynamics shifts between source and target domains. A key challenge lies in the lack of target-domain environment interaction and reward supervision, which prevents direct policy learning. To address this challenge, we propose Bridging Dynamics Gaps for Cross-Domain Reinforcement Learning (BDGxRL), a novel framework that leverages Diffusion Schrödinger Bridge (DSB) to align source transitions with target-domain dynamics encoded in offline demonstrations. Moreover, we introduce a reward modulation mechanism that estimates rewards based on state transitions, applying to DSB-aligned samples to ensure consistency between rewards and target-domain dynamics. BDGxRL performs target-oriented policy learning entirely within the source domain, without access to the target environment or its rewards. Experiments on MuJoCo cross-domain benchmarks demonstrate that BDGxRL outperforms state-of-the-art baselines and shows strong adaptability under transition dynamics shifts.

Figures & tables

Figure 1: Overview of the proposed BDGxRL framework. The agent first collects a dataset $\mathcal{D}_{S}$ from the source environment, which is used to train a transition-aware reward model $R(s_{t},s_{t+1})$ . Together with offline target demonstrations $\mathcal{D}_{T}$ , it also trains a DSB model for dynamics alignment. During online interactions, source transitions are translated into target-style transitions via $\tilde{s}_{t+1}\sim\mathrm{DSB}(s_{t},a_{t},s_{t+1})$ to mitigate dynamics mismatch. The modulated reward $\tilde{r}_{t}=R(s_{t},\tilde{s}_{t+1})$ is then used to learn a target-oriented policy entirely within the source domain, initialized via imitation from $\mathcal{D}_{T}$ .

Task	Expert	Domain Gap	D4RL	xTED (IL)	xTED (CD)	DARA	GAIL	DARC	DARAIL	BDGxRL (Ours)
HalfCheetah	Med	Gravity	$39.5\pm 2.4$	$40.6\pm 2.0$	$41.1\pm 2.0$	$40.6\pm 1.9$	$39.0\pm 1.8$	$40.1\pm 1.6$	$42.6\pm 2.4$	$\mathbf{46.3\pm 2.7}$
		Friction	$43.1\pm 1.5$	$43.2\pm 3.0$	$44.2\pm 1.8$	$42.3\pm 1.7$	$40.2\pm 2.0$	$41.7\pm 1.9$	$43.9\pm 2.3$	$\mathbf{47.8\pm 2.9}$
		Thigh Size	$39.5\pm 2.4$	$40.7\pm 2.4$	$41.5\pm 1.6$	$40.4\pm 2.1$	$39.5\pm 2.2$	$40.9\pm 1.7$	$42.7\pm 2.1$	$\mathbf{46.5\pm 2.5}$
	Med-R	Gravity	$26.2\pm 3.5$	$30.7\pm 1.9$	$30.7\pm 1.9$	$27.2\pm 3.2$	$28.2\pm 1.7$	$30.5\pm 2.2$	$33.6\pm 2.3$	$\mathbf{37.8\pm 2.8}$
		Friction	$26.2\pm 3.5$	$31.8\pm 3.1$	$32.4\pm 2.9$	$23.2\pm 4.7$	$30.0\pm 1.9$	$31.3\pm 2.5$	$35.0\pm 2.4$	$\mathbf{39.5\pm 2.6}$
		Thigh Size	$26.2\pm 3.5$	$30.3\pm 1.9$	$32.9\pm 2.7$	$29.1\pm 1.4$	$28.9\pm 2.1$	$30.7\pm 2.4$	$34.4\pm 2.1$	$\mathbf{38.6\pm 2.7}$

Table 1: Experimental results across three domain gaps (gravity, friction, thigh size) and three demonstration levels (Medium, Medium-Replay, Med-Expert) on two MuJoCo environments, averaged over five runs.

Figure 2: Overall average performance of each method across all tasks, domain gaps, and demonstration levels.

Task	Domain Gap	BDGxRL	w/o IL	w/o RM	w/o Alignment
HalfCheetah	Gravity	$46.3_{\pm 2.7}$	$39.5_{\pm 3.5}$	$42.1_{\pm 3.1}$	$33.8_{\pm 5.2}$
	Friction	$47.8_{\pm 2.9}$	$40.3_{\pm 3.3}$	$43.2_{\pm 2.9}$	$35.4_{\pm 4.8}$
	Thigh Size	$46.5_{\pm 2.5}$	$40.1_{\pm 3.0}$	$42.8_{\pm 2.8}$	$34.9_{\pm 5.1}$
Walker2d	Gravity	$63.9_{\pm 3.2}$	$55.5_{\pm 4.1}$	$58.2_{\pm 3.3}$	$44.7_{\pm 6.7}$
	Friction	$61.6_{\pm 3.4}$	$53.3_{\pm 3.9}$	$56.1_{\pm 3.1}$	$43.2_{\pm 5.8}$
	Thigh Size	$62.8_{\pm 3.0}$	$54.1_{\pm 4.0}$	$56.8_{\pm 3.2}$	$43.9_{\pm 6.0}$

Table 2: Ablation results under the Medium expert level across three domain gaps on two MuJoCo environments.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Task	Expert	Domain Gap	BDGxRL (Ours)	w/o IL	w/o RM	w/o Alignment
HalfCheetah	Med	Gravity	$46.3\pm 2.7$	$39.5\pm 3.5$	$42.1\pm 3.1$	$33.8\pm 5.2$
		Friction	$47.8\pm 2.9$	$40.3\pm 3.3$	$43.2\pm 2.9$	$35.4\pm 4.8$
		Thigh Size	$46.5\pm 2.5$	$40.1\pm 3.0$	$42.8\pm 2.8$	$34.9\pm 5.1$
	Med-R	Gravity	$37.8\pm 2.8$	$31.0\pm 4.0$	$33.2\pm 3.7$	$25.6\pm 6.1$
		Friction	$39.5\pm 2.6$	$32.5\pm 3.2$	$34.4\pm 3.4$	$27.2\pm 5.3$
		Thigh Size	$38.6\pm 2.7$	$31.8\pm 3.0$	$33.9\pm 3.2$	$26.4\pm 5.0$

Appendix

Table 3: Full ablation results across three domain gaps (gravity, friction, thigh size) and three demonstration levels (Medium, Medium-Replay, Med-Expert) on two MuJoCo environments, averaged over three runs.

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,717 papers

DADiff: Diffusion-Driven Cross-Domain Policy Adaptation for Reinforcement Learning
Jul 17, 2026Hanyang Chen, Anirudh Satheesh, Longchao Da +1Transfer Learning Generalization in RL
Arizona State University · University of Maryland, College Park
Target-Aligned Bellman Backup for Cross-domain Offline Reinforcement Learning
May 21, 2026Wei Liu, Ting LongOffline RL Cross-Domain Transfer Learning
School of Artificial Intelligence Jilin University China
Generative Support Realignment for Cross-Domain Offline Reinforcement Learning
May 13, 2026Minung Kim, Jeongmo Kim, Gwanwoo Choi +1Offline RL Cross-Domain Transfer Learning
Graduate School of Artificial Intelligence Ulsan National Institute of Science and Technology (UNIST)

Jul 17, 2026J/K move · Enter open · S save

DADiff: Diffusion-Driven Cross-Domain Policy Adaptation for Reinforcement Learning

Hanyang Chen, Anirudh Satheesh, Longchao Da, Hua Wei

Arizona State University · University of Maryland, College Park

Transfer Learning Generalization in RL

Open paper AskPDF arXiv

Transferring policies across domains poses a vital challenge in reinforcement learning, due to the dynamics mismatch between the source and target domains. In this paper, we consider the setting of online dynamics adaptation, where policies are trained in the source domain with sufficient data, while only limited interactions with the target domain are allowed. There are a few existing works that address the dynamics mismatch by employing domain classifiers, value-guided data filtering, or representation learning. Instead, we study the domain adaptation problem from a generative modeling perspective. Specifically, we introduce DADiff, a diffusion-based framework that leverages the discrepancy between source and target domain generative trajectories in the generation process of the next state to estimate the dynamics mismatch. Both reward modification and data selection variants are developed to adapt the policy to the target domain. We also provide a theoretical analysis to show that the performance difference of a given policy between the two domains is bounded by the generative trajectory deviation. More discussions on the applicability of the variants and the connection between our theoretical analysis and the prior work are further provided. We conduct extensive experiments in environments with various shifts to validate the effectiveness of our method. The results demonstrate that our method provides superior performance compared to existing approaches, effectively addressing the dynamics mismatch. We provide the code of our method at https://github.com/hanyang-chen/DADiff-release

Abstract

Reinforcement Learning

Explore similar work

DADiff: Diffusion-Driven Cross-Domain Policy Adaptation for Reinforcement Learning

Target-Aligned Bellman Backup for Cross-domain Offline Reinforcement Learning

Generative Support Realignment for Cross-Domain Offline Reinforcement Learning