Localized Dynamics-Aware Domain Adaption for Off-Dynamics Offline Reinforcement Learning

Zhangjie Xia; Yu Yang; Pan Xu

cs.LGFeb 24, 2026

Localized Dynamics-Aware Domain Adaption for Off-Dynamics Offline Reinforcement Learning

Authors: Zhangjie Xia, Yu Yang, Pan Xu

Organizations: New York University · Duke University

Abstract

Off-dynamics offline reinforcement learning (RL) aims to learn a policy for a target domain using limited target data and abundant source data collected under different transition dynamics. Existing methods typically address dynamics mismatch either globally over the state space or via pointwise data filtering; these approaches can miss localized cross-domain similarities or incur high computational cost. We propose Localized Dynamics-Aware Domain Adaptation (LoDADA), which exploits localized dynamics mismatch to better reuse source data. LoDADA clusters transitions from source and target datasets and estimates cluster-level dynamics discrepancy via domain discrimination. Source transitions from clusters with small discrepancy are retained, while those from clusters with large discrepancy are filtered out. This yields a fine-grained and scalable data selection strategy that avoids overly coarse global assumptions and expensive per-sample filtering. We provide theoretical insights and extensive experiments across environments with diverse global and local dynamics shifts. Results show that LoDADA consistently outperforms state-of-the-art off-dynamics offline RL methods by better leveraging localized distribution mismatch.

Figures & tables

Figure 1: An overview of our proposed framework. We first perform K-means clustering on the mixed source and target dataset. Then we estimate the local KL divergence between source and target dynamics in each cluster. We use cluster-wise classifiers to selectively retain source domain transitions for downstream offline RL algorithms. We further introduce a dataset regularization term to ensure policy consistency with the target dataset.

Env	Shift Level	BOSA	IQL	DARA	IGDF	OTDF	Ours
HalfCheetah Gravity	0.1	9.31 $\pm$ 1.94	12.90 $\pm$ 1.01	9.62 $\pm$ 4.27	7.65 $\pm$ 6.63	12.17 $\pm$ 3.58	14.09 $\pm$ 3.83
	0.5	39.92 $\pm$ 5.63	43.68 $\pm$ 2.96	40.87 $\pm$ 5.46	39.73 $\pm$ 4.62	37.12 $\pm$ 4.88	45.00 $\pm$ 2.95
	2.0	28.47 $\pm$ 1.23	31.12 $\pm$ 0.3	30.49 $\pm$ 0.54	30.16 $\pm$ 1.5	25.38 $\pm$ 3.12	28.02 $\pm$ 1.67
	5.0	11.27 $\pm$ 1.77	20.87 $\pm$ 21.45	32.42 $\pm$ 18.54	30.08 $\pm$ 34.02	24.01 $\pm$ 17.88	71.11 $\pm$ 1.63
HalfCheetah Friction	0.1	12.53 $\pm$ 3.61	23.69 $\pm$ 16.46	26.39 $\pm$ 11.35	20.56 $\pm$ 11.22	10.31 $\pm$ 1.68	9.97 $\pm$ 0.47
	0.5	63.65 $\pm$ 9.14	68.26 $\pm$ 1.42	68.41 $\pm$ 3.19	66.30 $\pm$ 1.79	66.32 $\pm$ 0.46	68.51 $\pm$ 0.44

Table 1: Performance comparison on MuJoCo tasks (HalfCheetah, Ant, Walker2D, Hopper) under a medium-level offline dataset with dynamics shifts in gravity and friction at levels 0.1, 0.5, 2.0, 5.0. Source domains remain unchanged; target domains are shifted. We report normalized target-domain scores (mean ± std over five seeds). Best and second-best scores are highlighted in green and blue , respectively.

Env	# Groups	Shift Level	BOSA	DARA	IQL	IGDF	OTDF	Ours
HalfCheetah	3	0.1-0.5-2.0	27.36 $\pm$ 2.22	19.86 $\pm$ 4.42	17.57 $\pm$ 3.41	14.59 $\pm$ 3.13	29.26 $\pm$ 3.10	34.97 $\pm$ 0.85
		0.2-1.0-4.0	25.28 $\pm$ 3.10	16.94 $\pm$ 3.31	16.07 $\pm$ 3.45	13.16 $\pm$ 4.83	24.21 $\pm$ 2.82	28.36 $\pm$ 2.76
		0.3-1.5-6.0	20.75 $\pm$ 2.66	6.59 $\pm$ 0.45	10.73 $\pm$ 2.88	8.73 $\pm$ 2.31	22.84 $\pm$ 0.54	27.24 $\pm$ 4.37
	5	0.1-0.2-0.3-0.4-0.5	19.30 $\pm$ 2.38	12.43 $\pm$ 1.95	10.83 $\pm$ 1.05	11.27 $\pm$ 2.35	28.99 $\pm$ 1.85	34.28 $\pm$ 2.91
		0.2-0.4-0.6-0.8-1.0	23.59 $\pm$ 5.81	13.18 $\pm$ 2.09	14.77 $\pm$ 2.41	11.43 $\pm$ 3.96	21.41 $\pm$ 3.95	25.73 $\pm$ 5.38
		0.3-0.6-0.9-1.2-1.5	23.95 $\pm$ 2.23	7.10 $\pm$ 0.97	8.98 $\pm$ 2.61	7.37 $\pm$ 3.03	11.53 $\pm$ 2.04	18.74 $\pm$ 1.55

Table 2: Performance comparison on MuJoCo tasks (Halfcheetah, Ant, Walker2d, Hopper) under a medium-level offline dataset. Target domains remain unchanged; source datasets are shifted by adding local perturbations with varying shift levels across regions. We report normalized target-domain scores (mean ± std over five seeds). Best and second-best scores are highlighted in green and blue , respectively.

Env	Shift Level	BOSA	DARA	IQL	IGDF	OTDF	Ours
Antmaze Medium	1	$43.6\pm 11.2$	$53.6\pm 9.9$	76.5 $\pm$ 8.4	$60.4\pm 9.1$	$51.6\pm 10.53$	$66.8\pm 4.6$
	2	$22.4\pm 5.5$	$33.6\pm 3.3$	$52.8\pm 7.9$	$45.2\pm 7.8$	$56.4\pm 4.34$	69.2 $\pm$ 8.9
	3	$39.6\pm 11.5$	$50.8\pm 3.0$	$57.6\pm 7.7$	$65.2\pm 6.6$	71.6 $\pm$ 5.90	73.2 $\pm$ 7.3
	4	$40.0\pm 3.2$	$48.8\pm 8.3$	$58.4\pm 4.8$	$60.4\pm 6.2$	$52.4\pm 8.53$	65.2 $\pm$ 5.9
	5	$21.2\pm 7.6$	$41.6\pm 12.7$	66.4 $\pm$ 8.3	$56.8\pm 7.6$	$57.2\pm 7.16$	66.8 $\pm$ 4.8
	6	$24.8\pm 14.0$	$59.2\pm 12.4$	$72.4\pm 10.7$	$73.6\pm 1.7$	$65.2\pm 9.86$	75.2 $\pm$ 6.7

Table 3: Performance comparison on AntMaze tasks under the diverse source dataset with dynamics shifts in the map. Source domains remain unchanged; target domains are shifted. We report normalized target-domain scores (mean ± std over five seeds) with the mean of 50 episodes. Best and second-best scores are highlighted in green and blue , respectively.

Panel of Figure 2 — Figure 2: Parameter sensitivity study.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 3 — Figure 3: Visualization of morphology shift environments for Ant.

Panel of Figure 4 — Figure 4: Visualization of morphology shift environments for HalfCheetah.

Panel of Figure 5 — Figure 5: Visualization of morphology shift environments for Hopper.

Panel of Figure 6 — Figure 6: Visualization of morphology shift environments for Walker2d.

Panel of Figure 7 — Figure 7: Visualization of the AntMaze map structure. We consider medium size map with 6 different map layouts.

Panel of Figure 8 — Figure 8: Visualization of morphology and kinematic shift environments for Adroit Pen.

Panel of Figure 9 — Figure 9: Visualization of morphology and kinematic shift environments for Adroit Door.

Task Name	Dynamics Shift Type	Reference Min Score ( $J_{r}$ )	Reference Max Score ( $J_{e}$ )
HalfCheetah	Gravity - 0.1	-280.18	2466.85
	Gravity - 0.5	-280.18	9509.15
	Gravity - 2.0	-280.18	9509.15
	Gravity - 5.0	-280.18	3756.24
	Friction - 0.1	-280.18	41696.55
	Friction - 0.5	-280.18	7357.07

Appendix

Table 4: The referenced minimum and maximum scores for MuJoCo datasets under various dynamics shift scenarios. These reference scores are used to compute normalized performance in the target domain.

Task Name	Dynamics Shift Type	Reference Min Score ( $J_{r}$ )	Reference Max Score ( $J_{e}$ )
AntMaze-medium	1	0.0	1.0
	2	0.0	1.0
	3	0.0	1.0
	4	0.0	1.0
	5	0.0	1.0
	6	0.0	1.0

Appendix

Table 5: The referenced minimum and maximum scores for AntMaze medium datasets under various dynamics shift scenarios. These reference scores are used to compute normalized performance in the target domain.

Task Name	Dynamics Shift Type	Reference Min Score ( $J_{r}$ )	Reference Max Score ( $J_{e}$ )
Pen	broken-joint-medium	-12.17	6408.38
	broken-joint-hard	-12.17	6408.38
	shrink-finger-medium	-12.17	6408.38
	shrink-finger-hard	-12.17	6408.38
Door	broken-joint-medium	-52.34	2880.57
	broken-joint-hard	-52.34	2880.57

Appendix

Table 6: The referenced minimum and maximum scores for Adroit datasets under various dynamics shift scenarios. These reference scores are used to compute normalized performance in the target domain.

Methods	Hyperparameter	Value
Shared among all methods	Actor network	(256, 256)
	Critic network	(256, 256)
	Learning rate	$3\times 10^{-4}$
	Optimizer	Adam
	Discount factor	0.99
	Replay buffer size	$10^{6}$

Appendix

Table 7: Hyperparameters of LoDADA and baselines.

\mathcal{L}_{Q}=\mathbb{E}_{D_{tar}}[(Q_{\theta}-y)^{2}]+\mathbb{E}_{(s,a,s^{\prime},d)\sim\tilde{D}_{src}}[\exp(-\alpha\cdot\hat{d_{i}})(Q_{\theta}-y)^{2}].

Appendix

Algorithm 2 LoDADA : Localized Dynamics-Aware Domain Adaptation

Env	Type	Level	BOSA	IQL	DARA	IGDF	OTDF	Ours
HalfCheetah	morph-thigh	M	25.12 $\pm$ 4.33	22.22 $\pm$ 1.13	20.68 $\pm$ 2.49	19.31 $\pm$ 4.35	21.94 $\pm$ 6.30	26.42 $\pm$ 3.44
		H	20.74 $\pm$ 3.22	15.90 $\pm$ 1.47	16.35 $\pm$ 0.97	15.57 $\pm$ 2.06	22.75 $\pm$ 2.76	25.81 $\pm$ 5.29
	morph-torso	M	0.66 $\pm$ 0.90	1.95 $\pm$ 1.69	1.40 $\pm$ 1.66	1.84 $\pm$ 0.37	3.38 $\pm$ 1.30	9.96 $\pm$ 4.07
		H	16.44 $\pm$ 12.82	30.87 $\pm$ 10.00	29.89 $\pm$ 5.88	28.85 $\pm$ 3.89	26.06 $\pm$ 6.00	23.44 $\pm$ 4.82
Ant	morph-halflegs	M	64.52 $\pm$ 7.46	67.15 $\pm$ 4.33	65.82 $\pm$ 5.31	69.07 $\pm$ 5.76	73.86 $\pm$ 0.79	77.48 $\pm$ 0.77
		H	37.21 $\pm$ 6.15	54.27 $\pm$ 5.60	61.31 $\pm$ 5.71	60.08 $\pm$ 2.09	59.76 $\pm$ 9.27	64.47 $\pm$ 4.62

Appendix

Table 8: Performance comparison on HalfCheetah, Ant, Walker2d, and Hopper environments under morphology shifts across 5 seeds. We use M and H to denote the medium and hard levels of the dynamics shift.

Env	Shift Level	BOSA	DARA	IQL	IGDF	OTDF	Ours
Antmaze Medium	1	$33.6\pm 8.0$	$52.4\pm 7.8$	76.4 $\pm$ 10.1	$58.0\pm 12.4$	$56.8\pm 6.87$	$52.8\pm 9.4$
	2	$25.2\pm 4.6$	$29.6\pm 6.2$	$46.8\pm 4.1$	$42.4\pm 8.2$	$63.2\pm 7.95$	67.6 $\pm$ 6.7
	3	$30.0\pm 8.1$	$56.4\pm 11.1$	$66.4\pm 6.7$	$64.8\pm 7.0$	$69.6\pm 5.90$	71.6 $\pm$ 3.8
	4	$34.0\pm 8.9$	$52.8\pm 6.3$	$48.8\pm 6.9$	$58.0\pm 1.4$	$44.8\pm 3.90$	71.6 $\pm$ 6.8
	5	$22.8\pm 13.9$	$34.4\pm 9.3$	73.6 $\pm$ 8.2	$54.4\pm 6.2$	$55.2\pm 3.03$	$60.8\pm 7.2$
	6	$12.8\pm 4.8$	$62.8\pm 3.0$	$66.8\pm 8.1$	74.8 $\pm$ 4.1	$70.8\pm 5.93$	$74.4\pm 3.0$

Appendix

Table 9: Performance comparison on Antmaze tasks under the play offline dataset with dynamics shifts in the map. Source domains remain unchanged; target domains are shifted. We report normalized target-domain scores (mean ± std over five seeds). Best and second-best scores are highlighted in green and blue , respectively.

Env	Type	Level	BOSA	IQL	DARA	IGDF	OTDF	Ours
Pen	kin-broken-joint	M	42.56 $\pm$ 11.31	45.46 $\pm$ 8.54	46.17 $\pm$ 6.26	50.74 $\pm$ 11.20	52.14 $\pm$ 6.51	64.31 $\pm$ 11.33
		H	19.36 $\pm$ 5.42	24.02 $\pm$ 5.16	22.41 $\pm$ 4.73	18.62 $\pm$ 10.47	20.95 $\pm$ 7.23	28.16 $\pm$ 18.04
	morph-shrink-finger	M	10.33 $\pm$ 2.79	15.64 $\pm$ 2.97	15.33 $\pm$ 2.39	13.19 $\pm$ 4.41	18.02 $\pm$ 5.64	19.88 $\pm$ 8.02
		H	19.47 $\pm$ 5.01	19.40 $\pm$ 5.05	18.24 $\pm$ 2.59	19.05 $\pm$ 3.87	20.34 $\pm$ 2.85	33.06 $\pm$ 11.48
Door	kin-broken-joint	M	20.65 $\pm$ 10.72	40.17 $\pm$ 7.10	36.82 $\pm$ 4.63	37.76 $\pm$ 7.02	39.82 $\pm$ 4.75	55.12 $\pm$ 12.50
		H	19.80 $\pm$ 15.61	52.37 $\pm$ 5.52	49.31 $\pm$ 5.53	52.61 $\pm$ 4.83	58.24 $\pm$ 6.37	59.81 $\pm$ 7.23

Appendix

Table 10: Performance comparison on the Door and Pen environments under morphology shifts and kinematic shifts across 5 seeds. We use M and H to denote the medium and hard levels of the dynamics shift.

# Sources	IGDF	OTDF	LoDADA
1000000	$\sim$ 41.49s	$\sim$ 4597.82s	$\sim$ 134.62s
100000	$\sim$ 38.71s	$\sim$ 628.16s	$\sim$ 80.18s
10000	$\sim$ 55.64s	$\sim$ 126.39s	$\sim$ 72.92s

Appendix

Table 11: Runtime comparison in Ant-Gravity-0.5 with different numbers of source samples.

Offline RL

Offline reinforcement learning learns decision-making policies from fixed datasets without interacting with the environment during training.

See all 298 papers

DADiff: Diffusion-Driven Cross-Domain Policy Adaptation for Reinforcement Learning
Jul 17, 2026Hanyang Chen, Anirudh Satheesh, Longchao Da +1Transfer Learning Generalization in RL
Arizona State University · University of Maryland, College Park
Cross-Domain Energy-Guided Diffusion Generation for Off-Dynamics Reinforcement Learning
May 24, 2026Yu Yang, Yihong Guo, Anqi Liu +1Diffusion Model Guidance Reinforcement Learning
Duke University · Johns Hopkins University
Target-Aligned Bellman Backup for Cross-domain Offline Reinforcement Learning
May 21, 2026Wei Liu, Ting LongOffline RL Cross-Domain Transfer Learning
School of Artificial Intelligence Jilin University China

Jul 17, 2026J/K move · Enter open · S save

DADiff: Diffusion-Driven Cross-Domain Policy Adaptation for Reinforcement Learning

Hanyang Chen, Anirudh Satheesh, Longchao Da, Hua Wei

Arizona State University · University of Maryland, College Park

Transfer Learning Generalization in RL

Open paper AskPDF arXiv

Transferring policies across domains poses a vital challenge in reinforcement learning, due to the dynamics mismatch between the source and target domains. In this paper, we consider the setting of online dynamics adaptation, where policies are trained in the source domain with sufficient data, while only limited interactions with the target domain are allowed. There are a few existing works that address the dynamics mismatch by employing domain classifiers, value-guided data filtering, or representation learning. Instead, we study the domain adaptation problem from a generative modeling perspective. Specifically, we introduce DADiff, a diffusion-based framework that leverages the discrepancy between source and target domain generative trajectories in the generation process of the next state to estimate the dynamics mismatch. Both reward modification and data selection variants are developed to adapt the policy to the target domain. We also provide a theoretical analysis to show that the performance difference of a given policy between the two domains is bounded by the generative trajectory deviation. More discussions on the applicability of the variants and the connection between our theoretical analysis and the prior work are further provided. We conduct extensive experiments in environments with various shifts to validate the effectiveness of our method. The results demonstrate that our method provides superior performance compared to existing approaches, effectively addressing the dynamics mismatch. We provide the code of our method at https://github.com/hanyang-chen/DADiff-release

Abstract

Offline RL

Explore similar work

DADiff: Diffusion-Driven Cross-Domain Policy Adaptation for Reinforcement Learning

Cross-Domain Energy-Guided Diffusion Generation for Off-Dynamics Reinforcement Learning

Target-Aligned Bellman Backup for Cross-domain Offline Reinforcement Learning