Cross-Embodiment Offline Reinforcement Learning for Heterogeneous Robot Datasets

Haruki Abe; Takayuki Osa; Yusuke Mukuta; Tatsuya Harada

cs.AIFeb 20, 2026

Cross-Embodiment Offline Reinforcement Learning for Heterogeneous Robot Datasets

Authors: Haruki Abe, Takayuki Osa, Yusuke Mukuta, Tatsuya Harada

Organizations: The University of Tokyo, Tokyo, Japan · RIKEN Center for Advanced Intelligence Project, Tokyo, Japan

Abstract

Scalable robot policy pre-training has been hindered by the high cost of collecting high-quality demonstrations for each platform. In this study, we address this issue by uniting offline reinforcement learning (offline RL) with cross-embodiment learning. Offline RL leverages both expert and abundant suboptimal data, and cross-embodiment learning aggregates heterogeneous robot trajectories across diverse morphologies to acquire universal control priors. We perform a systematic analysis of this offline RL and cross-embodiment paradigm, providing a principled understanding of its strengths and limitations. To evaluate this offline RL and cross-embodiment paradigm, we construct a suite of locomotion datasets spanning 16 distinct robot platforms. Our experiments confirm that this combined approach excels at pre-training with datasets rich in suboptimal trajectories, outperforming pure behavior cloning. However, as the proportion of suboptimal data and the number of robot types increase, we observe that conflicting gradients across morphologies begin to impede learning. To mitigate this, we introduce an embodiment-based grouping strategy in which robots are clustered by morphological similarity and the model is updated with a group gradient. This simple, static grouping substantially reduces inter-robot conflicts and outperforms existing conflict-resolution methods.

Figures & tables

Dataset	BC	IQL
Expert Forward	63.31 $\pm$ 0.10	63.39 $\pm$ 0.05
Expert Backward	67.17 $\pm$ 0.01	67.10 $\pm$ 0.01
Expert Replay Forward	49.71 $\pm$ 1.06	54.61 $\pm$ 0.12
Expert Replay Backward	42.87 $\pm$ 1.32	51.86 $\pm$ 1.56
70% Suboptimal Forward	30.52 $\pm$ 3.10	36.62 $\pm$ 1.02
70% Suboptimal Backward	41.42 $\pm$ 0.71	38.69 $\pm$ 0.89

Table 1: BC vs. IQL performance across datasets (mean

\pm

standard error over 5 seeds).

Figure 1: Comparison of learning curves between cross-embodiment pre-trained networks and networks trained without cross-embodiment pre-training for Badger, Unitree G1, and Cassie.

Panel of Figure 4 — Figure 4: (a) Embodiment-based similarity matrix (1 - min-max-normalized FGW distance between robot pairs); (b) Gradient cosine similarity matrix in Expert Forward dataset from Section 5.1 , with the color scale compressed for readability using a symlog normalization; (c) Scatter plot of embodiment-based similarity and mean gradient cosine similarity for all robot pairs.

Figure 5: Overview of Embodiment Grouping (EG) for cross-embodiment offline RL.

Dataset	BC	TD3+BC	IQL	IQL + SEL	IQL + PCGrad	BC + EG	TD3+BC + EG	IQL + EG (ours)
Expert Forward	63.31 $\pm$ 0.10	52.14 $\pm$ 1.89	63.39 $\pm$ 0.05	63.37 $\pm$ 0.07	63.37 $\pm$ 0.04	63.47 $\pm$ 0.04	59.34 $\pm$ 1.19	63.52 $\pm$ 0.04
Expert Backward	67.17 $\pm$ 0.01	47.94 $\pm$ 0.48	67.10 $\pm$ 0.01	67.24 $\pm$ 0.02	67.05 $\pm$ 0.02	67.24 $\pm$ 0.02	51.98 $\pm$ 1.26	67.24 $\pm$ 0.01
Expert Replay Forward	49.71 $\pm$ 1.06	55.66 $\pm$ 0.84	54.61 $\pm$ 0.12	55.01 $\pm$ 0.55	53.84 $\pm$ 0.67	51.89 $\pm$ 0.65	57.04 $\pm$ 0.46	54.62 $\pm$ 0.53
Expert Replay Backward	42.87 $\pm$ 1.32	52.31 $\pm$ 1.22	51.86 $\pm$ 1.56	55.73 $\pm$ 1.06	55.94 $\pm$ 1.14	48.64 $\pm$ 2.65	55.67 $\pm$ 1.30	57.58 $\pm$ 0.05
70% Suboptimal Forward	30.52 $\pm$ 3.10	35.74 $\pm$ 1.51	36.62 $\pm$ 1.02	44.59 $\pm$ 2.02	39.63 $\pm$ 1.95	42.99 $\pm$ 1.23	43.41 $\pm$ 1.54	51.19 $\pm$ 1.06
70% Suboptimal Backward	41.42 $\pm$ 0.71	34.79 $\pm$ 1.40	38.69 $\pm$ 0.89	44.45 $\pm$ 1.75	41.04 $\pm$ 1.10	46.30 $\pm$ 2.39	40.88 $\pm$ 0.83	49.60 $\pm$ 2.39

Table 3: Each Algorithm Performance across Dataset (

\pm

is Standard Error, 5 seeds)

Method	Final return	Relative %
IQL (baseline)	37.57 $\pm$ 0.78	0.00%
Random grouping	38.73 $\pm$ 2.03	+3.08%
Heuristic	34.45 $\pm$ 1.97	-8.31%
EG (ours)	51.98 $\pm$ 1.70	+38.34%

Table 4: Ablation on grouping strategy (70% Suboptimal Forward). Values are mean final return

\pm

SEM; Relative is vs. cross-embodiment IQL.

Figure 6: Effect of the number of Embodiment Grouping clusters

M

on final return and wall-clock training time (mean

\pm

s.e.).

Method	(mean $\pm$ SEM)
Normalized IQL	44.20 $\pm$ 2.22
IQL + EG	51.98 $\pm$ 1.70
$\Delta R$	+7.78

Table 5: Normalized IQL vs. EG (70% Suboptimal Forward).

\Delta R

= EG

-

normalized IQL.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Illustration of the Forward and Backward locomotion commands used in our datasets.

Term	Name	Equation
(T1)	Xy velocity tracking	$\exp\left(-{\|v_{xy}-c_{xy}\|^{2}}/{0.25}\right)$
(T2)	Yaw velocity tracking	$\exp\left(-{\|\omega_{\text{yaw}}-c_{\text{yaw}}\|^{2}}/{0.25}\right)$
(T3)	Z velocity penalty	$-\|v_{z}\|^{2}$
(T4)	Pitch-roll velocity penalty	$-\|\omega_{\text{pitch, roll}}\|^{2}$
(T5)	Pitch-roll position penalty	$-\|\theta_{\text{pitch, roll}}\|^{2}$
(T6)	Joint nominal differences penalty	$-\|q-q_{\text{nominal}}\|^{2}$

Appendix

Table 6: Reward terms composing the reward function. Coefficients for each robot are listed in Table 7 .

Robot	T1	T2	T3	T4	T5	T7	T8	T9	T10	T11	T12	T13	T14	T
ANYmalB	2.0	1.0	2.0	0.05	0.2	10.0	2.5e-7	2e-4	0.01	30.0	1.0	0.1	0.5	20e6
ANYmalC	2.0	1.0	2.0	0.05	0.2	10.0	2.5e-7	2e-4	0.01	30.0	1.0	0.1	0.5	20e6
Barkour v0	3.0	1.5	2.0	0.05	0.2	10.0	2.5e-7	2e-4	0.01	30.0	1.0	0.1	0.5	15e6
Barkour vB	2.0	1.0	2.0	0.05	0.2	10.0	2.5e-7	2e-4	0.01	30.0	1.0	0.1	0.5	15e6
Silver Badger	2.0	1.0	2.0	0.05	0.2	10.0	2.5e-7	2e-4	0.01	30.0	1.0	0.1	0.5	12e6
Bittle	5.0	2.5	2.0	0.05	0.2	10.0	2.5e-7	2e-4	0.01	30.0	1.0	0.1	0.5	40e6

Appendix

Table 7: Reward coefficients

r_{c}

and curriculum length

T

for each robot.

Figure 8: Overlaid histograms of per-episode total reward (x-axis) vs. episode proportion (y-axis) for Forward datasets across all robots. Each panel corresponds to a robot; colors denote Expert, Expert replay, and 70% Suboptimal. Expert concentrates at high returns, Replay spans a wide range, and 70% Suboptimal places substantial mass on low returns. Vertical solid line: median; vertical dashed lines: 25th and 75th percentiles.

Panel of Figure 9 — Figure 9: Model overview: (a) URMA encoder; (b) State-action value network.

Figure 10: All robots with their images and the corresponding graph representation. Nodes are torso joints and feet. Edges follow kinematic adjacency and the connection from each terminal joint to its foot.

Parameter	Setting
Feature ground metric	Euclidean distance
Structure ground metric	Shortest path distance
Fusion tradeoff $\alpha$	$0.5$
Regularization $\varepsilon$	$1\mathrm{e}{-3}$

Appendix

Table 8: Essential hyperparameters for Fused Gromov–Wasserstein distance with POT

Panel of Figure 11 — Figure 11: Embodiment-group assignments obtained by cutting the same hierarchical clustering tree at different numbers of groups.

Panel of Figure 12 — Figure 12: Histograms of pairwise cosine similarities $C[\tau_{i},\tau_{j}]$ aggregated over training. In both settings—(a) higher suboptimal-data ratios and (b) greater embodiment diversity—the fraction with $C<0$ increases; within $C<0$ , mass concentrates at more negative values. Solid lines indicate the mean $\bar{C}$ .

Panel of Figure 15 — Figure 15: (a) Embodiment-based similarity matrix (1 $-$ min-max-normalized FGW distance between robot pairs); (b) Average gradient cosine similarity matrix for TD3+BC on the Expert Forward dataset, with the color scale compressed for readability using a symlog normalization; (c) Scatter plot of embodiment-based similarity and mean gradient cosine similarity for all robot pairs under TD3+BC.

Group count $M$	EG on Actor only	EG on Actor + Critic
2	$48.53\pm 2.33$	$48.38\pm 2.25$
4	$51.98\pm 1.70$	$50.30\pm 0.22$

Appendix

Table 10: Ablation on applying Embodiment Grouping to the critic. Returns are mean

\pm

s.e. over 3 seeds on the 70% Suboptimal Replay Forward dataset.

Parameter	IQL	BC	TD3+BC
Batch size per robot	1024	1024	1024
Learning rate	3e-4	3e-4	3e-4
Offline updates	1e5	1e5	1e5
Max grad norm	0.5	0.5	0.5
Discount factor $\gamma$	0.99	–	0.99
Value expectile $\tau_{\text{exp}}$	0.7	–	–

Appendix

Table 11: IQL, BC, and TD3+BC hyperparameters (common)

Parameter	Value
Total batch size / update	522240 (48 envs $\times$ 10880 steps)
Minibatch size	32640
SGD epochs per update	10
Learning rate (init $\rightarrow$ final)	0.0004 $\rightarrow$ 0.0 (linear anneal over 100M steps)
Entropy coefficient	0.0 or 0.1
Discount factor $\gamma$	0.99

Appendix

Table 12: PPO hyperparameters (dataset collection)

Explore similar work

Latent Policy Steering: An Efficient and Flexible Framework for Cross-Embodiment Transfer
Sep 18, 2026Yiqi Wang, Mrinal Verghese, Jeff SchneiderCross-Embodiment Transfer Visuomotor Policy Learning
Robotics Institute, School of Computer Science, Carnegie Mellon University, 5000 Forbes Ave, Pittsburgh, United States
One-Policy-Fits-All: Geometry-Aware Action Latents for Cross-Embodiment Manipulation
Mar 15, 2026Juncheng Mu, Sizhe Yang, Hojin Bae +5Cross-Embodiment Robot Learning Cross-Embodiment Transfer
Shanghai AI Laboratory · Tsinghua University · The Chinese University of Hong Kong +2
Data Analogies Enable Efficient Cross-Embodiment Transfer
Mar 6, 2026Jonathan Yang, Chelsea Finn, Dorsa SadighCross-Embodiment Transfer Generalization in Robotic Manipulation
Stanford University

Sep 18, 2026J/K move · Enter open · S save

Latent Policy Steering: An Efficient and Flexible Framework for Cross-Embodiment Transfer

Yiqi Wang, Mrinal Verghese, Jeff Schneider

Robotics Institute, School of Computer Science, Carnegie Mellon University, 5000 Forbes Ave, Pittsburgh, United States

Cross-Embodiment Transfer Visuomotor Policy Learning Robot Policy Generalization Data-Efficient Robot Learning World Models for Robotics

Open paper AskPDF arXiv

The performance of learned robot visuomotor policies depends heavily on the size and quality of their training data, yet collecting high-quality demonstrations remains costly for robots in the real world. Although large-scale robot and human datasets are increasingly available, embodiment gaps and mismatched action spaces make them difficult to leverage directly. Cross-embodiment transfer, reusing experience from other embodiments to improve learning on a target embodiment, is therefore crucial for scaling robot learning beyond per-robot data collection. In this work, we find that efficient transfer can be achieved by learning from what is shared across embodiments, the visual dynamics of how the world responds to motion, and by effectively exploiting the scarce target-embodiment data at test time. The proposed framework, called Latent Policy Steering (LPS), implements an embodiment-agnostic pretraining phase, which trains an image-based World Model (WM) with optical flow across diverse embodiments. The resulting WM is finetuned on the target embodiment with robot actions. It then steers the base policy toward better actions by searching in the WM's latent space for plans that stay close to the finetuning data. LPS is a policy-agnostic framework: it can flexibly accommodate different policies without having to retrain them. In Robomimic and real-world evaluations, LPS improves the average performance of Diffusion Policy relatively by 16% and 62%, and Pi0.5 by 8% and 14%, with only 50 demonstrations on an unseen target embodiment.

Abstract

Explore similar work

Latent Policy Steering: An Efficient and Flexible Framework for Cross-Embodiment Transfer

One-Policy-Fits-All: Geometry-Aware Action Latents for Cross-Embodiment Manipulation

Data Analogies Enable Efficient Cross-Embodiment Transfer