Scalable robot policy pre-training has been hindered by the high cost of collecting high-quality demonstrations for each platform. In this study, we address this issue by uniting offline reinforcement learning (offline RL) with cross-embodiment learning. Offline RL leverages both expert and abundant suboptimal data, and cross-embodiment learning aggregates heterogeneous robot trajectories across diverse morphologies to acquire universal control priors. We perform a systematic analysis of this offline RL and cross-embodiment paradigm, providing a principled understanding of its strengths and limitations. To evaluate this offline RL and cross-embodiment paradigm, we construct a suite of locomotion datasets spanning 16 distinct robot platforms. Our experiments confirm that this combined approach excels at pre-training with datasets rich in suboptimal trajectories, outperforming pure behavior cloning. However, as the proportion of suboptimal data and the number of robot types increase, we observe that conflicting gradients across morphologies begin to impede learning. To mitigate this, we introduce an embodiment-based grouping strategy in which robots are clustered by morphological similarity and the model is updated with a group gradient. This simple, static grouping substantially reduces inter-robot conflicts and outperforms existing conflict-resolution methods.
Figures & tables
Dataset
BC
IQL
Expert Forward
63.31 ± 0.10
63.39 ± 0.05
Expert Backward
67.17 ± 0.01
67.10 ± 0.01
Expert Replay Forward
49.71 ± 1.06
54.61 ± 0.12
Expert Replay Backward
42.87 ± 1.32
51.86 ± 1.56
70% Suboptimal Forward
30.52 ± 3.10
36.62 ± 1.02
70% Suboptimal Backward
41.42 ± 0.71
38.69 ± 0.89
Table 1: BC vs. IQL performance across datasets (mean ± standard error over 5 seeds).
Figure 1: Comparison of learning curves between cross-embodiment pre-trained networks and networks trained without cross-embodiment pre-training for Badger, Unitree G1, and Cassie.
Figure 3
Figure 4: (a) Embodiment-based similarity matrix (1 - min-max-normalized FGW distance between robot pairs); (b) Gradient cosine similarity matrix in Expert Forward dataset from Section 5.1 , with the color scale compressed for readability using a symlog normalization; (c) Scatter plot of embodiment-based similarity and mean gradient cosine similarity for all robot pairs.
Figure 5: Overview of Embodiment Grouping (EG) for cross-embodiment offline RL.
Dataset
BC
TD3+BC
IQL
IQL + SEL
IQL + PCGrad
BC + EG
TD3+BC + EG
IQL + EG (ours)
Expert Forward
63.31 ± 0.10
52.14 ± 1.89
63.39 ± 0.05
63.37 ± 0.07
63.37 ± 0.04
63.47 ± 0.04
59.34 ± 1.19
63.52 ± 0.04
Expert Backward
67.17 ± 0.01
47.94 ± 0.48
67.10 ± 0.01
67.24 ± 0.02
67.05 ± 0.02
67.24 ± 0.02
51.98 ± 1.26
67.24 ± 0.01
Expert Replay Forward
49.71 ± 1.06
55.66 ± 0.84
54.61 ± 0.12
55.01 ± 0.55
53.84 ± 0.67
51.89 ± 0.65
57.04 ± 0.46
54.62 ± 0.53
Expert Replay Backward
42.87 ± 1.32
52.31 ± 1.22
51.86 ± 1.56
55.73 ± 1.06
55.94 ± 1.14
48.64 ± 2.65
55.67 ± 1.30
57.58 ± 0.05
70% Suboptimal Forward
30.52 ± 3.10
35.74 ± 1.51
36.62 ± 1.02
44.59 ± 2.02
39.63 ± 1.95
42.99 ± 1.23
43.41 ± 1.54
51.19 ± 1.06
70% Suboptimal Backward
41.42 ± 0.71
34.79 ± 1.40
38.69 ± 0.89
44.45 ± 1.75
41.04 ± 1.10
46.30 ± 2.39
40.88 ± 0.83
49.60 ± 2.39
Table 3: Each Algorithm Performance across Dataset ( ± is Standard Error, 5 seeds)
Method
Final return
Relative %
IQL (baseline)
37.57 ± 0.78
0.00%
Random grouping
38.73 ± 2.03
+3.08%
Heuristic
34.45 ± 1.97
-8.31%
EG (ours)
51.98 ± 1.70
+38.34%
Table 4: Ablation on grouping strategy (70% Suboptimal Forward). Values are mean final return ± SEM; Relative is vs. cross-embodiment IQL.
Figure 6: Effect of the number of Embodiment Grouping clusters M on final return and wall-clock training time (mean ± s.e.).
Figure 7: Illustration of the Forward and Backward locomotion commands used in our datasets.
Term
Name
Equation
(T1)
Xy velocity tracking
exp(−∣vxy−cxy∣2/0.25)
(T2)
Yaw velocity tracking
exp(−∣ωyaw−cyaw∣2/0.25)
(T3)
Z velocity penalty
−∣vz∣2
(T4)
Pitch-roll velocity penalty
−∣ωpitch, roll∣2
(T5)
Pitch-roll position penalty
−∣θpitch, roll∣2
(T6)
Joint nominal differences penalty
−∣q−qnominal∣2
Appendix
Table 6: Reward terms composing the reward function. Coefficients for each robot are listed in Table 7 .
Robot
T1
T2
T3
T4
T5
T6
T7
T8
T9
T10
T11
T12
T13
T14
T
ANYmalB
2.0
1.0
2.0
0.05
0.2
0.0
10.0
2.5e-7
2e-4
0.01
30.0
1.0
0.1
0.5
20e6
ANYmalC
2.0
1.0
2.0
0.05
0.2
0.0
10.0
2.5e-7
2e-4
0.01
30.0
1.0
0.1
0.5
20e6
Barkour v0
3.0
1.5
2.0
0.05
0.2
0.0
10.0
2.5e-7
2e-4
0.01
30.0
1.0
0.1
0.5
15e6
Barkour vB
2.0
1.0
2.0
0.05
0.2
0.0
10.0
2.5e-7
2e-4
0.01
30.0
1.0
0.1
0.5
15e6
Silver Badger
2.0
1.0
2.0
0.05
0.2
0.0
10.0
2.5e-7
2e-4
0.01
30.0
1.0
0.1
0.5
12e6
Bittle
5.0
2.5
2.0
0.05
0.2
0.0
10.0
2.5e-7
2e-4
0.01
30.0
1.0
0.1
0.5
40e6
Appendix
Table 7: Reward coefficients rc and curriculum length T for each robot.
Figure 8: Overlaid histograms of per-episode total reward (x-axis) vs. episode proportion (y-axis) for Forward datasets across all robots. Each panel corresponds to a robot; colors denote Expert, Expert replay, and 70% Suboptimal. Expert concentrates at high returns, Replay spans a wide range, and 70% Suboptimal places substantial mass on low returns. Vertical solid line: median; vertical dashed lines: 25th and 75th percentiles.
Figure 9: Model overview: (a) URMA encoder; (b) State-action value network.
Figure 10: All robots with their images and the corresponding graph representation. Nodes are torso joints and feet. Edges follow kinematic adjacency and the connection from each terminal joint to its foot.
Parameter
Setting
Feature ground metric
Euclidean distance
Structure ground metric
Shortest path distance
Fusion tradeoff α
0.5
Regularization ε
1e−3
Appendix
Table 8: Essential hyperparameters for Fused Gromov–Wasserstein distance with POT
Figure 11: Embodiment-group assignments obtained by cutting the same hierarchical clustering tree at different numbers of groups.
Figure 12: Histograms of pairwise cosine similarities C[τi,τj] aggregated over training. In both settings—(a) higher suboptimal-data ratios and (b) greater embodiment diversity—the fraction with C<0 increases; within C<0 , mass concentrates at more negative values. Solid lines indicate the mean Cˉ .
Figure 19
Figure 15: (a) Embodiment-based similarity matrix (1 − min-max-normalized FGW distance between robot pairs); (b) Average gradient cosine similarity matrix for TD3+BC on the Expert Forward dataset, with the color scale compressed for readability using a symlog normalization; (c) Scatter plot of embodiment-based similarity and mean gradient cosine similarity for all robot pairs under TD3+BC.
Group count M
EG on Actor only
EG on Actor + Critic
2
48.53±2.33
48.38±2.25
4
51.98±1.70
50.30±0.22
Appendix
Table 10: Ablation on applying Embodiment Grouping to the critic. Returns are mean ± s.e. over 3 seeds on the 70% Suboptimal Replay Forward dataset.
Parameter
IQL
BC
TD3+BC
Batch size per robot
1024
1024
1024
Learning rate
3e-4
3e-4
3e-4
Offline updates
1e5
1e5
1e5
Max grad norm
0.5
0.5
0.5
Discount factor γ
0.99
–
0.99
Value expectile τexp
0.7
–
–
Appendix
Table 11: IQL, BC, and TD3+BC hyperparameters (common)
The performance of learned robot visuomotor policies depends heavily on the size and quality of their training data, yet collecting high-quality demonstrations remains costly for robots in the real world. Although large-scale robot and human datasets are increasingly available, embodiment gaps and mismatched action spaces make them difficult to leverage directly. Cross-embodiment transfer, reusing experience from other embodiments to improve learning on a target embodiment, is therefore crucial for scaling robot learning beyond per-robot data collection. In this work, we find that efficient transfer can be achieved by learning from what is shared across embodiments, the visual dynamics of how the world responds to motion, and by effectively exploiting the scarce target-embodiment data at test time. The proposed framework, called Latent Policy Steering (LPS), implements an embodiment-agnostic pretraining phase, which trains an image-based World Model (WM) with optical flow across diverse embodiments. The resulting WM is finetuned on the target embodiment with robot actions. It then steers the base policy toward better actions by searching in the WM's latent space for plans that stay close to the finetuning data. LPS is a policy-agnostic framework: it can flexibly accommodate different policies without having to retrain them. In Robomimic and real-world evaluations, LPS improves the average performance of Diffusion Policy relatively by 16% and 62%, and Pi0.5 by 8% and 14%, with only 50 demonstrations on an unseen target embodiment.
Yiqi Wang, Mrinal Verghese, Jeff Schneider
Robotics Institute, School of Computer Science, Carnegie Mellon University, 5000 Forbes Ave, Pittsburgh, United States
Cross-embodiment manipulation is crucial for enhancing the scalability of robot manipulation and reducing the high cost of data collection. However, the significant differences between embodiments, such as variations in action spaces and structural disparities, pose challenges for joint training across multiple sources of data. To address this, we propose One-Policy-Fits-All (OPFA), a framework that enables learning a single, versatile policy across multiple embodiments. We first learn a Geometry-Aware Latent Representation (GaLR), which leverages 3D convolution networks and transformers to build a shared latent action space across different embodiments. Then we design a unified latent retargeting decoder that extracts embodiment-specific actions from the latent representations, without any embodiment-specific decoder tuning. OPFA enables end-to-end co-training of data from diverse embodiments, including various grippers and dexterous hands with arbitrary degrees of freedom, significantly improving data efficiency and reducing the cost of skill transfer. We conduct extensive experiments across 11 different end-effectors. The results demonstrate that OPFA significantly improves policy performance in diverse settings by leveraging heterogeneous embodiment data. For instance, cross-embodiment co-training can improve success rates by more than 50% compared to single-source training. Moreover, by adding only a few demonstrations from a new embodiment (e.g., eight), OPFA can achieve performance comparable to that of a well-trained model with 72 demonstrations.
Juncheng Mu, Sizhe Yang, Hojin Bae +5
Shanghai AI Laboratory · Tsinghua University · The Chinese University of Hong Kong +2
Generalist robot policies are trained on demonstrations collected across a wide variety of robots, scenes, and viewpoints. Yet it remains unclear how to best organize and scale such heterogeneous data so that it genuinely improves performance in a given target setting. In this work, we ask: what form of demonstration data is most useful for enabling transfer across robot set-ups? We conduct controlled experiments that vary end-effector morphology, robot platform appearance, and camera perspective, and compare the effects of simply scaling the number of demonstrations against systematically broadening the diversity in different ways. Our simulated experiments show that while perceptual shifts such as viewpoint benefit most from broad diversity, morphology shifts benefit far less from unstructured diversity and instead see the largest gains from data analogies, i.e. paired demonstrations that align scenes, tasks, and/or trajectories across different embodiments. Informed by the simulation results, we improve real-world cross-embodiment transfer success by an average of 22.5% over large-scale, unpaired datasets by changing only the composition of the data.