Robot datasets are costly to collect and are usually tied to a specific embodiment, making it difficult to reuse demonstrations across robots with different geometry, appearance, and kinematics. We introduce UniLatent, a cross-embodiment transfer framework that reduces the embodiment-induced observation gap in a shared latent space. UniLatent renders paired source-target robot observations in simulation and aligns a source visual encoder to a frozen target encoder. A policy trained on encoded source demonstrations can then run directly on target observations, with no pixel-level image translation. Across three simulation tasks and two target robot arms, UniLatent reaches up to 93% zero-shot success, exceeding the strongest pixel-translation baseline by over 30 points on average. In real-world xArm-to-Franka transfer, fine-tuning with only 10 target demonstrations per task yields 27/30 successful rollouts. Code, videos and data are available at https://selina2023.github.io/unilatent/.
Figures & tables
Figure 1: UniLatent aligns robot observations for cross-embodiment transfer. UniLatent renders paired source–target images and aligns a source encoder to a target encoder, forming a shared latent space that reduces embodiment-induced visual gaps. Top: source and target embeddings increasingly overlap during training. Bottom: cross-decoded source observations become more target-consistent as alignment improves.
Figure 2: Pipeline overview of UniLatent. We render paired (source, target) observations {(oi(S),oi(T))} for matched end-effector poses and use them to align encoders by training ES to match a fixed ET in a shared latent space. A latent policy πθ is then trained on source demonstrations and deployed on the target robot by encoding target observations with ET and calling πθ .
Figure 3: Paired image generation. Examples of simulation-rendered paired observations for three robots (top to bottom): Franka, xArm, and Sawyer. For each sampled end-effector pose, we render the robots under randomized backgrounds, lighting, and scene variations while preserving action-level correspondence between paired frames.
Figure 4: Simulation tasks. Example frames from our simulation experiments. Each row shows representative frames of a Franka demonstration under a fixed third-person camera.
Task & Setup
Close Box Lid
Stack Cup into Bowl
Kick Football
Franka
xArm
Sawyer
Franka
xArm
Sawyer
Franka
xArm
Sawyer
Direct Policy Transfer
97
2
5
80
0
0
90
1
1
DINOv2 w/o align.
97
40
16
89
3
0
93
11
1
RoVi-Aug*
97
93
54
89
15
5
93
60
40
UniLatent (ours)
97
93
81
89
65
51
93
89
77
Table 1: Zero-shot transfer results in simulation. Policies are trained on Franka demonstrations and evaluated on Franka (in-domain) and zero-shot on xArm and Sawyer. We compare direct policy transfer, a shared frozen DINOv2 encoder without alignment, RoVi-Aug* (our adaptation of RoVi-Aug), and the full UniLatent pipeline. Entries report success rates (%) over 100 rollouts.
Figure 5: Pixel-space translation examples. Each row shows source observation o(S) , its target pair o(T) , decoded source translation DT(ES(o(S))) used by RoVi-Aug*, and target reconstruction DT(ET(o(T))) .
Figure 7
Figure 7: t-SNE of latent embeddings during alignment. Each point denotes a latent embedding of a rendered image; colors indicate robot embodiments, and marker shapes indicate paired rendered samples. Across training (1 → 6), embeddings from different robots become more aligned in the latent space.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Qualitative comparison of pixel-space translation baselines for cross-embodiment transfer. Source is the input xArm observation; Target is the ground-truth Franka observation under the same end-effector pose. RoVi-Aug , Shadow ∗ , and RoVi-Aug ∗ show diffusion-based image generation, segmentation-masking, and cross-decoding translations, respectively.
Pooling
Close Box Lid
Stack Cup into Bowl
Kick Football
Mean
10
3
31
CLS token
6
0
5
Mean+CLS
15
3
29
Attn+CLS
97
89
93
Appendix
Table 4: Pooling ablation in a single-robot setting (Franka, simulation) with frozen DINOv2 backbone. We report task success rates (%).
Pooling
Close Box Lid
Stack Cup into Bowl
Kick Football
Mean
95
92
88
CLS token
80
69
79
Mean+CLS
95
91
89
Attn+CLS
15
2
18
Appendix
Table 5: Pooling ablation in a single-robot setting (Franka, simulation) with unfrozen DINOv2 backbone trained end-to-end with the policy. We report task success rates (%).
Close Box Lid
Stack Cup into Bowl
Kick Football
Avg.
Policy
Franka
xArm
Franka
xArm
Franka
xArm
Diffusion (DDPM)
89
85
79
49
84
76
77.0
Flow Matching
97
93
89
65
93
89
87.7
Appendix
Table 6: Policy type comparison in the aligned latent space. Both policies use the same aligned encoders, UNet backbone, training data, and hyperparameters; only the generative formulation differs. Franka is the source robot (in-domain); xArm is the transfer target (Franka → xArm). We report success rates (%) over 100 rollouts.
Pose generator for paired poses
xArm
Sawyer
Unconstrained random poses
60
32
Manipulation-relevant pose prior
93
81
Appendix
Table 7: Effect of pose sampling for paired rendering. We train the observation alignment stage with either unconstrained random TCP-pose samples or the proposed manipulation-relevant pose prior, and evaluate zero-shot Close Box Lid transfer from Franka to xArm and Sawyer. Entries report success rates (%) over 100 simulation rollouts per target robot.
Figure 9: Qualitative comparison of pose samplers for paired rendering. Unconstrained random pose sampling frequently produces implausible robot configurations, including twisted joints, collisions, and out-of-view grippers. In contrast, the manipulation-relevant pose prior produces task-plausible tabletop configurations with diverse gripper states and orientations, providing more useful paired observations for alignment training.
Franka (source)
xArm
Sawyer
Randomized background
88.6
85.6
80.0
Matched background
97
93
81
Appendix
Table 8: Background generalization via image randomization. Success rates (%) on the simulated Close Box Lid task, with Franka as the source robot and xArm and Sawyer as transfer targets. The first row trains with randomized source backgrounds and tests in 5 unseen scenes; the second uses matched train–test backgrounds.
The performance of learned robot visuomotor policies depends heavily on the size and quality of their training data, yet collecting high-quality demonstrations remains costly for robots in the real world. Although large-scale robot and human datasets are increasingly available, embodiment gaps and mismatched action spaces make them difficult to leverage directly. Cross-embodiment transfer, reusing experience from other embodiments to improve learning on a target embodiment, is therefore crucial for scaling robot learning beyond per-robot data collection. In this work, we find that efficient transfer can be achieved by learning from what is shared across embodiments, the visual dynamics of how the world responds to motion, and by effectively exploiting the scarce target-embodiment data at test time. The proposed framework, called Latent Policy Steering (LPS), implements an embodiment-agnostic pretraining phase, which trains an image-based World Model (WM) with optical flow across diverse embodiments. The resulting WM is finetuned on the target embodiment with robot actions. It then steers the base policy toward better actions by searching in the WM's latent space for plans that stay close to the finetuning data. LPS is a policy-agnostic framework: it can flexibly accommodate different policies without having to retrain them. In Robomimic and real-world evaluations, LPS improves the average performance of Diffusion Policy relatively by 16% and 62%, and Pi0.5 by 8% and 14%, with only 50 demonstrations on an unseen target embodiment.
Yiqi Wang, Mrinal Verghese, Jeff Schneider
Robotics Institute, School of Computer Science, Carnegie Mellon University, 5000 Forbes Ave, Pittsburgh, United States
Recent progress in large-scale imitation learning for robot manipulation has been driven by leveraging datasets across a wide range of robot embodiments. However, achieving significant cross-embodiment transfer is often still challenging. In this work, we study the role of using behavior-aligned representations (e.g., object bounding boxes, language motions, end-effector traces of robot motion) in vision-language-action (VLA) models to promote cross-embodiment transfer. We hypothesize that by possessing invariances across embodiments while being predictive of robot actions, these representations can help unify large-scale cross-embodiment data to enhance transfer. To assess our hypothesis, we develop a simulation-based benchmark designed to assess transfer with diverse cross-embodiment data to new embodiments. Using this benchmark, we compare different representations and ways of incorporating them. We identify that end-effector traces can be particularly beneficial for transfer, representations are generally more useful with larger prior datasets, and can be used to benefit from action-free data. We also demonstrate that they can enhance sim-to-real cross-embodiment transfer, improving task completion progress of real robot policies pre-trained on simulation data by 28%. We provide videos of our evaluations at our website: https://ajaysridhar.com/barx/.
Human demonstrations provide a scalable source of diverse manipulation data, but leveraging them for robot learning remains challenging due to differences in embodiment, viewpoint, action spaces, and temporal structure. We introduce BATCH, a cross-embodiment learning framework that explicitly aligns human and robot behaviors. BATCH learns self-supervised temporal representations with a cycle-consistency objective to establish cross-embodiment correspondences without manual annotation. We further introduce two complementary training objectives that leverage these correspondences within an autoregressive videoaction diffusion model: behavior-aligned co-training and human-to-robot style transfer. On real-world manipulation tasks, BATCH improves generalization to unseen task configurations compared with training on unaligned data and prior cross-embodiment baselines, with additional gains from the style-transfer objective. Our results demonstrate that explicit behavior alignment enables more effective transfer from human demonstrations to robot policies.