The performance of learned robot visuomotor policies depends heavily on the size and quality of their training data, yet collecting high-quality demonstrations remains costly for robots in the real world. Although large-scale robot and human datasets are increasingly available, embodiment gaps and mismatched action spaces make them difficult to leverage directly. Cross-embodiment transfer, reusing experience from other embodiments to improve learning on a target embodiment, is therefore crucial for scaling robot learning beyond per-robot data collection. In this work, we find that efficient transfer can be achieved by learning from what is shared across embodiments, the visual dynamics of how the world responds to motion, and by effectively exploiting the scarce target-embodiment data at test time. The proposed framework, called Latent Policy Steering (LPS), implements an embodiment-agnostic pretraining phase, which trains an image-based World Model (WM) with optical flow across diverse embodiments. The resulting WM is finetuned on the target embodiment with robot actions. It then steers the base policy toward better actions by searching in the WM's latent space for plans that stay close to the finetuning data. LPS is a policy-agnostic framework: it can flexibly accommodate different policies without having to retrain them. In Robomimic and real-world evaluations, LPS improves the average performance of Diffusion Policy relatively by 16% and 62%, and Pi0.5 by 8% and 14%, with only 50 demonstrations on an unseen target embodiment.
Figures & tables
Fig. 2: Optical flow reduces embodiment gaps . We observe that motions captured by optical flow across embodiments are similar in visual space. For example, picking up a cup will generate similar flow vectors in both the agent and the cup, regardless of the agent embodiment. By using optical flow as a transition representation of visual dynamics, we reduce the pretrained model’s dependency on specific embodiments, improving transferability to a new embodiment.
Fig. 3: Real-world experiment . The proposed method and baselines are evaluated on: (a) Put-radish-in-pot, (b) Sweep-salad-off-the-board, (c) Scoop-beads-with-spoon, and (d) Fold-towel-to-triangle.
Fig. 4: Pretraining data curation . This chart summarizes the pretraining data by quality (inner ring), domain (middle ring), and embodiment (outer ring). The dataset comprises roughly 55 hours of interaction data across 9 embodiments from different domains, including non-expert data.
Fig. 5: Scalability and sample efficiency . Left: LPS’s WM is pretrained on an increasing number of embodiments, and finetuned with 50 demonstrations. Thanks to the embodiment-agnostic pretraining, LPS’s performance scales effectively when provided with more data from diverse embodiments during pretraining. Right: LPS’s WM is pretrained on all embodiments and then receives increasing numbers of demonstrations (50–300) to finetune the WM and train the base policy. It takes at least twice the data for the DP baseline to outperform LPS-DP (LPS-DP with 50 demonstrations outperforms DP with 100 demonstrations), highlighting the sample efficiency of LPS in the low-data regime.
Task \ Setting
From scratch
Pretrain-and-finetune
DP
LPS-DP- scratch
HPT
Pi0.5
LPS- DP
LPS- Pi0.5
Lift
83.9±1.5
86.9±1.7
58.2±2.6
100 ±0.0
90.1±1.8
100 ±0.0
Can
77.8±2.2
81.0±3.2
79.4±3.5
87.8±0.8
88.0±1.9
94.4 ±1.1
Square
40.5±3.8
46.3±2.7
32.5±2.5
30.7±1.0
53.4 ±4.8
35.1±1.2
Transport
26.4±1.8
29.5±3.2
11.5±1.4
25.1±4.6
35.3 ±3.0
34.5±2.0
Average
57.2%
60.9%
45.4%
60.9%
66.7 %
66.0%
TABLE I: Latent Policy Steering in Robomimic
Task\Settings
From scratch
Pretrain-and-finetune
DP
LPS-DP- scratch
Pi0.5
LPS- DP
LPS- Pi0.5
Put-radish- in-pot
12/20
13/20
20 /20
18/20
20 /20
Sweep-salad- off-the-board
5/20
7/20
10/20
11/20
12 /20
Scoop-beads- with-spoon
13/20
15/20
16/20
18/20
20 /20
Fold-towel- to-triangle
7/20
7/20
9/20
13 /20
12/20
Average
46.2%
52.5%
68.8%
75.0%
80.0 %
TABLE II: Latent Policy Steering in the real world
Fig. 6: Ablations on LPS’s design . Pretraining the WM brings the largest improvement in this low-data regime (LPS-DP vs. no-pretrain). In addition, naively learning a value function for policy steering from expert states with binary rewards (no-pretrain-no-robust) provides only marginal benefit, and learning from extra states without the reward that penalizes distribution shift decreases performance (no-pretrain-no-reward).
Fig. 7: Effective horizons. LPS effectively improves policy performance for horizons up to length 16. When the horizon becomes very long (e.g., 20, 24), the reward used to capture distribution shift becomes noisy, making the value function less effective during inference.
Recent progress in large-scale imitation learning for robot manipulation has been driven by leveraging datasets across a wide range of robot embodiments. However, achieving significant cross-embodiment transfer is often still challenging. In this work, we study the role of using behavior-aligned representations (e.g., object bounding boxes, language motions, end-effector traces of robot motion) in vision-language-action (VLA) models to promote cross-embodiment transfer. We hypothesize that by possessing invariances across embodiments while being predictive of robot actions, these representations can help unify large-scale cross-embodiment data to enhance transfer. To assess our hypothesis, we develop a simulation-based benchmark designed to assess transfer with diverse cross-embodiment data to new embodiments. Using this benchmark, we compare different representations and ways of incorporating them. We identify that end-effector traces can be particularly beneficial for transfer, representations are generally more useful with larger prior datasets, and can be used to benefit from action-free data. We also demonstrate that they can enhance sim-to-real cross-embodiment transfer, improving task completion progress of real robot policies pre-trained on simulation data by 28%. We provide videos of our evaluations at our website: https://ajaysridhar.com/barx/.
As generalist robot policies gain vision and language from web-scale pretraining, demonstrations remain costly to collect and tied to the robot that recorded them. Latent action models (LAMs) address both by learning latent actions from action-free videos that can be shared across embodiments, however, in practice, LAMs are sensitive to background visual noise, and the same motion from two different robots may be encoded with different latents. One solution to the background visual noise is to add an auxiliary loss predicting the robot action from the latent action, further associating the latent action space to the embodiment specific robot action space. We study a different use of the same labels, through action-similarity supervision. The similarity between any two latent actions is trained to match the similarity of the two ground-truth robot action sequences. The ground-truth actions are never predicted by the LAM, so the latent action does not need to encode embodiment specifics. We evaluate cross-embodiment transfer on RoboTwin 2.0 in a controlled setup, two bimanual robots demonstrate disjoint task sets, a policy is trained on all the demonstrations, and each robot is evaluated closed-loop on the tasks only the other demonstrated. With the policy architecture and its hyperparameters, the dataset, and the evaluation protocol fixed, predicting latent actions instead of ground-truth actions more than doubles cross-embodiment success. Given the same ground-truth actions, similarity supervision transfers better than an auxiliary loss that predicts the ground-truth action during the LAM training. Computing the similarities on end-effector motion rather than joint-space motion, and letting the loss compare latent actions across the two robots, gives the best approach of the study.
Maxime Alvarez, Renzo Caballero, Tatsuya Matsushima +2
Graduate School of Engineering, The University of Tokyo
Generalist robot policies are trained on demonstrations collected across a wide variety of robots, scenes, and viewpoints. Yet it remains unclear how to best organize and scale such heterogeneous data so that it genuinely improves performance in a given target setting. In this work, we ask: what form of demonstration data is most useful for enabling transfer across robot set-ups? We conduct controlled experiments that vary end-effector morphology, robot platform appearance, and camera perspective, and compare the effects of simply scaling the number of demonstrations against systematically broadening the diversity in different ways. Our simulated experiments show that while perceptual shifts such as viewpoint benefit most from broad diversity, morphology shifts benefit far less from unstructured diversity and instead see the largest gains from data analogies, i.e. paired demonstrations that align scenes, tasks, and/or trajectories across different embodiments. Informed by the simulation results, we improve real-world cross-embodiment transfer success by an average of 22.5% over large-scale, unpaired datasets by changing only the composition of the data.