Organizations: Shanghai AI Laboratory · Tsinghua University · The Chinese University of Hong Kong · University of Science and Technology of China · NVIDIA
Cross-embodiment manipulation is crucial for enhancing the scalability of robot manipulation and reducing the high cost of data collection. However, the significant differences between embodiments, such as variations in action spaces and structural disparities, pose challenges for joint training across multiple sources of data. To address this, we propose One-Policy-Fits-All (OPFA), a framework that enables learning a single, versatile policy across multiple embodiments. We first learn a Geometry-Aware Latent Representation (GaLR), which leverages 3D convolution networks and transformers to build a shared latent action space across different embodiments. Then we design a unified latent retargeting decoder that extracts embodiment-specific actions from the latent representations, without any embodiment-specific decoder tuning. OPFA enables end-to-end co-training of data from diverse embodiments, including various grippers and dexterous hands with arbitrary degrees of freedom, significantly improving data efficiency and reducing the cost of skill transfer. We conduct extensive experiments across 11 different end-effectors. The results demonstrate that OPFA significantly improves policy performance in diverse settings by leveraging heterogeneous embodiment data. For instance, cross-embodiment co-training can improve success rates by more than 50% compared to single-source training. Moreover, by adding only a few demonstrations from a new embodiment (e.g., eight), OPFA can achieve performance comparable to that of a well-trained model with 72 demonstrations.
Figures & tables
Fig. 2 : The training pipeline of OPFA follows a two-stage paradigm. (1) We first construct a Geometry-Aware Latent Representation (GaLR) by encoding sampled reachable-state point clouds with 3D convolutions and geometric transformers for local/global feature extraction. A unified latent retargeting decoder then disentangles embodiment-specific actions from the latent space, enabling end-to-end training without manual annotations. (2) The pretrained encoder–decoder pair is integrated into any downstream policy (e.g., DP3), allowing cross-embodiment data to be jointly trained in a unified latent action space.
Fig. 3 : Spatial generalization evaluation on the spray-picking task. Data for each embodiment are collected in distinct regions, and we evaluate the policy for each embodiment to generalize to regions covered by data from the others.
Task
Kettle
Button
Sanitizer
Bucket
Banana
Pick Spray&Can
Spatial Generalization
Object Generalization
Inspire Hand Success Rate (%)
w/o Co-training
30.0
26.0
5.0
3.0
10.0
1.0
Naive Co-training
57.0
39.0
71.0
50.0
83.0
57.0
OPFA (ours)
83.0
60.0
82.0
75.0
98.0
83.0
Xhand Success Rate (%)
TABLE I : Zero-shot skill transfer experiments. In the table, Position Generalization refers to different data collection regions for different embodiments, while Object Generalization refers to different manipulation objects.
Fig. 4 : Few-shot learning curves with different demo numbers on the banana-picking task of (Left) Inspire Hand and (Right) XHand . Each embodiment is co-trained with 72 demonstrations from the other.
Fig. 5 : Few-shot learning performance across nine different embodiments on the spray-picking task. We collect eight trajectories for each of the nine end-effectors and co-train on the full dataset to evaluate the few-shot learning capability for each embodiment. OPFA yields a 20%+ average performance gain over the baselines.
Fig. 6 : Tasks for real world evaluation. Basket and Mango are one-stage pick-and-place tasks. Tissue and Pot are contact-intensive tasks, involving deformable object handling or precise manipulation. Broom and Drawer are long-horizon sequential tasks involving tool-use and multi-step coordination. Syringe is a fine-motor control task demanding precise force application.
Task
Basket
Tissue
Broom
Pot
Mango
Drawer
Syringe
Inspire Hand Success Rate (%)
w/o Co-training
90
70
70
80
60
60
-
Naive Co-training
90
70
80
70
80
70
-
OPFA (ours)
100
60
100
100
100
80
-
Leap Hand Success Rate (%)
w/o Co-training
90
70
80
40
60
70
60
TABLE II : Real-world experiments success rates. Best and second-best methods are highlighted with different colors.
Recent progress in large-scale imitation learning for robot manipulation has been driven by leveraging datasets across a wide range of robot embodiments. However, achieving significant cross-embodiment transfer is often still challenging. In this work, we study the role of using behavior-aligned representations (e.g., object bounding boxes, language motions, end-effector traces of robot motion) in vision-language-action (VLA) models to promote cross-embodiment transfer. We hypothesize that by possessing invariances across embodiments while being predictive of robot actions, these representations can help unify large-scale cross-embodiment data to enhance transfer. To assess our hypothesis, we develop a simulation-based benchmark designed to assess transfer with diverse cross-embodiment data to new embodiments. Using this benchmark, we compare different representations and ways of incorporating them. We identify that end-effector traces can be particularly beneficial for transfer, representations are generally more useful with larger prior datasets, and can be used to benefit from action-free data. We also demonstrate that they can enhance sim-to-real cross-embodiment transfer, improving task completion progress of real robot policies pre-trained on simulation data by 28%. We provide videos of our evaluations at our website: https://ajaysridhar.com/barx/.
Generalizing manipulation policies across robot embodiments remains difficult because standard policies entangle task reasoning with embodiment-specific motor control. We study zero-shot cross-embodiment manipulation, where a policy trained on source embodiments must be deployed on a structurally different target embodiment without additional task demonstrations. We introduce Kinematic Interaction Transfer across Embodiments (KITE), which decouples manipulation into embodiment-agnostic task reasoning and embodiment-specific motor control, connected through a learned latent representation of interaction intent based on contact patterns. Task reasoning is performed by a shared policy that predicts latent intents from source demonstrations, while motor control is performed by an intent-conditioned action decoder learned from each embodiment's kinematic model. With KITE, adaptation to a new embodiment requires only training a new action decoder using its kinematic model, without recollecting demonstration data. We evaluate KITE on three manipulation tasks spanning transfer between parallel grippers, dexterous hands, and composite embodiments. KITE consistently achieves zero-shot transfer to structurally different target embodiments, outperforming state-of-the-art baselines in transfer success and task-embodiment scope.
Reusing manipulation experience across robot embodiments is important for scaling robot learning and reducing repeated task-specific data collection. However, changes in embodiment alter visual appearance, action dimensionality and semantics, and the whole-body configurations that can realize the same tool pose. We present SkelWAM, a skeleton-guided world-action model that couples perception and control through one explicit geometric representation for single-source cross-embodiment manipulation. Arm centerline geometry, tool-center-point (TCP) pose, and parallel-jaw commands form a shared 25-D state. The same definition underlies canonical third-person and wrist observations and future whole-body action targets. Trained with predictive visual supervision, a video-action mixture of transformers predicts canonical skeleton action chunks, which embodiment-specific constrained decoders convert into joint or continuum-robot controls. This formulation requires no one-to-one joint correspondence and uses no target-task demonstrations or target policy updates. We introduce LIBERO-Cross10, a source-only cross-embodiment transfer benchmark covering ten tasks and ten target embodiments across four morphological groups. On this benchmark, Franka-trained SkelWAM achieves 43.3% success over 1,000 episodes, exceeding the best-performing evaluated baseline by 36.2 percentage points. We further deploy a JAKA mini2-trained policy on the Feagine A03 continuum robot for three tabletop manipulation tasks, illustrating the approach's potential for real-world cross-embodiment manipulation. Project page: http://www.liukepku.com/skelwam/index.html
Pengjun Niu, Yujia Xie, Rui Peng +2
School of Advanced Manufacturing and Robotics, Peking University, Beijing, 100871, China · Feagine · IIIS, Tsinghua University +1