Organizations: Peking University · D-Robotics · Shenzhen Institute for Advanced Study, University of Electronic Science and Technology of China · Hong Kong Embodied AI Lab, The Chinese University of Hong Kong (CUHK)
Robot demonstrations are collected on different embodiments, leaving many robot-skill combinations unobserved. Pooling these data shares visual knowledge, while joint labels remain tied to the demonstrating kinematic chain. We propose a geometry-supervised training framework for zero-shot composition of familiar robots and skills. The central insight is that a demonstrated command provides a tool-space target for every robot that can realize it. Calibrated forward kinematics supplies this supervision to foreign native action interfaces without their joint labels for that skill. Native reconstruction retains demonstrated configurations, and positive reachability checks select targets for cross-robot supervision. A visual policy learns to predict shared actions and executes through the target robot's native decoder. During policy learning, one prediction receives geometric supervision through both source and foreign native realizations. Matched supervision controls, repeated policy training and evaluations across three backbones and two simulators demonstrate improved transfer to missing robot-task combinations.
Figures & tables
Figure 2: Shared tool-space supervision. (A) A shared encoder connects native action decoders. Source commands define a calibrated expert tool target that supervises eligible foreign realizations. (B) A visual policy learns shared actions from observed pairings and executes through the target robot’s frozen native decoder. The lower dashed branch adds optional policy-stage FK supervision. Robot poses and the tool target are schematic.
Figure 3: Selected paired executions with matched initial scenes. Each side shows four recorded stages. Labels report the complete execution outcome.
RoboTwin
ManiSkill3 (Push)
Native joints
Ours (Full)
Native joints
Ours (Full)
Backbone
Seen
Held
Seen
Held
Seen
Held
Seen
Held
π0.5
97.1
17.0
97.4
33.6
82.8
3.1
79.7
35.9
OpenWAM
75.3
34.8
78.7
46.5
80.4
11.6
81.3
43.5
FastWAM
82.9
9.7
84.0
29.8
78.9
5.4
80.1
28.6
Table 1: Cross-robot skill composition across policy backbones. Task-averaged success (%) on seen and held-out robot–task combinations in RoboTwin and ManiSkill3.
Figure 4: Independent paired and interface studies. (a) Matched supervision effects. (b) Cross minus Source across three policy runs. (c) Paired scene improvements, regressions and unchanged outcomes. (d) Foreign-decoding errors after native-only (Source) or foreign-FK (Cross) interface training. Intervals in (a,b) resample task clusters.
Figure 5: Two successful cross-robot expert-command replays through frozen interfaces. Active-arm command FK compares source and foreign tool motion. Source frames match the target command index.
Figure 6: Four stages of held-out policy executions with matched initial scenes and crops. Labels report complete execution outcomes. The ManiSkill panel renders recorded simulator states.
OpenWAM
FastWAM
Training
Seen
Held
Seen
Held
Direct EEF + IK
75.8
41.5
83.1
25.2
Source
76.3
33.4
83.9
17.7
Cross
80.1
42.9
84.4
26.3
Full
78.7
46.5
84.0
29.8
Table 2: Geometric supervision and action realization. Task-averaged success (%) on seen and held-out robot–task combinations.
Cross-task generalization is a core challenge in open-world robotic manipulation, and the key lies in extracting transferable manipulation knowledge from seen tasks. Recent in-context learning approaches leverage seen task demonstrations to generate actions for unseen tasks without parameter updates. However, existing methods provide only low-level continuous action sequences as context, failing to capture composable skill knowledge and causing models to degenerate into superficial trajectory imitation. We propose Decompose and Recompose, a skill reasoning framework using atomic skill-action pairs as intermediate representations. Our approach decomposes seen demonstrations into interpretable skill--action alignments, enabling the model to recompose these skills for unseen tasks through compositional reasoning. Specifically, we construct a task-adaptive dynamic demonstration library via visual-semantic retrieval combined with skill sequences from a planning agent, complemented by a coverage-aware static library to fill missing skill patterns. Together, these yield skill-comprehensive demonstrations that explicitly elicit compositional reasoning for skill composition and execution ordering. Experiments on the AGNOSTOS benchmark and real-world environments validate our method's zero-shot cross-task generalization capability.
Xitie Zhang, Aming Wu, Yahong Han
School of Artificial Intelligence, College of Intelligence and Computing, Tianjin University, China · School of Computer Science and Information Engineering, Hefei University of Technology, China
Robot datasets are costly to collect and are usually tied to a specific embodiment, making it difficult to reuse demonstrations across robots with different geometry, appearance, and kinematics. We introduce UniLatent, a cross-embodiment transfer framework that reduces the embodiment-induced observation gap in a shared latent space. UniLatent renders paired source-target robot observations in simulation and aligns a source visual encoder to a frozen target encoder. A policy trained on encoded source demonstrations can then run directly on target observations, with no pixel-level image translation. Across three simulation tasks and two target robot arms, UniLatent reaches up to 93% zero-shot success, exceeding the strongest pixel-translation baseline by over 30 points on average. In real-world xArm-to-Franka transfer, fine-tuning with only 10 target demonstrations per task yields 27/30 successful rollouts. Code, videos and data are available at https://selina2023.github.io/unilatent/.
Yufei Ding, Yi Du, Weiduo Yuan +5
Stanford University · University of Southern California · UC Berkeley
Given a demonstration, a robot should be able to generalize a skill to any object it encounters-but existing approaches to skill transfer often fail to adapt to objects with unfamiliar shapes. Motivated by examples of improved transfer from compositional modeling, we propose a method for improving transfer by decomposing objects into their constituent semantic parts. We leverage data-efficient generative shape models to accurately transfer interaction points from the parts of a demonstration object to a novel object. We autonomously construct an objective to optimize the alignment of those points on skill-relevant object parts. Our method generalizes to a wider range of object geometries than existing work, and achieves successful one-shot transfer for a range of skills and objects from a single demonstration, in both simulated and real environments.