Zero-Shot Compositional Generalization for Embodied Manipulation
Organizations: Peking University · D-Robotics · Shenzhen Institute for Advanced Study, University of Electronic Science and Technology of China · Hong Kong Embodied AI Lab, The Chinese University of Hong Kong (CUHK)
Abstract
Robot demonstrations are collected on different embodiments, leaving many robot-skill combinations unobserved. Pooling these data shares visual knowledge, while joint labels remain tied to the demonstrating kinematic chain. We propose a geometry-supervised training framework for zero-shot composition of familiar robots and skills. The central insight is that a demonstrated command provides a tool-space target for every robot that can realize it. Calibrated forward kinematics supplies this supervision to foreign native action interfaces without their joint labels for that skill. Native reconstruction retains demonstrated configurations, and positive reachability checks select targets for cross-robot supervision. A visual policy learns to predict shared actions and executes through the target robot's native decoder. During policy learning, one prediction receives geometric supervision through both source and foreign native realizations. Matched supervision controls, repeated policy training and evaluations across three backbones and two simulators demonstrate improved transfer to missing robot-task combinations.
Figures & tables
| RoboTwin | ManiSkill3 (Push) | |||||||
|---|---|---|---|---|---|---|---|---|
| Native joints | Ours (Full) | Native joints | Ours (Full) | |||||
| Backbone | Seen | Held | Seen | Held | Seen | Held | Seen | Held |
| 97.1 | 17.0 | 97.4 | 33.6 | 82.8 | 3.1 | 79.7 | 35.9 | |
| OpenWAM | 75.3 | 34.8 | 78.7 | 46.5 | 80.4 | 11.6 | 81.3 | 43.5 |
| FastWAM | 82.9 | 9.7 | 84.0 | 29.8 | 78.9 | 5.4 | 80.1 | 28.6 |
| OpenWAM | FastWAM | |||
| Training | Seen | Held | Seen | Held |
| Direct EEF + IK | 75.8 | 41.5 | 83.1 | 25.2 |
| Source | 76.3 | 33.4 | 83.9 | 17.7 |
| Cross | 80.1 | 42.9 | 84.4 | 26.3 |
| Full | 78.7 | 46.5 | 84.0 | 29.8 |
Generalization in Robotic Manipulation
Generalization in robotic manipulation studies how robots transfer grasping and interaction skills to new objects, scenes, tasks, and embodiments.
See all 381 papers