PointPilot: Unified Robotic Manipulation via Object-Centric 3D Point Flow Prediction
Organizations: The Chinese University of Hong Kong, Shenzhen · DexForce Technology Co., Ltd. · Shenzhen University · Foshan University
Abstract
Generalization remains a central challenge in robotics, calling for policies that address domain variations from multiple dimensions such as diverse embodiments, camera poses, and/or scene layouts. In order to approach this problem, we introduce PointPilot, a task-focused and 3D aligned manipulation framework built on a 3D space point-flow prediction. Rather than modeling the global scene, PointPilot only represents task-focused objects (robot grippers treated as objects), avoiding the prediction of potentially redundant scene parts unrelated to the manipulation tasks. Moreover, PointPilot aligns 3D spatial representation in one unified coordinate and supports closed-loop and fully automatic action execution across different manipulation modes from object point flow without a separately trained action expert, while existing works typically need an extra action head or are (semi-)automatic and single-arm mode specific. Given sensor observations and language instructions, our model jointly learns subtask planning, task-relevant object grounding, and 3D point flow prediction in an end-to-end way. By processing heterogeneous data in a shared 3D point flow space, PointPilot is able to absorb demonstrations from different embodiments. Extensive simulative and real world experiments validate our method. Notably, we demonstrate zero-shot embodiment generalization with PointPilot. A single trained policy can be deployed on previously unseen embodiments and successfully complete manipulation tasks without any additional fine-tuning.
Figures & tables
| Method | Feat.1 | Feat.2 | Feat.3 | Feat.4 | Feat.5 |
|---|---|---|---|---|---|
| Im2Flow2Act ( Xu et al., 2025 ) | ✗ | ✓ | ✗ | ✓ | ✗ |
| PPI ( Yang et al., 2025 ) | ✗ | ✓ | ✗ | ✗ | ✓ |
| RoboFlow4D ( Lin et al., 2026 ) | ✗ | ✓ | ✗ | ✓ | ✗ |
| General Flow ( Yuan et al., 2025 ) | ✗ | ✗ | ✓ | ✓ | ✗ |
| Object-Part Scene Flow ( Tang et al., 2025 ) | ✓ | ✗ | ✓ | ✓ | ✗ |
| 3D Motion Field ( Yin et al., 2025 ) | ✗ | ✗ | ✓ | ✓ | ✗ |
| Task | ALOHA | ARX | Piper | |||
|---|---|---|---|---|---|---|
| PointPilot | X-VLA | PointPilot | X-VLA | PointPilot | X-VLA | |
| Grab roller | 98.0 | 100.0 | 85.0 | 79.0 | 79.0 | 21.0 |
| Shake bottle | 100.0 | 100.0 | 93.0 | 90.0 | 75.0 | 41.0 |
| Shake bottle horizontally | 94.0 | 99.0 | 90.0 | 90.0 | 83.0 | 41.0 |
| Click alarm clock | 75.0 | 17.0 | 69.0 | 62.0 | – | – |
| Click bell | 79.0 | 99.0 | 75.0 | 97.0 | 77.0 | 1.0 |
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Tensor | Shape |
|---|---|
| Per-view visual/PETR tokens | |
| Grounding memory | |
| Box outputs / role logits | / |
| Anchors / object tokens | / |
| DiT input | |
| Point velocity |
| Setting | Trials per task | Execution budget |
|---|---|---|
| LIBERO Spatial / Object | 50 | 500 |
| LIBERO Goal / Long | 50 | 1000 |
| RoboTwin clean | 100 | 400 |
| Paired cross-embodiment | 100 per embodiment | 300 |
| Real world | 20 | 600 |
Robotic Manipulation
Robotic manipulation studies how robots perceive, plan, and control physical interactions to grasp, move, assemble, and use objects.
See all 1,805 papers