cs.RONov 2, 2026

PointPilot: Unified Robotic Manipulation via Object-Centric 3D Point Flow Prediction

Authors: Yihan Fang, Yuxin Tian, Ruixiang Wang, Zhanqi Zhang, Xing Hu, Zhen Yang, Huayi Zhou, Yongyi Su, 

Organizations: The Chinese University of Hong Kong, Shenzhen · DexForce Technology Co., Ltd. · Shenzhen University · Foshan University

Abstract

Generalization remains a central challenge in robotics, calling for policies that address domain variations from multiple dimensions such as diverse embodiments, camera poses, and/or scene layouts. In order to approach this problem, we introduce PointPilot, a task-focused and 3D aligned manipulation framework built on a 3D space point-flow prediction. Rather than modeling the global scene, PointPilot only represents task-focused objects (robot grippers treated as objects), avoiding the prediction of potentially redundant scene parts unrelated to the manipulation tasks. Moreover, PointPilot aligns 3D spatial representation in one unified coordinate and supports closed-loop and fully automatic action execution across different manipulation modes from object point flow without a separately trained action expert, while existing works typically need an extra action head or are (semi-)automatic and single-arm mode specific. Given sensor observations and language instructions, our model jointly learns subtask planning, task-relevant object grounding, and 3D point flow prediction in an end-to-end way. By processing heterogeneous data in a shared 3D point flow space, PointPilot is able to absorb demonstrations from different embodiments. Extensive simulative and real world experiments validate our method. Notably, we demonstrate zero-shot embodiment generalization with PointPilot. A single trained policy can be deployed on previously unseen embodiments and successfully complete manipulation tasks without any additional fine-tuning.

Figures & tables

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Beyond Point-Attached Semantics: Stable Object-Centric Semantic Fields for Robust Manipulation

    Jul 3, 2026Zheng Sun, Lerong Zhang, Zhihao Li +33D Part SegmentationBimanual Robotic Manipulation

  2. 3PoinTr: From Human Videos to Robot Policies with 3D Point-Track Plans

    Mar 9, 2026Adam Hung, Bardienus Pieter Duisterhof, Jeffrey IchnowskiHuman-to-Robot Motion ImitationRobot Policy Learning

  3. UMR: Universal Manipulation Representation

    Sep 28, 2026Song Liu, Linyi Li, Yanshun Zhao +11Zero-Shot Robotic ManipulationGeneralization in Robotic Manipulation