cs.RONov 17, 2026

TCP4D: A Unified Tool-Center-Point Tracking and Generation Model for Robot Manipulation

Authors: Haosong Peng, Xiangyang Liu, Zhengshen Zhang, Yunfeng Fan, Tianyu Qi, Xin Xia, Yufeng Zhan, Wenchao Xu

Organizations: The Hong Kong University of Science and Technology · Beijing Institute of Technology · National University of Singapore · The Hong Kong Polytechnic University · 

Abstract

Robot manipulation videos provide a rich visual experience, yet recovering robot action and gripper-state annotations from RGB observations still remains challenging. Using the embodiment-agnostic tool-center-point (TCP) pose as the action space, we present TCP4D, a unified model in which a shared 4D scene representation both tracks observed TCP motion and generates future TCP trajectories. Given a monocular video and initial 2D TCP queries, the tracker directly recovers metric TCP poses in SE(3) and gripper states alongside scene geometry and camera parameters. The generator samples local features from the same 4D representation along the recovered TCP tracks, then combines them with robot observations and language instructions to predict future TCP states. Our new cross-embodiment augmentation pipeline converts existing RGB videos into demonstrations for alternative embodiments by retargeting the recovered motion and synthesizing the corresponding observations, without recorded source-robot states. Experiments show that the tracker achieves absolute trajectory errors of 7.60 and 16.07 mm on DROID and RoboLab and adapts to a new embodiment with only ten demonstrations. The generator achieves 69.19% mean success on RoboTwin and outperforms well-established policy models on two real-world tasks, while joint training on source and augmented demonstrations improves target-robot success by 41.5% over augmented-only training.

Figures & tables

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. GEM-4D: Geometry-Enhanced Video World Models for Robot Manipulation

    May 20, 2026Kaichen Zhou, Yuzhen Chen, Fangneng Zhan +8Physical Consistency in Video GenerationRobot Motion Generation

  2. PointAction: 3D Points as Universal Action Representations for Robot Control

    Jun 2, 2026Mutian Tong, Han Jiang, Qiao Feng +2Generalization in Robotic ManipulationCross-Embodiment Robot Learning

  3. MVISTA-4D: View-Consistent 4D World Model with Test-Time Action Inference for Robotic Manipulation

    Feb 10, 2026Jiaxu Wang, Yicheng Jiang, Tianlun He +7World Models for Robotics4D Scene Generation