Organizations: The Hong Kong University of Science and Technology · Beijing Institute of Technology · National University of Singapore · The Hong Kong Polytechnic University · Sun Yat-sen University · Zhejiang University
Robot manipulation videos provide a rich visual experience, yet recovering robot action and gripper-state annotations from RGB observations still remains challenging. Using the embodiment-agnostic tool-center-point (TCP) pose as the action space, we present TCP4D, a unified model in which a shared 4D scene representation both tracks observed TCP motion and generates future TCP trajectories. Given a monocular video and initial 2D TCP queries, the tracker directly recovers metric TCP poses in SE(3) and gripper states alongside scene geometry and camera parameters. The generator samples local features from the same 4D representation along the recovered TCP tracks, then combines them with robot observations and language instructions to predict future TCP states. Our new cross-embodiment augmentation pipeline converts existing RGB videos into demonstrations for alternative embodiments by retargeting the recovered motion and synthesizing the corresponding observations, without recorded source-robot states. Experiments show that the tracker achieves absolute trajectory errors of 7.60 and 16.07 mm on DROID and RoboLab and adapts to a new embodiment with only ten demonstrations. The generator achieves 69.19% mean success on RoboTwin and outperforms well-established policy models on two real-world tasks, while joint training on source and augmented demonstrations improves target-robot success by 41.5% over augmented-only training.
Figures & tables
Figure 1: We propose TCP4D , a unified model that tracks tool-center-point (TCP) poses and gripper states from RGB videos and generates task-conditioned future TCP trajectories and gripper states, enabling robot manipulation and cross-embodiment data augmentation.
Figure 2: Overview of TCP4D. \raisebox{0.2pt}{\scriptsize1}⃝ TCP4D Tracker takes a monocular video as input and constructs a 4D scene representation. Given the TCP’s 2D location in the initial frame as a query, it decodes the scene geometry, the TCP’s SE(3) motion, and the gripper state. \raisebox{0.2pt}{\scriptsize2}⃝ TCP4D Generator extracts local features along the TCP tracks to form queries and incorporates the robot states. Conditioned on the current 4D scene representation, the wrist RGB view, and language instruction, it predicts the TCP motion and gripper state over the next h steps.
Figure 3
DROID (Eval set)
Depth
TCP Tracking
SE(3)
Gripper
Method
AbsRel ↓
MAE ↓
δ1↑
APD ↑
MPE ↓
ATE ↓
RRE ↓
RTE ↓
Acc. ↑
SR ↑
–
(mm)
(%)
(%)
(mm)
(mm)
( ∘ )
(mm)
(%)
(%)
VGGT- Ω Wang et al. (2026b)
0.2279
224.10
66.19
-
-
-
-
-
-
-
DA3-GIANT Lin et al. (2026)
0.2271
224.35
64.98
-
-
-
-
-
-
-
MapAnything Keetha et al. (2026)
0.1994
187.42
71.90
-
-
-
-
-
-
-
Table 1: TCP4D Tracker evaluation on DROID and Robolab. All results are camera-wise macro averages. Depth is evaluated within 3 m with per-frame median alignment. SE(3) errors use Sim(3) alignment. CoTracker3 uses VGGT- Ω depth. The top-2 results are highlighted as best and second , respectively. The metric definitions are given in Appx. D .
Figure 4: Qualitative comparison of TCP tracking. We compare TCP tracking by TCP4D against dense tracking baselines followed by post-processing. The left side shows the input video. The red point in the first frame marks the query used by TCP4D, while the green points indicate the rigid point group tracked by the baselines. More visualizations are provided in Appx. D.8 .
Figure 5: Example of our cross-embodiment augmented data. Given an RGB video, TCP4D reconstructs the TCP’s 3D trajectory and gripper state, enabling augmentation on any embodiment that satisfies reachability . More visualizations in Appx. B .
Figure 6: Cross-embodiment policy evaluation. Task success rates under three training-data settings on three embodiments. For each task, we collect 50 source demonstrations on Aloha-AgileX and augment them to ARX-X5 and UR5-WSG using our pipeline.
Figure 7: Real-world experiments setup and task rollouts. The right panel shows the rollout of tasks 1) placing pens and 2) stacking bowls , with the predicted TCP trajectory over the next 32 steps displayed in each inference window. The color gradient from blue to orange indicates the temporal order of future actions. Complete rollout visualizations are in Appx. F.2 .
Type
Method
View
Easy (C2C)
Hard (C2R)
VLA&WAM
EventVLA Yang et al. (2026a)
head
65.71
14.50
AHA-WAM Cai et al. (2026a)
head
64.38
2.77
Xiaomi Robotics-0 Cai et al. (2026b)
head
62.79
17.85
GalaxeaVLA Liu et al. (2026)
head
62.73
12.31
Discrete Forcing Wang et al. (2026c)
head
59.25
11.04
StarVLA Community (2026)
head
46.73
3.21
Table 3: Average success rates (%) on RoboTwin 2.0 over 48 tasks. “View” indicates the primary input camera view used by each method. The top-2 results in each column are highlighted as best and second , respectively.
Figure 10
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 11
Resource / signal
RoVi-Aug Chen et al. (2024b)
OXE-AugE Ji et al. (2025)
TCP4D
Source robot URDF
×
×
×
Target robot URDF
×
✓
✓
Recorded camera intrinsics
×
×
Pred.
Recorded camera extrinsics
×
×
Pred.
Source TCP trajectory
✓
✓
Pred.
Source joint configurations
✓
×
×
Appendix
Table 4: Resources and motion signals for cross-embodiment augmentation. ✓ : supplied as input or recorded metadata (required); × : not required; Pred. : inferred from RGB by TCP4D. Source URDFs are unnecessary with image-based segmentation.
Module
Architecture / setting
Specification
Shared visual and TCP modules
Visual backbone and geometry head
4RC backbone and Dual-DPT; trainable
Sparse TCP modules
Query encoder, decoder, and TCP head; trainable
TCP query neighborhood
3×3 patches per arm and frame
Geometry decoding
Enabled
Camera head
Excluded from optimization in both stages
Conditioning encoders
Robot-state encoder
MLP: 14→1024→1280
Appendix
Table 5: Model architecture for the TCP4D Stage 1&2 experiments.
Module
Parameters (M)
Visual backbone
1,136.491
Motion decoder
283.340
TCP query encoder
0.808
TCP trajectory prediction head
0.800
Depth and ray geometry head
50.115
Camera parameter decoder
18.908
Appendix
Table 6: Model parameter counts. Counts include frozen modules and retained heads; the wrist encoder is shared across both cameras.
Scope
Hyperparameter
Value
Common
Optimizer
AdamW
Adam coefficients (β1,β2)
(0.9,0.95)
Adam ϵ
10−8
Weight decay
0.01
Mixed precision
bfloat16
Gradient accumulation steps
2
Appendix
Table 7: Training hyperparameters. Common settings apply to both DROID Stage 1 and the RoboTwin Stage 2 experiments.
Figure 10: Sliding-window TCP tracking inference. TCP4D supports inference on sequences of varying lengths. To track a long sequence, only the TCP’s 2D location in the first frame needs to be specified. The predicted 3D TCP position, (x,y,z) , at the final frame can then be projected onto the camera plane and used as the new query for the next window.
Figure 11: Query points on DROID and Robolab for baseline evaluation. For baselines evaluated by rigid fitting of point tracks, we select 16 point groups on the gripper in the first frame of each window using ground-truth annotations, track them throughout the window, and fit the TCP pose from the tracked points.
Metric
DROID
RoboLab
Extrinsics
Translation (mm)
12.96
129.80
Rotation ( ∘ )
1.26
2.42
Camera center (mm)
14.80
130.99
Intrinsics
fx rel. error (%)
0.248
0.477
Appendix
Table 11: Camera pose evaluation for TCP4D.
Figure 12: Visualization of TCP4D for few-shot cross-embodiment adaptation. We fine-tune TCP4D using only three and ten randomly selected episodes for three-shot and ten-shot transfer, respectively. The orange and green paths represent the predicted and ground-truth TCP trajectories, respectively. Our method can be readily transferred to new environments and robot embodiments.
Figure 13: Qualitative results of TCP4D Tracker on real-world task. Methods such as 4RC and MOSE3 can experience tracking failures.
Figure 14: Qualitative comparison of TCP tracking. We compare TCP tracking by TCP4D against dense tracking baselines followed by post-processing. The left side shows the input video. The red point in the first frame marks the query used by TCP4D, while the green points indicate the rigid point group used by the baselines.
Table 13: Downstream policy configuration. Settings are shared across tasks and training-data regimes.
Figure 15: Qualitative results of our cross-embodiment augmented data. Given an RGB-only video, TCP4D reconstructs the TCP’s 3D trajectory and gripper state, enabling augmentation on any embodiment that satisfies reachability . The right side shows the resulting augmented trajectories rendered on different target embodiments.
Task
Ordered subtasks
Placing_Pens ( R=4 )
(1/4) right arm picks up the pen on the right side of the table; (2/4) right arm places the pen in the right pen holder; (3/4) left arm picks up the pen on the left side of the table; (4/4) left arm places the pen in the left pen holder.
Stacking_bowls ( R=4 )
(2/4) both arms simultaneously pick up two bowls; (3/4) right arm places its bowl on the plate; (4/4) left arm stacks its bowl on top of the first bowl.
Appendix
Table 15: Real-world task progress-score rubric. Parentheses indicate the cumulative score after completing each ordered subtask.
Figure 16: Visualizations of TCP4D on Task Placing Pens . Each window shows the future TCP trajectory predicted by TCP4D. The motion is then planned using the cuRobo Sundaralingam et al. (2026) library.
Figure 17: Visualizations of TCP4D on Task Stack Bowls . Each window shows the future TCP trajectory predicted by TCP4D. The motion is then planned using the cuRobo Sundaralingam et al. (2026) library.
Video world models can generate realistic futures from a single instruction, but they often fail to track the same physical points consistently across time. As a result, the generated videos appear plausible, yet lack the physical grounding required for reliable action execution, such as robot manipulation. We present GEM-4D, a geometry-grounded video world model that resolves this limitation by injecting dense 4D correspondence supervision distilled from a pretrained geometry foundation model into the video generative backbone during training. This supervision enables the model to jointly capture appearance and geometric structure while retaining a single-stream architecture with no additional inference cost. We further introduce an inverse dynamics module that converts correspondence-consistent video rollouts into executable robot trajectories, enabling direct deployment in both real-world and simulated manipulation. GEM-4D achieves state-of-the-art performance on both video prediction and geometric consistency across both simulation and realistic scenarios and improves real-world manipulation success from 61% to 81%. Additional results are available at https://gem-4d.github.io/.
Kaichen Zhou, Yuzhen Chen, Fangneng Zhan +8
Harvard AI and Robotics Lab, Harvard University · Media Lab and EECS, MIT · MIT-IBM Watson AI Lab +1
Video-Action Models (VAMs) leverage the broad visual dynamics captured by pre-trained video diffusion models, offering a promising path toward generalizable robot manipulation. However, RGB-only video rollouts are not directly actionable: they leave metric 3D motion, contact geometry, and fine-grained spatial constraints under-specified, making action grounding ambiguous. Meanwhile, scaling action supervision across diverse tasks and embodiments remains costly. We present PointAction, a framework that bridges video predictions to robot actions through explicit point-based 4D modeling. PointAction fine-tunes a foundation video generation model to jointly predict future RGB frames and dynamic 3D pointmaps, producing temporally consistent 3D motion of task-relevant scene geometry. These point dynamics serve as a structured, embodiment-agnostic action interface, which a diffusion-based action decoder maps to executable robot actions. By using metric 3D point dynamics as the interface between video prediction and control, PointAction reduces the ambiguity of RGB-only action grounding and supports transfer across tasks and embodiments with limited action supervision. Experiments show that PointAction achieves state-of-the-art 4D generation quality on robot scenes, outperforms existing baselines in simulation, and generalizes to two real robot arms unseen during pretraining.
Mutian Tong, Han Jiang, Qiao Feng +2
University of Pennsylvania · Work done during internship at University of Pennsylvania.
World-model-based imagine-then-act becomes a promising paradigm for robotic manipulation, yet existing approaches typically support either purely image-based forecasting or reasoning over partial 3D geometry, limiting their ability to predict complete 4D scene dynamics. This work proposes a novel embodied 4D world model that enables geometrically consistent, arbitrary-view RGBD generation: given only a single-view RGBD observation as input, the model imagines the remaining viewpoints, which can then be back-projected and fused to assemble a more complete 3D structure across time. To efficiently learn the multi-view, cross-modality generation, we explicitly design cross-view and cross-modality feature fusion that jointly encourage consistency between RGB and depth and enforce geometric alignment across views. Beyond prediction, converting generated futures into actions is often handled by inverse dynamics, which is ill-posed because multiple actions can explain the same transition. We address this with a test-time action optimization strategy that backpropagates through the generative model to infer a trajectory-level latent best matching the predicted future, and a residual inverse dynamics model that turns this trajectory prior into accurate executable actions. Experiments on three datasets demonstrate strong performance on both 4D scene generation and downstream manipulation, and ablations provide practical insights into the key design choices.
Jiaxu Wang, Yicheng Jiang, Tianlun He +7
MMLab, The Chinese University of Hong Kong, Hong Kong SAR · The Hong Kong University of Science and Technology, Hong Kong · The University of Hong Kong +2