3PoinTr: From Human Videos to Robot Policies with 3D Point-Track Plans

Adam Hung; Bardienus Pieter Duisterhof; Jeffrey Ichnowski

cs.ROMar 9, 2026

3PoinTr: From Human Videos to Robot Policies with 3D Point-Track Plans

Authors: Adam Hung, Bardienus Pieter Duisterhof, Jeffrey Ichnowski

Organizations: Carnegie Mellon University

Abstract

Human videos could reduce the reliance on expensive robot demonstrations, but extracting task-relevant features remains challenging. We introduce 3PoinTr, a method that provides robot policies with a global 3D point-track plan predicted from human videos. Given an initial point cloud, 3PoinTr predicts dense 3D point tracks spanning the full task and compresses them into a small set of tokens that remain fixed throughout execution. At each policy query, the policy combines these plan tokens with the current observation to predict robot actions. This formulation enables the policy to extract useful motion cues from human videos. We find this formulation transfers exceptionally well, even when human demonstration motions are infeasible for robots. In controlled simulation and real-world evaluations, 3PoinTr achieves an average success rate 25.0 percentage points higher than the strongest behavior-cloning and video-pretraining baselines on real-world tasks, and an improvement of 29.6 percentage points in simulation. Ablations establish the benefits of our method, including predicting non-embodiment 3D tracks once as a global plan, Perceiver-IO compression, and a residual cross-attention mechanism. Our point-track prediction network also outperforms a strong 3D-flow baseline by retaining supervision for temporarily occluded points. Project page: https://adamhung60.github.io/3PoinTr/

Figures & tables

Using embodiment tracks as robot actions
Method	3D tracks	Scene-dense tracks	Excludes embodiment points	No object segmentation	Learned policy	Tracks predicted once
Point Policy [ 1 ]	$\checkmark$	$\times$	$\times$	$\checkmark$	$\checkmark$	$\times$
Motion Track Policy [ 2 ]	$\times$	$\times$	$\times$	$\times$	$\checkmark$	$\times$
Geometrically deriving robot actions from scene motion
Track2Act [ 3 ]	$\times$	$\checkmark$	$\times$	$\checkmark$	$\checkmark$	$\checkmark$
General Flow [ 4 ]	$\checkmark$	$\times$	$\checkmark$	$\times$	$\times$	$\times$

TABLE I: Point-track methods for learning robot control from human videos. ✓/

\times

indicate presence/absence of each property.

Fig. 2: Diagram of the 3PoinTr network architecture. Given an initial point cloud, a transformer predicts dense 3D point tracks spanning the full task execution. A Perceiver-IO point-track encoder compresses these tracks into a small set of learned plan tokens, which remain fixed throughout the episode. At each action step, the policy conditions on the plan tokens together with the current point cloud and robot state, enabling closed-loop action prediction. Beyond global feature conditioning, residual track-token cross-attention in every U-Net block gives the action head a direct path to the predicted object motion, encouraging a simpler mapping from point tracks to robot actions.

Fig. 3: Simulation and real-world tasks used for evaluation. Simulation tasks use procedurally generated robot trajectories for both video pretraining and policy learning; real-world tasks use human videos for video pretraining and robot teleoperation data for policy learning. Green tracks show 2D projections of the 3D point-track prediction targets (only tracks for moving points are visualized).

Simulation Tasks					Real-World Tasks
Task	General Flow		3PoinTr		Task	General Flow		3PoinTr
	ADE $\downarrow$	5% ADE $\downarrow$	ADE $\downarrow$	5% ADE $\downarrow$		ADE $\downarrow$	5% ADE $\downarrow$	ADE $\downarrow$	5% ADE $\downarrow$
Block Stack	0.47	2.56	0.85	5.98	Open Drawer	2.54	16.37	2.18	13.19
Open Microwave	0.68	6.04	1.79	6.79	Right Glass	3.34	30.38	2.29	18.55
Right Glass	1.96	28.82	1.66	15.86	Throw Away Paper	2.40	22.87	1.47	8.17
Pot Lid	1.18	6.89	0.77	4.83	Fold Sock	1.56	10.09	1.14	4.68

TABLE II: Point-track Average Displacement Error (ADE) and ADE of the 5 % of points that move the most (5% ADE) for 3PoinTr and General Flow [ 4 ] ; lower is better and units are millimeters. On real-world tasks, 3PoinTr lowers average ADE by 28 % and 5% ADE by 44 % , outperforming General Flow on every task and metric by retaining supervision for temporarily occluded points. In simulation, where there is no occlusion, the methods are competitive.

Task	AMPLIFY [ 8 ]			DP [ 14 ]			ATM [ 7 ]			DP3 [ 16 ]			3PoinTr
# Demonstrations	20	50	100	20	50	100	20	50	100	20	50	100	20	50	100
Block Stack	0 0.5	0 3.5	0 2.0	17.0	30.5	60.5	11.0	47.5	67.0	25.5	41.5	57.0	66.0	84.5	94.0
Right Glass	0 2.0	0 6.0	0 9.0	16.5	27.5	76.5	0 8.0	25.0	32.5	21.0	43.0	70.0	51.0	77.0	97.0
Pot Lid	0 3.0	0 4.0	0 7.0	42.5	51.0	58.0	27.5	51.0	54.5	51.5	60.0	77.5	67.5	91.0	93.0
Open Microwave	0 7.0	0 2.5	0 8.0	17.5	35.0	81.5	70.0	93.5	99.0	43.0	48.0	56.0	75.0	78.0	89.0
Average	0 3.1	0 4.0	0 6.5	23.4	36.0	69.1	29.1	54.3	63.3	35.3	48.1	65.1	64.9	82.6	93.3

TABLE III: Simulation success rates (%) over 200 rollouts per task. Policies use 20, 50, or 100 action-labeled demonstrations; video-pretrained methods additionally use 100 actionless videos per task. With 20 demonstrations, 3PoinTr achieves the highest success rate on every task and improves average success by 29.6 percentage points over the strongest baseline; it also achieves the highest average success rate at 50 and 100 demonstrations.

Fig. 4: Example human vs. robot trajectories: 3PoinTr learns scene motion from human videos (left) and derives a global point-track plan, while robot demonstrations teach the closed-loop policy how to realize that plan through embodiment-specific actions (right). This separation lets the human and robot use different grasps, trajectories, and even strategies.

Method	Open Drawer	Right Glass	Throw Away Paper	Fold Sock	Average
ATM	6/20	3/20	0/20	7/20	4.0/20
DP3	14/20	18/20	9/20	14/20	13.8/20
3PoinTr	20/20	20/20	18/20	17/20	18.8/20

TABLE IV: Real-world success rates over 20 rollouts per task, using 20 robot demonstrations and 50 human videos per task. 3PoinTr succeeds on 18.8/20 rollouts on average, outperforming the strongest baseline, DP3, by 25.0 percentage points and achieving the highest success rate on every task.

(a) Simulation
Task	2D Tracks	No Perc.	No Xattn	No Videos	3PoinTr
Block Stack	35.0	24.5	56.0	69.0	66.0
Right Glass	26.5	25.0	41.0	19.5	51.0
Pot Lid	44.0	55.5	62.5	57.5	67.5
Open Microwave	37.5	29.0	70.0	60.0	75.0
Average	35.8	33.5	57.4	51.5	64.9

TABLE V: Ablation of 3PoinTr design choices defined in Section IV-E . Simulation variants (a) replace 3D tracks with 2D tracks, replace Perceiver-IO with mean pooling, remove residual U-Net cross-attention, or train the track predictor without extra actionless videos. These are trained on 20 robot demonstrations and 100 actionless videos per task, and evaluated on 200 rollouts per task. Real-world variants (b) include embodiment points in the track targets or re-predict tracks at every policy query. These are trained on 20 robot demonstrations and 50 human videos per task, and evaluated on 20 rollouts per task. 3PoinTr achieves the highest average success rate in both domains, demonstrating the contribution of each design choice.

Explore similar work

PointAction: 3D Points as Universal Action Representations for Robot Control
Jun 2, 2026Mutian Tong, Han Jiang, Qiao Feng +2Generalization in Robotic Manipulation Cross-Embodiment Robot Learning
University of Pennsylvania · Work done during internship at University of Pennsylvania.
Dexterous Point Policy: Learning Point-based Dexterous Hand Policies from Human Demonstrations
Jun 9, 2026Beomjun Kim, Seong Hyeon Park, Seunghoon Sim +3Dexterous Manipulation Cross-Embodiment Robot Learning
KAIST
$μ_0$ : A Scalable 3D Interaction-Trace World Model
Jun 11, 2026Seungjae Lee, Yoonkyo Jung, Jusuk Lee +6World Models for Robotics Generalization in Robotic Manipulation
University of Maryland, College Park · Seoul National University

Jun 2, 2026J/K move · Enter open · S save

PointAction: 3D Points as Universal Action Representations for Robot Control

Mutian Tong, Han Jiang, Qiao Feng, Lingjie Liu+1

University of Pennsylvania · Work done during internship at University of Pennsylvania.

Generalization in Robotic Manipulation Cross-Embodiment Robot Learning Robotic Manipulation Vision-Based Robot Control

Open paper AskPDF arXiv

Video-Action Models (VAMs) leverage the broad visual dynamics captured by pre-trained video diffusion models, offering a promising path toward generalizable robot manipulation. However, RGB-only video rollouts are not directly actionable: they leave metric 3D motion, contact geometry, and fine-grained spatial constraints under-specified, making action grounding ambiguous. Meanwhile, scaling action supervision across diverse tasks and embodiments remains costly. We present PointAction, a framework that bridges video predictions to robot actions through explicit point-based 4D modeling. PointAction fine-tunes a foundation video generation model to jointly predict future RGB frames and dynamic 3D pointmaps, producing temporally consistent 3D motion of task-relevant scene geometry. These point dynamics serve as a structured, embodiment-agnostic action interface, which a diffusion-based action decoder maps to executable robot actions. By using metric 3D point dynamics as the interface between video prediction and control, PointAction reduces the ambiguity of RGB-only action grounding and supports transfer across tasks and embodiments with limited action supervision. Experiments show that PointAction achieves state-of-the-art 4D generation quality on robot scenes, outperforms existing baselines in simulation, and generalizes to two real robot arms unseen during pretraining.

Abstract

Explore similar work

PointAction: 3D Points as Universal Action Representations for Robot Control

Dexterous Point Policy: Learning Point-based Dexterous Hand Policies from Human Demonstrations

μ0μ_0μ0​: A Scalable 3D Interaction-Trace World Model

$μ_0$ : A Scalable 3D Interaction-Trace World Model