Learning the 4D World through Point Trajectory Forecasting

Byungwoo Jeon; Sunghwan Hong; Dohyeon Kim; Jonghoon Lee; Jungwoo Park; Inhee Lee; Doosan Baek; Jinil Kim

cs.CVOct 20, 2026

Learning the 4D World through Point Trajectory Forecasting

Authors: Byungwoo Jeon, Sunghwan Hong, Dohyeon Kim, Jonghoon Lee, Jungwoo Park, Inhee Lee, Doosan Baek, Jinil Kim,

Organizations: KAIST · ETH Zürich · Seoul National University · Physics Sim Lab ·

Abstract

Reliable multi-view forecasting requires videos from different cameras to describe the same evolving 3D scene. To this end, we introduce Video-Point Model (VPM), a world model that conditions on multi-view video histories and a language instruction to jointly forecast future videos from arbitrary viewpoints and the 3D trajectories of persistent scene points. We extend a pretrained world model with camera-conditioned multi-view processing that integrates complementary observations across cameras, and add a trajectory head that predicts how physical points move through time. Each trajectory tracks a single point across all views, and supervising these trajectories alongside video generation therefore anchors appearance changes in every camera to a shared description of 3D motion. To support training and evaluation, we build VPM-Data, a simulation corpus pairing synchronized multi-view videos with persistent point trajectories, and VP-WorldBench, a forecasting benchmark spanning simulated and real environments. We further extend VPM to robot control by adding robot actions as a token modality in the generator. Action tokens share the point-supervised backbone and are denoised jointly with future videos, thereby grounding the policy in the same 3D-aware representation. On VP-WorldBench and existing 4D generation benchmarks, point supervision improves both video fidelity and 3D motion accuracy over a video-only counterpart, and in simulated and real-robot experiments, it also raises the manipulation success of the action-enabled model.

Figures & tables

Figure 1: Teaser. We introduce simulated and real-world multi-view videos paired with 3D point tracks (left), jointly forecast future videos and point trajectories (center), and show that point supervision improves visual forecasting (right).

Figure 2: Overview of VPM. From posed multi-view histories and a task instruction, VPM forecasts future videos and the trajectories of points queried from observed frames. The 3D point tracker iteratively refines point positions and depths before lifting them to 3D. Joint point supervision shapes the generator’s representation, and the action-enabled extension also outputs robot action chunks.

Domain	Samples	Views	fps	Frames	3D tracks/clip	Hard
VPM-Data
Tabletop manipulation	8,000	3	20	81–181	768–2,048	-
Urban dynamics	2,000	4	15	81–181	256	-
VP-WorldBench (Sim)
Tabletop manipulation	500	3	20	81–680	768–1,536	27
Urban dynamics	500	4	15	81–181	256	95

Table 1: Overview of VPM-Data and VP-WorldBench.

Figure 3: Qualitative examples of VPM-Data and VP-WorldBench.

		Simulation								Real-world
		Robot (500)				General (500)				Robot (100)				General (100)
Method	MV	FVD $\downarrow$	PSNR $\uparrow$	MEt3R $\downarrow$	$\delta^{\text{3D}}_{\text{avg}}$ $\uparrow$	FVD $\downarrow$	PSNR $\uparrow$	MEt3R $\downarrow$	$\delta^{\text{3D}}_{\text{avg}}$ $\uparrow$	FVD $\downarrow$	PSNR $\uparrow$	MEt3R $\downarrow$	$\delta^{\text{3D}}_{\text{avg}}$ $\uparrow$	FVD $\downarrow$	PSNR $\uparrow$	MEt3R $\downarrow$	$\delta^{\text{3D}}_{\text{avg}}$ $\uparrow$
HV-1.5	✗	98.34	20.21	0.023	41.62	115.63	11.45	0.061	23.10	426.76	16.04	0.102	36.17	105.83	20.00	0.014	21.35
Wan2.2-5B	✗	315.13	15.84	0.108	29.51	205.73	10.43	0.076	18.00	550.55	14.58	0.159	29.65	115.52	16.97	0.030	16.82
X-WAM	✓	203.83	19.97	0.038	38.80	-	-	-	-	492.31	17.63	0.091	50.86	-	-	-	-
Cosmos3-N	✗	104.74	20.05	0.040	39.03	167.02	10.99	0.062	22.12	326.11	16.37	0.099	41.50	66.24	20.89	0.015	20.47

Table 2: Results on VP-WorldBench. Numbers in parentheses denote clips per split. V and T denote video and point-trajectory supervision.

Table 6

VLA-based Methods
			Simulation			Real-world
Method	MV	Backbone	LIBERO	LIBERO-Plus	RoboCasa	PnP Banana	PnP Cup	PnP Cube
$\pi_{0.5}$	$\triangle$	-	96.8	68.0	62.1	41.7	47.9	12.5
GR00T N1.5	$\triangle$	-	86.6	66.4	65.6	-	-	-
GR00T N1.7	$\triangle$	-	-	-	70.0	31.3	27.1	4.2
WAM-based Methods

Table 5: Robot manipulation results. A denotes action.

\triangle

denotes methods that observe multiple cameras without camera conditioning, either as separate input images or tiled into a single frame. ✓ denotes camera-conditioned multi-view generation.

	Simulation								Real-world
	Robot (500)				General (500)				Robot (100)				General (100)
Method	FVD $\downarrow$	PSNR $\uparrow$	MEt3R $\downarrow$	$\delta^{\text{3D}}_{\text{avg}}$ $\uparrow$	FVD $\downarrow$	PSNR $\uparrow$	MEt3R $\downarrow$	$\delta^{\text{3D}}_{\text{avg}}$ $\uparrow$	FVD $\downarrow$	PSNR $\uparrow$	MEt3R $\downarrow$	$\delta^{\text{3D}}_{\text{avg}}$ $\uparrow$	FVD $\downarrow$	PSNR $\uparrow$	MEt3R $\downarrow$	$\delta^{\text{3D}}_{\text{avg}}$ $\uparrow$
Cosmos3-Nano	104.74	20.05	0.040	39.03	167.02	10.99	0.062	22.12	326.11	16.37	0.099	41.50	66.24	20.89	0.015	20.47
VPM (V)	78.52	21.41	0.022	43.62	101.37	16.95	0.057	28.71	296.83	18.42	0.074	50.83	62.85	20.96	0.010	26.54
VPM (V+A)	74.29	21.59	0.021	44.27	98.08	17.42	0.054	30.20	291.05	18.87	0.069	53.21	61.31	21.24	0.009	28.06
VPM (V+T)	67.35	22.08	0.016	50.05	81.92	19.16	0.050	36.80	255.19	20.57	0.064	64.75	56.48	22.10	0.007	33.92

Table 6: Modality-wise ablation on VP-WorldBench.

Figure 4: Predicted 3D point trajectories guide generation and action. Left: Predicted trajectories alone and overlaid on generated videos. Right: Zero-shot real-robot rollout obtained by converting predicted trajectories into commands with inverse kinematics.

Appendix figures & tables35 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Stage 1	Stage 2	Stage 3
Trainable modules	Generator, Camera emb.	Traj. head	Generator, Traj. head
Objective	$\mathcal{L}_{\mathrm{FM}}$	$\mathcal{L}_{\mathrm{track}}$	$\mathcal{L}_{\mathrm{forecast}}$ ( $+\lambda_{\mathrm{act}}\mathcal{L}_{\mathrm{act}}$ )
Optimizer	AdamW
Learning rate	1e-4	1e-4	2e-5
Tokens	60,000	60,000	24,000
Training iterations	5,000	2,000	5,000

Appendix

Table 7: Hyperparameters for each training stage.

Figure 5: Real-world hardware platform. A 7-DoF Franka Research 3 robot arm equipped with a Robotiq 2F-85 gripper and a wrist-mounted Zed Mini stereo camera. Two external Zed 2i stereo cameras (view 1 and view 2) observe the workspace from different viewpoints, providing synchronized multi-view observations.

Figure 6: Examples of the robot manipulation tasks.

Figure 7: From predicted point trajectories to joint-space actions. Predicted 2D trajectories from both views are lifted to 3D (Stage 1) and converted into a joint trajectory by fitting the robot’s kinematic model to the lifted points (Stage 2).

Source	Domain	Real/Sim	Samples	Views	3D traj.	Action
VPM-Data	Tabletop manipulation	Sim	8,000	3	✓	✗
VPM-Data	Urban dynamics	Sim	2,000	4	✓	✗
DROID	Tabletop manipulation	Real	89,312	3	✓	✓
HRDexDB	Tabletop manipulation	Real	1,601	2	✓	✓
MolmoMotion-1M	Mixed domain	Real	480,000	1-2	✓	✗
Ego-Exo4D	Human activities	Real	24,000	1-4	✓	✗

Appendix

Table 8: Statistics of the training data.

Setting	Tabletop	Urban
Motion source	Recorded manipulation	Live traffic and crowds
Views / frame rate	3 / 20 fps	4 / 15 fps
Clip length	81–181 frames	81, 100, 121, 144, or 181 frames
Intrinsics	Authored	Sampled per camera and clip
Camera motion	Static or planned hard-clip paths	Static, arc, translation, random polyline, or pan–tilt
Visibility	Rendered depth, $3$ cm tolerance	Surface-point ray casting

Appendix

Table 9: Training-data capture settings. Both domains use

1280{\times}1280

images and distortion-free pinhole cameras.

Criterion	Tabletop	Urban
Invisible interval	$\geq 30$ consecutive frames per camera	$\geq 30$ per camera, followed by re-acquisition
Collective coverage	At least one visible camera per frame	At least one visible camera per frame
Motion in interval	Target displacement $\geq 5$ cm	Target moving in $\geq 40\%$ of interval frames
Additional condition	$\geq 20$ occluded frames summed over cameras	Two-phase departure schedule

Appendix

Table 10: Hard-clip criteria for the training pipelines. An invisible interval comprises frames labeled occluded or out of frustum. Benchmark definitions and measured statistics are reported separately in Appendix D .

Check	Threshold
Target visibility	Minimum visible ratio $\geq 0.1$ over all cameras
Survival and motion	Target present throughout; travel $\geq 1.0$ m per 81 frames, with a $0.5$ m floor
Track completeness	No track breaks; completeness $\geq 0.99$
Depth validity	Sky, near, and constant-value pixel ratios each $\leq 0.25$
Camera clearance	At most two cameras with blocked paths; view-block ratio $\leq 0.6$
Caption	Present, at least 20 characters, and passes format checks

Appendix

Table 11: Configured acceptance checks. Departure–re-acquisition is recorded but not required by the training profile.

Property	Tabletop	Urban
Clips (hard)	500 (27)	500 (95)
Views / frame rate	3 / 20 fps	4 / 15 fps
Clip length	81–680 frames	81–181 frames
Synchronized timesteps	129,441	68,187
RGB images	388,323	272,748
3D point trajectories	588,544	128,000

Appendix

Table 12: Simulation benchmark composition and capture settings. Timesteps are counted once per synchronized camera set; images count all views. Trajectory counts sum persistent point identities over clips. All images are

1280{\times}1280

, with distortion-free pinhole cameras.

Property	Tabletop	Urban
Hard clips	27	95
Cameras with $\geq 30$ consecutive invisible frames	All 3	2 or 3 †
Target motion in interval	Displacement $\geq 5$ cm	Moving fraction $\geq 0.4$
Additional occlusion requirement	$\geq 20$ frames summed over views	None
Frames with no visible camera	0	0 (strict)

Appendix

Table 13: Hard-subset criteria and composition. An invisible interval includes occluded and out-of-frustum frames. Urban collective coverage is evaluated on the strict array.

	Tabletop ( $V=3$ )		Urban ( $V=4$ )
Visible cameras	Normal	Hard	Normal	Hard
4	N/A	N/A	71.81	36.48
3	89.67	28.60	21.63	24.41
2	9.65	48.49	5.78	29.34
1	0.67	22.91	0.67	9.77
0	0.012	0.000	0.12	$0.01^{\dagger}$

Appendix

Table 14: Percentage of frames by the number of cameras seeing the target. Urban columns use bounding-box visibility; strict-array collective coverage is reported separately.

Check	Threshold
Target visibility	Minimum visible ratio $\geq 0.1$ over cameras without scheduled departures
Survival and motion	Target present throughout; travel $\geq 1.0$ m per 81 frames, with a 0.5 m floor
Track completeness	Non-empty points, no track breaks, completeness $\geq 0.99$
Depth validity	Sky, near, and constant-value pixel ratios individually $\leq 0.25$
Camera clearance	At most two cameras with blocked paths; view-block ratio $\leq 0.6$
Caption	Present, at least 20 characters, and passes style checks

Appendix

Table 15: Urban benchmark acceptance thresholds. The caption check is applied after selection.

		Simulation								Real-world
		Robot (500)				General (500)				Robot (100)				General (100)
Method	MV	FVD $\downarrow$	PSNR $\uparrow$	MEt3R $\downarrow$	$\delta^{\text{3D}}_{\text{avg}}$ $\uparrow$	FVD $\downarrow$	PSNR $\uparrow$	MEt3R $\downarrow$	$\delta^{\text{3D}}_{\text{avg}}$ $\uparrow$	FVD $\downarrow$	PSNR $\uparrow$	MEt3R $\downarrow$	$\delta^{\text{3D}}_{\text{avg}}$ $\uparrow$	FVD $\downarrow$	PSNR $\uparrow$	MEt3R $\downarrow$	$\delta^{\text{3D}}_{\text{avg}}$ $\uparrow$
HV-1.5	✗	98.34	20.21	0.023	41.62	115.63	11.45	0.061	23.10	426.76	16.04	0.102	36.17	105.83	20.00	0.014	21.35
Wan2.2-5B	✗	315.13	15.84	0.108	29.51	205.73	10.43	0.076	18.00	550.55	14.58	0.159	29.65	115.52	16.97	0.030	16.82
X-WAM	✓	203.83	19.97	0.038	38.80	-	-	-	-	492.31	17.63	0.091	50.86	-	-	-	-
Cosmos3-N	✗	104.74	20.05	0.040	39.03	167.02	10.99	0.062	22.12	326.11	16.37	0.099	41.50	66.24	20.89	0.015	20.47

Appendix

Table 16: Results on VP-WorldBench (I2V with all-view). V and T denote video and point-trajectory supervision. Best results are in bold . The upper bound is obtained by applying the same evaluation pipeline to the ground-truth videos.

		Simulation								Real-world
		Robot (500)				General (500)				Robot (100)				General (100)
Method	MV	FVD $\downarrow$	PSNR $\uparrow$	MEt3R $\downarrow$	$\delta^{\text{3D}}_{\text{avg}}$ $\uparrow$	FVD $\downarrow$	PSNR $\uparrow$	MEt3R $\downarrow$	$\delta^{\text{3D}}_{\text{avg}}$ $\uparrow$	FVD $\downarrow$	PSNR $\uparrow$	MEt3R $\downarrow$	$\delta^{\text{3D}}_{\text{avg}}$ $\uparrow$	FVD $\downarrow$	PSNR $\uparrow$	MEt3R $\downarrow$	$\delta^{\text{3D}}_{\text{avg}}$ $\uparrow$
HV-1.5	✗	85.91	20.72	0.021	39.22	141.01	11.66	0.062	20.46	426.76	16.04	0.102	36.17	117.21	20.17	0.016	19.54
Wan2.2-5B	✗	294.52	16.42	0.089	27.62	180.34	9.96	0.070	17.33	550.55	14.58	0.159	29.65	151.03	16.76	0.033	16.20
X-WAM	✓	154.26	20.03	0.033	36.73	-	-	-	-	492.31	17.63	0.091	50.86	-	-	-	-
Cosmos3-N	✗	86.05	20.17	0.037	34.66	202.55	10.06	0.058	19.99	326.11	16.37	0.099	41.50	88.21	20.08	0.018	19.48

Appendix

Table 17: Results on VP-WorldBench (I2V with 2-view). V and T denote video and point-trajectory supervision. Best results are in bold . The upper bound is obtained by applying the same evaluation pipeline to the ground-truth videos.

		Simulation								Real-world
		Robot (500)				General (500)				Robot (100)				General (100)
Method	MV	FVD $\downarrow$	PSNR $\uparrow$	MEt3R $\downarrow$	$\delta^{\text{3D}}_{\text{avg}}$ $\uparrow$	FVD $\downarrow$	PSNR $\uparrow$	MEt3R $\downarrow$	$\delta^{\text{3D}}_{\text{avg}}$ $\uparrow$	FVD $\downarrow$	PSNR $\uparrow$	MEt3R $\downarrow$	$\delta^{\text{3D}}_{\text{avg}}$ $\uparrow$	FVD $\downarrow$	PSNR $\uparrow$	MEt3R $\downarrow$	$\delta^{\text{3D}}_{\text{avg}}$ $\uparrow$
Cosmos3-N	✗	111.92	20.06	0.044	40.91	140.65	11.05	0.067	23.27	296.25	16.34	0.099	43.53	77.55	21.03	0.018	21.72
+ camera cond.	✗	110.23	19.98	0.040	43.27	145.96	10.46	0.067	19.08	288.63	17.49	0.095	44.92	72.59	21.17	0.014	22.80
VPM (V)	✓	73.31	21.44	0.028	43.75	125.83	17.53	0.060	28.92	243.62	18.32	0.073	53.12	66.96	21.21	0.012	28.09
VPM (V+T)	✓	63.77	22.18	0.021	50.17	81.49	19.36	0.055	38.31	239.61	20.56	0.064	68.73	48.25	22.11	0.010	34.94

Appendix

Table 18: Results on VP-WorldBench (V2V with all-view). V and T denote video and point-trajectory supervision. Best results are in bold . The upper bound is obtained by applying the same evaluation pipeline to the ground-truth videos.

		Simulation		Real-world
		Robot (500)	General (500)	Robot (100)	General (100)
Method	Tracker	$\delta^{\text{3D}}_{\text{avg}}$ $\uparrow$	$\delta^{\text{3D}}_{\text{avg}}$ $\uparrow$	$\delta^{\text{3D}}_{\text{avg}}$ $\uparrow$	$\delta^{\text{3D}}_{\text{avg}}$ $\uparrow$
VPM (V)	MVTracker	43.62	28.71	50.83	26.54
VPM (V+T)	MVTracker	50.05	36.80	64.75	33.92
+ camera cond.	MVTracker	55.05	40.04	66.75	38.46
VPM (V)	Ours	45.06	29.92	51.78	28.39

Appendix

Table 19: Results on VP-WorldBench with VPM trajectory head. V and T denote video and point-trajectory supervision. Best results are in bold .

Table 26

Per-frame geometry (no temporal identity)
	Simulation				Real-world
	Robot (500)		General (500)		Robot (100)		General (100)
Supervision	FVD $\downarrow$	$\delta^{\text{3D}}_{\text{avg}}\!\uparrow$	FVD $\downarrow$	$\delta^{\text{3D}}_{\text{avg}}\!\uparrow$	FVD $\downarrow$	$\delta^{\text{3D}}_{\text{avg}}\!\uparrow$	FVD $\downarrow$	$\delta^{\text{3D}}_{\text{avg}}\!\uparrow$
None	78.52	43.62	101.37	28.71	296.83	50.83	62.85	26.54
Depth	77.07	45.21	100.02	29.88	285.13	53.73	62.08	26.98
Pointmap	75.48	44.90	99.35	29.69	281.67	54.50	61.37	29.46

Appendix

Table 22: Per-frame geometry vs. 3D point trajectory supervision. All variants share the backbone, data, feature layer, and training schedule of VPM (V+T). Per-frame variants are supervised with the same 3D point positions at every frame without correspondence across frames.

Paradigm	Model	Inputs		HOT3D			WorldTrack			DAVIS
Paradigm	Model	Frames	Text	ADE $\downarrow$	FDE $\downarrow$	PWT $\uparrow$	ADE $\downarrow$	FDE $\downarrow$	PWT $\uparrow$	ADE $\downarrow$	FDE $\downarrow$	PWT $\uparrow$
Non- parametric	Static	1	✗	0.180	0.316	0.293	0.167	0.317	0.390	2.281	4.360	0.085
Non- parametric	Extrapolate	3	✗	0.159	0.309	0.351	0.184	0.432	0.436	2.683	5.741	0.104
Pixel-space	Wan2.2-5B	1	✓	0.200	0.308	0.253	0.852	1.046	0.090	3.074	5.192	0.051
Pixel-space	Cosmos-Predict	5	✓	0.225	0.294	0.199	0.831	0.988	0.072	4.191	6.368	0.033
3D model	ObjectForesight	3	✗	0.129	0.192	0.353	–	–	–	–	–	–

Appendix

Table 23: Results on 3D point forecasting. We evaluate the models on PointMotionBench ( Zhang et al., 2026b ) . Best results are in bold , second best are underlined .

	PAIBench-G ( $\uparrow$ )			RBench ( $\uparrow$ )
Model	Overall	Domain	Quality	Score (%)
Wan2.2-A14B	81.3	85.3	77.3	50.7
HunyuanVideo-1.5	81.7	85.9	77.6	46.0
Cosmos-Predict2.5-2B	81.2	84.6	77.9	46.4
Cosmos-Predict2.5-14B	81.1	84.0	78.1	-
Wan2.1-14B	80.2	83.6	76.8	-

Appendix

Table 24: Results on single-view I2V world modeling.

Model	Mode	Score ( $\uparrow$ )
VideoPoet	V2V	29.5
VideoGPT	V2V	35.0
Magi-1	V2V	56.0
Cosmos3-Super	V2V	59.7
Cosmos3-Nano	V2V	50.2
VPM (ours)	V2V	52.1

Appendix

Table 25: Results on single-view V2V world modeling.

		ReCamMaster (100)		CMU Panoptic (100)
Method	MV	FVD $\downarrow$	PSNR $\uparrow$	FVD $\downarrow$	PSNR $\uparrow$	MEt3R $\downarrow$	$\delta^{\text{3D}}_{\text{avg}}$ $\uparrow$
Cosmos3-Nano	✗	125.70	13.99	162.36	20.72	0.060	30.12
VPM (V)	✓	103.12	15.09	150.63	20.85	0.037	34.72
VPM (V+A)	✓	101.35	15.63	149.06	21.14	0.035	36.51
VPM (V+T)	✓	79.63	17.06	128.64	23.24	0.029	42.03
VPM (V+T+A)	✓	76.12	17.48	118.05	24.08	0.029	44.33

Appendix

Table 26: Modality-wise ablation on ReCamMaster and CMU Panoptic. V, T, and A denote video, point-trajectory, and action supervision. The upper bound applies the evaluation pipeline to the ground-truth videos.

Figure 8: Visualization of cross-view attention map.

Method	Output	1 view	2 views	4 views
Cosmos3-Nano	Video	15.9	31.8	63.5
VPM (V)	Video	16.3 (1.03 $\times$ )	38.7 (1.22 $\times$ )	101.4 (1.60 $\times$ )
VPM (V+T)	Video + 3D traj.	16.5 (1.04 $\times$ )	39.1 (1.23 $\times$ )	102.3 (1.61 $\times$ )
VPM (V+T+A)	Video + 3D traj. + action	16.6 (1.04 $\times$ )	39.2 (1.23 $\times$ )	102.5 (1.61 $\times$ )

Appendix

Table 27: Clock-time analysis. Wall-clock time (sec) to generate one clip of 64 frames at 480p per view on one GH200 GPU. Cosmos3-Nano generates the views independently, whereas VPM generates them jointly. Parentheses show the time relative to Cosmos3-Nano with the same number of views.

Figure 9: Data scalability. FVD (top) and $\delta^{\text{3D}}_{\text{avg}}$ (bottom) of VPM (V) and VPM (V+T) trained on 10K, 100K, and 600K samples. The shaded area marks the gain from point supervision.

Camera cond.			Robot (500)		General (500)		Robot (100)		General (100)
			Simulation				Real-world
Add.	Rot.	View	FVD $\downarrow$	$\delta^{\text{3D}}_{\text{avg}}\!\uparrow$	FVD $\downarrow$	$\delta^{\text{3D}}_{\text{avg}}\!\uparrow$	FVD $\downarrow$	$\delta^{\text{3D}}_{\text{avg}}\!\uparrow$	FVD $\downarrow$	$\delta^{\text{3D}}_{\text{avg}}\!\uparrow$
✗	✗	✗	91.22	33.29	140.06	24.64	299.51	45.79	67.80	23.94
✗	✗	✓	75.17	41.70	128.41	29.08	281.67	49.25	61.43	26.99
✓	✗	✓	70.70	48.18	84.60	33.94	268.34	59.00	59.01	30.05
✓	✓	✗	70.01	48.59	85.15	33.40	265.37	61.88	58.90	31.55

Appendix

Table 28: Ablation on camera conditioning. Add. and Rot. denote the additive Plücker embedding and the camera-dependent rotation of queries and keys in Equation 2 , and View denotes the view axis of the 4D RoPE in Equation 3 .

Objective			Robot (500)		General (500)		Robot (100)		General (100)
			Simulation				Real-world
$\mathcal{L}_{\text{seq}}$	$\mathcal{L}_{\text{depth}}$	$\mathcal{L}_{\text{corr}}$	FVD $\downarrow$	$\delta^{\text{3D}}_{\text{avg}}\!\uparrow$	FVD $\downarrow$	$\delta^{\text{3D}}_{\text{avg}}\!\uparrow$	FVD $\downarrow$	$\delta^{\text{3D}}_{\text{avg}}\!\uparrow$	FVD $\downarrow$	$\delta^{\text{3D}}_{\text{avg}}\!\uparrow$
✗	✗	✗	78.52	43.62	101.37	28.71	296.83	50.83	62.85	26.54
✗	✗	✓	78.44	42.96	100.45	29.90	284.17	52.00	61.57	27.50
✓	✗	✗	74.82	43.90	91.92	30.00	275.24	54.69	59.71	27.99
✓	✓	✗	69.88	48.54	85.73	31.54	271.01	59.94	59.33	30.75

Appendix

Table 29: Ablation on training objectives.

\mathcal{L}_{\text{FM}}

is used in all rows, and

\mathcal{L}_{\text{conf}}

is used together with

\mathcal{L}_{\text{seq}}

. The first row corresponds to VPM (V) in Table 2 .

Stage			Robot (500)		General (500)		Robot (100)		General (100)
			Simulation				Real-world
1	2	3	FVD $\downarrow$	$\delta^{\text{3D}}_{\text{avg}}\!\uparrow$	FVD $\downarrow$	$\delta^{\text{3D}}_{\text{avg}}\!\uparrow$	FVD $\downarrow$	$\delta^{\text{3D}}_{\text{avg}}\!\uparrow$	FVD $\downarrow$	$\delta^{\text{3D}}_{\text{avg}}\!\uparrow$
✗	✗	✓	70.45	48.50	85.34	34.29	262.40	60.86	58.11	31.75
✗	✓	✓	69.83	49.15	83.27	34.51	259.42	62.00	58.02	32.80
✓	✗	✓	77.50	39.51	99.81	29.38	292.53	49.77	62.49	27.04
✓	✓	✓	67.35	50.05	81.92	36.80	255.19	64.75	56.48	33.92

Appendix

Table 30: Ablation on training stages. Iterations of a removed stage are reassigned to Stage 3, keeping the total number of iterations fixed.

	Simulation				Real-world
	Robot (500)		General (500)		Robot (100)		General (100)
Layer $l$	FVD $\downarrow$	$\delta^{\text{3D}}_{\text{avg}}\!\uparrow$	FVD $\downarrow$	$\delta^{\text{3D}}_{\text{avg}}\!\uparrow$	FVD $\downarrow$	$\delta^{\text{3D}}_{\text{avg}}\!\uparrow$	FVD $\downarrow$	$\delta^{\text{3D}}_{\text{avg}}\!\uparrow$
6	72.81	45.92	96.28	31.10	277.11	57.04	60.26	28.75
12	69.45	48.30	85.76	34.90	266.16	61.21	58.30	31.88
18	67.35	50.05	81.92	36.80	255.19	64.75	56.48	33.92
24	68.76	49.16	84.96	35.28	259.84	63.80	58.65	31.60

Appendix

Table 31: Ablation on the generator layer. The trajectory head reads query and key features from the

l

-th self-attention layer of the generator.

(a) Refinement iterations
	Simulation				Real-world
	Robot (500)		General (500)		Robot (100)		General (100)
Setting	FVD $\downarrow$	$\delta^{\text{3D}}_{\text{avg}}\!\uparrow$	FVD $\downarrow$	$\delta^{\text{3D}}_{\text{avg}}\!\uparrow$	FVD $\downarrow$	$\delta^{\text{3D}}_{\text{avg}}\!\uparrow$	FVD $\downarrow$	$\delta^{\text{3D}}_{\text{avg}}\!\uparrow$
$M=1$	69.46	47.11	87.23	33.44	267.42	61.05	59.83	30.19
$M=2$	68.77	49.00	83.25	35.75	260.49	62.18	57.07	33.20
$M=4$	67.35	50.05	81.92	36.80	255.19	64.75	56.48	33.92

Appendix

Table 32: Ablation on the trajectory refiner. (a) Number of refinement iterations

M

. (b) Attention types of the refiner, with

M=4

Figure 10: Ablation on training data sources. Relative FVD increase (a) and

\delta^{\text{3D}}_{\text{avg}}

decrease (b) of VPM (V+T) when one source is removed from training, measured against the model trained on all sources. Higher is worse.

Figure 11: Qualitative results of multi-view prediction. Predicted video with overlaid predicted point track for two-view (top) and three-view (bottom) scenes. Conditioning frames are omitted.

Figure 12: Additional qualitative results of multi-view prediction. Predicted video with overlaid predicted point track on robot and general scenes from VP-WorldBench (Real, top) and VP-WorldBench (Sim, bottom).

Figure 13: Predicted vs. ground-truth videos. Each view shows the prediction next to the ground truth. Conditioning frames are omitted.

Figure 14: Example rollouts of robot manipulation tasks. Successful rollouts of our policy on six tasks, shown from two exterior cameras and a wrist camera. Rows are time steps sampled uniformly over each episode of length $T$ .

3D Scene Prediction

See all 16 papers

PointAction: 3D Points as Universal Action Representations for Robot Control
Jun 2, 2026Mutian Tong, Han Jiang, Qiao Feng +2Generalization in Robotic Manipulation Cross-Embodiment Robot Learning
University of Pennsylvania · Work done during internship at University of Pennsylvania.
GEM-4D: Geometry-Enhanced Video World Models for Robot Manipulation
May 20, 2026Kaichen Zhou, Yuzhen Chen, Fangneng Zhan +8Physical Consistency in Video Generation Robot Motion Generation
Harvard AI and Robotics Lab, Harvard University · Media Lab and EECS, MIT · MIT-IBM Watson AI Lab +1
MolmoMotion: Forecasting Point Trajectories in 3D with Language Instruction
Jun 17, 2026Jianing Zhang, Chenhao Zheng, Yajun Yang +10Trajectory Prediction
Allen Institute for AI · University of Washington · UNC-Chapel Hill

Jun 2, 2026J/K move · Enter open · S save

PointAction: 3D Points as Universal Action Representations for Robot Control

Mutian Tong, Han Jiang, Qiao Feng, Lingjie Liu+1

University of Pennsylvania · Work done during internship at University of Pennsylvania.

Generalization in Robotic Manipulation Cross-Embodiment Robot Learning Robotic Manipulation Vision-Based Robot Control

Open paper AskPDF arXiv

Video-Action Models (VAMs) leverage the broad visual dynamics captured by pre-trained video diffusion models, offering a promising path toward generalizable robot manipulation. However, RGB-only video rollouts are not directly actionable: they leave metric 3D motion, contact geometry, and fine-grained spatial constraints under-specified, making action grounding ambiguous. Meanwhile, scaling action supervision across diverse tasks and embodiments remains costly. We present PointAction, a framework that bridges video predictions to robot actions through explicit point-based 4D modeling. PointAction fine-tunes a foundation video generation model to jointly predict future RGB frames and dynamic 3D pointmaps, producing temporally consistent 3D motion of task-relevant scene geometry. These point dynamics serve as a structured, embodiment-agnostic action interface, which a diffusion-based action decoder maps to executable robot actions. By using metric 3D point dynamics as the interface between video prediction and control, PointAction reduces the ambiguity of RGB-only action grounding and supports transfer across tasks and embodiments with limited action supervision. Experiments show that PointAction achieves state-of-the-art 4D generation quality on robot scenes, outperforms existing baselines in simulation, and generalizes to two real robot arms unseen during pretraining.

Abstract

3D Scene Prediction

Explore similar work

PointAction: 3D Points as Universal Action Representations for Robot Control

GEM-4D: Geometry-Enhanced Video World Models for Robot Manipulation

MolmoMotion: Forecasting Point Trajectories in 3D with Language Instruction