WorldFlow: Joint Video and 3D Motion Prediction via Large-Scale Pretraining

Yuhao Lu; Jiongxu Chen; Hexiang Wei; Jiajin Wu; Juelin Li; Mingjian Liang; Fan Yang; Zhuowei Liang

cs.CVOct 29, 2026

WorldFlow: Joint Video and 3D Motion Prediction via Large-Scale Pretraining

Authors: Yuhao Lu, Jiongxu Chen, Hexiang Wei, Jiajin Wu, Juelin Li, Mingjian Liang, Fan Yang, Zhuowei Liang,

Organizations: RoboScience · National University of Singapore

Abstract

Predicting future physical states stands as a central challenge in embodied intelligence. Existing video world models leave 3D dynamics implicit, action-based formulations remain tied to specific robot configurations, and 3D motion prediction models use flow-only supervision, providing a limited learning signal for physical dynamics. To circumvent these limitations, we introduce \method{}, a generative world model that jointly predicts future video and query-based 3D trajectories. By parameterizing 3D motion as continuous displacement tokens aligned with the video grid, a unified Mixture-of-Transformers (MoT) backbone jointly denoises both modalities under a rectified-flow objective. This formulation couples explicit geometric dynamics with visual evolution, effectively unlocking the physical commonsense embedded within pretrained video models. To empirically scale and validate this unified architecture, we train \method{} on a heterogeneous mixture of 1.44 million object-level samples spanning simulated and real-world domains. We further introduce WorldFlowBench, comprising 1,678 evaluation samples from 11 data sources across simulated and real-world interactions, including spatial reasoning tasks. Across all three evaluation sets, WorldFlow achieves the lowest average displacement error (ADE) and final displacement error (FDE) among the evaluated learning-based methods. Notably, on the challenging SpatialCog set, WorldFlow achieves an ADE of 0.060 m, a 54.2% reduction compared with the strongest external baseline. It also reduces ADE and FDE by 55.9% and 68.5%, respectively, compared with its flow-only variant. Project page: https://worldflow-rbs.github.io/.

Figures & tables

Figure 2: Overview of WorldFlow . Conditioned on text tokens $t_{j}$ through the pretrained reasoner, clean observed-frame latents $v_{0}$ , and query-point geometry, the generator jointly denoises video and flow from noise $\epsilon_{j}$ and $\delta_{k}$ through shared multimodal attention. Generated video latents $v_{j}$ are decoded by the VAE, while generated 3D flow $f_{t}$ is rescaled and accumulated from the initial query positions to recover metric trajectories. The 2D flow overlay shows generated 3D trajectories projected onto generated video frames. The 3D flow overlay shows generated 3D trajectories overlaid on a scene point cloud reconstructed from the initial frame’s ground-truth depth.

Figure 3: Qualitative comparison on diverse real and simulated scenes. We visualize predicted 3D query-point trajectories from $\mu_{0}$ , MolmoMotion, and WorldFlow across SpatialCog, AgiBot, DROID, Stereo4D, Meta-World, RLBench, RoboTwin, and MetaFold. The trajectories are projected onto the corresponding input frames for visualization.

Paradigm	Model	Sim7			Real3			SpatialCog
Paradigm	Model	ADE $\downarrow$	FDE $\downarrow$	PWT $\uparrow$	ADE $\downarrow$	FDE $\downarrow$	PWT $\uparrow$	ADE $\downarrow$	FDE $\downarrow$	PWT $\uparrow$
Non-parametric	Static	0.096	0.201	0.573	0.062	0.086	0.464	0.143	0.278	0.412
Video generation + 3D tracking	NovaFlow	0.196	0.318	0.289	0.095	0.129	0.293	0.165	0.295	0.320
	Cosmos3-Nano + T4W	0.101	0.202	0.451	0.088	0.133	0.310	0.141	0.257	0.398
	Cosmos3-Nano + 4RC	0.096	0.196	0.531	0.085	0.122	0.332	0.141	0.263	0.401
	LingBot-Video-MoE + T4W	0.105	0.207	0.425	0.097	0.145	0.267	0.134	0.239	0.364

Table 1: 3D motion forecasting on WorldFlowBench. ADE and FDE measure average and final position errors in meters, respectively; lower is better. PWT is the mean point–frame hit rate over distance thresholds of 1, 2, and 5 cm; higher is better. Bold and underline mark the best and second-best results in each column among non-static methods, including ablations. Static is reported as a reference and excluded from ranking. T4W denotes Track4World. For WorldFlow + T4W/4RC, trajectories are extracted from the videos generated by WorldFlow using the corresponding tracker.

Group	Task Caption	Flow Prediction		Robot Execution
Group		WorldFlow	$\mu_{0}$	WorldFlow	$\mu_{0}$
Pick and Place	Place the watermelon doll into the small pink plate.	3/3	0/3	3/3	0/3
	Place the oolong tea into the large blue tray.	3/3	1/3	2/3	0/3
	Place the potato chips into the dusty pink basket.	3/3	1/3	2/3	0/3
	Place the yellow toy inside the red circle.	3/3	1/3	2/3	1/3
Stacking	Place the green biscuit box on the translucent box.	3/3	1/3	3/3	0/3

Table 2: Real-world manipulation with different flow predictors. Each task is evaluated over three trials, yielding 36 trials per method. Flow prediction and robot execution are evaluated separately for each trial. Both methods use matched initial layouts, identical grasp results, and the same trajectory optimization and execution settings; only the flow predictor changes. Bold indicates the higher success rate.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure I : Initial data volume and retention across sources before the training/evaluation split. (a) Number of candidate object records entering the reported processing stage, shown on a logarithmic scale. (b) Fraction retained after formatting, quality control, and manifest validation, before benchmark exclusion. For SpatialCog, 82,831 candidate records yield 82,588 valid records, corresponding to a retention rate of 99.7%. Initial counts describe source-specific candidate pools rather than the full original datasets. Colors indicate the four collections defined in the main paper. All counts refer to object records rather than unique videos.

Source	Collection	Description	Object clips	Standardized duration (h)
Sim7	Sim7	Simulated robot manipulation drawn from seven simulation datasets.	190,267	161.9
AgiBot World	Real3	Real-world robot manipulation across diverse environments and tasks.	631,873	537.5
STH-V2	Real3	Human–object interactions covering a broad range of object motions.	86,811	73.8
EPIC-KITCHENS	Real3	Egocentric human activities and object interactions in kitchens.	5,337	4.5
SpatialCog	SpatialCog	Tabletop and household interactions emphasizing spatial relations.	82,188	69.9
MolmoSpaces	MolmoMotion-1M	Simulated interactive scenes with released 3D motion annotations.	164,970	140.3

Appendix

Table I : Composition of the processed data inventory. Real3 and SpatialCog counts exclude WorldFlowBench evaluation records; Sim7 uses its QC inventory before runtime benchmark exclusion. Each object clip contains 49 frames standardized to 16 FPS. Standardized duration in hours is computed as

N_{\mathrm{clips}}\times 49/(16\times 3600)

, where

N_{\mathrm{clips}}

denotes the reported number of object clips. This measures standardized data volume rather than deduplicated source-video duration, since different object records may share the same underlying video interval.

Figure II : Native temporal coverage of the six retained MolmoMotion-1M subsets before conversion to 49 frames. (a) Native interval-length distributions; thin lines indicate P5–P95, thick bars indicate the interquartile range, and circles indicate the median. The dashed line marks the target length of 49 frames. (b) Percentage of records requiring tail padding. Intervals longer than 49 frames are uniformly subsampled.

Figure III : Pre-normalization motion-scale statistics. Left: median, 90th percentile, and 99th percentile of the per-record component RMS. Right: source-level aggregate component RMS. Both horizontal axes use logarithmic scales. SpatialCog statistics use 82,188 records after excluding all 400 evaluation samples; Real3 statistics also exclude benchmark records. Sim7, marked with an asterisk, uses its QC inventory before runtime benchmark exclusion and provides aggregate moments but no per-record percentile summary. Statistics include zero-motion transitions introduced by temporal padding. YT-VIS values follow the non-metric scale of its annotations.

Interval	Steps	Simulation	Molmo-real	Outdoor	Real3	SpatialCog
Simulation-focused	$[0,12{,}000)$	80%	5%	0%	10%	5%
Molmo-real adaptation	$[12{,}000,24{,}000)$	10%	70%	0%	15%	5%
Outdoor adaptation	$[24{,}000,24{,}100)$	5%	20%	50%	25%	0%
Real3 adaptation	$[24{,}100,48{,}100)$	10%	10%	0%	75%	5%
SpatialCog refinement	$[48{,}100,51{,}100)$	5%	10%	0%	35%	50%

Appendix

Table II : Target domain-sampling probabilities for the second-stage curriculum. Adjacent distributions are linearly interpolated near interval boundaries using the windows specified in the text. Within each domain, sources are sampled according to Eq. equation 12 .

Figure IV : Learning-rate schedule for second-stage training. Background bands indicate the curriculum intervals. The narrow gold band marks the 100-step Outdoor interval, during which the learning rate remains at

8\times 10^{-5}

Video World Models

Video world models generate and predict action-conditioned visual futures to simulate changing scenes for embodied agents.

See all 264 papers

MotionForesight: Re-purposing Video Models for Future 3D Scene-Flow Prediction
Jul 17, 2026Homanga Bharadhwaj, Yash JangirScene Flow Estimation Trajectory Prediction
Department of Computer Science, Johns Hopkins University
World Models Need a Physical Registry: Learning Internal Predictive States for Physically Plausible World Models
Oct 12, 2026Shinseong Kim, Jaeseok Jeong, Jianlong Fu +1Video World Models Physical Consistency in Video Generation
Yonsei University · Microsoft Research Asia
Future Dynamic 3D Reconstruction: A 3D World Model with Disentangled Ego-Motion
Jun 16, 2026Nils Morbitzer, Jonathan Evers, Artem Savkin +4World Model Learning Monocular 3D Reconstruction
Technical University of Munich (TUM) · Munich Center for Machine Learning (MCML) · BMW Group +1

Jul 17, 2026J/K move · Enter open · S save

MotionForesight: Re-purposing Video Models for Future 3D Scene-Flow Prediction

Homanga Bharadhwaj, Yash Jangir

Department of Computer Science, Johns Hopkins University

Scene Flow Estimation Trajectory Prediction 3D Scene Understanding Human-Object Interaction 3D Scene Prediction

Open paper AskPDF arXiv

Humans can infer how objects are likely to move from passive observation: a cup may be lifted, a drawer may slide, and a lid may rotate shut. Such predictions expose the physical consequences of interaction needed to act in the real world. We study how to learn this anticipation from ordinary monocular videos of human-object interaction. Given a short observed video context, MotionForesight predicts future 3D trajectories for points on the manipulated object. This casts interaction prediction as object-centered 3D motion forecasting without any assumptions on the object properties. Our key insight is that video prediction models already encode rich priors about how objects move during human interactions. We redirect these priors from pixel prediction toward future 3D scene flow. We start from a dense 3D tracker built on a pretrained video model, generate pseudo-ground-truth tracks from complete clips, and train the forecaster using only the observed frames. We replace future RGB and geometry with learned mask latents and train a lightweight adapter to turn the retrospective tracking representation into a forward predictor, while freezing the large video and tracking components. Using just 40k human videos and no auxiliary inputs such as language, MotionForesight generalizes across diverse out-of-distribution objects, environments, viewpoints, and interactions. It also outperforms substantially larger models that use over a million training videos. These results show that we can efficiently re-purpose video priors into explicit geometric forecasts for embodied intelligence. https://motionforesight.github.io/

Abstract

Video World Models

Explore similar work

MotionForesight: Re-purposing Video Models for Future 3D Scene-Flow Prediction

World Models Need a Physical Registry: Learning Internal Predictive States for Physically Plausible World Models

Future Dynamic 3D Reconstruction: A 3D World Model with Disentangled Ego-Motion