Predicting future physical states stands as a central challenge in embodied intelligence. Existing video world models leave 3D dynamics implicit, action-based formulations remain tied to specific robot configurations, and 3D motion prediction models use flow-only supervision, providing a limited learning signal for physical dynamics. To circumvent these limitations, we introduce \method{}, a generative world model that jointly predicts future video and query-based 3D trajectories. By parameterizing 3D motion as continuous displacement tokens aligned with the video grid, a unified Mixture-of-Transformers (MoT) backbone jointly denoises both modalities under a rectified-flow objective. This formulation couples explicit geometric dynamics with visual evolution, effectively unlocking the physical commonsense embedded within pretrained video models. To empirically scale and validate this unified architecture, we train \method{} on a heterogeneous mixture of 1.44 million object-level samples spanning simulated and real-world domains. We further introduce WorldFlowBench, comprising 1,678 evaluation samples from 11 data sources across simulated and real-world interactions, including spatial reasoning tasks. Across all three evaluation sets, WorldFlow achieves the lowest average displacement error (ADE) and final displacement error (FDE) among the evaluated learning-based methods. Notably, on the challenging SpatialCog set, WorldFlow achieves an ADE of 0.060 m, a 54.2% reduction compared with the strongest external baseline. It also reduces ADE and FDE by 55.9% and 68.5%, respectively, compared with its flow-only variant. Project page: https://worldflow-rbs.github.io/.
Figures & tables
Figure 2: Overview of WorldFlow . Conditioned on text tokens tj through the pretrained reasoner, clean observed-frame latents v0 , and query-point geometry, the generator jointly denoises video and flow from noise ϵj and δk through shared multimodal attention. Generated video latents vj are decoded by the VAE, while generated 3D flow ft is rescaled and accumulated from the initial query positions to recover metric trajectories. The 2D flow overlay shows generated 3D trajectories projected onto generated video frames. The 3D flow overlay shows generated 3D trajectories overlaid on a scene point cloud reconstructed from the initial frame’s ground-truth depth.
Figure 3: Qualitative comparison on diverse real and simulated scenes. We visualize predicted 3D query-point trajectories from μ0 , MolmoMotion, and WorldFlow across SpatialCog, AgiBot, DROID, Stereo4D, Meta-World, RLBench, RoboTwin, and MetaFold. The trajectories are projected onto the corresponding input frames for visualization.
Paradigm
Model
Sim7
Real3
SpatialCog
ADE ↓
FDE ↓
PWT ↑
ADE ↓
FDE ↓
PWT ↑
ADE ↓
FDE ↓
PWT ↑
Non-parametric
Static
0.096
0.201
0.573
0.062
0.086
0.464
0.143
0.278
0.412
Video generation + 3D tracking
NovaFlow
0.196
0.318
0.289
0.095
0.129
0.293
0.165
0.295
0.320
Cosmos3-Nano + T4W
0.101
0.202
0.451
0.088
0.133
0.310
0.141
0.257
0.398
Cosmos3-Nano + 4RC
0.096
0.196
0.531
0.085
0.122
0.332
0.141
0.263
0.401
LingBot-Video-MoE + T4W
0.105
0.207
0.425
0.097
0.145
0.267
0.134
0.239
0.364
Table 1: 3D motion forecasting on WorldFlowBench. ADE and FDE measure average and final position errors in meters, respectively; lower is better. PWT is the mean point–frame hit rate over distance thresholds of 1, 2, and 5 cm; higher is better. Bold and underline mark the best and second-best results in each column among non-static methods, including ablations. Static is reported as a reference and excluded from ranking. T4W denotes Track4World. For WorldFlow + T4W/4RC, trajectories are extracted from the videos generated by WorldFlow using the corresponding tracker.
Group
Task Caption
Flow Prediction
Robot Execution
WorldFlow
μ0
WorldFlow
μ0
Pick and Place
Place the watermelon doll into the small pink plate.
3/3
0/3
3/3
0/3
Place the oolong tea into the large blue tray.
3/3
1/3
2/3
0/3
Place the potato chips into the dusty pink basket.
3/3
1/3
2/3
0/3
Place the yellow toy inside the red circle.
3/3
1/3
2/3
1/3
Stacking
Place the green biscuit box on the translucent box.
3/3
1/3
3/3
0/3
Table 2: Real-world manipulation with different flow predictors. Each task is evaluated over three trials, yielding 36 trials per method. Flow prediction and robot execution are evaluated separately for each trial. Both methods use matched initial layouts, identical grasp results, and the same trajectory optimization and execution settings; only the flow predictor changes. Bold indicates the higher success rate.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure I : Initial data volume and retention across sources before the training/evaluation split. (a) Number of candidate object records entering the reported processing stage, shown on a logarithmic scale. (b) Fraction retained after formatting, quality control, and manifest validation, before benchmark exclusion. For SpatialCog, 82,831 candidate records yield 82,588 valid records, corresponding to a retention rate of 99.7%. Initial counts describe source-specific candidate pools rather than the full original datasets. Colors indicate the four collections defined in the main paper. All counts refer to object records rather than unique videos.
Source
Collection
Description
Object clips
Standardized duration (h)
Sim7
Sim7
Simulated robot manipulation drawn from seven simulation datasets.
190,267
161.9
AgiBot World
Real3
Real-world robot manipulation across diverse environments and tasks.
631,873
537.5
STH-V2
Real3
Human–object interactions covering a broad range of object motions.
86,811
73.8
EPIC-KITCHENS
Real3
Egocentric human activities and object interactions in kitchens.
5,337
4.5
SpatialCog
SpatialCog
Tabletop and household interactions emphasizing spatial relations.
82,188
69.9
MolmoSpaces
MolmoMotion-1M
Simulated interactive scenes with released 3D motion annotations.
164,970
140.3
Appendix
Table I : Composition of the processed data inventory. Real3 and SpatialCog counts exclude WorldFlowBench evaluation records; Sim7 uses its QC inventory before runtime benchmark exclusion. Each object clip contains 49 frames standardized to 16 FPS. Standardized duration in hours is computed as Nclips×49/(16×3600) , where Nclips denotes the reported number of object clips. This measures standardized data volume rather than deduplicated source-video duration, since different object records may share the same underlying video interval.
Figure II : Native temporal coverage of the six retained MolmoMotion-1M subsets before conversion to 49 frames. (a) Native interval-length distributions; thin lines indicate P5–P95, thick bars indicate the interquartile range, and circles indicate the median. The dashed line marks the target length of 49 frames. (b) Percentage of records requiring tail padding. Intervals longer than 49 frames are uniformly subsampled.
Figure III : Pre-normalization motion-scale statistics. Left: median, 90th percentile, and 99th percentile of the per-record component RMS. Right: source-level aggregate component RMS. Both horizontal axes use logarithmic scales. SpatialCog statistics use 82,188 records after excluding all 400 evaluation samples; Real3 statistics also exclude benchmark records. Sim7, marked with an asterisk, uses its QC inventory before runtime benchmark exclusion and provides aggregate moments but no per-record percentile summary. Statistics include zero-motion transitions introduced by temporal padding. YT-VIS values follow the non-metric scale of its annotations.
Interval
Steps
Simulation
Molmo-real
Outdoor
Real3
SpatialCog
Simulation-focused
[0,12,000)
80%
5%
0%
10%
5%
Molmo-real adaptation
[12,000,24,000)
10%
70%
0%
15%
5%
Outdoor adaptation
[24,000,24,100)
5%
20%
50%
25%
0%
Real3 adaptation
[24,100,48,100)
10%
10%
0%
75%
5%
SpatialCog refinement
[48,100,51,100)
5%
10%
0%
35%
50%
Appendix
Table II : Target domain-sampling probabilities for the second-stage curriculum. Adjacent distributions are linearly interpolated near interval boundaries using the windows specified in the text. Within each domain, sources are sampled according to Eq. equation 12 .
Figure IV : Learning-rate schedule for second-stage training. Background bands indicate the curriculum intervals. The narrow gold band marks the 100-step Outdoor interval, during which the learning rate remains at 8×10−5 .
Humans can infer how objects are likely to move from passive observation: a cup may be lifted, a drawer may slide, and a lid may rotate shut. Such predictions expose the physical consequences of interaction needed to act in the real world. We study how to learn this anticipation from ordinary monocular videos of human-object interaction. Given a short observed video context, MotionForesight predicts future 3D trajectories for points on the manipulated object. This casts interaction prediction as object-centered 3D motion forecasting without any assumptions on the object properties. Our key insight is that video prediction models already encode rich priors about how objects move during human interactions. We redirect these priors from pixel prediction toward future 3D scene flow. We start from a dense 3D tracker built on a pretrained video model, generate pseudo-ground-truth tracks from complete clips, and train the forecaster using only the observed frames. We replace future RGB and geometry with learned mask latents and train a lightweight adapter to turn the retrospective tracking representation into a forward predictor, while freezing the large video and tracking components. Using just 40k human videos and no auxiliary inputs such as language, MotionForesight generalizes across diverse out-of-distribution objects, environments, viewpoints, and interactions. It also outperforms substantially larger models that use over a million training videos. These results show that we can efficiently re-purpose video priors into explicit geometric forecasts for embodied intelligence. https://motionforesight.github.io/
Homanga Bharadhwaj, Yash Jangir
Department of Computer Science, Johns Hopkins University
Video world models must generate physically plausible futures, not just realistic frames. Existing approaches typically use semantic, physical, or predictive information to guide the visual generation stream, leaving a fundamental question: where inside the generator should predictions of scene evolution live? Standard video tokens jointly encode appearance and dynamics, but provide no distinct state for learning and reading predictive dynamics. We answer this question with the Physical Registry: temporally grounded registry slots inside a video DiT that are initialized from an observation-conditioned forecast, co-evolve with video tokens, and receive future-representation supervision, making predictive dynamics an explicit, readable state of the generator. In a controlled comparison of predictive-state interfaces on Physics-IQ Verified, Physical Registry improves vanilla fine-tuning by 9.9/8.7 points on I2V/V2V, the largest gain among the interfaces we compare. Across all Cosmos and WAN backbones, Physical Registry improves Physics-IQ Verified scores. With only 0.45% additional parameters, it enables Cosmos 3 Nano 16B to surpass the published I2V score of Cosmos 3 Super 64B (43.56 vs. 42.7). On WorldModelBench, it also improves physical adherence and the total score for all four evaluated backbones. Because the registry becomes candidate-specific while remaining readable, the Physical Registry Selector (PRS) ranks emerging futures by how far their intermediate registry readouts depart from the shared forecast before full denoising is complete. Across four backbones, this selection improves I2V scores by 5.6-15.9% without completed-video external scoring. Together, these results support a broader principle: dynamics prediction should live as an explicit model state that co-evolves with the video representation. Project page: https://grainsack.github.io/physical-registry/
Forecasting the evolution of dynamic environments is crucial for autonomous agents. While generative world models have recently achieved high photorealism in 2D video synthesis by mixing ego-motion and environmental dynamics within the image plane, they exhibit physical inconsistencies, such as morphing or vanishing objects, especially over long time horizons. In this paper, we propose FR3D, a world model that predicts a persistent 3D latent representation for future dynamic 3D reconstruction. Unlike prior works that treat the world as a sequence of image-based features, FR3D explicitly decouples the 3D evolution of the scene from the agent's trajectory, treating the inferred ego-motion as a latent proxy for action. This disentanglement resolves the ambiguities between self-motion and world-motion, ensuring geometric consistency into the future. Furthermore, we introduce a teacher-student distillation strategy that leverages the spatial "common sense" of off-the-shelf foundation models, leading to robust zero-shot generalization. Extensive experiments demonstrate FR3D's strong performance for future dynamic 3D reconstruction from monocular observations across multiple datasets, even 2 seconds into the future. Project page: https://fr3d-wm.github.io.
Nils Morbitzer, Jonathan Evers, Artem Savkin +4
Technical University of Munich (TUM) · Munich Center for Machine Learning (MCML) · BMW Group +1