Pretrained video generation models distill internet-scale motion priors into robot policies, but full video is a redundant signal for action prediction. We argue that dense 3D motion is the medium through which video pretraining best transfers to robot actions, and that predicting motion rather than video affords better sample-efficiency for action prediction. FloMo is a 5B video-generation model fine-tuned to predict 3D scene-flow and action tokens. Rendering scene flow as RGB and encoding it into the same latent space as video enables a pretrained video generation backbone to efficiently adapt to predict 3D scene flow. Co-trained on action-free human video and action-labeled robot teleoperation data, our model benefits from human-video pretraining and generalizes to unseen objects, motion primitives, and tasks. Replacing motion prediction with a video prediction objective degrades out-of-distribution generalization, while adding a video prediction target alongside motion lowers policy success rather than improving it. When motion, video, and action are predicted jointly, action tokens attend far more to motion than to video, suggesting that predicting video wastes model capacity. Together, these results indicate that motion effectively mediates the transfer from video pretraining to action. More information can be found at flomo-wam.github.io.
Figures & tables
Figure 1: Given an image and text prompt, FloMo jointly predicts dense 3D motion (scene flow) and robot actions using a pretrained video generation model using flow-matching. To tokenize scene flow, we transform cumulative 3D velocities into RGB videos that are encoded by the pretrained video backbone’s video encoder.
Figure 2: In-distribution success rates. Across 15 rollouts FloMo performs best on every task; the matched video–motion–action (VMA) and video–action (VA) variants lower success.
Table I: Out-of-distribution success rates and human-video ablation. We evaluate generalization across unseen motion primitives, tasks, and objects. The main comparison uses 10 rollouts per condition ( 30 total); the no-human-video ablation uses 5 rollouts per condition ( 15 total). SR = successful rollouts / total; TP = avg. fraction of subtasks completed (%).
Figure 3: (a) RoboTwin single-task evaluation. Recorded RoboTwin Easy success rates across five tasks, with 100 evaluations per task and method. Across all tasks, FloMo performs the best. (b) Inverse-dynamics scaling. Held-out action error versus action-data scale for two ground-truth conditioning signals (log-log). Recovering actions from motion is more sample-efficient than from video: the gap is largest at 1% of the data and narrows as data grows.
Figure 4: Action attention concentrates on motion tokens. Share of action-query attention to each input modality, averaged over attention heads and robot validation rollouts, as a function of DiT layer. Attention to motion rises through the trunk and dominates at the late-middle layers, while attention to video stays lowest throughout; averaged over layers, action queries attend to motion roughly 3× more than to video.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Mid-training
Fine-tuning
Learning rate (constant)
2.8×10−4
5.6×10−5
Global batch size
512
256
Action loss weight λa
0.1
1.0
Optimizer steps
≈14,000
≈1,500
Appendix
Table II: Per-stage training settings.
Train
Validation
Dataset
Tasks
Hours
Clips
Tasks
Hours
Clips
EgoVerse
3,660
81.0
4,303
35
1.0
46
MolmoAct2-BimanualYAM
25
10.0
428
22
1.0
46
In-house YAM
4
3.7
1,029
4
0.4
113
In-house Human Data
11
0.7
710
11
0.1
83
In-house Play YAM
162
0.4
173
19
0.05
19
Appendix
Table III: Dataset composition. “Tasks” counts distinct task labels (EgoVerse uses fine-grained activity categories, the others coarse manipulation tasks); “Clips” counts clips, videos, or demonstrations (one recording each); “Hours” is the span of unique footage used in training: the number of distinct frames appearing in any 16 -frame window, divided by the source frame rate. All figures are for the subset actually used in each split.
Dataset
Mid-training weight
Fine-tuning weight
EgoVerse
0.65
0.00
MolmoAct2-BimanualYAM
0.125
0.00
In-house YAM
0.125
0.75
In-house Human Data
0.05
0.00
In-house Play YAM
0.05
0.25
Appendix
Table IV: Dataset sampling weights across training stages .
Predicting future physical states stands as a central challenge in embodied intelligence. Existing video world models leave 3D dynamics implicit, action-based formulations remain tied to specific robot configurations, and 3D motion prediction models use flow-only supervision, providing a limited learning signal for physical dynamics. To circumvent these limitations, we introduce \method{}, a generative world model that jointly predicts future video and query-based 3D trajectories. By parameterizing 3D motion as continuous displacement tokens aligned with the video grid, a unified Mixture-of-Transformers (MoT) backbone jointly denoises both modalities under a rectified-flow objective. This formulation couples explicit geometric dynamics with visual evolution, effectively unlocking the physical commonsense embedded within pretrained video models. To empirically scale and validate this unified architecture, we train \method{} on a heterogeneous mixture of 1.44 million object-level samples spanning simulated and real-world domains. We further introduce WorldFlowBench, comprising 1,678 evaluation samples from 11 data sources across simulated and real-world interactions, including spatial reasoning tasks. Across all three evaluation sets, WorldFlow achieves the lowest average displacement error (ADE) and final displacement error (FDE) among the evaluated learning-based methods. Notably, on the challenging SpatialCog set, WorldFlow achieves an ADE of 0.060 m, a 54.2% reduction compared with the strongest external baseline. It also reduces ADE and FDE by 55.9% and 68.5%, respectively, compared with its flow-only variant. Project page: https://worldflow-rbs.github.io/.
Humans can infer how objects are likely to move from passive observation: a cup may be lifted, a drawer may slide, and a lid may rotate shut. Such predictions expose the physical consequences of interaction needed to act in the real world. We study how to learn this anticipation from ordinary monocular videos of human-object interaction. Given a short observed video context, MotionForesight predicts future 3D trajectories for points on the manipulated object. This casts interaction prediction as object-centered 3D motion forecasting without any assumptions on the object properties. Our key insight is that video prediction models already encode rich priors about how objects move during human interactions. We redirect these priors from pixel prediction toward future 3D scene flow. We start from a dense 3D tracker built on a pretrained video model, generate pseudo-ground-truth tracks from complete clips, and train the forecaster using only the observed frames. We replace future RGB and geometry with learned mask latents and train a lightweight adapter to turn the retrospective tracking representation into a forward predictor, while freezing the large video and tracking components. Using just 40k human videos and no auxiliary inputs such as language, MotionForesight generalizes across diverse out-of-distribution objects, environments, viewpoints, and interactions. It also outperforms substantially larger models that use over a million training videos. These results show that we can efficiently re-purpose video priors into explicit geometric forecasts for embodied intelligence. https://motionforesight.github.io/
Homanga Bharadhwaj, Yash Jangir
Department of Computer Science, Johns Hopkins University
World Action Models (WAMs) are able to leverage pretrained video generators for both world modeling and action prediction. However, directly leveraging such video generators for control raises a new challenge: how to represent actions in a suitable form that aligns with pretrained video generators while carrying enough motion cues for accurate control. Existing numerical actions fail to satisfy the former, and prior visual action representations overlook the temporal motion structure across frames. We address this issue with FlowWAM, a dual-stream diffusion framework that adopts optical flow as a unified, video-native action representation. Flow videos share the same format as RGB videos and encode rich per-pixel displacement. By jointly modeling them within a shared pretrained video generator, FlowWAM can naturally implement two modes of WAMs. In policy mode, FlowWAM generates flow for action prediction, while in world-model mode, it uses target flow sequences to guide future video generation. Moreover, since flow can be easily extracted from raw videos without action labels, FlowWAM can leverage large-scale action-unlabeled video datasets for pretraining. We empirically find that our flow-based action representation delivers gains across both modes. On RoboTwin manipulation, FlowWAM raises the success rate to 92.94% on the Clean setting and 92.14% on Random, outperforming both VLA and WAM baselines. On WorldArena world modeling, it achieves the best overall EWMScore (63.71) with an 18.4% relative improvement in trajectory accuracy. More results can be found on our project website: https://flow-wam.github.io .
Yixiang Chen, Peiyan Li, Yuan Xu +13
New Laboratory of Pattern Recognition (NLPR), Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · MBZUAI +2