FloMo: 3D Scene Flow as a World Action Model Intermediate for Learning from Human Video

Jeremy Collins; Namra Patel; Ayush Agarwal; Shitij Govil; Animesh Garg

cs.CVNov 3, 2026

FloMo: 3D Scene Flow as a World Action Model Intermediate for Learning from Human Video

Authors: Jeremy Collins, Namra Patel, Ayush Agarwal, Shitij Govil, Animesh Garg

Organizations: Georgia Institute of Technology

Abstract

Pretrained video generation models distill internet-scale motion priors into robot policies, but full video is a redundant signal for action prediction. We argue that dense 3D motion is the medium through which video pretraining best transfers to robot actions, and that predicting motion rather than video affords better sample-efficiency for action prediction. FloMo is a 5B video-generation model fine-tuned to predict 3D scene-flow and action tokens. Rendering scene flow as RGB and encoding it into the same latent space as video enables a pretrained video generation backbone to efficiently adapt to predict 3D scene flow. Co-trained on action-free human video and action-labeled robot teleoperation data, our model benefits from human-video pretraining and generalizes to unseen objects, motion primitives, and tasks. Replacing motion prediction with a video prediction objective degrades out-of-distribution generalization, while adding a video prediction target alongside motion lowers policy success rather than improving it. When motion, video, and action are predicted jointly, action tokens attend far more to motion than to video, suggesting that predicting video wastes model capacity. Together, these results indicate that motion effectively mediates the transfer from video pretraining to action. More information can be found at flomo-wam.github.io.

Figures & tables

Figure 1: Given an image and text prompt, FloMo jointly predicts dense 3D motion (scene flow) and robot actions using a pretrained video generation model using flow-matching. To tokenize scene flow, we transform cumulative 3D velocities into RGB videos that are encoded by the pretrained video backbone’s video encoder.

Figure 2: In-distribution success rates. Across 15 rollouts FloMo performs best on every task; the matched video–motion–action (VMA) and video–action (VA) variants lower success.

Panel of Table 3 — Table I: Out-of-distribution success rates and human-video ablation. We evaluate generalization across unseen motion primitives, tasks, and objects. The main comparison uses $10$ rollouts per condition ( $30$ total); the no-human-video ablation uses $5$ rollouts per condition ( $15$ total). SR = successful rollouts / total; TP = avg. fraction of subtasks completed (%).

Figure 3: (a) RoboTwin single-task evaluation. Recorded RoboTwin Easy success rates across five tasks, with 100 evaluations per task and method. Across all tasks, FloMo performs the best. (b) Inverse-dynamics scaling. Held-out action error versus action-data scale for two ground-truth conditioning signals (log-log). Recovering actions from motion is more sample-efficient than from video: the gap is largest at

1\%

of the data and narrows as data grows.

Figure 4: Action attention concentrates on motion tokens. Share of action-query attention to each input modality, averaged over attention heads and robot validation rollouts, as a function of DiT layer. Attention to motion rises through the trunk and dominates at the late-middle layers, while attention to video stays lowest throughout; averaged over layers, action queries attend to motion roughly $3\times$ more than to video.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

	Mid-training	Fine-tuning
Learning rate (constant)	$2.8\times 10^{-4}$	$5.6\times 10^{-5}$
Global batch size	$512$	$256$
Action loss weight $\lambda_{a}$	$0.1$	$1.0$
Optimizer steps	${\approx}14{,}000$	${\approx}1{,}500$

Appendix

Table II: Per-stage training settings.

	Train			Validation
Dataset	Tasks	Hours	Clips	Tasks	Hours	Clips
EgoVerse	$3{,}660$	$81.0$	$4{,}303$	$35$	$1.0$	$46$
MolmoAct2-BimanualYAM	$25$	$10.0$	$428$	$22$	$1.0$	$46$
In-house YAM	$4$	$3.7$	$1{,}029$	$4$	$0.4$	$113$
In-house Human Data	$11$	$0.7$	$710$	$11$	$0.1$	$83$
In-house Play YAM	$162$	$0.4$	$173$	$19$	$0.05$	$19$

Appendix

Table III: Dataset composition. “Tasks” counts distinct task labels (EgoVerse uses fine-grained activity categories, the others coarse manipulation tasks); “Clips” counts clips, videos, or demonstrations (one recording each); “Hours” is the span of unique footage used in training: the number of distinct frames appearing in any

16

-frame window, divided by the source frame rate. All figures are for the subset actually used in each split.

Dataset	Mid-training weight	Fine-tuning weight
EgoVerse	0.65	0.00
MolmoAct2-BimanualYAM	0.125	0.00
In-house YAM	0.125	0.75
In-house Human Data	0.05	0.00
In-house Play YAM	0.05	0.25

Appendix

Table IV: Dataset sampling weights across training stages .

Robot Learning from Video

See all 67 papers

WorldFlow: Joint Video and 3D Motion Prediction via Large-Scale Pretraining
Oct 29, 2026Yuhao Lu, Jiongxu Chen, Hexiang Wei +9Video World Models Multimodal Trajectory Prediction
RoboScience · National University of Singapore
MotionForesight: Re-purposing Video Models for Future 3D Scene-Flow Prediction
Jul 17, 2026Homanga Bharadhwaj, Yash JangirScene Flow Estimation Trajectory Prediction
Department of Computer Science, Johns Hopkins University
FlowWAM: Optical Flow as a Unified Action Representation for World Action Models
Jul 14, 2026Yixiang Chen, Peiyan Li, Yuan Xu +13Action-Conditioned Video Generation Video Diffusion Models
New Laboratory of Pattern Recognition (NLPR), Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · MBZUAI +2

Oct 29, 2026J/K move · Enter open · S save

WorldFlow: Joint Video and 3D Motion Prediction via Large-Scale Pretraining

Yuhao Lu, Jiongxu Chen, Hexiang Wei, Jiajin Wu+8

RoboScience · National University of Singapore

Video World Models Multimodal Trajectory Prediction 3D Scene Prediction

Open paper AskPDF arXiv

Predicting future physical states stands as a central challenge in embodied intelligence. Existing video world models leave 3D dynamics implicit, action-based formulations remain tied to specific robot configurations, and 3D motion prediction models use flow-only supervision, providing a limited learning signal for physical dynamics. To circumvent these limitations, we introduce \method{}, a generative world model that jointly predicts future video and query-based 3D trajectories. By parameterizing 3D motion as continuous displacement tokens aligned with the video grid, a unified Mixture-of-Transformers (MoT) backbone jointly denoises both modalities under a rectified-flow objective. This formulation couples explicit geometric dynamics with visual evolution, effectively unlocking the physical commonsense embedded within pretrained video models. To empirically scale and validate this unified architecture, we train \method{} on a heterogeneous mixture of 1.44 million object-level samples spanning simulated and real-world domains. We further introduce WorldFlowBench, comprising 1,678 evaluation samples from 11 data sources across simulated and real-world interactions, including spatial reasoning tasks. Across all three evaluation sets, WorldFlow achieves the lowest average displacement error (ADE) and final displacement error (FDE) among the evaluated learning-based methods. Notably, on the challenging SpatialCog set, WorldFlow achieves an ADE of 0.060 m, a 54.2% reduction compared with the strongest external baseline. It also reduces ADE and FDE by 55.9% and 68.5%, respectively, compared with its flow-only variant. Project page: https://worldflow-rbs.github.io/.

Abstract

Robot Learning from Video

Explore similar work

WorldFlow: Joint Video and 3D Motion Prediction via Large-Scale Pretraining

MotionForesight: Re-purposing Video Models for Future 3D Scene-Flow Prediction

FlowWAM: Optical Flow as a Unified Action Representation for World Action Models