Generative world modeling from large-scale visual data offers a promising path toward scalable robot manipulation. This calls for an intermediate representation that can be supervised at scale and remains actionable for downstream control. We propose 3D full-object flow, an object-centric motion representation that assigns dense 3D trajectories to points across the complete surface of a target object, including its occluded regions. By extending visible-surface object flow to the full object geometry, this representation makes complete object motion explicit and applies uniformly to both rigid and deformable objects. To realize the potential of this representation at scale, we introduce a 1210-hour full-object flow dataset that addresses the scarcity of large-scale supervision for complete 3D object motion. The dataset integrates diverse real-world videos and physical simulations, with a scalable annotation pipeline for recovering full-object motion from real-world observations and quality metrics for systematically assessing annotation quality. We then train a world model that takes an image and a task description as input and predicts dense full-object flow over the reconstructed object geometry. Experiments show reliable annotation quality, data-scalable flow prediction, and effective downstream manipulation across novel scenes and robot embodiments. Project website with additional results and videos: https://nus-lins-lab.github.io/ObjectWorld/.
Figures & tables
Fig. 2 : Automatic data annotation pipeline. Given an in-the-wild manipulation video, a VLM identifies the target object, describes the task, and segments the video into atomic action clips. For each clip, our pipeline extracts the target-object masks, reconstructs the complete object mesh and the point clouds, and aligns the mesh with the point clouds across frames to recover the object motion. The motion is propagated over the complete object surface to produce 3D full-object flow.
Fig. 3 : Object-centric full-object flow prediction framework. Given an initial RGB-D observation and task instruction, Qwen2.5-VL [ 39 ] extracts task-conditioned visual-language features, while a joint PointTransformer V3 [ 40 ] (PTv3) encodes reconstructed object and scene geometry. These features are fused into object tokens and refined by self-attention. The object queries attend to scene and language context through a cross-attention decoder, conditioned on noisy flow and time t , before a flow head predicts dense trajectories over the complete object surface.
ID
Study
Supports
A1
Pipeline validation against simulator ground truth
Q1
A2
Filter-gate validation against expert labels
Q1
A3
Full-object vs. partial-object flow annotation
Q1
B1
Flow predictor comparison with baselines
Q2
B2
Robustness of flow predictor
Q2
B3
Data scaling experiment
Q3
TABLE I : Experiment map. A: annotation validation; B: flow prediction and scaling; C: robot execution and generalization.
Module / Stage
Evaluated Output
Protocol
Metric
Median
Time (s)
Gemini 2.5 Pro (VLM)
Localization
Against GT box
Box IoU ↑
0.908
12.04 †
SeC
Mask propagation
Against GT masks
Mask IoU ↑
0.873
11.20
Track4World (DA3)
Relative depth
Median-scale aligned
AbsRel ↓ / δ<1.25↑
0.089 / 0.826
8.25
SAM 3D Objects
Complete geometry
Shape aligned
CD/ D↓
2.61%
7.41
Hidden geometry
Visible-only Sim(3)
Vis./hid. surface error/ D↓
4.22/7.28%
Hidden coverage
Visible-only Sim(3)
Recall@ 5%D↑
0.462
TABLE II : A1. Ground-truth validation and runtime of the annotation pipeline on simulation. Quality metrics are medians over 97 successfully annotated RoboTwin clips. D denotes the ground-truth object diameter. Times are measured on standardized 3-second clips. † Amortized time per call; ‡ CPU flow construction.
Fig. 4 : A2. Expert validation of the filtering gates. Confusion matrices for the Golden and Silver gates on 200 expert-labeled real-world clips.
Representation
EPE/ D↓
Surface Cov. ↑
Hidden Cov. ↑
Track4World partial
37.22%
28.4%
9.8%
Full-object flow
30.74%
45.8%
26.6%
TABLE III : A3. Full-object versus partial motion supervision. All results are medians. EPE is evaluated on identical GT material points; Coverage@ 5%D is measured over the complete GT surface, with unsuccessful cases counted as 0.
TABLE IV : Flow predictor evaluation. EPE is reported in centimeters, and Acc@ τ denotes per-point accuracy (%). Subscript T denotes the final frame. † im2Flow2Act reports lateral 2D-flow error.
Fig. 5 : Qualitative full-object flow predictions. Given an initial RGB observation and task instruction (top), our model predicts dense 3D trajectories over the complete target-object surface (bottom) across real-world and simulated tasks.
Caption
AgiBot
SthV2
Overall
Placeholder
16.39
17.48
13.76
Task-specific
14.86
15.27
13.01
Δ
−1.53
−2.21
−0.75
TABLE V : B4. Task-language ablation. Final-frame EPE.
Fig. 6 : B3. Data scaling.
Method
ManiSkill
LIBERO
PushCube
PlaceSphere
Insert
Stack-Arrange
TraceGen
5/10
4/10
7/10
7/10
ObjectForesight
6/10
7/10
7/10
6/10
Ours
10/10
10/10
9/10
10/10
TABLE VI : C1. Simulation execution results.
Method
Banana
Water
Flashlight
Cabbage
Dream2Flow
2/5
3/5
5/5
4/5
Ours
4/5
4/5
5/5
5/5
TABLE VII : C2. Real-world execution results.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Source
Samples
Avg. frames
Hours
Simulation
ManiSkill 3
8,421
111.9
16.4
MIKASA
4,599
91.0
7.3
Meta-World
52,500
76.5
69.7
MetaFold
2,014
101.0
3.5
RLBench
3,386
156.2
9.2
Appendix
TABLE VIII : Dataset composition. Average frame counts are measured before fixed-length sampling. Sth-V2 denotes Something-Something V2.
Source
Samples
Avg. frames
Hours
Golden
AgiBot World
99,360
158.0
272.5
Sth-V2
20,573
43.7
15.6
EPIC-KITCHENS
336
101.2
0.6
Subtotal
120,269
138.3
288.7
Silver (including Golden)
Appendix
TABLE IX : Real-world quality subsets. Golden is a subset of Silver. Average frame counts are measured before fixed-length sampling.
Fig. 7 : Visualization of real-world full-object flow annotations. Examples from AgiBot World [ 31 ] , EPIC-KITCHENS [ 22 ] , and Something-Something V2 [ 25 ] . Colored trajectories represent the estimated motion of points on the reconstructed complete object surface, with colors progressing from red at earlier timesteps to blue at later timesteps. Each example corresponds to a different manipulation clip processed by our automatic annotation pipeline.
All Frames
Final Frame
Configuration
EPE ↓
Acc@5 ↑
Acc@10 ↑
EPE ↓
Acc@5 ↑
Acc@10 ↑
Full model
6.84
57.5
76.5
12.69
19.6
52.0
3D only (w/o VLM)
8.61
49.6
67.6
16.90
9.2
31.5
Separate PTv3
7.24
56.6
74.4
13.46
17.0
45.9
Appendix
TABLE X : Component ablations. EPE is reported in centimeters, while Acc@5 and Acc@10 denote pooled per-point accuracies (%). All configurations use the same evaluation set.
Predicting object dynamics (i.e., world modeling) is a fundamental challenge for robotic manipulation, and modeling deformable objects presents a particularly difficult case due to their high-dimensional state spaces and complex material properties. While current world models approach this through two distinct paradigms: learning the dynamics over the 2D pixel space or more explicit 3D geometric space. A systematic understanding of their relative strengths and limitations remains elusive due to the lack of diverse, large-scale real-world data. To address this, we present Deform360, a large-scale visuotactile dataset featuring 198 daily-life objects, 1,980 interaction sequences, and over 215 hours of observations from 41 surround-view cameras and bimanual tactile grippers to capture both global motion and contact-induced local deformations. Leveraging a novel markerless visuotactile 3D tracking pipeline to extract dense geometry and motion, we systematically evaluate current state-of-the-art world models, comparing 2D video models against 3D particle models. Finally, we provide a preliminary demonstration indicating the real-world applicability of our dataset by performing robot planning tasks on deformable objects. Our analysis reveals key insights into the trade-offs between structural priors and scalability, providing a solid benchmark for future research in generalizable deformable object-centric world modeling. Project website: https://deform360.lhy.xyz
Hongyu Li, Wanjia Fu, Xiaoyan Cong +11
Brown University, Providence, RI · Columbia University, New York, NY · Massachusetts Institute of Technology, Cambridge, MA
Predicting future physical states stands as a central challenge in embodied intelligence. Existing video world models leave 3D dynamics implicit, action-based formulations remain tied to specific robot configurations, and 3D motion prediction models use flow-only supervision, providing a limited learning signal for physical dynamics. To circumvent these limitations, we introduce \method{}, a generative world model that jointly predicts future video and query-based 3D trajectories. By parameterizing 3D motion as continuous displacement tokens aligned with the video grid, a unified Mixture-of-Transformers (MoT) backbone jointly denoises both modalities under a rectified-flow objective. This formulation couples explicit geometric dynamics with visual evolution, effectively unlocking the physical commonsense embedded within pretrained video models. To empirically scale and validate this unified architecture, we train \method{} on a heterogeneous mixture of 1.44 million object-level samples spanning simulated and real-world domains. We further introduce WorldFlowBench, comprising 1,678 evaluation samples from 11 data sources across simulated and real-world interactions, including spatial reasoning tasks. Across all three evaluation sets, WorldFlow achieves the lowest average displacement error (ADE) and final displacement error (FDE) among the evaluated learning-based methods. Notably, on the challenging SpatialCog set, WorldFlow achieves an ADE of 0.060 m, a 54.2% reduction compared with the strongest external baseline. It also reduces ADE and FDE by 55.9% and 68.5%, respectively, compared with its flow-only variant. Project page: https://worldflow-rbs.github.io/.
Recent advances in 3D reconstruction enable the extraction of temporally consistent 3D motion from RGB observations; however, current Vision-Language-Action (VLA) models still predominantly rely on 2D image-text priors. Therefore, they do not accurately represent the fundamental dynamics of the 3D world associated with physical actions. This over-reliance on static 2D visual cues limits the capacity of VLA models in achieving robust robot manipulation. To address these limitations, we introduce FlowVLA, a reconstruction-guided, pluggable 3D flow learning framework aimed at empowering VLA models with an inherent understanding of action-relevant dynamics. Instead of requiring explicit 3D sensor inputs or altering the inference-time architecture, our method is designed to utilize temporally aligned 3D flow only during training. This allows the model to learn the dynamic evolution of the physical scene while preserving the computational efficiency of conventional 2D VLA backbones. To bridge scene-level dynamics with robotic manipulation, we propose a modular 3D flow learning framework with dual-level flow supervision. Specifically, point-level flow guides the model to forecast the future displacements of task-critical 3D points, while TCP-level flow aggregates these motions to align with the robot's TCP trajectories. This dual-level flow supervision encourages the model to focus on semantically meaningful dynamic regions, effectively coupling scene-level motion with precise control signals. Extensive evaluations across simulation and real-world experiments demonstrate that FlowVLA consistently improves manipulation performance, zero-shot robustness, and optimizes data efficiency. FlowVLA adds only lightweight auxiliary heads during training and discards them at inference. Therefore, FlowVLA introduces no additional 3D sensor inputs, architectural modifications, or inference-time computation.