Generative world modeling from large-scale visual data offers a promising path toward scalable robot manipulation. This calls for an intermediate representation that can be supervised at scale and remains actionable for downstream control. We propose 3D full-object flow, an object-centric motion representation that assigns dense 3D trajectories to points across the complete surface of a target object, including its occluded regions. By extending visible-surface object flow to the full object geometry, this representation makes complete object motion explicit and applies uniformly to both rigid and deformable objects. To realize the potential of this representation at scale, we introduce a 1210-hour full-object flow dataset that addresses the scarcity of large-scale supervision for complete 3D object motion. The dataset integrates diverse real-world videos and physical simulations, with a scalable annotation pipeline for recovering full-object motion from real-world observations and quality metrics for systematically assessing annotation quality. We then train a world model that takes an image and a task description as input and predicts dense full-object flow over the reconstructed object geometry. Experiments show reliable annotation quality, data-scalable flow prediction, and effective downstream manipulation across novel scenes and robot embodiments. Project website with additional results and videos: https://nus-lins-lab.github.io/ObjectWorld/.
Figures & tables
Fig. 2 : Automatic data annotation pipeline. Given an in-the-wild manipulation video, a VLM identifies the target object, describes the task, and segments the video into atomic action clips. For each clip, our pipeline extracts the target-object masks, reconstructs the complete object mesh and the point clouds, and aligns the mesh with the point clouds across frames to recover the object motion. The motion is propagated over the complete object surface to produce 3D full-object flow.
Fig. 3 : Object-centric full-object flow prediction framework. Given an initial RGB-D observation and task instruction, Qwen2.5-VL [ 39 ] extracts task-conditioned visual-language features, while a joint PointTransformer V3 [ 40 ] (PTv3) encodes reconstructed object and scene geometry. These features are fused into object tokens and refined by self-attention. The object queries attend to scene and language context through a cross-attention decoder, conditioned on noisy flow and time t , before a flow head predicts dense trajectories over the complete object surface.
ID
Study
Supports
A1
Pipeline validation against simulator ground truth
Q1
A2
Filter-gate validation against expert labels
Q1
A3
Full-object vs. partial-object flow annotation
Q1
B1
Flow predictor comparison with baselines
Q2
B2
Robustness of flow predictor
Q2
B3
Data scaling experiment
Q3
TABLE I : Experiment map. A: annotation validation; B: flow prediction and scaling; C: robot execution and generalization.
Module / Stage
Evaluated Output
Protocol
Metric
Median
Time (s)
Gemini 2.5 Pro (VLM)
Localization
Against GT box
Box IoU ↑
0.908
12.04 †
SeC
Mask propagation
Against GT masks
Mask IoU ↑
0.873
11.20
Track4World (DA3)
Relative depth
Median-scale aligned
AbsRel ↓ / δ<1.25↑
0.089 / 0.826
8.25
SAM 3D Objects
Complete geometry
Shape aligned
CD/ D↓
2.61%
7.41
Hidden geometry
Visible-only Sim(3)
Vis./hid. surface error/ D↓
4.22/7.28%
Hidden coverage
Visible-only Sim(3)
Recall@ 5%D↑
0.462
TABLE II : A1. Ground-truth validation and runtime of the annotation pipeline on simulation. Quality metrics are medians over 97 successfully annotated RoboTwin clips. D denotes the ground-truth object diameter. Times are measured on standardized 3-second clips. † Amortized time per call; ‡ CPU flow construction.
Fig. 4 : A2. Expert validation of the filtering gates. Confusion matrices for the Golden and Silver gates on 200 expert-labeled real-world clips.
Representation
EPE/ D↓
Surface Cov. ↑
Hidden Cov. ↑
Track4World partial
37.22%
28.4%
9.8%
Full-object flow
30.74%
45.8%
26.6%
TABLE III : A3. Full-object versus partial motion supervision. All results are medians. EPE is evaluated on identical GT material points; Coverage@ 5%D is measured over the complete GT surface, with unsuccessful cases counted as 0.
TABLE IV : Flow predictor evaluation. EPE is reported in centimeters, and Acc@ τ denotes per-point accuracy (%). Subscript T denotes the final frame. † im2Flow2Act reports lateral 2D-flow error.
Fig. 5 : Qualitative full-object flow predictions. Given an initial RGB observation and task instruction (top), our model predicts dense 3D trajectories over the complete target-object surface (bottom) across real-world and simulated tasks.
Caption
AgiBot
SthV2
Overall
Placeholder
16.39
17.48
13.76
Task-specific
14.86
15.27
13.01
Δ
−1.53
−2.21
−0.75
TABLE V : B4. Task-language ablation. Final-frame EPE.
Fig. 6 : B3. Data scaling.
Method
ManiSkill
LIBERO
PushCube
PlaceSphere
Insert
Stack-Arrange
TraceGen
5/10
4/10
7/10
7/10
ObjectForesight
6/10
7/10
7/10
6/10
Ours
10/10
10/10
9/10
10/10
TABLE VI : C1. Simulation execution results.
Method
Banana
Water
Flashlight
Cabbage
Dream2Flow
2/5
3/5
5/5
4/5
Ours
4/5
4/5
5/5
5/5
TABLE VII : C2. Real-world execution results.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Source
Samples
Avg. frames
Hours
Simulation
ManiSkill 3
8,421
111.9
16.4
MIKASA
4,599
91.0
7.3
Meta-World
52,500
76.5
69.7
MetaFold
2,014
101.0
3.5
RLBench
3,386
156.2
9.2
Appendix
TABLE VIII : Dataset composition. Average frame counts are measured before fixed-length sampling. Sth-V2 denotes Something-Something V2.
Source
Samples
Avg. frames
Hours
Golden
AgiBot World
99,360
158.0
272.5
Sth-V2
20,573
43.7
15.6
EPIC-KITCHENS
336
101.2
0.6
Subtotal
120,269
138.3
288.7
Silver (including Golden)
Appendix
TABLE IX : Real-world quality subsets. Golden is a subset of Silver. Average frame counts are measured before fixed-length sampling.
Fig. 7 : Visualization of real-world full-object flow annotations. Examples from AgiBot World [ 31 ] , EPIC-KITCHENS [ 22 ] , and Something-Something V2 [ 25 ] . Colored trajectories represent the estimated motion of points on the reconstructed complete object surface, with colors progressing from red at earlier timesteps to blue at later timesteps. Each example corresponds to a different manipulation clip processed by our automatic annotation pipeline.
All Frames
Final Frame
Configuration
EPE ↓
Acc@5 ↑
Acc@10 ↑
EPE ↓
Acc@5 ↑
Acc@10 ↑
Full model
6.84
57.5
76.5
12.69
19.6
52.0
3D only (w/o VLM)
8.61
49.6
67.6
16.90
9.2
31.5
Separate PTv3
7.24
56.6
74.4
13.46
17.0
45.9
Appendix
TABLE X : Component ablations. EPE is reported in centimeters, while Acc@5 and Acc@10 denote pooled per-point accuracies (%). All configurations use the same evaluation set.