World action models jointly predict video and actions conditioned on language. Existing studies focus primarily on architectures and training mechanisms, typically treating language-video alignment acquired through video pretraining as an established foundation. However, introducing action learning may impair this alignment. Principled guidance on data and supervision design for maintaining language-video alignment during action learning remains limited. To investigate this, we curate a pool of approximately 47,000 hours of general videos, egocentric videos, UMI demonstrations, and robot demonstrations. Building on this, we systematically study how language-video supervision, language-video-action supervision, and the joint use of these two forms affect WAM performance. In particular, we examine how language annotation quality affects pretrained language-video capabilities, and how action supervision from UMI and robot demonstrations extends these capabilities to language-video-action modeling. To better scale the model, we further conduct detailed studies on joint training recipe and model architecture. Under a fixed training budget, mixing egocentric videos and UMI demonstrations improves control generalization. Model architecture affects how effectively these experiences are used, while scaling pretraining under the selected configuration brings further gains. The resulting recipe, post-trained on RoboTwin Clean demonstrations only, achieves success rates of 88.10% and 61.56% under Easy (Clean-to-Clean) and Hard (Clean-to-Random) conditions, respectively. Our model achieves 40.00% SR across three real-world tasks, compared with 23.33% for the strongest baseline, indicating better generalization under unseen conditions. We release the curated data, training code, and model weights.
Figures & tables
Figure 2: Multisource data and model overview. (a) Composition of the 47k-hour pretraining data pool. (b) Our world action model architecture, with video and action branches connected through shared masked attention.
Table 2
Figure 3: World action model architecture and attention masks. (a) The video and action DiT branches are connected through shared masked attention, with text cross-attention in the video branch only. Attention masks for (b) training with clean video history under teacher forcing and (c) generating video and actions chunk by chunk at inference.
Figure 4: Multisource training data and curation pipeline. The upper panel shows representative examples from four data sources. The lower panel illustrates visual checks, action checks, format conversion, subtask segmentation, and caption generation. Captions include action descriptions at three levels of detail, along with scene and lighting information.
Detail level
Annotation requirements
Example language annotation
Brief
Summarize the main action and core object with a verb–object phrase.
Pick up the snack.
Medium
Summarize the manipulation in one sentence, adding object attributes, spatial relations, or visible outcomes.
Pick up the snack in the light-blue package.
Detailed
Describe the manipulation chronologically, including discernible hand actions, object interactions, and necessary adjustments.
The right hand first touches and grasps the snack in the light-blue package, then lifts it from the table, while the left hand does not participate.
Table 3: Language annotation requirements and example descriptions at three levels of detail. All examples describe the same manipulation segment.
Table 6
Data
Steps
Easy (C2C)
Hard (C2R)
General videos
UMI/Ego 20k
20k
82.64
53.84
+ General 25k
45k
82.64
56.48
Egocentric videos
UMI only
10k
83.04
49.68
+ Ego 10k
20k
82.64
53.84
Table 7: Different data sources strengthen language–video and language–video–action alignment. We separately increase pretraining exposure to general videos ( General ), egocentric videos ( Ego ), and UMI data ( UMI ).
Ratio
Easy (C2C)
Hard (C2R)
1:0
77.04
42.56
2:1
82.72
52.64
1:1
82.64
53.84
1:2
83.20
53.28
0:1
83.28
51.28
Table 8: Ego and UMI provide complementary supervision. We compare five Ego-to-UMI sampling ratios under a fixed 20k-step pretraining budget.
Easy (C2C)
Hard (C2R)
Architecture
Single-stream
86.48
52.56
Dual-stream
84.96
52.48
Ours
84.80
57.36
Extra features
Native
83.44
49.52
Table 9: Model architecture affects control generalization. We compare video–action architectures (top) and extra image features (bottom).
Figure 5: Scaling pretraining improves control generalization. We vary training video hours on a fixed corpus of multisource experience, using the same downstream post-training protocol within each domain. (a) RoboTwin SR under the Easy (C2C) and Hard (C2R) settings. (b) Real-world SR and PSR under the Hard setting, averaged across three tasks. The 0-hour setting uses the video-model initialization without additional pretraining and achieves no success in real-world evaluation.
Stage 1
Stage 2
Easy (C2C)
Background
Lighting
Clutter
Height
Hard (C2R)
0k
0k
76.16
59.12
76.24
51.28
57.68
34.56
20k
0k
82.64
72.16
84.08
68.56
70.32
53.84
75k
0k
85.68
77.92
85.60
78.48
75.44
59.60
75k
25k
88.08
82.16
88.08
80.72
76.96
61.52
Table 10: Scaling pretraining on human experience and robot data improves generalization across unseen environmental conditions. We first pretrain on egocentric videos and UMI demonstrations ( Stage 1 ), then continue training on robot data ( Stage 2 ). The middle four columns evaluate single-factor environmental changes.
Model
RoboTwin
Real-world
Easy
Hard
Avg.
Easy SR
Easy PSR
Hard SR
Hard PSR
VLA
StarVLA [ 10 ]
60.03
14.06
37.05
31.11
36.87
4.44
21.55
π0.5 [ 18 ]
70.66
46.00
58.33
50.00
71.87
20.00
37.78
Xiaomi Robotics-0 [ 5 ]
62.86
18.22
40.54
–
–
–
–
X-VLA [ 51 ]
67.96
20.95
44.46
–
–
–
–
Table 11: Our model outperforms VLA and WAM baselines in simulation and real-world evaluation. We compare Easy (C2C) and Hard (C2R) settings, reporting SR in simulation and SR/PSR on real-world tasks.
Figure 6: Our method achieves the best performance under the Hard setting in real-world evaluation. Results are averaged equally across three real-world tasks.
Figure 7: Real-world execution comparisons. Our model completes cup stacking and towel folding under the Easy setting and gift-in-box under the Hard setting, while the corresponding baseline executions fail. Light red dashed boxes highlight the baselines’ main execution errors.
Figure 8: Generalization examples from head and external camera views. Fruit placement and rainbow stack demonstrate generalization to new objects, containers, and scene appearances.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Configuration
Value
Optimizer
AdamW
Learning rate
4×10−5
Adam (β1,β2)
(0.95,0.999)
Adam ϵ
10−8
Weight decay
10−5
Gradient clip
1.0
Appendix
Table 12: Shared optimization settings. Values shared across pretraining and post-training.
Configuration
Pretraining
Post-training
Scaling study
Simulation
Real world
Global batch size
512
128
128
Training steps
100,000
20,000
50,000
Learning-rate schedule
Constant
Cosine
Cosine
Appendix
Table 13: Comparison of training configurations.
Task
Episodes
Hours
Cup stacking
2,748
25.788
Towel folding
2,852
15.112
Gift-in-box
4,399
21.715
Appendix
Table 14: Task-specific post-training data for real-world evaluation.
Figure 9: Real-world evaluation layouts under Easy and Hard settings. All models use the same ten layouts for each task and setting, with three trials per layout.
Figure 10: Subtask illustrations for the three real-world tasks.
Task
Easy
Hard
Task
Easy
Hard
Adjust Bottle
97
81
Place Can Basket
84
36
Beat Block Hammer
97
55
Place Cans Plasticbox
99
74
Blocks Ranking RGB
98
80
Place Container Plate
96
87
Blocks Ranking Size
87
30
Place Dual Shoes
56
33
Click Alarmclock
100
75
Place Empty Cup
100
77
Click Bell
100
85
Place Fan
87
48
Appendix
Table 15: Per-task success rates on RoboTwin 2.0 under Easy and Hard settings. Each task is evaluated over 100 episodes per setting. All values are percentages.
Figure 11: Real-world comparison of our model and baseline methods.
Figure 12: Real-world performance with increasing pretraining video hours.