World action models jointly predict video and actions conditioned on language. Existing studies focus primarily on architectures and training mechanisms, typically treating language-video alignment acquired through video pretraining as an established foundation. However, introducing action learning may impair this alignment. Principled guidance on data and supervision design for maintaining language-video alignment during action learning remains limited. To investigate this, we curate a pool of approximately 47,000 hours of general videos, egocentric videos, UMI demonstrations, and robot demonstrations. Building on this, we systematically study how language-video supervision, language-video-action supervision, and the joint use of these two forms affect WAM performance. In particular, we examine how language annotation quality affects pretrained language-video capabilities, and how action supervision from UMI and robot demonstrations extends these capabilities to language-video-action modeling. To better scale the model, we further conduct detailed studies on joint training recipe and model architecture. Under a fixed training budget, mixing egocentric videos and UMI demonstrations improves control generalization. Model architecture affects how effectively these experiences are used, while scaling pretraining under the selected configuration brings further gains. The resulting recipe, post-trained on RoboTwin Clean demonstrations only, achieves success rates of 88.10% and 61.56% under Easy (Clean-to-Clean) and Hard (Clean-to-Random) conditions, respectively. Our model achieves 40.00% SR across three real-world tasks, compared with 23.33% for the strongest baseline, indicating better generalization under unseen conditions. We release the curated data, training code, and model weights.
Figures & tables
Figure 2: Multisource data and model overview. (a) Composition of the 47k-hour pretraining data pool. (b) Our world action model architecture, with video and action branches connected through shared masked attention.
Table 2
Figure 3: World action model architecture and attention masks. (a) The video and action DiT branches are connected through shared masked attention, with text cross-attention in the video branch only. Attention masks for (b) training with clean video history under teacher forcing and (c) generating video and actions chunk by chunk at inference.
Figure 4: Multisource training data and curation pipeline. The upper panel shows representative examples from four data sources. The lower panel illustrates visual checks, action checks, format conversion, subtask segmentation, and caption generation. Captions include action descriptions at three levels of detail, along with scene and lighting information.
Detail level
Annotation requirements
Example language annotation
Brief
Summarize the main action and core object with a verb–object phrase.
Pick up the snack.
Medium
Summarize the manipulation in one sentence, adding object attributes, spatial relations, or visible outcomes.
Pick up the snack in the light-blue package.
Detailed
Describe the manipulation chronologically, including discernible hand actions, object interactions, and necessary adjustments.
The right hand first touches and grasps the snack in the light-blue package, then lifts it from the table, while the left hand does not participate.
Table 3: Language annotation requirements and example descriptions at three levels of detail. All examples describe the same manipulation segment.
Table 6
Data
Steps
Easy (C2C)
Hard (C2R)
General videos
UMI/Ego 20k
20k
82.64
53.84
+ General 25k
45k
82.64
56.48
Egocentric videos
UMI only
10k
83.04
49.68
+ Ego 10k
20k
82.64
53.84
Table 7: Different data sources strengthen language–video and language–video–action alignment. We separately increase pretraining exposure to general videos ( General ), egocentric videos ( Ego ), and UMI data ( UMI ).
Ratio
Easy (C2C)
Hard (C2R)
1:0
77.04
42.56
2:1
82.72
52.64
1:1
82.64
53.84
1:2
83.20
53.28
0:1
83.28
51.28
Table 8: Ego and UMI provide complementary supervision. We compare five Ego-to-UMI sampling ratios under a fixed 20k-step pretraining budget.
Easy (C2C)
Hard (C2R)
Architecture
Single-stream
86.48
52.56
Dual-stream
84.96
52.48
Ours
84.80
57.36
Extra features
Native
83.44
49.52
Table 9: Model architecture affects control generalization. We compare video–action architectures (top) and extra image features (bottom).
Figure 5: Scaling pretraining improves control generalization. We vary training video hours on a fixed corpus of multisource experience, using the same downstream post-training protocol within each domain. (a) RoboTwin SR under the Easy (C2C) and Hard (C2R) settings. (b) Real-world SR and PSR under the Hard setting, averaged across three tasks. The 0-hour setting uses the video-model initialization without additional pretraining and achieves no success in real-world evaluation.
Stage 1
Stage 2
Easy (C2C)
Background
Lighting
Clutter
Height
Hard (C2R)
0k
0k
76.16
59.12
76.24
51.28
57.68
34.56
20k
0k
82.64
72.16
84.08
68.56
70.32
53.84
75k
0k
85.68
77.92
85.60
78.48
75.44
59.60
75k
25k
88.08
82.16
88.08
80.72
76.96
61.52
Table 10: Scaling pretraining on human experience and robot data improves generalization across unseen environmental conditions. We first pretrain on egocentric videos and UMI demonstrations ( Stage 1 ), then continue training on robot data ( Stage 2 ). The middle four columns evaluate single-factor environmental changes.
Model
RoboTwin
Real-world
Easy
Hard
Avg.
Easy SR
Easy PSR
Hard SR
Hard PSR
VLA
StarVLA [ 10 ]
60.03
14.06
37.05
31.11
36.87
4.44
21.55
π0.5 [ 18 ]
70.66
46.00
58.33
50.00
71.87
20.00
37.78
Xiaomi Robotics-0 [ 5 ]
62.86
18.22
40.54
–
–
–
–
X-VLA [ 51 ]
67.96
20.95
44.46
–
–
–
–
Table 11: Our model outperforms VLA and WAM baselines in simulation and real-world evaluation. We compare Easy (C2C) and Hard (C2R) settings, reporting SR in simulation and SR/PSR on real-world tasks.
Figure 6: Our method achieves the best performance under the Hard setting in real-world evaluation. Results are averaged equally across three real-world tasks.
Figure 7: Real-world execution comparisons. Our model completes cup stacking and towel folding under the Easy setting and gift-in-box under the Hard setting, while the corresponding baseline executions fail. Light red dashed boxes highlight the baselines’ main execution errors.
Figure 8: Generalization examples from head and external camera views. Fruit placement and rainbow stack demonstrate generalization to new objects, containers, and scene appearances.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Configuration
Value
Optimizer
AdamW
Learning rate
4×10−5
Adam (β1,β2)
(0.95,0.999)
Adam ϵ
10−8
Weight decay
10−5
Gradient clip
1.0
Appendix
Table 12: Shared optimization settings. Values shared across pretraining and post-training.
Configuration
Pretraining
Post-training
Scaling study
Simulation
Real world
Global batch size
512
128
128
Training steps
100,000
20,000
50,000
Learning-rate schedule
Constant
Cosine
Cosine
Appendix
Table 13: Comparison of training configurations.
Task
Episodes
Hours
Cup stacking
2,748
25.788
Towel folding
2,852
15.112
Gift-in-box
4,399
21.715
Appendix
Table 14: Task-specific post-training data for real-world evaluation.
Figure 9: Real-world evaluation layouts under Easy and Hard settings. All models use the same ten layouts for each task and setting, with three trials per layout.
Figure 10: Subtask illustrations for the three real-world tasks.
Task
Easy
Hard
Task
Easy
Hard
Adjust Bottle
97
81
Place Can Basket
84
36
Beat Block Hammer
97
55
Place Cans Plasticbox
99
74
Blocks Ranking RGB
98
80
Place Container Plate
96
87
Blocks Ranking Size
87
30
Place Dual Shoes
56
33
Click Alarmclock
100
75
Place Empty Cup
100
77
Click Bell
100
85
Place Fan
87
48
Appendix
Table 15: Per-task success rates on RoboTwin 2.0 under Easy and Hard settings. Each task is evaluated over 100 episodes per setting. All values are percentages.
Figure 11: Real-world comparison of our model and baseline methods.
Figure 12: Real-world performance with increasing pretraining video hours.
World Action Models (WAMs) generate both future video and robot actions, offering two connected outputs for post-training supervision. How can a pretrained WAM learn from a stronger Teacher on the histories it encounters during execution? We present WAM-OPD, which collects Student rollout histories and queries a Teacher for paired video and action targets. The Student learns from both targets while retaining its one-step video and action generation at deployment. Across 12 RoboTwin 2.0 tasks, WAM-OPD improves average success from 33.8% to 65.7%; across four real-robot tasks, it improves average success from 51.4% to 64.6%. With the collected Student histories held fixed, joint video-action supervision achieves the highest observed success on all three ablation tasks, while either modality alone also improves performance. A separate comparison with Teacher-generated histories finds task-dependent differences between the two history sources. These results demonstrate the value of paired video-action supervision for improving WAM policies without increasing their deployed sampling budget.
Vision-language-action models benefit from the understanding and reasoning capabilities of pretrained vision-language models, but action-only supervision provides limited grounding in world dynamics. Conversely, world-action models inherit spatiotemporal priors from video generation models, yet remain limited in semantic understanding and reasoning under distribution shifts. We introduce UniWAM, a unified architecture that integrates a physical reasoner, a world generator, and an action predictor to jointly learn semantic understanding of the physical world, visual generation, and action prediction. To ensure the quality of the training data, we developed a rigorous data cleaning and annotation pipeline for both human egocentric data and robot data. To adapt the vision-language component to embodied tasks while preserving its inherited language capabilities, we represent low-level actions in natural language and introduce a pre-training recipe that assigns complementary supervision from visual question answering (VQA) data, human egocentric data, and robot demonstrations to the appropriate model components. During post-training, future visual noise augmentation reduces reliance on precise future predictions, while history-conditioned flow matching uses encoded action history to initialize action generation. Together, these designs significantly reduce denoising steps while maintaining performance. UniWAM achieves state-of-the-art (SOTA) performance across multiple evaluations, including in-distribution performance, robustness, generalization, instruction following, and long-horizon task execution. Furthermore, we uncover a log-linear scaling law of unified human-robot co-training, demonstrating the effectiveness of large-scale pre-training on a mixture of human and robot data.
WALL-WM is a World Action Model that shifts video-action learning from chunk-centric optimization to event-grounded Vision-Language-Action pretraining, using semantically coherent action events as the atomic unit of learning. Existing WAMs commonly initialize from multimodal or video foundation models and then optimize fixed-length action chunks conditioned directly on the current observation and instruction. Although convenient, this chunk-centric formulation creates a fundamental granularity mismatch. Language describes semantic goals and events, vision evolves through continuous scene dynamics, and actions operate at control-level timescales; forcing all three into the same fixed-length prediction window turns VLA training into short-horizon correlation fitting. WALL-WM addresses this mismatch by organizing both supervision and data around semantic events. Specifically, it pairs event-grounded VLA pretraining with a data ecosystem built from event-level captions and cluster-balanced sampling, enabling scalable learning over diverse behaviors, scenes, and task structures. From the same event-pretrained backbone, WALL-WM supports two complementary inference modes. The event mode consumes next-event descriptions and enables variable-length execution chunks, while the unified mode uses a VLM with Staircase Decoding to condition conventional fixed-length chunk inference while preserving a gradient-continuous VLA path. Together with Muon-optimizer-based large-scale pretraining infrastructure, WALL-WM provides a practical scale-up recipe for general-purpose WAMs. Experiments show that WALL-WM generalizes broadly across language, scenes, and tasks, achieving state-of-the-art performance in large-scale real-world generalization evaluation.