PrismWAM: Decoupling State Representations in World Action Models
Organizations: Tsinghua University · NVIDIA
Abstract
World Action Models (WAMs) typically represent the world through latent embeddings from pretrained video models, which entangle heterogeneous aspects of the world, from how it looks (e.g., texture and color) to how it is physically configured (e.g., occupancy and pose). This raises an underexplored question: We argue that the world state should separate how the world appears from how the world evolves under action: the former captures semantic context, while the latter is governed by physical state. To this end, we propose , which decouples physics from semantics while jointly modeling state and action. PrismWAM factorizes an embodied task into a pipeline: given only the initial RGB observation, semantic planning learns to predict the final goal image, while physical rollout predicts the fine-grained evolution toward this goal in depth space; executable actions are then generated from the depth rollout. Without any embodied action pretraining or additional data augmentation, PrismWAM achieves strong manipulation performance in both simulation and the real world. Notably, PrismWAM even surpasses a broad range of models with large-scale embodied action pretraining (e.g., OpenWAM-). More broadly, we reframe World Action Models from modeling how the world looks to how it responds to action, paving the way for action-centric world models grounded in occupancy, flow, tactile sensing, proprioception, and beyond.
Figures & tables
| RoboTwin-C2R | RoboDojo | ||||||||
| Method | Clean | Rand. | Avg. | Gen. | Precision | Long-H. | Memory | Open | Overall |
| w/ Embodied Action Pretraining | |||||||||
| 70.70 | 46.00 | 58.35 | 13.38/8.17 | 12.40/5.50 | 23.54/14.67 | 5.89/4.67 | 1.98 / 1.67 | 11.44/6.93 | |
| OpenWAM- | 89.40 | 48.70 | 69.05 | 20.71/14.83 | 18.45/9.25 | 34.93/ 25.33 | 10.41/9.11 | 1.41/1.08 | 17.18/11.92 |
| w/o Embodied Action Pretraining | |||||||||
| starVLA | 46.52 | 3.16 | 24.84 | 3.94/2.34 | 9.90/4.33 | 14.15/6.50 | 3.34/2.44 | 0.67/0.58 | 6.40/3.24 |
| OpenWAM | PrismWAM | |||
|---|---|---|---|---|
| Task | ID | OOD | ID | OOD |
| Pick Up Plate | 4/5 (80.0) | 10/15 (66.7) | 5/5 (100.0) | 13/15 (86.7) |
| Ranking Block Size | 7/12 (58.3) | 3/10 (30.0) | 11/12 (91.7) | 6/10 (60.0) |
| Open Microwave | 9/10 (90.0) | 4/10 (40.0) | 8/10 (80.0) | 5/10 (50.0) |
| Pooled (3 tasks) | 20/27 (74.1) | 17/35 (48.6) | 24/27 (88.9) | 24/35 (68.6) |
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| PrismWAM | PrismWAM ∗ | |||
|---|---|---|---|---|
| Task | Clean | Rand. | Clean | Rand. |
| Adjust Bottle | 98 | 54 | 100 | 97 |
| Beat Block Hammer | 81 | 48 | 98 | 58 |
| Blocks Ranking RGB | 87 | 32 | 93 | 72 |
| Blocks Ranking Size | 56 | 27 | 70 | 55 |
| Click Alarmclock | 100 | 86 | 100 | 98 |
| PrismWAM | PrismWAM ∗ | |||
|---|---|---|---|---|
| Task | Score | SR | Score | SR |
| Generalization – standard | ||||
| stack bowls | 80.53 | 77.33 | 88.67 | 86.67 |
| pour liquid into cup | 54.67 | 54.67 | 60.00 | 60.00 |
| push T | 0.00 | 0.00 | 1.33 | 1.33 |
| make toast | 40.67 | 22.67 | 68.67 | 60.00 |
World Models for Robotics
World models for robotics predict how scenes and robot states change under actions to support planning, control, and learning.
See all 384 papers