From World Models to World Action Models: Rethinking Next-State Prediction

Tingyu Yuan; Ziming Ji; Biaoliang Guan; Wen Ye; Wenrui Tian; Zhaopeng Gu; Feihong Zhang; Xu Yang

cs.ROSep 28, 2026

From World Models to World Action Models: Rethinking Next-State Prediction

Authors: Tingyu Yuan, Ziming Ji, Biaoliang Guan, Wen Ye, Wenrui Tian, Zhaopeng Gu, Feihong Zhang, Xu Yang, +4 more

Organizations: CASIA · UCAS · BUPT · Yinwang Intelligent Technology Co. Ltd. · XJTU · WHU · THU

Abstract

Predicting the next state is a core paradigm of World Models for modeling physical dynamics, emphasizing prediction fidelity. As World Models evolve into World-Action Models (WAMs), existing methods still fix the next state before training as RGB, a single latent feature, or a static combination of predefined targets, thereby constraining action learning to the inductive biases preserved by a particular representation. To address this limitation, we propose CF-WAM, a dynamic next-state prediction framework that samples visual, semantic, geometric, and interaction projections of the same future, standardizes them into a common video form, and supervises a unified WAM across these projections. The action-relevant constraints exposed by these projections accumulate across training steps, forcing WAM to capture the underlying state-transition structure that supports multiple projections of the same action-conditioned future. This dynamic mechanism also provides a natural cross-embodiment dynamics reference frame for Human and Robot learning. By jointly learning across different next-state parameterizations, heterogeneous Human and Robot experience can bypass appearance differences and directly contribute to shared state-transition learning, improving cross-embodiment generalization. Experiments show that CF-WAM improves both training efficiency and final control performance, while translating Human experience effectively into policy gains. CF-WAM achieves state-of-the-art performance on RoboCasa-GR1 with an average success rate of 82.50%, while reaching 82.65% on LIBERO-Plus and up to 84.00% in real-world evaluations.

Figures & tables

Figure 1: Overview of CF-WAM. Human and Robot data supervise multiple future-state projections of a shared underlying manifold, enabling effective Human–Robot learning and strong performance.

Figure 2: Overview of CF-WAM. CF-WAM dynamically samples multiple future-state projections to jointly model world dynamics and actions from unpaired Human and Robot experience.

Table 4

Group	Method	Camera	Robot	Language	Light	Background	Noise	Layout	Avg.
VLA	GWM-VLA	57.90	54.70	89.80	95.40	90.80	72.50	77.10	76.90
VLA	FoMoVLA	64.00	62.20	94.00	94.10	96.20	82.20	79.60	80.50
WAM	FastWAM	16.26	44.13	66.82	79.77	52.60	38.54	61.38	51.36
WAM	GaussianWAM	79.11	56.52	92.18	89.84	66.17	86.26	70.55	77.30
Ours	CF-WAM	66.92	84.90	90.50	98.25	67.84	88.94	81.11	82.65

Table 3: OOD generalization results on LIBERO-Plus. We report success rates (%).

Figure 5: Representative Human and Human-like manipulation trajectories in real-world scenes.

Method	Chemistry	Folding	Organization	Fruit	Insertion	Wiping	Avg.
$\pi_{0.5}$	48.00	72.00	56.00	74.00	42.00	82.00	62.33
GR00T-N1.6	32.00	44.00	38.00	72.00	34.00	64.00	47.33
FastWAM	16.00	62.00	40.00	46.00	22.00	64.00	41.67
LingBot-VA	28.00	12.00	42.00	58.00	24.00	58.00	37.00
CF-WAM (Visual)	76.00	92.00	74.00	88.00	76.00	96.00	83.67
CF-WAM (Semantic)	64.00	78.00	76.00	90.00	62.00	88.00	76.33

Table 4: Real-world manipulation results. We report success rates (%) across six manipulation tasks.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Configuration	Video Expert	Action Expert
Backbone	Wan2.2-TI2V-5B DiT	ActionDiT
Transformer layers	30	30
Residual width	3072	1024
Attention width	3072	3072
Attention heads	$24\times 128$	$24\times 128$
Positional encoding	3D RoPE $(t,h,w)$	1D RoPE

Appendix

Table 5: Expert configurations in CF-WAM. The attention width denotes the feature dimension used for cross-expert mixed attention.

Stream	Input and processing	Tokens
Video input	5 frames, $224\times 224\times 3$	–
VAE latent	48 channels, $2\times 28\times 28$	–
Video tokens	Patchify $(1,2,2)$	392
Action chunk	16 steps, 47 dimensions per step	16
Joint self-attention	Video + action tokens	408
Language context	umT5-XXL, up to 128 tokens	128

Appendix

Table 6: Per-sample tokenization and sequence budget.

Item	Configuration
Optimizer	AdamW
Initial learning rate	$1\times 10^{-4}$
Learning-rate schedule	Cosine decay
Weight decay	$10^{-2}$
Numerical precision	bf16
Distributed optimization	ZeRO-1

Appendix

Table 7: Training and inference configuration of CF-WAM on RoboCasa-GR1.

Source	Data type	Physical episodes	Modality episodes	Representations	FPS	Action
EgoDex	Real egocentric human	23,849	95,396	RGB/Depth/Seg/DINOv3-PCA	20	18 valid, padded to 47
Teleop-GR1	Simulated robot teleoperation	24,000	96,000	RGB/Depth/Seg/DINOv3-PCA	20	47 dimensions
Total	–	47,849	191,396	8 modality subsets	20	Unified to 47

Appendix

Table 8: Composition of the core multimodal training mixture. The number of modality episodes counts the four future-state representations associated with each physical trajectory.

Task	ABot-M0	JoyAI-RA	FastWAM	WALA	CF-WAM
CupToDrawerClose	48.0	48.0	32.0	86.0	60.0
PotatoToMicrowaveClose	50.0	70.0	72.0	78.0	72.0
MilkToMicrowaveClose	46.0	84.0	68.0	78.0	72.0
BottleToCabinetClose	86.0	84.0	92.0	82.0	86.0
WineToCabinetClose	66.0	54.0	56.0	62.0	68.0
CanToDrawerClose	74.0	90.0	84.0	96.0	92.0

Appendix

Table 9: Full evaluation results on the RoboCasa GR1 TableTop benchmark. We report success rates (%) over 50 rollouts per task.

Figure 6: Additional closed-loop manipulation trajectories of CF-WAM on RoboCasa-GR1.

Figure 7: Visualization of action-to-video attention for CF-WAM and Static Multi-State across representative RoboCasa-GR1 manipulation scenes.

Setting	Human Action	Robot Action	Future State	Avg.
Joint-only	Joint	Joint	Visual	76.8
EE+Joint	EE + Joint	EE + Joint	Visual	75
Human–Robot	EE	EE + Joint	Visual	78.25
CF-WAM	EE	EE + Joint	Dynamic	82.50

Appendix

Table 10: Ablation of Human–Robot action-space designs on RoboCasa-GR1. Success rates (%) are reported across 24 tasks.

Setting	Inference	Human-like	Bg. Texture	Lighting	Obj. Inst.	Obj. Pos.	Avg.
CF-WAM	Visual	82.00	64.00	74.00	70.00	88.00	75.60
	Semantic	74.00	70.00	74.00	64.00	84.00	73.20
	Geometric	76.00	78.00	76.00	62.00	88.00	76.00
	Interaction	80.00	82.00	80.00	68.00	86.00	79.20
w/o Human	Visual	38.00	42.00	50.00	36.00	62.00	45.60

Appendix

Table 11: Real-world OOD analysis under different inference projections and Human experience. We report success rates (%) under five OOD settings and their average.

Figure 8: Additional real-world manipulation trajectories of CF-WAM under in-domain, Human-like, and OOD settings.

Explore similar work

World Action Models: The Next Frontier in Embodied AI
May 12, 2026Siyin Wang, Junhao Shi, Zhaoyang Fu +11Recent World-Action Models World Models
1Fudan University · 2Shanghai Innovation Institute · 3National University of Singapore
DreamWAM: Beyond RGB Future Prediction for World Action Models
Aug 5, 2026Shanglin Yuan, Weiheng Zhao, Xin Shi +6Efficient World-Action Model World Models
Huazhong University of Science and Technology · D-Robotics · Wuhan University +1
Foresight Without Seeing: Latent Futures for World Action Models
Aug 12, 2026Jiakai Huang, Zhongbo Wu, Siyu Xu +5Action Prediction Foresight
Shanghai Jiao Tong University · ACE Robotics · Nanyang Technological University

May 12, 2026, cs.ROJ/K move · Enter open · S save

World Action Models: The Next Frontier in Embodied AI

Siyin Wang, Junhao Shi, Zhaoyang Fu, Xinzhe He+10

1Fudan University · 2Shanghai Innovation Institute · 3National University of Singapore

Recent World-Action Models World Models Embodied Artificial Intelligence Action Generation Next-State Prediction Egocentric Video+1

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models have achieved strong semantic generalization for embodied policy learning, yet they learn reactive observation-to-action mappings without explicitly modeling how the physical world evolves under intervention. A growing body of work addresses this limitation by integrating world models, predictive models of environment dynamics, into the action generation pipeline. We term this emerging paradigm World Action Models (WAMs): embodied foundation models that unify predictive state modeling with action generation, targeting a joint distribution over future states and actions rather than actions alone. However, the literature remains fragmented across architectures, learning objectives, and application scenarios, lacking a unified conceptual framework. We formally define WAMs and disambiguate them from related concepts, and trace the foundations and early integration of VLA and world model research that gave rise to this paradigm. We organize existing methods into a structured taxonomy of Cascaded and Joint WAMs, with further subdivision by generation modality, conditioning mechanism, and action decoding strategy. We systematically analyze the data ecosystem fueling WAMs development, spanning robot teleoperation, portable human demonstrations, simulation, and internet-scale egocentric video, and synthesize emerging evaluation protocols organized around visual fidelity, physical commonsense, and action plausibility. Overall, this survey provides the first systematic account of the WAMs landscape, clarifies key architectural paradigms and their trade-offs, and identifies open challenges and future opportunities for this rapidly evolving field.

Abstract

Explore similar work

World Action Models: The Next Frontier in Embodied AI

DreamWAM: Beyond RGB Future Prediction for World Action Models

Foresight Without Seeing: Latent Futures for World Action Models