World-Action Models for Robot Learning and Control: A Survey

Zuxing Lu; Hongjia Zhai; Guanzhi Wang; Huajian Zeng; Jiaqi Yang; Jingyu Liu; Lei Cheng; Yuantai Zhang

cs.ROSep 13, 2026

World-Action Models for Robot Learning and Control: A Survey

Authors: Zuxing Lu, Hongjia Zhai, Guanzhi Wang, Huajian Zeng, Jiaqi Yang, Jingyu Liu, Lei Cheng, Yuantai Zhang, +8 more

Organizations: MBZUAI · Caltech · Amazon FAR · University of Virginia · Georgia Tech · New York University · UC Berkeley

Abstract

Robots operating in open environments act under partial observability, physical constraints, and dynamic task contexts. Beyond mapping observations and language instructions to actions, they must anticipate how candidate actions may affect future states and task-relevant outcomes. Recent advances in world models, video generation, and Vision-Language-Action (VLA) policies have motivated the development of World-Action Models (WAMs), which couple future world prediction with executable action generation. This survey provides a robotics-oriented review of WAMs. We clarify their scope relative to conventional world models, model-based reinforcement learning, action-conditioned video generation, and reactive VLA policies, and organize existing methods through a unified taxonomy covering representations, transition modeling, action interfaces, architectures, training pipelines, data modalities, and scaling strategies. We further review applications of WAMs in manipulation, navigation, and autonomous driving, and we summarize the datasets, benchmarks, metrics, and protocols used to evaluate WAM systems. Finally, we discuss key challenges in action alignment, world-action factorization, spatial and multi-view consistency, long-horizon memory, neural simulation for closed-loop policy learning, and efficient inference. Taken together, this survey aims to provide a concise technical foundation for integrating predictive world modeling with action generation, toward more reliable embodied robot intelligence. Project page: https://rcl-robotics.github.io/Awesome-World-Action-Models.

Figures & tables

Fig. 2: Conceptual overview of four foundational threads behind WAMs: (a) world models learn latent dynamics by predicting the next state from current state and action; (b) video generative models predict future visual observations conditioned on visual history and language instruction; (c) latent action pretraining infers latent actions from consecutive frames and uses them to model latent transitions; and (d) VLAs generate future action sequences directly from state and language instructions. (The third-row illustrations are adapted from the corresponding papers cited in each thread.)

Fig. 3: Component-level taxonomy for World-Action Models. The architecture can be viewed through four functional modules: a language interface that provides task conditioning, a visual encoder that defines the predictive state, a predictive backbone that carries dynamics, and an action decoder that converts predicted futures into executable controls.

Fig. 4: Transition Modeling Paradigms. Joint prediction couples future observation and action generation within a shared process, whereas the inverse dynamics paradigm factorizes this transition by first imagining future states and then inferring executable actions.

Fig. 5: Taxonomy of WAM Architectures. Existing models can be categorized along two orthogonal axes: architectural design and transition modeling paradigm. Structurally, end-to-end models (a, b) process all modalities within a unified backbone, whereas dual systems (c, d) explicitly separate visual and action experts. In terms of transition modeling, joint prediction approaches (a, c) generate future observations and actions simultaneously, while inverse dynamics approaches (b, d) follow a factorized, predict-then-act sequential generation.

Fig. 6: Policy training pipeline for WAMs. (a) Pre-training leverages internet-scale video data to learn spatiotemporal and action representations and align video with actions; (b) post-training adapts the pretrained model through policy fine-tuning, data augmentation, and reinforcement learning. Images adapted from [ fast , lapa , dreamzero ] .

Fig. 7: Data modality and scaling structure for World-Action Models. The left panel illustrates how different sensing modalities expose complementary action-relevant state variables, while the right panel shows a data-pyramid view in which large action-free video corpora provide broad world priors and smaller embodied trajectories provide action grounding.

Method	Paradigm	Backbone	Eval / Note	Method	Paradigm	Backbone	Eval / Note
Manipulation
GR-1 [ gr1 ]	VAM	GPT	LIBERO, CALVIN	EVA [ eva_vwm ]	RL	video WM	RoboTwin 2.0
GR-2 [ gr2 ]	VAM	GPT, 38M	LIBERO, CALVIN	RISE [ rise ]	RL	latent	contact-rich
PAD [ pad ]	Uni	DiT	LIBERO	VLA-MBPO [ vla_mbpo ]	RL	video WM	LIBERO-Plus
UWM [ uwm ]	Uni	4-mode DiT	LIBERO	GPC [ gpc ]	RL	rollout WM	Push-T
VPP [ vpp ]	VAM	video pred.	dexterous	WMPO [ wmpo ]	RL	pixel WM	LIBERO

TABLE I: Unified summary of WAMs in Sec. IV , grouped by application domain. Category labels: VAM Video-Action Models, Uni Unified WAMs (joint observation and action generation), WM-Plan WMs-guided Planning (predictions from WMs and action generated by external planner), RL WM-RL (imagination-based RL), 3D 3D/Geometry-based.

Metric (Full name)	Formula / definition
Task completion and navigation
SR (Success Rate)	$\operatorname{SR}=N^{-1}\sum_{i=1}^{N}s_{i}$ , where $s_{i}\in\{0,1\}$ indicates task success.
SPL (Success weighted by Path Length) [ spl_eval ]	$\operatorname{SPL}=N^{-1}\sum_{i=1}^{N}s_{i}\ell_{i}/\max(p_{i},\ell_{i})$ , where $\ell_{i}$ and $p_{i}$ are the shortest and executed path lengths.
NE (Navigation Error) [ r2r ]	$\operatorname{NE}=N^{-1}\sum_{i=1}^{N}d_{\mathcal{G}}(v_{i}^{\mathrm{stop}},v_{i}^{\mathrm{goal}})$ , where $d_{\mathcal{G}}$ is the geodesic shortest-path distance.
nDTW (normalized Dynamic Time Warping) [ ndtw ]	$\operatorname{nDTW}_{i}=\exp[-\operatorname{DTW}_{d_{\mathcal{G}}}(\mathbf{P}_{i}^{\star},\hat{\mathbf{P}}_{i})/(\|\mathbf{P}_{i}^{\star}\|d_{\mathrm{th}})]$ ; report $N^{-1}\sum_{i}\operatorname{nDTW}_{i}$ .
Trajectory and localization

TABLE II: Compact definitions of metrics used for WAMs.

Name	Reported Scale	Type	Eval Metrics	Name	Reported Scale	Type	Eval Metrics
Robot demonstration and human video corpora
DROID [ droid ]	76k trajectories	Real	SR, FVD, FID	Ego4D [ ego4d ]	3.67k hours	Real	SR, ATE, RPE
BridgeData V2 [ bridgev2 ]	60,096 trajectories	Real	SR, Accuracy	EgoDex [ egodex ]	829 hours / 194 tasks	Real	SR (downstream)
Open X-Embodiment [ oxe ]	1M+ trajectories	Mixed	SR, FVD, LPIPS	Kinetics-700 [ kinetics ]	700 classes	Real	SR (downstream)
AgiBot-World [ agibotworld ]	1M+ trajectories	Real	SR, FVD, LPIPS	HowTo100M [ howto100m ]	136M clips	Real	SR (downstream)
SSv2 V2 [ ssv2 ]	220k clips	Real	SR (downstream)

TABLE III: Summary of surveyed datasets, simulators, and benchmarks. Columns report Name , Reported Scale , Type , and Eval Metrics . Scales retain the units used by their primary sources and are not directly comparable. Here, “k” and “M” denote thousand and million, respectively. N/A denotes an open-ended simulator, whereas “—” indicates unavailable scale information.

Explore similar work

World Action Models: The Next Frontier in Embodied AI
May 12, 2026Siyin Wang, Junhao Shi, Zhaoyang Fu +11Vision-Language-Action Models World Action Models
1Fudan University · 2Shanghai Innovation Institute · 3National University of Singapore
From World Models to World Action Models: A Concise Tutorial for Robotics
Jul 1, 2026Xiaoxiong Zhang, Xiong Zeng, Wei ZhangWorld Model Learning World Models for Robotics
School of Automation and Intelligent Manufacturing, Southern University of Science and Technology, Shenzhen, China · LimX Dynamics, Shenzhen, China
World Action Models: A Survey
Jun 18, 2026Qiuhong Shen, Shihua Zhang, Yue Liao +5Action-Conditioned Video Generation Action-Conditioned World Models
National University of Singapore

May 12, 2026J/K move · Enter open · S save

World Action Models: The Next Frontier in Embodied AI

Siyin Wang, Junhao Shi, Zhaoyang Fu, Xinzhe He+10

1Fudan University · 2Shanghai Innovation Institute · 3National University of Singapore

Vision-Language-Action Models World Action Models World Model Evaluation Embodied World Models

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models have achieved strong semantic generalization for embodied policy learning, yet they learn reactive observation-to-action mappings without explicitly modeling how the physical world evolves under intervention. A growing body of work addresses this limitation by integrating world models, predictive models of environment dynamics, into the action generation pipeline. We term this emerging paradigm World Action Models (WAMs): embodied foundation models that unify predictive state modeling with action generation, targeting a joint distribution over future states and actions rather than actions alone. However, the literature remains fragmented across architectures, learning objectives, and application scenarios, lacking a unified conceptual framework. We formally define WAMs and disambiguate them from related concepts, and trace the foundations and early integration of VLA and world model research that gave rise to this paradigm. We organize existing methods into a structured taxonomy of Cascaded and Joint WAMs, with further subdivision by generation modality, conditioning mechanism, and action decoding strategy. We systematically analyze the data ecosystem fueling WAMs development, spanning robot teleoperation, portable human demonstrations, simulation, and internet-scale egocentric video, and synthesize emerging evaluation protocols organized around visual fidelity, physical commonsense, and action plausibility. Overall, this survey provides the first systematic account of the WAMs landscape, clarifies key architectural paradigms and their trade-offs, and identifies open challenges and future opportunities for this rapidly evolving field.

Abstract

Explore similar work

World Action Models: The Next Frontier in Embodied AI

From World Models to World Action Models: A Concise Tutorial for Robotics

World Action Models: A Survey