World-Action Models for Robot Learning and Control: A Survey

Zuxing Lu; Hongjia Zhai; Guanzhi Wang; Huajian Zeng; Jiaqi Yang; Jingyu Liu; Lei Cheng; Yuantai Zhang

cs.ROSep 13, 2026

World-Action Models for Robot Learning and Control: A Survey

Authors: Zuxing Lu, Hongjia Zhai, Guanzhi Wang, Huajian Zeng, Jiaqi Yang, Jingyu Liu, Lei Cheng, Yuantai Zhang, +8 more

Organizations: MBZUAI · Caltech · Amazon FAR · University of Virginia · Georgia Tech · New York University · UC Berkeley

Abstract

Robots operating in open environments act under partial observability, physical constraints, and dynamic task contexts. Beyond mapping observations and language instructions to actions, they must anticipate how candidate actions may affect future states and task-relevant outcomes. Recent advances in world models, video generation, and Vision-Language-Action (VLA) policies have motivated the development of World-Action Models (WAMs), which couple future world prediction with executable action generation. This survey provides a robotics-oriented review of WAMs. We clarify their scope relative to conventional world models, model-based reinforcement learning, action-conditioned video generation, and reactive VLA policies, and organize existing methods through a unified taxonomy covering representations, transition modeling, action interfaces, architectures, training pipelines, data modalities, and scaling strategies. We further review applications of WAMs in manipulation, navigation, and autonomous driving, and we summarize the datasets, benchmarks, metrics, and protocols used to evaluate WAM systems. Finally, we discuss key challenges in action alignment, world-action factorization, spatial and multi-view consistency, long-horizon memory, neural simulation for closed-loop policy learning, and efficient inference. Taken together, this survey aims to provide a concise technical foundation for integrating predictive world modeling with action generation, toward more reliable embodied robot intelligence. Project page: https://rcl-robotics.github.io/Awesome-World-Action-Models.

Figures & tables

Fig. 2: Conceptual overview of four foundational threads behind WAMs: (a) world models learn latent dynamics by predicting the next state from current state and action; (b) video generative models predict future visual observations conditioned on visual history and language instruction; (c) latent action pretraining infers latent actions from consecutive frames and uses them to model latent transitions; and (d) VLAs generate future action sequences directly from state and language instructions. (The third-row illustrations are adapted from the corresponding papers cited in each thread.)

Fig. 3: Component-level taxonomy for World-Action Models. The architecture can be viewed through four functional modules: a language interface that provides task conditioning, a visual encoder that defines the predictive state, a predictive backbone that carries dynamics, and an action decoder that converts predicted futures into executable controls.

Fig. 4: Transition Modeling Paradigms. Joint prediction couples future observation and action generation within a shared process, whereas the inverse dynamics paradigm factorizes this transition by first imagining future states and then inferring executable actions.

Fig. 5: Taxonomy of WAM Architectures. Existing models can be categorized along two orthogonal axes: architectural design and transition modeling paradigm. Structurally, end-to-end models (a, b) process all modalities within a unified backbone, whereas dual systems (c, d) explicitly separate visual and action experts. In terms of transition modeling, joint prediction approaches (a, c) generate future observations and actions simultaneously, while inverse dynamics approaches (b, d) follow a factorized, predict-then-act sequential generation.

Fig. 6: Policy training pipeline for WAMs. (a) Pre-training leverages internet-scale video data to learn spatiotemporal and action representations and align video with actions; (b) post-training adapts the pretrained model through policy fine-tuning, data augmentation, and reinforcement learning. Images adapted from [ fast , lapa , dreamzero ] .

Fig. 7: Data modality and scaling structure for World-Action Models. The left panel illustrates how different sensing modalities expose complementary action-relevant state variables, while the right panel shows a data-pyramid view in which large action-free video corpora provide broad world priors and smaller embodied trajectories provide action grounding.

Method	Paradigm	Backbone	Eval / Note	Method	Paradigm	Backbone	Eval / Note
Manipulation
GR-1 [ gr1 ]	VAM	GPT	LIBERO, CALVIN	EVA [ eva_vwm ]	RL	video WM	RoboTwin 2.0
GR-2 [ gr2 ]	VAM	GPT, 38M	LIBERO, CALVIN	RISE [ rise ]	RL	latent	contact-rich
PAD [ pad ]	Uni	DiT	LIBERO	VLA-MBPO [ vla_mbpo ]	RL	video WM	LIBERO-Plus
UWM [ uwm ]	Uni	4-mode DiT	LIBERO	GPC [ gpc ]	RL	rollout WM	Push-T
VPP [ vpp ]	VAM	video pred.	dexterous	WMPO [ wmpo ]	RL	pixel WM	LIBERO

TABLE I: Unified summary of WAMs in Sec. IV , grouped by application domain. Category labels: VAM Video-Action Models, Uni Unified WAMs (joint observation and action generation), WM-Plan WMs-guided Planning (predictions from WMs and action generated by external planner), RL WM-RL (imagination-based RL), 3D 3D/Geometry-based.

Metric (Full name)	Formula / definition
Task completion and navigation
SR (Success Rate)	$\operatorname{SR}=N^{-1}\sum_{i=1}^{N}s_{i}$ , where $s_{i}\in\{0,1\}$ indicates task success.
SPL (Success weighted by Path Length) [ spl_eval ]	$\operatorname{SPL}=N^{-1}\sum_{i=1}^{N}s_{i}\ell_{i}/\max(p_{i},\ell_{i})$ , where $\ell_{i}$ and $p_{i}$ are the shortest and executed path lengths.
NE (Navigation Error) [ r2r ]	$\operatorname{NE}=N^{-1}\sum_{i=1}^{N}d_{\mathcal{G}}(v_{i}^{\mathrm{stop}},v_{i}^{\mathrm{goal}})$ , where $d_{\mathcal{G}}$ is the geodesic shortest-path distance.
nDTW (normalized Dynamic Time Warping) [ ndtw ]	$\operatorname{nDTW}_{i}=\exp[-\operatorname{DTW}_{d_{\mathcal{G}}}(\mathbf{P}_{i}^{\star},\hat{\mathbf{P}}_{i})/(\|\mathbf{P}_{i}^{\star}\|d_{\mathrm{th}})]$ ; report $N^{-1}\sum_{i}\operatorname{nDTW}_{i}$ .
Trajectory and localization

TABLE II: Compact definitions of metrics used for WAMs.

Name	Reported Scale	Type	Eval Metrics	Name	Reported Scale	Type	Eval Metrics
Robot demonstration and human video corpora
DROID [ droid ]	76k trajectories	Real	SR, FVD, FID	Ego4D [ ego4d ]	3.67k hours	Real	SR, ATE, RPE
BridgeData V2 [ bridgev2 ]	60,096 trajectories	Real	SR, Accuracy	EgoDex [ egodex ]	829 hours / 194 tasks	Real	SR (downstream)
Open X-Embodiment [ oxe ]	1M+ trajectories	Mixed	SR, FVD, LPIPS	Kinetics-700 [ kinetics ]	700 classes	Real	SR (downstream)
AgiBot-World [ agibotworld ]	1M+ trajectories	Real	SR, FVD, LPIPS	HowTo100M [ howto100m ]	136M clips	Real	SR (downstream)
SSv2 V2 [ ssv2 ]	220k clips	Real	SR (downstream)

TABLE III: Summary of surveyed datasets, simulators, and benchmarks. Columns report Name , Reported Scale , Type , and Eval Metrics . Scales retain the units used by their primary sources and are not directly comparable. Here, “k” and “M” denote thousand and million, respectively. N/A denotes an open-ended simulator, whereas “—” indicates unavailable scale information.

Explore similar work

May 12, 2026

World Action Models: The Next Frontier in Embodied AI

Vision-Language-Action (VLA) models have achieved strong semantic generalization for embodied policy learning, yet they learn reactive observation-to-action mappings without explicitly modeling how the physical world evolves under intervention. A growing body of work addresses this limitation by integrating world models, predictive models of environment dynamics, into the action generation pipeline. We term this emerging paradigm World Action Models (WAMs): embodied foundation models that unify predictive state modeling with action generation, targeting a joint distribution over future states and actions rather than actions alone. However, the literature remains fragmented across architectures, learning objectives, and application scenarios, lacking a unified conceptual framework. We formally define WAMs and disambiguate them from related concepts, and trace the foundations and early integration of VLA and world model research that gave rise to this paradigm. We organize existing methods into a structured taxonomy of Cascaded and Joint WAMs, with further subdivision by generation modality, conditioning mechanism, and action decoding strategy. We systematically analyze the data ecosystem fueling WAMs development, spanning robot teleoperation, portable human demonstrations, simulation, and internet-scale egocentric video, and synthesize emerging evaluation protocols organized around visual fidelity, physical commonsense, and action plausibility. Overall, this survey provides the first systematic account of the WAMs landscape, clarifies key architectural paradigms and their trade-offs, and identifies open challenges and future opportunities for this rapidly evolving field.

Siyin Wang, Junhao Shi, Zhaoyang Fu +11

1Fudan University · 2Shanghai Innovation Institute · 3National University of Singapore

Vision-Language-Action Models World Action Models World Model Evaluation Embodied World Models

Ask PDF

Jul 1, 2026

From World Models to World Action Models: A Concise Tutorial for Robotics

World models are increasingly used in embodied intelligence and generative simulation, yet their scope remains ambiguous across communities. This tutorial presents a design-space view of world models as action-conditioned predictive models that estimate the future evolution of task-relevant observations or states. We categorize existing methods into observation-space and state-space world models, comparing their trade-offs in visual fidelity, spatial structure, physical interpretability, and control usability. We further introduce world action models, which connect predicted futures with executable robot actions, and summarize four representative paradigms: imagine-then-execute, video-feature-conditioned action prediction, joint video-action modeling, and auxiliary video prediction for policy learning. The goal of this tutorial is to clarify the conceptual scope of world (action) models and provide a structured taxonomy for embodied prediction and control.

Xiaoxiong Zhang, Xiong Zeng, Wei Zhang

School of Automation and Intelligent Manufacturing, Southern University of Science and Technology, Shenzhen, China · LimX Dynamics, Shenzhen, China

World Model Learning World Models for Robotics Action-Conditioned World Models World Action Models Embodied World Models

Ask PDF

Jun 18, 2026

World Action Models: A Survey

World Action Models (WAMs) are embodied predictive-action models that make a forecast of the future available to action. Recent WAMs repurpose large video generation models, and a parallel line relies on language or vision-language backbones without a video-generation core. This rapid expansion has blurred the boundary among broad world models, video generation models, action-grounded video world models, Vision-Language-Action policies, and WAMs. This survey gives the field a common account. It first clarifies these boundaries, then organizes existing works through two complementary views. The first view asks what each method is required to generate, spanning rendered futures, latent futures, and video-generation-free action reasoning. The second view decomposes each method by predictive substrate, backbone, action coupling, and deployment regime. This anatomy supports a unified discussion of interactability, causality, persistence, physical plausibility, and generalization, followed by data, evaluation, and open challenges. Across these axes, a consistent design pattern emerges: WAMs are not simply video generators with action heads, but predictive-action methods whose design choices trade representational richness against compute, memory, latency, and action-label cost. The field is moving toward methods that generate less of the future while preserving what control requires. The survey homepage is available at https://world-action-models.github.io/.

Qiuhong Shen, Shihua Zhang, Yue Liao +5

National University of Singapore

Action-Conditioned Video Generation Action-Conditioned World Models Vision-Language-Action Models World Action Models Embodied World Models Video World Models

Ask PDF