Humanoid World Action Model With Joint State--Action Generation

Yan Yang; Jikun Rong; Minzhao Zhu; Zheyi Zhao; Qirui Hu; Zihan Lan; Weixin Mao; Yinhao Li

cs.ROOct 8, 2026

Humanoid World Action Model With Joint State--Action Generation

Authors: Yan Yang, Jikun Rong, Minzhao Zhu, Zheyi Zhao, Qirui Hu, Zihan Lan, Weixin Mao, Yinhao Li, +2 more

Organizations: LimX Dynamics · Southern University of Science and Technology

Abstract

Humanoid robots are a promising platform for general-purpose manipulation. Recent Vision-Language-Action (VLA) policies learn actions directly from multimodal observations, while World Action Models (WAMs) further incorporate future visual prediction to improve action generation. However, in hierarchical humanoid systems, VLA and WAM policies output reference actions that are subsequently realized through whole-body control, robot dynamics, balance, and contact. This hierarchy creates an action--execution gap: the reference produced by the policy can differ from the motion realized by the robot. Without explicitly modeling the realized body state, future visual prediction must jointly explain scene evolution and discrepancies between reference actions and executed motion, making it difficult to associate an action with its physical outcome. We propose HWAM, a Humanoid World Action Model with joint state--action generation, which makes the robot's post-execution proprioceptive state an explicit prediction target. By jointly generating reference actions and their realized body states, HWAM directly incorporates supervision of executed motion into action learning. HWAM is trained through three complementary conditional paths. The Policy path jointly denoises state--action trajectories conditioned only on current observations, matching deployment conditions. Forward Dynamics Modeling (FDM) predicts future visual observations conditioned on actions and post-execution states, while Inverse Dynamics Modeling (IDM) reconstructs the joint trajectory from visual transitions. Together, these paths connect policy references, realized body motion, and visual outcomes. HWAM achieves the highest success rate among evaluated baselines on three real-robot tasks on the LimX OLI humanoid. On Candy Picking, HWAM achieves a 70.6% success rate, compared with 43.3% for Fast-WAM.

Figures & tables

Figure 1: Hierarchical control interface in HWAM. The high-level VLA/WAM/HWAM policy produces reference actions, which are adapted by low-level whole-body tracking and joint servoing before physical execution. The measured post-execution proprioceptive state provides feedback to the high-level policy.

Figure 2: Overview of HWAM. The high-level action $a_{t}$ is executed through the low-level whole-body controller, producing the measured post-execution state $s_{t+1}$ . HWAM represents the joint state–action pair as a single token. A frozen causal video VAE provides current and future visual latents, while the video and state–action experts interact through path-dependent mixed attention.

Figure 3: Path-dependent corruption and attention. FDM predicts future visual observations from the joint state–action trajectory, whereas IDM reconstructs the joint trajectory from visual transitions, providing complementary bidirectional grounding between visual dynamics and state–action trajectories. Policy matches deployment by conditioning only on current observations.

Figure 4: Examples from the OLI evaluation suite. The stationary manipulation example shows repeated tabletop object collection, while the mobile manipulation example combines object handling with changes in body posture and position. Each row follows a separate video timeline; backgrounds are simplified and panels are cropped for visualization.

Method	Robot-data pretraining	Candy Picking	Object Collection	Plush-Toy Picking
Cosmos3-Edge	Yes	15.8	0.0	56.7
GR00T N1.5	Yes	40.5	42.5	60.0
$\pi_{0.5}$	Yes	62.5	45.0	60.7
Fast-WAM	No	43.3	0.0	60.0
DiT4DiT	No	52.5	15.0	—
HWAM (Ours)	No	70.6	46.7	73.3

Table 1: Real-robot evaluation results (%) on the three OLI tasks. An em dash denotes an unreported or unverified task result.

Figure 5: Open-loop action (a) and state (b) MAE on Candy Picking over a 32-step horizon. Errors are averaged over 31 shared joints on the held-out episodes and reported in degrees. Dashed and solid curves denote action-only and joint state–action prediction, respectively. Lower is better.

Configuration	Prediction target	Training recipe	Success (%)
Action + VGM/Policy	Action	VGM/Policy	60.0
State–Action + VGM/Policy	Joint state–action	VGM/Policy	45.5
Action + FDM/IDM/Policy	Action	FDM/IDM/Policy	55.0
HWAM	Joint state–action	FDM/IDM/Policy	73.3

Table 2: Controlled ablation on Plush-Toy Picking.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Platform	Direct MAE	Steady-state MAE
LimX OLI (mobile humanoid)	4.420	2.885
ALOHA (stationary bimanual)	0.648	0.402

Appendix

Table 3: Action–execution gap measured by mean absolute error (MAE) in degrees. Direct MAE compares the joint reference

a_{t}

with the measured post-execution joint state

s_{t+1}

. Steady-state MAE uses the same pairing under approximately constant joint references.

Task	Control demands	Training episodes	Completion criterion
Candy Picking	Precise grasping and spatially targeted placement	1,783	Pick the instructed candy and place it in its assigned tray region.
Object Collection	Repeated reaching and grasping across objects	4,220	Place all four tabletop objects in the basket.
Plush-Toy Picking	Mobile object handling with whole-body coordination	404	Complete the specified plush-toy handling sequence.

Appendix

Table 4: OLI task suite, training data, and completion criteria.

Configuration	PSNR	SSIM
VGM + Action	20.690	0.7420
VGM + State–Action	20.673	0.7409
Joint denoising + Action	20.844	0.7453
Joint denoising + State–Action	20.670	0.7440
FDM/IDM/Policy + Action	20.345	0.7361
HWAM	20.501	0.7370

Appendix

Table 5: End-to-end future-video prediction without ground-truth future state or action conditions. Higher is better.

Model	Condition	PSNR
Action-only	Ground-truth actions	20.559
HWAM	Ground-truth states and actions	23.336

Appendix

Table 6: Future-video prediction with ground-truth trajectory conditions on held-out Candy Picking episodes. Higher is better.

Method	Successes / trials	SR (%)	95% CI (%)
GR00T N1.5	49/121	40.5	[32.2, 49.4]
$\pi_{0.5}$	75/120	62.5	[53.6, 70.6]
Fast-WAM	52/120	43.3	[34.8, 52.3]
HWAM	84/119	70.6	[61.9, 78.0]

Appendix

Table 7: Candy Picking trial counts and 95% Wilson confidence intervals for methods with available counts.

(a) Motion directions
Method	CH	CV	X-P	X-S
GR00T N1.5	50.0	25.8	40.0	46.7
$\pi_{0.5}$	60.0	66.7	80.0	43.3
Fast-WAM	33.3	43.3	46.7	50.0
HWAM	82.8	63.3	76.7	60.0

Appendix

Table 8: Candy Picking success rates (%) by (a) motion direction and (b) workspace region. CH/CV: cross-horizontal/vertical; X-P/X-S: primary/secondary diagonal.

Figure 6: ALOHA Clothes Folding task. The sequence shows the robot performing successive bimanual folds on a spread garment, from the initial grasp (Start, 0.4 s) through three intermediate folds to the final fold and release (49.2 s).

Method	Clothes Folding score
GR00T N1.5	78.75
GR00T N1.7	58.75
$\pi_{0}$	65.00
$\pi_{0.5}$	77.00
Fast-WAM	66.25
SmolVLA	51.25

Appendix

Table 9: ALOHA Clothes Folding task-completion scores on a 0–100 scale. Scores include partial credit for completed folding stages. Higher is better.

Explore similar work

Sep 28, 2026cs.RO

From World Models to World Action Models: Rethinking Next-State Prediction

Predicting the next state is a core paradigm of World Models for modeling physical dynamics, emphasizing prediction fidelity. As World Models evolve into World-Action Models (WAMs), existing methods still fix the next state before training as RGB, a single latent feature, or a static combination of predefined targets, thereby constraining action learning to the inductive biases preserved by a particular representation. To address this limitation, we propose CF-WAM, a dynamic next-state prediction framework that samples visual, semantic, geometric, and interaction projections of the same future, standardizes them into a common video form, and supervises a unified WAM across these projections. The action-relevant constraints exposed by these projections accumulate across training steps, forcing WAM to capture the underlying state-transition structure that supports multiple projections of the same action-conditioned future. This dynamic mechanism also provides a natural cross-embodiment dynamics reference frame for Human and Robot learning. By jointly learning across different next-state parameterizations, heterogeneous Human and Robot experience can bypass appearance differences and directly contribute to shared state-transition learning, improving cross-embodiment generalization. Experiments show that CF-WAM improves both training efficiency and final control performance, while translating Human experience effectively into policy gains. CF-WAM achieves state-of-the-art performance on RoboCasa-GR1 with an average success rate of 82.50%, while reaching 82.65% on LIBERO-Plus and up to 84.00% in real-world evaluations.

Tingyu Yuan, Ziming Ji, Biaoliang Guan +9

CASIA · UCAS · BUPT +4

Robot Dynamics Learning Action-Conditioned World Models Cross-Embodiment Robot Learning World Action Models

Ask PDF

May 12, 2026cs.RO

World Action Models: The Next Frontier in Embodied AI

Vision-Language-Action (VLA) models have achieved strong semantic generalization for embodied policy learning, yet they learn reactive observation-to-action mappings without explicitly modeling how the physical world evolves under intervention. A growing body of work addresses this limitation by integrating world models, predictive models of environment dynamics, into the action generation pipeline. We term this emerging paradigm World Action Models (WAMs): embodied foundation models that unify predictive state modeling with action generation, targeting a joint distribution over future states and actions rather than actions alone. However, the literature remains fragmented across architectures, learning objectives, and application scenarios, lacking a unified conceptual framework. We formally define WAMs and disambiguate them from related concepts, and trace the foundations and early integration of VLA and world model research that gave rise to this paradigm. We organize existing methods into a structured taxonomy of Cascaded and Joint WAMs, with further subdivision by generation modality, conditioning mechanism, and action decoding strategy. We systematically analyze the data ecosystem fueling WAMs development, spanning robot teleoperation, portable human demonstrations, simulation, and internet-scale egocentric video, and synthesize emerging evaluation protocols organized around visual fidelity, physical commonsense, and action plausibility. Overall, this survey provides the first systematic account of the WAMs landscape, clarifies key architectural paradigms and their trade-offs, and identifies open challenges and future opportunities for this rapidly evolving field.

Siyin Wang, Junhao Shi, Zhaoyang Fu +11

1Fudan University · 2Shanghai Innovation Institute · 3National University of Singapore

Vision-Language-Action Models World Action Models World Model Evaluation Embodied World Models

Ask PDF

Jun 17, 2026cs.CV

ImageWAM: Do World Action Models Really Need Video Generation, or Just Image Editing?

World Action Models (WAMs) commonly rely on video generation to bridge visual world modeling and robot control. However, video-based WAMs face three coupled limitations: dense multi-frame future tokens make inference costly, full video prediction spends capacity on action-irrelevant temporal and appearance details, and long-horizon future imagination may introduce errors that mislead action prediction. These issues raise a simple question: Does world action model really need video generation? We propose ImageWAM, a simple WAM framework that repurposes pretrained image editing models for robot action prediction. In contrast to video generation, image editing provides a better-matched prior: it only needs to model a target-frame transformation, focuses on action-relevant current-to-target visual differences, and grounds task instructions to localized visual changes through edit pretraining. In practice, ImageWAM does not decode the target frame at inference time; instead, it conditions a flow-matching action expert on the KV caches produced by image-editing denoising, using them as a compact world-action context. ImageWAM outperforms standard VLA baselines and matching competitive WAMs without additional policy pretraining across different simulator and real-world experiments. It also reduces FLOPs to 1/6 and latency to 1/4 of video-based WAMs. Attention analysis further shows that editing caches focus on task-relevant change regions, supporting image editing as an effective alternative to video-based world-action modeling.

Yuyang Zhang, Wenyao Zhang, Zekun Qi +7

Shanghai Jiao Tong University · Tsinghua University · Tencent Robotics X +1

Action-Conditioned World Models World Action Models Embodied World Models

Ask PDF