What Does World Modeling Actually Improve in Robot Action Learning?

Hanwen Cui; Yuanyuan Li; Qing Guo; Zhida Zhang; Chao Chen; Xi Chen; Jie Cao

cs.ROOct 25, 2026

What Does World Modeling Actually Improve in Robot Action Learning?

Authors: Hanwen Cui, Yuanyuan Li, Qing Guo, Zhida Zhang, Chao Chen, Xi Chen, Jie Cao

Organizations: MAIS & NLPR, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Gengine · School of Computer Science and Engineering, Central South University

Abstract

World Action Models (WAMs) combine future-video prediction with robot action learning, yet improvements in task success alone do not reveal how predictive supervision manifests in execution behavior or whether its benefits persist under task and environmental shifts. We conduct controlled future-video prediction loss ablations within FastWAM and OpenWAM, keeping model architecture, initialization, robot training data, and training steps fixed within each model family. Under clean conditions on seen tasks, predictive supervision increases success rates by 24.38 and 72.91 percentage points, respectively. Trajectory clustering further reveals higher success rates within several coarse trajectory clusters, including cases where cluster frequencies change little, indicating that the gains are not solely reflected in changes in the prevalence of trajectory patterns. However, these benefits do not consistently persist under distribution shifts: improvements on unseen tasks vary across models and tasks, while both models exhibit low success rates under the evaluated environmental randomizations, without a consistent advantage from predictive supervision. We further introduce future-video prediction as an auxiliary training objective for $π_{0.5}$ , where a lightweight Transformer predictor used only during training increases success from 82.94% to 89.19%. These findings support future-video prediction as effective auxiliary supervision for action learning, while distinguishing improved execution on familiar tasks from reliable transfer and showing that such gains need not rely on a large pretrained video diffusion transformer (DiT) in the evaluated setting.

Figures & tables

Figure 1: Overview and main findings. (a) Controlled comparison with and without future-video prediction loss, using identical architectures within each WAM family. (b) Representative behavior-mode analysis on FastWAM’s stack_blocks_two task: the dominant mode M0 retains a similar frequency, while its within-mode success rate increases from 50.2% to 89.5%. Scatter layouts and point colors are schematic; numerical annotations report measured statistics. (c) Success-rate differences between $\lambda_{\mathrm{video}}=1$ and $\lambda_{\mathrm{video}}=0$ for FastWAM and OpenWAM under Seen/Unseen $\times$ Clean/Randomized conditions (Experiment 3). The Seen task set differs from Experiment 1. (d) Adding a training-only lightweight video predictor improves $\pi_{0.5}$ success rate from 82.94% to 89.19%, compared with 86.97% using a pretrained Wan2.2-TI2V-5B branch.

Figure 2: Training configurations for auxiliary video prediction in $\pi_{0.5}$ . (a) A randomly initialized six-layer Transformer predicts future-video latent flow conditioned on projected shared VLM representations, with targets encoded by a frozen Wan2.2 VAE. (b) A pretrained Wan2.2-TI2V-5B DiT predicts future-video latent flow conditioned on adapted VLM representations and frozen T5 text features; the DiT is trainable, while the VAE and T5 encoder are frozen. Both variants retain the original action pathway and use the video branch only during training, removing it at inference. The video objective updates the shared vision-language pathway and the corresponding prediction branch, but does not directly update the action expert, which is trained by the action objective.

	FastWAM			OpenWAM
Task	$\lambda=0$	$\lambda=1$	$\Delta$	$\lambda=0$	$\lambda=1$	$\Delta$
place_fan	7.19	39.38	+32.19	2.81	80.00	+77.19
place_a2b_right	7.50	54.06	+46.56	6.56	81.88	+75.31
put_bottles_dustbin	9.69	6.25	-3.44	4.38	91.56	+87.19
place_mouse_pad	0.94	9.69	+8.75	0.00	57.50	+57.50
place_dual_shoes	16.88	53.12	+36.25	2.50	80.31	+77.81

Table 1: Experiment 1 success rates (%) on ten Seen+Clean tasks. Here

\lambda=\lambda_{\mathrm{video}}

, and

\Delta

denotes the success-rate difference between

\lambda=1

and

\lambda=0

in percentage points (pp).

Model	Task / Mode	Mode Share (%)	Within-Mode SR (%)
FastWAM	place_object_stand / M0	50.0 $\rightarrow$ 50.0	67.8 $\rightarrow$ 77.7
FastWAM	place_object_stand / M1	50.0 $\rightarrow$ 50.0	84.5 $\rightarrow$ 97.3
FastWAM	stack_blocks_two / M0	90.4 $\rightarrow$ 90.0	50.2 $\rightarrow$ 89.5
OpenWAM	open_laptop / M0	69.0 $\rightarrow$ 47.1	22.7 $\rightarrow$ 78.4
OpenWAM	open_laptop / M1	31.0 $\rightarrow$ 52.9	22.9 $\rightarrow$ 94.4

Table 2: Representative behavior-mode results. Mode share is the fraction of trajectories assigned to each mode; within-mode SR is the success rate within that mode. Values compare

\lambda_{\mathrm{video}}=0\rightarrow 1

Model	Evaluation Setting	$\lambda=0$	$\lambda=1$	$\Delta$ (pp)
FastWAM	Seen + Clean	18.44	54.81	+36.37
FastWAM	Seen + Randomized	0.50	0.38	-0.13
FastWAM	Unseen + Clean	15.81	16.09	+0.28
FastWAM	Unseen + Randomized	3.19	1.56	-1.62
OpenWAM	Seen + Clean	1.81	91.06	+89.25
OpenWAM	Seen + Randomized	0.38	0.12	-0.25

Table 3: Experiment 3 success rates (%) under task- and environment-level distribution shifts, using five Seen tasks and ten held-out Unseen tasks. Here

\lambda=\lambda_{\mathrm{video}}

, and

\Delta

is the difference in percentage points.

Variant	Successes	SR (%)	$\Delta$ (pp)
Original $\pi_{0.5}$	2,654 / 3,200	82.94	–
+ Lightweight Video Pred.	2,854 / 3,200	89.19	+6.25
+ Wan2.2-TI2V-5B	2,783 / 3,200	86.97	+4.03

Table 4: Success rates of the

\pi_{0.5}

variants.

\Delta

is measured relative to the original

\pi_{0.5}

baseline.

Task	No-WM (%)	WM (%)	$\Delta$ (pp)
place_fan	7.19	39.38	+32.19
place_a2b_right	7.50	54.06	+46.56
put_bottles_dustbin	9.69	6.25	-3.44
place_mouse_pad	0.94	9.69	+8.75
place_dual_shoes	16.88	53.12	+36.25
pick_diverse_bottles	7.19	36.56	+29.37

Table 1: Per-task FastWAM results on Experiment 1’s ten-task Seen+Clean suite.

\Delta

denotes WM minus No-WM in percentage points.

Task	No-WM Success	No-WM (%)	WM Success	WM (%)	$\Delta$ (pp)
beat_block_hammer	36/320	11.25	307/320	95.94	+84.69
open_laptop	67/320	20.94	228/320	71.25	+50.31
pick_diverse_bottles	40/320	12.50	248/320	77.50	+65.00
place_a2b_right	21/320	6.56	262/320	81.88	+75.31
place_dual_shoes	8/320	2.50	257/320	80.31	+77.81
place_fan	9/320	2.81	256/320	80.00	+77.19

Table 2: Per-task OpenWAM results on Experiment 1’s ten-task Seen+Clean suite. Each task contains 320 rollouts per method.

Task	Mode	Scene Coverage	Mode Share (%)	Within-Mode SR (%)
open_laptop	M0	6/10	59.8 $\rightarrow$ 50.1	66.8 $\rightarrow$ 80.0
open_laptop	M1	5/10	40.2 $\rightarrow$ 49.9	100.0 $\rightarrow$ 96.7
place_bread_basket	M0	4/10	31.6 $\rightarrow$ 30.0	86.1 $\rightarrow$ 99.0
	M1	2/10	20.0 $\rightarrow$ 20.0	82.0 $\rightarrow$ 94.9
	M2	2/10	10.0 $\rightarrow$ 10.1	47.7 $\rightarrow$ 58.9
	M3	2/10	10.1 $\rightarrow$ 10.0	47.3 $\rightarrow$ 93.0

Table 3: Complete FastWAM behavior-mode results. Entries report No-WM

\rightarrow

WM. A dash indicates that no WM trajectory was assigned to that mode.

Task	Mode	Scene Coverage	Mode Share (%)	Within-Mode SR (%)
open_laptop	M0	10/10	69.0 $\rightarrow$ 47.1	22.7 $\rightarrow$ 78.4
open_laptop	M1	10/10	31.0 $\rightarrow$ 52.9	22.9 $\rightarrow$ 94.4
place_bread_basket	M0	5/10	39.1 $\rightarrow$ 40.0	26.2 $\rightarrow$ 99.6
	M1	5/10	37.4 $\rightarrow$ 40.0	26.3 $\rightarrow$ 94.7
	M2	4/10	20.9 $\rightarrow$ 20.0	44.2 $\rightarrow$ 92.2
	M3	3/10	2.7 $\rightarrow$ 0.0	0.0 $\rightarrow$ –

Table 4: Complete OpenWAM behavior-mode results. Entries report No-WM

\rightarrow

WM. A dash indicates that no WM trajectory was assigned to that mode.

Condition	Task	No-WM (%)	WM (%)	$\Delta$ (pp)
Seen + Clean	place_a2b_left	24.38	44.06	+19.69
Seen + Clean	place_can_basket	21.25	56.56	+35.31
Seen + Clean	place_phone_stand	1.25	48.44	+47.19
Seen + Clean	place_shoe	29.38	67.19	+37.81
Seen + Clean	pick_dual_bottles	15.94	57.81	+41.88
Seen + Randomized	place_a2b_left	0.31	0.62	+0.31

Table 5: Per-task FastWAM results for Seen tasks in Experiment 3.

Condition	Task	No-WM (%)	WM (%)	$\Delta$ (pp)
Unseen + Clean	adjust_bottle	0.00	0.00	0.00
Unseen + Clean	grab_roller	0.00	0.00	0.00
Unseen + Clean	place_container_plate	52.50	61.25	+8.75
Unseen + Clean	move_pillbottle_pad	0.00	0.00	0.00
Unseen + Clean	move_can_pot	0.00	0.00	0.00
Unseen + Clean	open_microwave	13.75	2.50	-11.25

Table 6: Per-task FastWAM results for Unseen tasks in Experiment 3.

Condition	Task	No-WM (%)	WM (%)	$\Delta$ (pp)
Seen + Clean	pick_dual_bottles	0.94	98.75	+97.81
Seen + Clean	place_a2b_left	1.56	86.25	+84.69
Seen + Clean	place_can_basket	0.94	86.25	+85.31
Seen + Clean	place_phone_stand	0.00	87.19	+87.19
Seen + Clean	place_shoe	5.62	96.88	+91.25
Seen + Randomized	pick_dual_bottles	0.00	0.62	+0.62

Table 7: Per-task OpenWAM results for Seen tasks in Experiment 3.

Condition	Task	No-WM (%)	WM (%)	$\Delta$ (pp)
Unseen + Clean	adjust_bottle	0.00	0.62	+0.62
Unseen + Clean	grab_roller	1.25	0.00	-1.25
Unseen + Clean	handover_block	0.00	0.00	0.00
Unseen + Clean	move_can_pot	1.25	0.31	-0.94
Unseen + Clean	move_pillbottle_pad	1.56	0.00	-1.56
Unseen + Clean	open_microwave	1.25	73.12	+71.88

Table 8: Per-task OpenWAM results for Unseen tasks in Experiment 3.

Component	Action Loss	Video Loss	Used at Inference
SigLIP + shared VLM backbone	Yes	Yes	Yes
Action expert + action head	Yes	No	Yes
Video head / Wan DiT + adapters	No	Yes	No
Frozen VAE / frozen T5	No	No	No

Table 9: Gradient and inference roles of the main components in the

\pi_{0.5}

video-auxiliary variants.

Task	Original $\pi_{0.5}$	Lightweight Video Predictor	Wan2.2-TI2V-5B
beat_block_hammer	289/320 (90.31%)	306/320 (95.63%)	303/320 (94.69%)
open_laptop	236/320 (73.75%)	292/320 (91.25%)	284/320 (88.75%)
pick_diverse_bottles	190/320 (59.38%)	236/320 (73.75%)	188/320 (58.75%)
place_a2b_right	279/320 (87.19%)	286/320 (89.38%)	296/320 (92.50%)
place_dual_shoes	284/320 (88.75%)	297/320 (92.81%)	300/320 (93.75%)
place_fan	267/320 (83.44%)	282/320 (88.13%)	284/320 (88.75%)

Table 10: Per-task success rates of the three

\pi_{0.5}

variants in Experiment 4. Each task contains 320 rollouts per model.

Robot Learning

Robot learning develops methods for robots to acquire skills and adapt from demonstrations, interaction, data, and physical models.

See all 323 papers

De-Diffusing WAMs: Action-Sufficient Direct Prediction for Robot Control
Oct 20, 2026Shaoyi Zheng, Zhongtao Jiang, Yue Peng +4Efficient Inference for World Action Models World Models for VLA Models
ByteDance. · New York University (NYU).
RAP-WAM: Learning World-Action Models through Predictive Rehearsal and Action Perturbations
Oct 12, 2026Yuanpeng He, Ceyao Zhang, Fangjing Li +8Visuomotor Policy Learning World Action Models
Peking University · PKU-PsiBot Joint Lab · Beijing Jiaotong University +1
WAM-OPD: Joint Video-Action Supervision for World Action Model Post-Training with On-Policy Distillation
Aug 23, 2026Liuhaichen Yang, Zhengyang Zhong, Hanshang Zhu +13World Action Models Action-Conditioned World Models
University College London · Shanghai Jiao Tong University · The University of Manchester +3

Oct 20, 2026J/K move · Enter open · S save

De-Diffusing WAMs: Action-Sufficient Direct Prediction for Robot Control

Shaoyi Zheng, Zhongtao Jiang, Yue Peng, Wenbo Lu+3

ByteDance. · New York University (NYU).

Efficient Inference for World Action Models World Models for VLA Models Robotic Manipulation Knowledge Distillation

Open paper AskPDF arXiv

Cascade world-action models (WAMs) use predicted visual futures to condition robot actions, but inherit costly iterative inference from diffusion-pretrained experts. We ask what a single-step predicted future must preserve to support accurate single-step action prediction. In the evaluated WAM, resampling video noise has little effect on downstream actions, yet naively truncating both streams to one step substantially reduces task success. Video-space supervision only partially closes this gap, and a local sensitivity analysis shows that vision prediction errors have highly unequal consequences for action accuracy. These findings motivate DirectWAM, which converts pretrained video and action experts into single-step predictors without stochastic sampling through action-sufficient distillation, while retaining the explicit future-to-action cascade. After learning a one-step action model, we train the future predictor by decoding actions from its own predicted futures and backpropagating demonstration action error through the expert, with an optional future-latent reconstruction loss. The resulting objective evaluates the student-generated future through its downstream control consequences rather than relying on representation matching alone. DirectWAM achieves 97.1% average success across four LIBERO suites and 85.4% on RoboTwin 2.0, outperforming Flash-WAM at the same one-evaluation-per-stream budget. This advantage has also extended to three real-world tasks that we designed. Compared with the full iterative teacher, it reduces inference latency by

28.0\times

Abstract

Robot Learning

Explore similar work

De-Diffusing WAMs: Action-Sufficient Direct Prediction for Robot Control

RAP-WAM: Learning World-Action Models through Predictive Rehearsal and Action Perturbations

WAM-OPD: Joint Video-Action Supervision for World Action Model Post-Training with On-Policy Distillation