World Action Models (WAMs) have emerged as a promising approach to learning generalist robot policies using rich spatiotemporal priors from pre-trained video diffusion models. A recent line of WAMs improves inference efficiency by conditioning a separate action decoder on intermediate future video representations extracted from a single denoising step, avoiding costly iterative denoising required for video generation. This design makes action prediction depend directly on the quality of these future representations. In this paper, we introduce Future REpresentation learning via self-Distillation with temporal context (FRED), a framework that trains WAMs to improve the quality of future video representations from noise in a single denoising step. Specifically, FRED uses self-distillation to predict future representations from noise, using representations extracted from ground-truth future observations as targets. We further condition the video model on historical observations, providing temporal context for predicting future representations better. Across diverse real-world manipulation scenarios, FRED consistently outperforms competing WAMs. It achieves particularly large gains on long-horizon and out-of-domain tasks under zero-shot environment generalization, and substantially improves human-to-robot skill transfer, reaching 50.0% task success on tasks absent from robot training data and demonstrated only in human videos, compared with 19.4% for the best competing WAM.
Figures & tables
Figure 1: VDM learns future representations during WAM training, and FRED further improves them. With DiT4DiT as the base WAM, (a) future-motion probing accuracy of hidden states generated from noise ( i . e ., τ=1.0 ) increases with training but remains below that of representations extracted from clean future observations ( i . e ., τ=0.0 ); and (b) their structural similarity to representations extracted from clean future observations, measured by CKNNA ( Huh et al., 2024 ) , also increases during training. The detailed evaluation setup is provided in Appendix D .
Figure 2: Overview of FRED. We interpret the usage of a VDM in WAMs as a future representation generator, i . e ., h=fθ(ϵ∣ot,c) . From this perspective, we enhance h through self-distillation to predict target representations hτ extracted from noisy future observations ot+1τ by a momentum teacher, while using historical observations ot−H:t as conditioning input to exploit richer temporal context.
Figure 3: Visualization of the training and evaluation setup. (a) For zero-shot environment generalization, we train FRED on the DROID ( Khazatsky et al., 2024 ) dataset and evaluate it on diverse tasks in a new real-world environment without task-specific fine-tuning. (b) For human-robot co-training, we train FRED either on robot demonstrations alone or jointly on robot demonstrations and human videos, and evaluate it on both in-domain robot tasks and human-to-robot skill transfer tasks.
Table 2: Results on humanoid control with and without human-robot co-training. Performance of policies trained on the OpenArm humanoid using robot demonstrations only or with additional human videos. We report the average success rate (%) on in-domain atomic tasks, the average task progress rate (%) on in-domain composite tasks, and both task success (%) and instruction-following performance (%) on human skill transfer tasks.
Figure 4: Policy performance versus video generation and representation quality. Using DiT4DiT as the base WAM, we compare variants with different VDM training objectives in terms of task success rate, video generation quality, and representation quality on RoboCasa Kitchen ( Nasiriany et al., 2024 ) . We find that the task success rates are not strongly correlated with (a) one-step video generation quality and (b) representation quality from clean future observations, but more strongly correlated with (c) the quality of one-step generated future representations from noise.
Table 5: Analysis of FRED design choices. We examine the effects of (a) teacher noise level, (b) feature extraction layer, (c) self-distillation loss coefficient, and (d) blocking action gradients to the video backbone on RoboCasa Kitchen ( Nasiriany et al., 2024 ) . Default settings are highlighted in gray .
Table 7
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: DROID evaluation setup. The setup consists of a Franka Research 3 robot arm with a Robotiq 2F-85 gripper, a wrist camera, and left and right exterior cameras.
Figure 6: DROID evaluation tasks. Task visualization for four task suites: (a) in-domain easy, (b) in-domain long-horizon, (c) out-of-domain, and (d) object grounding. Clear Table repeats the pick-and-place step until the table is empty.
Figure 7: OpenArm evaluation setup. It consists of the two 7-DoF OpenArm arms and a ZED 2i stereo camera mounted on the 2-DoF neck, together with the objects used in the evaluation tasks.
Figure 8: Distribution of skills and object categories in the OpenArm robot dataset.
Figure 9: Distribution of skills and object categories in the human video dataset.
Figure 10: OpenArm evaluation tasks. The seven tasks, grouped into (a) in-domain atomic, (b) in-domain composite, and (c) human skill transfer. Each task is shown by the initial and final egocentric frames of one episode.
Primitive
(%)
Primitive
(%)
Primitive
(%)
Forward
21.82
Down
13.48
Left
12.91
Right
12.27
Backward
10.22
Up
0 9.64
Stop
0 5.16
Close gripper
0 4.17
Open gripper
0 3.11
Roll right
0 2.39
Tilt up
0 1.26
Rotate CW
0 1.32
Roll left
0 0.76
Tilt down
0 0.70
Rotate CCW
0 0.80
Appendix
Table 10: Distribution of motion-primitive labels in the probing validation dataset. CW and CCW denote clockwise and counterclockwise rotation, respectively.
Table 12: Per-task results on humanoid control with and without human-robot co-training. Performance of policies trained on the OpenArm humanoid using robot demonstrations only or with additional human videos. We report the average success rate (%) on in-domain atomic tasks, the average task progress rate (%) on in-domain composite tasks, and both task success (%) and instruction-following (IF) performance (%) on human skill transfer tasks.
World Action Models (WAMs) improve robot policy learning by jointly modeling future visual dynamics and actions. However, their scalability and generalization remain constrained by their reliance on costly expert demonstrations. We challenge this by asking whether future supervision for WAMs must originate from target-task expert trajectories. In this paper, we propose Vid2WAM, an offline distillation framework that transfers visual diffusion priors from a large video foundation model into a compact WAM student. Given an observation and language instruction, Vid2WAM distills supervision through two complementary channels: task-conditioned future rollouts directly supervise the student's future prediction branch, while an inverse dynamics model recovers embodiment-specific pseudo-actions for action learning. To robustly integrate synthetic and real supervision, we introduce source-aware residual action adaptation that learns source-specific corrections around a shared action backbone and mitigates interference from noisy pseudo-actions. During inference, both the video teacher and inverse dynamics model are discarded, leaving only the WAM student for efficient deployment. Simulation and real-world experiments demonstrate that Vid2WAM improves novel-task generalization and data efficiency under limited expert demonstrations while preserving low-latency inference.
Chenhao Qiu, Ruixiang Wang, Runyi Zhao +7
Fudan University · The Chinese University of Hong Kong, Shenzhen · Shanghai Innovation Institute
World Action Models (WAMs) improve robot policy learning by jointly modeling actions and future observations. However, conditioning future prediction only on the task prompt and observation context risks capturing generic task progression rather than the action-specific consequences of the executed action. We introduce SelfWAM, a unified self-grounded WAM built on a modality-specialized Mixture-of-Transformers (MoT) architecture that jointly predicts actions, action-conditioned future RGB frames, and robot self-masks, thereby grounding future prediction in the robot's visible body and its action-induced motion. During joint training, SelfWAM allows future visual queries to attend to a clean copy of the demonstrated action, turning the video branch into an action-specific consequence model while leaving the fast action-only inference path unchanged. To focus video learning on action-relevant visual changes, we use prompt-specific objectives for future robot self-mask prediction, which removes appearance details and provides a target whose temporal evolution is tightly coupled with the conditioning action. Together, clean-action conditioning and future self-mask supervision make future predictions more directly reflect how the executed action changes the robot's visible motion and the surrounding scene. Experiments on RoboTwin 2.0 and real-world manipulation tasks show that SelfWAM produces more action-sensitive futures and preserves fast policy inference, while improving policy performance.
World action models (WAMs) predict the future alongside actions during training. Due to the heavy computation cost of video denoising, whether the future must still be generated during inference is disputed: Explicit WAMs denoise it into clean frames along with every action chunk, whereas Latent WAMs discard it entirely for acceleration. We find that latent WAMs, despite matching explicit ones on in-distribution tasks, fail to retain the generalization benefits that originally motivated WAMs. To demonstrate this, we evaluate generalization along three axes: environmental perturbation, data efficiency, and task generalization. Controlled comparisons with a matched backbone, training data, and budget reveal consistent degradation across all three axes when the action expert no longer conditions on future representations. Further analysis shows that the gap arises almost entirely from the first denoising step: the benefit comes from preparing the future, not generating it. We therefore propose Simple-WAM, which simplifies future modeling into a single forward pass of fully noised video tokens and adapts the training-time noise schedule to this inference behavior. Across simulation and real-world tasks, Simple-WAM achieves the best of both worlds, leading explicit WAMs in generalization performance with efficiency comparable to Latent WAMs. Project Page: https://zrporz.github.io/Simple-WAM-Web/
Renping Zhou, Zanlin Ni, Zihao Fan +8
Leap Lab, Tsinghua University · University of Science and Technology of China · Beijing Institute of Technology