World Action Models (WAMs) use the predictive knowledge of pretrained video diffusion models for robot control through future video prediction. However, iterative video denoising is costly, and inaccurate visual predictions can compromise action prediction. Our key idea is to make this predictive knowledge available through visual representations computed from the current observation, enabling action prediction without generating future videos. We introduce DreamFree, a WAM that repurposes a pretrained video diffusion model as a predictive visual encoder through representation distillation. The visual encoder learns to predict a frozen video diffusion model's representation using only the current observation. An action module is then trained to predict actions by extracting action-relevant information from the visual encoder. Across real-world and simulated tasks, DreamFree achieves higher success rates than state-of-the-art VLAs and WAMs, while performing real-time closed-loop control at 11Hz with a 14B video diffusion backbone. Experiments show that DreamFree can translate stronger video priors and additional video data into better robot control and enable learning new robot tasks from video demonstrations, including videos from a different embodiment, without action annotation for the new tasks.
Figures & tables
Figure 1: Overview . DreamFree distills predictive knowledge from a frozen video diffusion model into a predictive visual encoder, enabling action prediction from the current observation without generating future videos. In real-world evaluations, DreamFree achieves higher success than DreamZero and π0.5 with latency comparable to that of π0.5 and a 17.8 × inference speedup over DreamZero. DreamFree generalizes across diverse real-world manipulation tasks.
Figure 2: Model architecture of DreamFree . Training is separated into two stages: (a) In the first stage, a pretrained video diffusion backbone is repurposed as a visual encoder through video generative training with a structured attention mask and distillation of predictive representations from the frozen video diffusion model. (b) In the second stage, the action readout module and action decoder are trained to predict actions by extracting action-relevant information from the encoder.
Figure 3: Representation analysis . We evaluate visual representations by measuring how accurately action sequences (Action probing) and future-frame VAE latents (Future probing) can be decoded from them. Future representation distillation makes the encoder’s features more useful for action prediction and improves their ability to capture future visual information from the current observation.
Table 4
Method
LIBERO
RoboCasa
Ours
98.6
71.7
w/o Lgen
98.0
55.6
w/o Ldistill
94.6
47.2
Table 3: Ablation of visual encoder training objectives .
Figure 4: Real-world evaluation on DROID . We evaluate policies out of the box on 10 tasks with five trials per task. Task descriptions and the progress scoring criteria are provided in Appendix C.2 .
Figure 6: Examples of real-world bimanual manipulation task executions . We show selected frames from successful rollouts of DreamFree .
Figure 7: Latency and completion time comparison .
Figure 8: Task transfer from video demonstrations . We report success rates and progress over 20 episodes per task. Task descriptions are provided in Appendix C.3 .
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 10
Task
Instruction
Setup and objective
Stand bottle upright
“Stand the bottle upright.”
A bottle lies on its side. Stand it on its base and release it.
Fold glove
“Fold the glove.”
A glove lies unfolded. Make a regular fold that turns back more than one-third of its original length and remains after release.
Cover strawberry with towel
“Use the towel to cover the strawberry.”
A toy strawberry and grape lie on opposite sides of a towel. Lift the towel and release it over the strawberry, fully covering it while leaving the grape uncovered.
Put red marker in blue cup
“Put the red marker inside the blue cup.”
Red and black markers lie beside blue and green cups. Release the red marker inside the upright blue cup; the black marker and green cup are distractors.
Move cup from bowl to cup
“Pick up the cup from the bowl and put it in the other cup.”
One cup sits inside a bowl, with another cup and a distractor can nearby. Remove the source cup and release it inside the other cup.
Press pump
“Press the pump.”
An upright pump bottle stands beside a cup holding a toothbrush. Depress the pump head while keeping the bottle upright.
Appendix
Table 4: Real-world evaluation tasks on DROID.
Task
Scoring criteria
Stand bottle upright
0.5 : bottle axis exceeds 45∘ above horizontal; upright is 90∘ . 1 : bottle stands on its base after release.
Fold glove
0 : no fold. 0.5 : a nonzero regular fold turns back at most one-third of the original glove length, or an irregular fold occurs, regardless of folded length. 1 : a regular fold turns back more than one-third of the original length and remains after release.
Cover strawberry with towel
Requires lifting, then partial coverage, then full coverage. 1/3 : towel lifted. 2/3 : any partial strawberry coverage after lifting, including slight coverage. 1 : strawberry fully covered after towel release, with the grape uncovered. Coverage without first lifting does not qualify.
Put red marker in blue cup
0.25 : red marker picked up. 0.5 : red marker contacts the blue cup. 0.75 : marker enters the cup, but the cup falls or tips over. 1 : marker released inside the upright blue cup. Picking up the black marker before picking up the red marker makes the entire rollout score 0 , even after later completion; incidental contact does not trigger this penalty.
Move cup from bowl to cup
1/3 : correct source cup picked up. 2/3 : source cup contacts the receiving cup without successful insertion. 1 : source cup inserted and released inside the receiving cup.
Press pump
0.5 : contact with the movable pump head. 1 : pump head depressed while the bottle remains upright. Spring return after release is allowed.
Appendix
Table 5: Task-progress rubrics for real-world evaluation tasks on DROID.
Setting
Task
Description and ordered subgoals
Atomic-to-composite
Arrange vegetables
Transfer the first vegetable from the sink to the counter → transfer the second vegetable from the sink to the counter.
Atomic-to-composite
Microwave thawing
Open the microwave door → transfer the object from the counter into the microwave → close the door → turn on the microwave.
Atomic-to-composite
Restock pantry
Transfer the first object from the counter into the cabinet → transfer the second object from the counter into the cabinet.
Cross embodiment
Place can in basket
Place the can inside the basket → lift the basket. Can placement alone constitutes partial completion.
Cross embodiment
Place cans in box
Place the first can inside the plastic box → place the second can inside the box.
Cross embodiment
Stack three bowls
Establish the base bowl → stack the second bowl on the base → stack the third bowl on the stack.
Appendix
Table 6: Task descriptions and ordered subgoals for task transfer from video demonstrations. Subgoals are evaluated in the listed order.
Method
Spatial
Object
Goal
Long
Average
LingBot-VA [ 16 ]
98.5
99.6
97.2
98.5
98.5
Fast-WAM [ 30 ]
98.2
100.0
97.0
95.2
97.6
DreamFree (Ours)
99.0
99.8
98.8
96.6
98.6
Appendix
Table 7: LIBERO success rates (%) of DreamFree across the four suites.
Task
SR (%)
Task
SR (%)
CloseDoubleDoor
84
PnPCounterToSink
72
CloseDrawer
98
PnPCounterToStove
74
CloseSingleDoor
98
PnPMicrowaveToCounter
58
CoffeePressButton
96
PnPSinkToCounter
76
CoffeeServeMug
72
PnPStoveToCounter
72
CoffeeSetupMug
38
TurnOffMicrowave
90
Appendix
Table 8: RoboCasa-Kitchen success rates (%) for DreamFree with the 14B backbone and additional video data. Each task is evaluated over 50 trials.
Cascade world-action models (WAMs) use predicted visual futures to condition robot actions, but inherit costly iterative inference from diffusion-pretrained experts. We ask what a single-step predicted future must preserve to support accurate single-step action prediction. In the evaluated WAM, resampling video noise has little effect on downstream actions, yet naively truncating both streams to one step substantially reduces task success. Video-space supervision only partially closes this gap, and a local sensitivity analysis shows that vision prediction errors have highly unequal consequences for action accuracy. These findings motivate DirectWAM, which converts pretrained video and action experts into single-step predictors without stochastic sampling through action-sufficient distillation, while retaining the explicit future-to-action cascade. After learning a one-step action model, we train the future predictor by decoding actions from its own predicted futures and backpropagating demonstration action error through the expert, with an optional future-latent reconstruction loss. The resulting objective evaluates the student-generated future through its downstream control consequences rather than relying on representation matching alone. DirectWAM achieves 97.1% average success across four LIBERO suites and 85.4% on RoboTwin 2.0, outperforming Flash-WAM at the same one-evaluation-per-stream budget. This advantage has also extended to three real-world tasks that we designed. Compared with the full iterative teacher, it reduces inference latency by 28.0×.
World Action Models (WAMs) have emerged as a promising approach to learning generalist robot policies using rich spatiotemporal priors from pre-trained video diffusion models. A recent line of WAMs improves inference efficiency by conditioning a separate action decoder on intermediate future video representations extracted from a single denoising step, avoiding costly iterative denoising required for video generation. This design makes action prediction depend directly on the quality of these future representations. In this paper, we introduce Future REpresentation learning via self-Distillation with temporal context (FRED), a framework that trains WAMs to improve the quality of future video representations from noise in a single denoising step. Specifically, FRED uses self-distillation to predict future representations from noise, using representations extracted from ground-truth future observations as targets. We further condition the video model on historical observations, providing temporal context for predicting future representations better. Across diverse real-world manipulation scenarios, FRED consistently outperforms competing WAMs. It achieves particularly large gains on long-horizon and out-of-domain tasks under zero-shot environment generalization, and substantially improves human-to-robot skill transfer, reaching 50.0% task success on tasks absent from robot training data and demonstrated only in human videos, compared with 19.4% for the best competing WAM.
World Action Models (WAMs) improve robot policy learning by jointly modeling future visual dynamics and actions. However, their scalability and generalization remain constrained by their reliance on costly expert demonstrations. We challenge this by asking whether future supervision for WAMs must originate from target-task expert trajectories. In this paper, we propose Vid2WAM, an offline distillation framework that transfers visual diffusion priors from a large video foundation model into a compact WAM student. Given an observation and language instruction, Vid2WAM distills supervision through two complementary channels: task-conditioned future rollouts directly supervise the student's future prediction branch, while an inverse dynamics model recovers embodiment-specific pseudo-actions for action learning. To robustly integrate synthetic and real supervision, we introduce source-aware residual action adaptation that learns source-specific corrections around a shared action backbone and mitigates interference from noisy pseudo-actions. During inference, both the video teacher and inverse dynamics model are discarded, leaving only the WAM student for efficient deployment. Simulation and real-world experiments demonstrate that Vid2WAM improves novel-task generalization and data efficiency under limited expert demonstrations while preserving low-latency inference.
Chenhao Qiu, Ruixiang Wang, Runyi Zhao +7
Fudan University · The Chinese University of Hong Kong, Shenzhen · Shanghai Innovation Institute