World Action Models (WAMs) have recently shown great promise in robotic control, benefiting from the future imagination capability of large-scale pretrained video models. However, most existing WAMs require explicitly imagining future states at test time, incurring substantial computational overhead and latency that make them unsuitable for real-time robotic control. In this paper, we propose W2A, a two-stage distillation framework for real-time robotic manipulation that distills intermediate denoising representations from a video diffusion model into an action model during training, while entirely discarding the video model at test time. We hypothesize that although video latent representations are high-dimensional, a compressed low-dimensional subspace is sufficient to provide effective supervision for action generation. This makes it possible to transfer such information effectively into an action model. Extensive experiments in both simulation and real-world settings demonstrate competitive success rates while significantly improving inference efficiency. On LIBERO, W2A achieves a 95.9% average success rate. More importantly, W2A achieves about 21x faster inference than the representative WAM baseline.
Figures & tables
Figure 1: Overview of W2A compared with mainstream world action models. While existing WAMs usually keep a video branch at test time, W2A distills world priors from a video DiT into an action DiT during training and removes the video DiT entirely at test time. This yields competitive performance in simulation and real-world manipulation while substantially improving inference efficiency.
Figure 2: The W2A training and inference framework.
Method
Ext. RA
Spatial (%)
Object (%)
Goal (%)
Long (%)
Average (%)
Diffusion Policy (scratch) [ 8 ]
✗
78.3
92.5
68.3
50.5
72.4
Octo (finetuned)
✓
78.9
85.7
84.6
51.1
75.1
OpenVLA (finetuned)
✓
84.7
88.4
79.2
53.7
76.5
π0.5 (finetuned)
✓
98.8
98.2
98.0
92.4
96.9
mimic-video (scratch)
✗
94.2
96.8
90.6
84.9
91.6
DreamZero-5B (scratch) [ 49 ]
✗
90.4
94.8
87.2
82.2
88.7
Table 1: Results on LIBERO. “finetuned” denotes models trained on large-scale external robot action data and then finetuned on the target benchmark; “scratch” denotes models trained with robot action data only from the target evaluation benchmark, without extra robot action data. Bold indicates the best result overall, and underline indicates the best result among “scratch” models. Ext. RA denotes external robot-action data (✓: yes; ✗: no).
Table 4
Method
PnP
Cube
Long
Towel folding
Table clearing
USB insertion
Diffusion Policy
29.2
10.0
2.5
23.3
6.7
0.0
OpenVLA
50.8
35.0
18.3
50.0
26.7
20.0
W2A
78.3
65.0
54.2
76.7
63.3
50.0
Table 3: Results on real-world tasks (% success rate).
Figure 4: Representative real-world task cases.
Figure 7
Figure 7: RGB visualizations of PCA-projected and raw teacher video latents across five temporal tokens, alongside the input image. PCA-projected latents show clearer spatial structure.
Method
Spatial
Object
Goal
Long
Avg.
w/o distill.
85.9 ( -10.4 )
87.7 ( -11.1 )
83.6 ( -11.4 )
80.5 ( -13.1 )
84.4 ( -11.5 )
w/o PCA
87.0 ( -9.3 )
92.3 ( -6.5 )
90.2 ( -4.8 )
84.8 ( -8.8 )
88.6 ( -7.3 )
Table 4: Ablation results on LIBERO, shown as SR drops (%) relative to W2A.
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 10
Figure 10: Effect of teacher latent extraction depth on LIBERO success rate.
Suite
w/o distill.
Shuffled Teacher
Random Teacher
Token0 Teacher
Current Feature
Future Feature
Spatial
85.9 ( -10.4 )
80.5 ( -15.8 )
72.5 ( -23.8 )
88.9 ( -7.4 )
83.3 ( -13.0 )
89.8 ( -6.5 )
Object
87.7 ( -11.1 )
81.6 ( -17.2 )
79.8 ( -19.0 )
91.0 ( -7.8 )
86.6 ( -12.2 )
92.5 ( -6.3 )
Goal
83.6 ( -11.4 )
77.7 ( -17.3 )
65.2 ( -29.8 )
84.4 ( -10.6 )
81.7 ( -13.3 )
86.4 ( -8.6 )
Long
80.5 ( -13.1 )
73.1 ( -20.5 )
59.3 ( -34.3 )
82.3 ( -11.3 )
77.1 ( -16.5 )
82.7 ( -10.9 )
Avg.
84.4 ( -11.5 )
78.2 ( -17.7 )
69.2 ( -26.7 )
86.7 ( -9.2 )
82.2 ( -13.7 )
87.8 ( -8.1 )
Appendix
Table 5: Ablation results on teacher signals on LIBERO, shown as success rate and performance drops (%) relative to W2A.
Suite
Global MLP
Grouped MLP
Spatial
83.7 ( -12.6 )
89.5 ( -6.8 )
Object
85.1 ( -13.7 )
90.4 ( -8.4 )
Goal
82.2 ( -12.8 )
87.2 ( -7.8 )
Long
79.8 ( -13.8 )
83.8 ( -9.8 )
Avg.
82.7 ( -13.2 )
87.7 ( -8.2 )
Appendix
Table 6: Ablation results about Q-Former on LIBERO, shown as performance drops (%) relative to W2A.
Component
Configuration
RGB observations
Front-view and wrist-view images
Image resolution
224×224
Vision encoder
ResNet-18 [ 14 ]
Observation history length
2
Action horizon
16
Executed horizon
8
Appendix
Table 7: Main model configuration of W2A.
Item
Configuration
Teacher latent
Output hidden state of the 14th video DiT block at τv=0.5
Student latent
Output hidden state of the 5th action DiT block
PCA dimension K
482
Teacher latent shape
[B,24,14,14,2048]
Teacher latent shape after PCA
[B,24,14,14,482]
Student latent shape
[B,16,768]
Appendix
Table 8: Latent distillation configuration.
Category
Method
Ext. RA
Generic video
Target action
Target video
Scratch
Diffusion Policy
✗
✗
✓
✗
Scratch
mimic-video
✗
✓
✓
✓
Scratch
DreamZero-5B
✗
✓
✓
✓
Scratch
W2A
✗
✓
✓
✓
Finetuned
Octo
✓
✗
✓
✗
Finetuned
OpenVLA
✓
✗
✓
✗
Appendix
Table 9: Training-data sources for representative methods. ✓ indicates use of the corresponding data source, and ✗ indicates no use.
Item
Configuration
Batch size
128
Training epochs
100
Optimizer
AdamW
Learning rate
1×10−4
Weight decay
1×10−4
Gradient clipping
1.0
Appendix
Table 10: Training configuration.
Item
Configuration
Number of flow steps
100
Flow-time sampling
Logit-normal
Inference denoising steps
20
Distillation weight λdist
1.0
Gaussian weighting center
τc=0.5
Gaussian weighting std.
σw=0.1
Appendix
Table 11: Flow matching and distillation settings.
Item
Configuration
Observation history
2 steps
Predicted action chunk
16 actions
Executed actions per prediction
8 actions
Inference denoising steps
20
Warmstart time
τws=0.5
Appendix
Table 12: Inference configuration.
Figure 11: Comparison of raw and PCA-projected video latents visualized as RGB feature maps.
Figure 12: Examples of task-relevant masks and action-video alignment maps used for world-prior analysis. SAM2 provides frame-level masks, and warmer colors denote higher latent similarity.
Method
Mask enrichment ↑
Mask-bg contrast ↑
Action DiT w/o distill.
0.98 ×
0.01
W2A
3.87 ×
1.34
Appendix
Table 13: Quantitative evidence of distilled task-relevant world priors. We measure whether action-video latent similarity is concentrated in task-relevant object regions. Mask enrichment is area-normalized, where a uniform similarity map has an expected value of 1.0× . Mask-bg contrast is computed on standardized similarity maps.
Method
Pos endpoint error (EPE) (px) ↓
t
t+5
t+10
Action DiT w/o distill.
6.4
13.7
23.8
W2A
5.9
8.2
13.1
Appendix
Table 14: Results of the linear probe test for predicting future gripper and object image-plane coordinates. Pos EPE is measured in pixels ( ↓ ).
Figure 13: Real-world hardware setup. We use a Franka Research 3 robot for policy execution and an Omega 7 haptic device for teleoperated demonstration collection.
Seed
LIBERO
CALVIN
Spatial
Object
Goal
Long
Avg. Len
0
95.4
99.0
91.6
91.0
4.34
5
98.6
98.2
97.4
94.2
4.26
42
95.0
99.2
96.0
95.6
4.30
Avg.
96.3 ± 1.97
98.8 ± 0.53
95.0 ± 3.03
93.6 ± 2.36
4.30 ± 0.04
Appendix
Table 15: Results across different random seeds on LIBERO and CALVIN.
Figure 14: Seed-wise simulation results. Error bars denote standard deviation over three seeds.
Task Type
Subtask
Diffusion Policy
OpenVLA
W2A
PnP
Pick up the carrot and place it in the basket
12/30
15/30
25/30
Pick up the banana and place it in the basket
6/30
15/30
28/30
Pick up the white bowl and place it on the plate
9/30
22/30
19/30
Pick up the blue cup and place it on the plate
8/30
9/30
22/30
Cube
Stack the green cube on top of the red cube
0/30
10/30
21/30
Stack the purple cube on top of the red cube
4/30
10/30
15/30
Appendix
Table 16: Per-subtask results on real-world tasks. Each entry is reported as the number of successful rollouts out of 30. The overall row reports the average success rate over all subtasks.
Figure 15: Additional real-world task cases.
Figure 16: Examples of compositional generalization in real-world evaluation.
Figure 17: Representative failure cases in real-world evaluation. W2A may fail when facing unseen tasks or objects, or when the initial scene configuration deviates substantially from the training distribution.