W2A: Distilling World Priors into Action Models for Real-Time Robotic Control
Organizations: Harbin Institute of Technology, Shenzhen
Abstract
World Action Models (WAMs) have recently shown great promise in robotic control, benefiting from the future imagination capability of large-scale pretrained video models. However, most existing WAMs require explicitly imagining future states at test time, incurring substantial computational overhead and latency that make them unsuitable for real-time robotic control. In this paper, we propose W2A, a two-stage distillation framework for real-time robotic manipulation that distills intermediate denoising representations from a video diffusion model into an action model during training, while entirely discarding the video model at test time. We hypothesize that although video latent representations are high-dimensional, a compressed low-dimensional subspace is sufficient to provide effective supervision for action generation. This makes it possible to transfer such information effectively into an action model. Extensive experiments in both simulation and real-world settings demonstrate competitive success rates while significantly improving inference efficiency. On LIBERO, W2A achieves a 95.9% average success rate. More importantly, W2A achieves about 21x faster inference than the representative WAM baseline.
Figures & tables
| Method | Ext. RA | Spatial (%) | Object (%) | Goal (%) | Long (%) | Average (%) |
|---|---|---|---|---|---|---|
| Diffusion Policy (scratch) [ 8 ] | ✗ | 78.3 | 92.5 | 68.3 | 50.5 | 72.4 |
| Octo (finetuned) | ✓ | 78.9 | 85.7 | 84.6 | 51.1 | 75.1 |
| OpenVLA (finetuned) | ✓ | 84.7 | 88.4 | 79.2 | 53.7 | 76.5 |
| (finetuned) | ✓ | 98.8 | 98.2 | 98.0 | 92.4 | 96.9 |
| mimic-video (scratch) | ✗ | 94.2 | 96.8 | 90.6 | 84.9 | 91.6 |
| DreamZero-5B (scratch) [ 49 ] | ✗ | 90.4 | 94.8 | 87.2 | 82.2 | 88.7 |
| Method | PnP | Cube | Long | Towel folding | Table clearing | USB insertion |
|---|---|---|---|---|---|---|
| Diffusion Policy | 29.2 | 10.0 | 2.5 | 23.3 | 6.7 | 0.0 |
| OpenVLA | 50.8 | 35.0 | 18.3 | 50.0 | 26.7 | 20.0 |
| W2A | 78.3 | 65.0 | 54.2 | 76.7 | 63.3 | 50.0 |
| Method | Spatial | Object | Goal | Long | Avg. |
|---|---|---|---|---|---|
| w/o distill. | 85.9 ( -10.4 ) | 87.7 ( -11.1 ) | 83.6 ( -11.4 ) | 80.5 ( -13.1 ) | 84.4 ( -11.5 ) |
| w/o PCA | 87.0 ( -9.3 ) | 92.3 ( -6.5 ) | 90.2 ( -4.8 ) | 84.8 ( -8.8 ) | 88.6 ( -7.3 ) |
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
| Suite | w/o distill. | Shuffled Teacher | Random Teacher | Token0 Teacher | Current Feature | Future Feature |
|---|---|---|---|---|---|---|
| Spatial | 85.9 ( -10.4 ) | 80.5 ( -15.8 ) | 72.5 ( -23.8 ) | 88.9 ( -7.4 ) | 83.3 ( -13.0 ) | 89.8 ( -6.5 ) |
| Object | 87.7 ( -11.1 ) | 81.6 ( -17.2 ) | 79.8 ( -19.0 ) | 91.0 ( -7.8 ) | 86.6 ( -12.2 ) | 92.5 ( -6.3 ) |
| Goal | 83.6 ( -11.4 ) | 77.7 ( -17.3 ) | 65.2 ( -29.8 ) | 84.4 ( -10.6 ) | 81.7 ( -13.3 ) | 86.4 ( -8.6 ) |
| Long | 80.5 ( -13.1 ) | 73.1 ( -20.5 ) | 59.3 ( -34.3 ) | 82.3 ( -11.3 ) | 77.1 ( -16.5 ) | 82.7 ( -10.9 ) |
| Avg. | 84.4 ( -11.5 ) | 78.2 ( -17.7 ) | 69.2 ( -26.7 ) | 86.7 ( -9.2 ) | 82.2 ( -13.7 ) | 87.8 ( -8.1 ) |
| Suite | Global MLP | Grouped MLP |
|---|---|---|
| Spatial | 83.7 ( -12.6 ) | 89.5 ( -6.8 ) |
| Object | 85.1 ( -13.7 ) | 90.4 ( -8.4 ) |
| Goal | 82.2 ( -12.8 ) | 87.2 ( -7.8 ) |
| Long | 79.8 ( -13.8 ) | 83.8 ( -9.8 ) |
| Avg. | 82.7 ( -13.2 ) | 87.7 ( -8.2 ) |
| Component | Configuration |
|---|---|
| RGB observations | Front-view and wrist-view images |
| Image resolution | |
| Vision encoder | ResNet-18 [ 14 ] |
| Observation history length | 2 |
| Action horizon | 16 |
| Executed horizon | 8 |
| Item | Configuration |
|---|---|
| Teacher latent | Output hidden state of the 14th video DiT block at |
| Student latent | Output hidden state of the 5th action DiT block |
| PCA dimension | 482 |
| Teacher latent shape | |
| Teacher latent shape after PCA | |
| Student latent shape |
| Category | Method | Ext. RA | Generic video | Target action | Target video |
|---|---|---|---|---|---|
| Scratch | Diffusion Policy | ✗ | ✗ | ✓ | ✗ |
| Scratch | mimic-video | ✗ | ✓ | ✓ | ✓ |
| Scratch | DreamZero-5B | ✗ | ✓ | ✓ | ✓ |
| Scratch | W2A | ✗ | ✓ | ✓ | ✓ |
| Finetuned | Octo | ✓ | ✗ | ✓ | ✗ |
| Finetuned | OpenVLA | ✓ | ✗ | ✓ | ✗ |
| Item | Configuration |
|---|---|
| Batch size | 128 |
| Training epochs | 100 |
| Optimizer | AdamW |
| Learning rate | |
| Weight decay | |
| Gradient clipping | 1.0 |
| Item | Configuration |
|---|---|
| Number of flow steps | 100 |
| Flow-time sampling | Logit-normal |
| Inference denoising steps | 20 |
| Distillation weight | 1.0 |
| Gaussian weighting center | |
| Gaussian weighting std. |
| Item | Configuration |
|---|---|
| Observation history | 2 steps |
| Predicted action chunk | 16 actions |
| Executed actions per prediction | 8 actions |
| Inference denoising steps | 20 |
| Warmstart time |
| Method | Mask enrichment | Mask-bg contrast |
|---|---|---|
| Action DiT w/o distill. | 0.98 | 0.01 |
| W2A | 3.87 | 1.34 |
| Method | Pos endpoint error (EPE) (px) | ||
|---|---|---|---|
| Action DiT w/o distill. | 6.4 | 13.7 | 23.8 |
| W2A | 5.9 | 8.2 | 13.1 |
| Seed | LIBERO | CALVIN | |||
|---|---|---|---|---|---|
| Spatial | Object | Goal | Long | Avg. Len | |
| 0 | 95.4 | 99.0 | 91.6 | 91.0 | 4.34 |
| 5 | 98.6 | 98.2 | 97.4 | 94.2 | 4.26 |
| 42 | 95.0 | 99.2 | 96.0 | 95.6 | 4.30 |
| Avg. | 96.3 1.97 | 98.8 0.53 | 95.0 3.03 | 93.6 2.36 | 4.30 0.04 |
| Task Type | Subtask | Diffusion Policy | OpenVLA | W2A |
| PnP | Pick up the carrot and place it in the basket | 12/30 | 15/30 | 25/30 |
| Pick up the banana and place it in the basket | 6/30 | 15/30 | 28/30 | |
| Pick up the white bowl and place it on the plate | 9/30 | 22/30 | 19/30 | |
| Pick up the blue cup and place it on the plate | 8/30 | 9/30 | 22/30 | |
| Cube | Stack the green cube on top of the red cube | 0/30 | 10/30 | 21/30 |
| Stack the purple cube on top of the red cube | 4/30 | 10/30 | 15/30 |
World Action Models
World Action Models connect predictions of how scenes change under action to executable policies for robots, vehicles, and other embodied agents.
See all 239 papers