What Limits Cross-Task Generalization in World Action Models?
Organizations: Renmin University of China, Beijing 100872, China · Key Laboratory of Data Engineering and KnowledgeEngineering, MOE, China · Meituan, Beijing 100102, China
Abstract
Cross-task generalization remains difficult for world action models (WAMs), even when their predicted futures depict plausible manipulation. This gap can be attributed directly: we make this gap directly testable by decoupling a WAM into a world model and an inverse-dynamics model (IDM), then holding the IDM fixed while replacing the source of future images. We introduce Align Once, Evaluate Everywhere (AoE), a camera-space inverse-dynamics model pretrained on large-scale robot data, which provides this recovery interface for different future sources across tasks and environments. Through it, we find that actions recovered from recorded futures of unseen tasks execute almost as well as replayed demonstrations, whereas futures predicted by world models collapse across benchmarks. The failures are not generic: they concentrate on the gripper and interaction geometry while generation capacity is spent on task-irrelevant background. This diagnosis suggests that supervision should target exactly these regions. We therefore introduce the vision-to-action (VITA) loss, which passes generated futures through the frozen AoE and penalizes discrepancies in the recovered actions, redirecting generation toward action-relevant detail. Across two world models and three manipulation benchmarks, this simple change raises the execution success of generated futures from near zero to 47.0%. The same loss also transfers to WAMs and yields consistent gains in native control.
Figures & tables
| LIBERO (%) | RoboTwin (%) | |||||||||
| Model | IDM action | Source | Inference | Spatial | Object | Goal | Long | Avg. | Easy | Hard |
| Dataset action replay | Replay | – | 80.4 | 88.8 | 62.0 | 57.8 | 72.3 | 66.7 | 70.0 | |
| Dataset image replay + AoE | ✓ | Replay | – | 71.6 | 74.0 | 53.0 | 29.2 | 57.0 | 65.3 | 60.7 |
| Cosmos + AoE | ✓ | Zero-shot | pass@1 | 0.4 | 0.6 | 0.4 | 0.0 | 0.4 | 1.6 | 1.1 |
| ✓ | Zero-shot | pass@50 | 5.2 | 3.8 | 4.0 | 10.6 | 5.9 | 5.3 | 4.7 | |
| WAN2.2 + AoE | ✓ | Zero-shot | pass@1 | 0.2 | 0.8 | 0.2 | 0.0 | 0.3 | 1.9 | 1.3 |
| Model | IDM action | Source | Inference | Background | Light | Language | Noise | Layout | Camera | Robot Init. | Overall |
| Dataset action replay | Replay | – | 71.9 | 75.3 | 78.0 | 83.8 | 78.8 | 67.7 | – | 76.2 | |
| Dataset image replay + AoE | ✓ | Replay | – | 60.7 | 60.3 | 59.6 | 0.0 | 53.6 | 57.0 | – | 47.0 |
| Cosmos + AoE | ✓ | Zero-shot | pass@1 | 1.1 | 0.4 | 0.5 | 0.0 | 0.1 | 0.4 | 0.3 | 0.4 |
| ✓ | Zero-shot | pass@50 | 8.7 | 2.8 | 4.2 | 0.2 | 2.7 | 5.4 | 2.1 | 3.5 | |
| WAN2.2 + AoE | ✓ | Zero-shot | pass@1 | 0.9 | 0.5 | 0.5 | 0.0 | 0.5 | 0.7 | 0.3 | 0.5 |
| ✓ | Zero-shot | pass@50 | 5.7 | 3.8 | 5.1 | 0.4 | 3.0 | 6.4 | 2.5 | 3.8 |
| LIBERO (%) | RoboTwin (%) | ||||||||
| Image source | IDM action | Source | Spatial | Object | Goal | Long | Avg. | Easy | Hard |
| Dataset image | ✓ | Replay | 71.6 | 74.0 | 53.0 | 29.2 | 57.0 | 65.3 | 60.7 |
| WAN2.2 | ✓ | Zero-shot | 0.2 | 0.8 | 0.2 | 0.0 | 0.3 | 1.9 | 1.3 |
| Cosmos | ✓ | Zero-shot | 0.4 | 0.6 | 0.4 | 0.0 | 0.4 | 1.6 | 1.1 |
| WAN2.2 + VITA | ✓ | Zero-shot | 64.4 (+64.2) | 65.2 (+64.4) | 36.8 (+36.6) | 21.4 (+21.4) | 47.0 (+46.7) | 59.6 (+57.7) | 40.4 (+39.1) |
| Cosmos + VITA | ✓ | Zero-shot | 63.6 (+63.2) | 79.2 (+78.6) | 40.6 (+40.2) | 23.2 (+23.2) | 51.7 (+51.3) | 57.7 (+56.1) | 36.4 (+35.3) |
| Image source | IDM action | Source | Background | Light | Language | Noise | Layout | Camera | Robot Init. | Overall |
|---|---|---|---|---|---|---|---|---|---|---|
| Dataset image | ✓ | Replay | 60.7 | 60.3 | 59.6 | 0.0 | 53.6 | 57.0 | – | 47.0 |
| WAN2.2 | ✓ | Zero-shot | 0.9 | 0.5 | 0.5 | 0.0 | 0.5 | 0.7 | 0.3 | 0.5 |
| Cosmos | ✓ | Zero-shot | 1.1 | 0.4 | 0.5 | 0.0 | 0.1 | 0.4 | 0.3 | 0.4 |
| WAN2.2 + VITA | ✓ | Zero-shot | 38.5 (+37.6) | 45.3 (+44.8) | 37.1 (+36.6) | 0.1 (+0.1) | 39.7 (+39.2) | 26.6 (+25.9) | 13.4 (+13.1) | 29.9 (+29.4) |
| Cosmos + VITA | ✓ | Zero-shot | 40.6 (+39.5) | 48.2 (+47.8) | 34.1 (+33.6) | 0.0 (+0.0) | 33.8 (+33.7) | 33.8 (+33.4) | 28.6 (+28.3) | 30.3 (+29.9) |
| LIBERO | RoboTwin: C+R C+R | |||||||
|---|---|---|---|---|---|---|---|---|
| Method | Long | Goal | Object | Spatial | Avg. | Easy | Hard | Avg. |
| FastWAM | 95.2 | 97.0 | 100.0 | 98.2 | 97.6 | 86.4 | 88.2 | 87.3 |
| LingBot-VA | 98.5 | 97.2 | 99.6 | 98.5 | 98.5 | 85.6 | 84.2 | 84.9 |
| FastWAM + VITA | 97.2 (+2.0) | 98.8 (+1.8) | 99.6 (-0.4) | 99.0 (+0.8) | 98.7 (+1.1) | 90.7 (+4.3) | 91.9 (+3.7) | 91.3 (+4.0) |
| LingBot-VA + VITA | 98.8 (+0.3) | 98.4 (+1.2) | 99.8 (+0.2) | 99.0 (+0.5) | 99.0 (+0.5) | 92.4 (+6.8) | 91.2 (+7.0) | 91.8 (+6.9) |
| LIBERO-Plus | RoboTwin: C C+R | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Method | Background | Light | Language | Noise | Layout | Robot Init. | Camera | Overall | Easy | Hard | Avg. |
| FastWAM | 53.7 | 78.2 | 68.9 | 37.7 | 60.7 | 44.5 | 16.4 | 51.5 | 69.5 | 9.3 | 39.4 |
| LingBot-VA | 56.3 | 46.5 | 61.7 | 11.4 | 34.8 | 36.2 | 9.5 | 35.1 | 74.3 | 19.2 | 46.8 |
| FastWAM + VITA | 74.6 (+20.9) | 80.5 (+2.3) | 75.6 (+6.7) | 27.4 (-10.3) | 80.9 (+20.2) | 53.4 (+8.9) | 14.2 (-2.2) | 55.9 (+4.4) | 73.8 (+4.3) | 17.5 (+8.2) | 45.7 (+6.3) |
| LingBot-VA + VITA | 77.9 (+21.6) | 71.2 (+24.7) | 69.9 (+8.2) | 15.0 (+3.6) | 51.2 (+16.4) | 49.3 (+13.1) | 12.7 (+3.2) | 47.2 (+12.1) | 82.5 (+8.2) | 31.6 (+12.4) | 57.1 (+10.3) |
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
| Data source | Duration (h) | Type |
|---|---|---|
| DROID-all-aligned | 164.0 | Real-robot |
| AgibotWorld-Beta-all | 2174.0 | Real-robot |
| LET-all-aligned | 171.0 | Real-robot |
| Xingzhi | 260.0 | Real-robot |
| Bodeng-Xinghai | 1431.0 | Real-robot |
| Songling | 957.0 | Real-robot |
| Category | Setting |
|---|---|
| Hardware | 64 H800 GPUs (8 nodes 8 GPUs) |
| Precision | BF16 |
| Per-GPU batch size | 16 |
| Global batch size | 1024 |
| Training steps | 600K |
| Warmup steps | 3K |
| Component | Cosmos + VITA | Cosmos |
|---|---|---|
| Hardware | 4 nodes 8 H800 GPUs | 4 nodes 8 H800 GPUs |
| IDM supervision | Frozen AoE teacher | None |
| Dataloader batch size | 4 | 4 |
| Video resolution | ||
| Temporal window | 93 continuous frames | 93 continuous frames |
| Window stride | 92 | 92 |
| Category | Setting |
|---|---|
| Task split | 25 training tasks and 25 held-out tasks |
| Cross-validation | Two folds, with the two task groups exchanged between training and evaluation |
| Number of task–configuration pairs | 100 |
| Replay procedure | AoE predicts actions from image pairs, followed by RoboTwin joint-position replay |
| IDM replay checkpoint | No additional training for the replay IDM |
| LingBot-VA | FastWAM | |||
| Setting | LIBERO | RoboTwin | LIBERO | RoboTwin |
| Hardware | 32 H800 | 32 H800 | 32 H800 | 32 H800 |
| Micro batch | 4 | 4 | 8 | 8 |
| Gradient accumulation | 2 | 2 | 2 | 2 |
| Global batch | 256 | 256 | 512 | 512 |
| Training steps | 5,000 | 50,000 | 5,000 | 50,000 |
| ID | Group 1 | ID | Group 2 |
|---|---|---|---|
| 1 | adjust_bottle | 1 | blocks_ranking_size |
| 2 | beat_block_hammer | 2 | dump_bin_bigbin |
| 3 | blocks_ranking_rgb | 3 | handover_mic |
| 4 | click_alarmclock | 4 | lift_pot |
| 5 | move_can_pot | 5 | click_bell |
| 6 | grab_roller | 6 | move_pillbottle_pad |
| Input and recovery path | Success (%) |
|---|---|
| Dataset action replay | 100.0 |
| GT camera actions | 98.0 |
| Dataset image replay + AoE | 93.7 |
| VAE-reconstructed dataset image + AoE | 91.3 |
| Generated image + AoE | 2.0 |
| Generated image + geometry | 3.0 |
| Image source | Source | Inference | Plausible | Looks done | Execute | Execute P | Looks done Fail |
|---|---|---|---|---|---|---|---|
| Dataset image | Replay | – | 97.0 | 94.5 | 57.5 | 58.8 | 90.6 |
| Cosmos | Zero-shot | pass@1 | 77.0 | 67.5 | 0.5 | 0.6 | 67.3 |
| WAN2.2 | Zero-shot | pass@1 | 75.0 | 65.0 | 0.5 | 0.7 | 64.8 |
| Cosmos + VITA | Zero-shot | pass@1 | 89.5 | 81.0 | 52.0 | 56.4 | 65.6 |
| WAN2.2 + VITA | Zero-shot | pass@1 | 88.0 | 77.5 | 47.0 | 51.7 | 61.3 |
| Offline score | Within-task AUROC | Spearman |
|---|---|---|
| SSIM | 0.55 | 0.08 |
| DINO feature similarity | 0.61 | 0.17 |
| Negative LPIPS | 0.60 | 0.15 |
| AoE action agreement | 0.79 | 0.49 |
| Geometric consistency | 0.82 | 0.55 |
| Model | IDM action | Source | Inference | Success (%) |
|---|---|---|---|---|
| Cosmos + AoE | ✓ | Zero-shot | pass@1 | 0.35 |
| ✓ | Zero-shot | pass@5 | 1.45 | |
| ✓ | Zero-shot | pass@10 | 2.35 | |
| ✓ | Zero-shot | pass@25 | 4.25 | |
| ✓ | Zero-shot | pass@50 | 5.90 | |
| WAN2.2 + AoE | ✓ | Zero-shot | pass@1 | 0.30 |
| Model | Target demonstrations | Inference | Success (%) |
|---|---|---|---|
| Cosmos | 0 | pass@1 | 3.6 |
| 5 | pass@1 | 18.2 | |
| 10 | pass@1 | 29.6 | |
| 25 | pass@1 | 43.4 | |
| 50 | pass@1 | 54.2 | |
| 100 | pass@1 | 55.6 |
| Training objective | Held-out tasks (%) | LIBERO-Plus (%) |
|---|---|---|
| Native baseline | 11.2 | 51.5 |
| Matched-budget native | 11.5 | 50.9 |
| Future DINO alignment | 11.8 | 54.0 |
| AoE smoothness | 14.5 | 51.5 |
| VITA (Match + GT + ROI) | 23.6 | 55.9 |
| Match + GT | 24.1 | 56.4 |
| RoboTwin: C+R C+R | RoboTwin: C C+R | ||||
|---|---|---|---|---|---|
| Method | Easy | Hard | Avg. | Easy | Hard |
| FastWAM | 86.4 | 88.2 | 87.3 | 69.5 | 9.3 |
| FastWAM + VITA | 90.7 | 91.9 | 91.3 | 73.8 | 17.5 |
| LingBot-VA | 85.6 | 84.2 | 84.9 | 74.3 | 19.2 |
| LingBot-VA + VITA | 92.4 | 91.2 | 91.8 | 82.5 | 31.6 |
| LIBERO-Plus | RoboTwin | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| IDM evaluator | IDM zero-shot | Background | Light | Language | Noise | Layout | Camera | Robot Init. | Overall | Easy | Hard |
| WAN2.2 + AoE | ✓ | 38.5 | 45.3 | 37.1 | 0.1 | 39.7 | 26.6 | 13.4 | 29.9 | 59.6 | 40.4 |
| WAN2.2 + J-IDM | 19.5 | 31.6 | 42.8 | 0.0 | 39.2 | 27.7 | – | 26.8 | 65.5 | 66.7 | |
| WAN2.2 + DreamGen-IDM | 39.4 | 46.1 | 36.2 | 9.4 | 35.2 | 26.1 | – | 30.8 | 61.9 | 57.8 | |
| Replay setting | Background | Light | Language | Noise | Layout | Camera | Robot Init. | Overall |
|---|---|---|---|---|---|---|---|---|
| Dataset action replay (Oracle) | 71.9 | 75.3 | 78.0 | 83.8 | 78.8 | 67.7 | – | 76.2 |
| Dataset image replay + AoE | 60.7 | 60.3 | 59.6 | 0.0 | 53.6 | 57.0 | – | 47.0 |
| Dataset image replay + J-IDM | 65.6 | 74.5 | 66.1 | 0.9 | 60.1 | 75.4 | – | 55.5 |
| Dataset image replay + DreamGen-IDM | 66.4 | 80.0 | 71.9 | 3.6 | 67.2 | 52.2 | – | 54.8 |
| IDM evaluator | IDM zero-shot | Background | Light | Language | Noise | Layout | Camera | Robot Init. | Overall |
|---|---|---|---|---|---|---|---|---|---|
| WAN2.2 + AoE | 0.9 | 0.5 | 0.5 | 0.0 | 0.5 | 0.7 | 0.3 | 0.5 | |
| WAN2.2 + J-IDM | 5.5 | 1.4 | 2.4 | 0.4 | 1.1 | 0.9 | – | 1.8 | |
| WAN2.2 + DreamGen-IDM | 2.7 | 0.7 | 0.9 | 0.1 | 0.5 | 1.2 | – | 0.9 |