Agentic World Action Models
Organizations: UC Berkeley · NVIDIA · University of Pennsylvania
Abstract
Recent world action models provide visual foresight and future prediction for robot manipulation, but lack explicit mechanisms for compositional skill reuse. Agentic robotics combines code-based planning and reasoning with reusable skills, but often lacks predictive grounding in physical dynamics. We introduce AW-0, an agentic world action model framework that integrates code-based planning with joint video and action prediction. We jointly model code, future video, and robot actions as three coupled generative streams within a unified mixture-of-transformers architecture. Within this architecture a code expert generates parameterized skill routines and control logic while video and action experts generate future visual states and low-level motor commands. Code serves two complementary roles: it structures multi-step prediction and control within the world model and represents skills as reusable programs for subsequent agentic execution. We further introduce an automated curation pipeline that labels robot and human video trajectories with aligned programs and consolidates recurring routines into reusable skill libraries. Evaluations on RoboTwin 2.0 and physical bimanual manipulation tasks show improved task performance over VLA and WAM baselines, alongside gains in language following, semantic reasoning and robustness, demonstrating that agentic code enables connection between planning, prediction and execution within WAMs.
Figures & tables
| YAM (parallel grippers) | Tianji–Sharpa (dexterous hands) | ||||||||
| Method | Arrange Fruits | Pack Backpack | Replace Lightbulb | Fold Towels | Fruits w/ Tong | Deal Poker | Flip Scoreboard | Solve Cube | Avg. |
| 70.8 | 49.7 | 63.8 | 81.7 | 7.3 | 13.1 | 26.3 | 36.1 | 43.6 | |
| DreamZero | 53.3 | 36.7 | 67.4 | 79.8 | 22.6 | 37.1 | 28.4 | 42.5 | 46.0 |
| EgoScale | 37.5 | 27.3 | 60.2 | 53.3 | 17.6 | 25.6 | 40.8 | 24.9 | 35.9 |
| WLA | 39.6 | 29.6 | 56.0 | 74.6 | 14.2 | 22.9 | 29.7 | 52.3 | 39.9 |
| AW-0 (ours) | 91.7 | 64.6 | 75.0 | 84.4 | 54.2 | 58.3 | 68.8 | 72.1 | 71.1 |
| Variant | Arrange Fruits | Pack Backpack | Replace Lightbulb | Fold Towels | Fruits w/ Tong | Deal Poker | Flip Scoreboard | Solve Cube | Avg. |
|---|---|---|---|---|---|---|---|---|---|
| ViT (ours) | 91.7 | 64.6 | 75.0 | 84.4 | 54.2 | 58.3 | 68.8 | 72.1 | 71.1 |
| VAE | 83.3 | 39.6 | 66.7 | 61.3 | 41.7 | 47.9 | 50.0 | 53.9 | 55.6 |
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
| YAM | Egocentric Human | Total | |
| Episodes | 73,101 | 175,541 | 248,642 |
| Durations (h) | 1,919.30 | 993.91 | 2,913.22 |
| Tasks | 279 | 327 | 606 |
| Total subtask slices | 986,314 | 1,089,059 | 2,075,373 |
| Mean subtask slices per episode | 13.5 | 6.2 | 8.3 |
| Skill | Definition |
|---|---|
| adjust_in_place | Grip an object where it lies, slide it a few centimetres without lifting, and let go. |
| alternating_smoothing_pats | Press the fabric flat with one arm and come off, then the other — the alternating pats of a smoothing slice. |
| approach_best_grasp | Sample grasp candidates on an object, pick the best one reachable by this arm, and drive the end-effector onto it. |
| brace_and_pull_open | Brace the body with one arm, take its lid, flap or handle with the other, draw it open, then release the brace. |
| brace_and_push_in | Grip the same part twice, low to keep it aligned and high to push, and drive it home into its receptacle. |
| flip_object_over | Lift a held flat object clear, roll the wrist a half turn, release it inverted, then grasp it again to pin it down. |
| Skill | Definition |
|---|---|
| align_and_seat_braced | One hand pins the target so it cannot shift while the other pushes the part home and lets go. |
| brace_and_pick | Hold the source still with one hand and pick the object off or out of it with the other. |
| brace_and_take_tool | Set up the two-handed working stance: the brace hand closes on the workpiece, the working hand on the tool. |
| close_closure | Swing a closure shut with both hands on it and come off it; nothing is held afterwards. |
| flatten_by_pulling_taut | Pin a slack flat item with one hand, drag it taut with the other, re-bite further along, and drag again. |
| flip_over_between_hands | Turn an object face-over by passing it between the hands through a half turn. |
| TF loss | TF accuracy | Grounding gap | Parse rate | Entity F1 | Primitive F1 | Skill-call rate |
|---|---|---|---|---|---|---|
| 0.535 | 0.897 | 1.774 | 0.875 | 0.534 | 0.365 | 0.292 |
| Task | EgoScale | DreamZero | WLA | AW-0 (ours) | AW-0 (hybrid) | |||||||
| Clean | Rand. | Clean | Rand. | Clean | Rand. | Clean | Rand. | Clean | Rand. | Clean | Rand. | |
| Adjust Bottle | 62 | 18 | 26 | 0 | 0 | 6 | 100 | 0 | 100 | 50 | 90 | 48 |
| Beat Block Hammer | 48 | 16 | 22 | 0 | 26 | 12 | 82 | 0 | 94 | 28 | 46 | 24 |
| Blocks Ranking RGB | 74 | 16 | 0 | 0 | 0 | 0 | 54 | 0 | 96 | 32 | 94 | 82 |
| Blocks Ranking Size | 32 | 0 | 0 | 0 | 16 | 0 | 32 | 0 | 56 | 0 | 64 | 50 |
| Click Alarmclock | 44 | 56 | 88 | 16 | 100 | 24 | 100 | 42 | 96 | 56 | 100 | 62 |
| Share of sim. steps | ||||
|---|---|---|---|---|
| Setting | Episode set | Code exec | Direct action | Steps / ep. |
| Clean | All | 8.5% | 91.5% | 26,720 |
| Successful | 14.5% | 85.5% | 11,549 | |
| Failed | 6.3% | 93.7% | 52,687 | |
| Random | All | 5.6% | 94.4% | 41,716 |
| Successful | 35.0% | 65.0% | 8,400 | |