cs.ROOct 21, 2026
SaveAgentic World Action Models
Organizations: UC Berkeley · NVIDIA · University of Pennsylvania
Abstract
Recent world action models provide visual foresight and future prediction for robot manipulation, but lack explicit mechanisms for compositional skill reuse. Agentic robotics combines code-based planning and reasoning with reusable skills, but often lacks predictive grounding in physical dynamics. We introduce AW-0, an agentic world action model framework that integrates code-based planning with joint video and action prediction. We jointly model code, future video, and robot actions as three coupled generative streams within a unified mixture-of-transformers architecture. Within this architecture a code expert generates parameterized skill routines and control logic while video and action experts generate future visual states and low-level motor commands. Code serves two complementary roles: it structures multi-step prediction and control within the world model and represents skills as reusable programs for subsequent agentic execution. We further introduce an automated curation pipeline that labels robot and human video trajectories with aligned programs and consolidates recurring routines into reusable skill libraries. Evaluations on RoboTwin 2.0 and physical bimanual manipulation tasks show improved task performance over VLA and WAM baselines, alongside gains in language following, semantic reasoning and robustness, demonstrating that agentic code enables connection between planning, prediction and execution within WAMs.
Figures & tables
Figure 2: AW-0 Framework. AW-0 includes a Mixture-of-Transformers backbone with three experts generating code, video and action, respectively, which are denoised jointly, with video and action under continuous flow matching and the code under discrete masked diffusion.
Figure 3: Automatic Annotation Pipeline. Coding agents generate subtask-level programs using predefined primitive APIs and available skills, while a coordinator distills recurring patterns into a library of reusable skills across tasks. With the library frozen, we annotate the corpus by reusing cached code when contact and motion evidence match and generating new labels otherwise.
| YAM (parallel grippers) | Tianji–Sharpa (dexterous hands) | ||||||||
| Method | Arrange Fruits | Pack Backpack | Replace Lightbulb | Fold Towels | Fruits w/ Tong | Deal Poker | Flip Scoreboard | Solve Cube | Avg. |
| 70.8 | 49.7 | 63.8 | 81.7 | 7.3 | 13.1 | 26.3 | 36.1 | 43.6 | |
| DreamZero | 53.3 | 36.7 | 67.4 | 79.8 | 22.6 | 37.1 | 28.4 | 42.5 | 46.0 |
| EgoScale | 37.5 | 27.3 | 60.2 | 53.3 | 17.6 | 25.6 | 40.8 | 24.9 | 35.9 |
| WLA | 39.6 | 29.6 | 56.0 | 74.6 | 14.2 | 22.9 | 29.7 | 52.3 | 39.9 |
| AW-0 (ours) | 91.7 | 64.6 | 75.0 | 84.4 | 54.2 | 58.3 | 68.8 | 72.1 | 71.1 |
Table 2: Real-world Results on 8 Tasks across 2 Embodiments. Average rubric score (%) over rollouts/task (Appendix E ).
Figure 5: Impact of Code Expert and Attention Structure. Avg. rubric score (%) over rollouts/task. , , and denote the code, video, and action streams.
| Variant | Arrange Fruits | Pack Backpack | Replace Lightbulb | Fold Towels | Fruits w/ Tong | Deal Poker | Flip Scoreboard | Solve Cube | Avg. |
|---|---|---|---|---|---|---|---|---|---|
| ViT (ours) | 91.7 | 64.6 | 75.0 | 84.4 | 54.2 | 58.3 | 68.8 | 72.1 | 71.1 |
| VAE | 83.3 | 39.6 | 66.7 | 61.3 | 41.7 | 47.9 | 50.0 | 53.9 | 55.6 |
Table 3: Visual Conditioning for Code Expert. Avg. rubric score (%) over rollouts/task.
Figure 7: Impact of Code Label Content. Average rubric score (%) over rollouts per task.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
| YAM | Egocentric Human | Total | |
| Episodes | 73,101 | 175,541 | 248,642 |
| Durations (h) | 1,919.30 | 993.91 | 2,913.22 |
| Tasks | 279 | 327 | 606 |
| Total subtask slices | 986,314 | 1,089,059 | 2,075,373 |
| Mean subtask slices per episode | 13.5 | 6.2 | 8.3 |
Table 7: Pretraining Corpus Statistics.
| Skill | Definition |
|---|---|
| adjust_in_place | Grip an object where it lies, slide it a few centimetres without lifting, and let go. |
| alternating_smoothing_pats | Press the fabric flat with one arm and come off, then the other — the alternating pats of a smoothing slice. |
| approach_best_grasp | Sample grasp candidates on an object, pick the best one reachable by this arm, and drive the end-effector onto it. |
| brace_and_pull_open | Brace the body with one arm, take its lid, flap or handle with the other, draw it open, then release the brace. |
| brace_and_push_in | Grip the same part twice, low to keep it aligned and high to push, and drive it home into its receptacle. |
| flip_object_over | Lift a held flat object clear, roll the wrist a half turn, release it inverted, then grasp it again to pin it down. |
Table 8: The Frozen YAM Skill Library ( skills).
| Skill | Definition |
|---|---|
| align_and_seat_braced | One hand pins the target so it cannot shift while the other pushes the part home and lets go. |
| brace_and_pick | Hold the source still with one hand and pick the object off or out of it with the other. |
| brace_and_take_tool | Set up the two-handed working stance: the brace hand closes on the workpiece, the working hand on the tool. |
| close_closure | Swing a closure shut with both hands on it and come off it; nothing is held afterwards. |
| flatten_by_pulling_taut | Pin a slack flat item with one hand, drag it taut with the other, re-bite further along, and drag again. |
| flip_over_between_hands | Turn an object face-over by passing it between the hands through a half turn. |
Table 9: The Frozen Egocentric Human Skill Library ( skills).
Figure 20: Real-world Robot Setups. (a) The YAM station: two arms with linear grippers, with a RealSense D405 camera overhead and one on each wrist. The bottom camera is not used. (b) The Tianji–Sharpa station: a Tianji torso with two Sharpa Wave dexterous hands, observed by an SHW5G camera on the head and one on each wrist.
| TF loss | TF accuracy | Grounding gap | Parse rate | Entity F1 | Primitive F1 | Skill-call rate |
|---|---|---|---|---|---|---|
| 0.535 | 0.897 | 1.774 | 0.875 | 0.534 | 0.365 | 0.292 |
Table 10: Validation Metrics for the Pretrained Code Expert. TF denotes teacher-forced evaluation. The grounding gap is the absolute increase in code loss when visual observations are shuffled across validation samples.
Figure 29: Pretrained Code Generation on a YAM Validation Example. The generated program is compared with a reference program from another labeled episode of the same task.
Figure 30: Pretrained Code Generation on an Egocentric Human Validation Example. Under the instruction “Squeeze toothpaste,” the model generates the cap-removal subtask, matching the annotated program for the held-out window.
Figure 31: Online Code Generation during Real-world Deployment on Pack Backpack. The panel labeled “Annotated Program” shows a reference program from a separate demonstration of the same task and subtask. “Generated Program” is the model output recorded online during deployment. The comparison illustrates semantic correspondence in skill selection and arguments.
Figure 32: Online Code Generation during Real-world Deployment on Solve Rubik’s Cube . The panel labeled “Annotated Program” shows a reference program from a separate demonstration of the same task and subtask. “Generated Program” is the model output recorded online during deployment. The comparison illustrates semantic correspondence in skill selection and arguments.
Figure 33: Future-frame Prediction on Arrange Fruits for Three Models.
Figure 34: Future-frame Prediction on Deal Poker for Three Models.
Figure 35: Qualitative Example of Hybrid Actions for the “Fold and Stack Towels” Task. A single episode continues from the upper to the lower row, with green and purple indicating code execution and direct actions, respectively.
Figure 36: Failure Case Analysis. Each row shows one representative failure case. Frames are ordered from left to right by execution progress. The red box on the last frame marks the region enlarged in the rightmost panel, where the failure occurs.
| Task | EgoScale | DreamZero | WLA | AW-0 (ours) | AW-0 (hybrid) | |||||||
| Clean | Rand. | Clean | Rand. | Clean | Rand. | Clean | Rand. | Clean | Rand. | Clean | Rand. | |
| Adjust Bottle | 62 | 18 | 26 | 0 | 0 | 6 | 100 | 0 | 100 | 50 | 90 | 48 |
| Beat Block Hammer | 48 | 16 | 22 | 0 | 26 | 12 | 82 | 0 | 94 | 28 | 46 | 24 |
| Blocks Ranking RGB | 74 | 16 | 0 | 0 | 0 | 0 | 54 | 0 | 96 | 32 | 94 | 82 |
| Blocks Ranking Size | 32 | 0 | 0 | 0 | 16 | 0 | 32 | 0 | 56 | 0 | 64 | 50 |
| Click Alarmclock | 44 | 56 | 88 | 16 | 100 | 24 | 100 | 42 | 96 | 56 | 100 | 62 |
Table 11: Per-task RoboTwin 2.0 Results under the Clean and Random Evaluation Settings (50 Tasks). Success rate (%) over 50 rollouts per task per setting. Best result per task and setting is shown in bold.
| Share of sim. steps | ||||
|---|---|---|---|---|
| Setting | Episode set | Code exec | Direct action | Steps / ep. |
| Clean | All | 8.5% | 91.5% | 26,720 |
| Successful | 14.5% | 85.5% | 11,549 | |
| Failed | 6.3% | 93.7% | 52,687 | |
| Random | All | 5.6% | 94.4% | 41,716 |
| Successful | 35.0% | 65.0% | 8,400 | |
Table 12: Share of simulation steps driven by each control channel in the hybrid evaluation of RoboTwin tasks in Table 1 . Code exec denotes steps driven by executing the generated code from the code expert; Direct action denotes steps directly generated by the action expert.
Figure 37: Qualitative examples of hybrid actions for RoboTwin 2.0 tasks. Each row shows one successful episode from the hybrid evaluation in Table 1 . Top: blocks_ranking_rgb . Bottom: pick_dual_bottles .
Figure 38: Declaration-only APIs for parallel grippers and dexterous hands.