World Action Models are Zero-shot Policies

Seonghyeon Ye; Yunhao Ge; Kaiyuan Zheng; Shenyuan Gao; Sihyun Yu; George Kurian; Suneel Indupuru; You Liang Tan

cs.ROFeb 17, 2026

World Action Models are Zero-shot Policies

Authors: Seonghyeon Ye, Yunhao Ge, Kaiyuan Zheng, Shenyuan Gao, Sihyun Yu, George Kurian, Suneel Indupuru, You Liang Tan,

Organizations: NVIDIA

Abstract

State-of-the-art Vision-Language-Action (VLA) models excel at semantic generalization but struggle to generalize to unseen physical motions in novel environments. We introduce DreamZero, a World Action Model (WAM) built upon a pretrained video diffusion backbone. Unlike VLAs, WAMs learn physical dynamics by predicting future world states and actions, using video as a dense representation of how the world evolves. By jointly modeling video and action, DreamZero learns diverse skills effectively from heterogeneous robot data without relying on repetitive demonstrations. This results in over 2x improvement in generalization to new tasks and environments compared to state-of-the-art VLAs in real robot experiments. Crucially, through model and system optimizations, we enable a 14B autoregressive video diffusion model to perform real-time closed-loop control at 7Hz. Finally, we demonstrate two forms of cross-embodiment transfer: video-only demonstrations from other robots or humans yield a relative improvement of over 42% on unseen task performance with just 10-20 minutes of data. More surprisingly, DreamZero enables few-shot embodiment adaptation, transferring to a new embodiment with only 30 minutes of play data while retaining zero-shot generalization.

Figures & tables

Figure 1: Overview . By jointly predicting video and action, World Action Models (WAMs) inherit world physics priors that enable 1) effective learning from diverse, non-repetitive data, 2) open-world generalization, 3) cross-embodiment learning from video-only data, and 4) few-shot adaptation to new robots.

Figure 2: Joint Video and Action Prediction . DreamZero jointly generates video and action. We observe that the predicted actions closely align with the generated video. The examples are from totally unseen tasks.

Figure 3: Free-form Evaluation. DreamZero performs a diverse range of tasks when conditioned on natural language instructions, including object manipulation, tool use, and human-robot interaction.

Figure 4: Model Architecture of DreamZero . The model takes three inputs: visual context (encoded via a VAE), language instructions (via a text encoder), and proprioceptive state (via a state encoder). These are processed by an autoregressive DiT backbone using flow matching, which jointly predicts future video frames and actions through separate decoders. During training (left), for each chunk, the model denoises noisy video and action latents conditioned on clean video context. During inference (right), predictions are executed asynchronously in the real world, and ground-truth observations are fed back into the KV cache to prevent error accumulation.

Figure 5: Decoupled Noise Schedules. DreamZero (blue) uses coupled noise for video and action (both uniform). DreamZero -Flash (red) biases video toward high-noise states via a Beta distribution while keeping action noise uniform, training the model to predict clean actions from noisy visual context.

Optimization	H100	GB200
Baseline	1 $\times$	1.1 $\times$
System-level
+ CFG Parallelism	1.9 $\times$	1.8 $\times$
+ DiT Caching	5.5 $\times$	5.4 $\times$
Implementation-level
+ Torch Compile + CUDA Graphs	8.9 $\times$	10.9 $\times$

Table 1: Cumulative inference speedups. Each row includes all optimizations above it. Entries marked “—” indicate features not applicable to that hardware.

Panel of Figure 6 — Figure 6: Distribution statistics for the AgiBot pretraining corpus: episode durations, subtask density, and skill coverage across 7.2K episodes ( $\sim$ 500 hours).

Figure 7: AgiBot Evaluation Set-up. We are first-citizens of generalization evals, where the default setting is unseen environment and unseen objects.

Figure 8: Seen Task Evaluation. DreamZero effectively learns from diverse data and generalizes to new environments, outperforming VLAs across all task categories. VLAs trained from scratch achieve near-zero success, while pretrained VLAs show modest performance—likely benefiting from embodiment-specific knowledge acquired through repetitive demonstrations during pretraining.

Figure 9: Zero-shot Generalization to Unseen Tasks. DreamZero achieves non-trivial task progress on 10 tasks absent from training, while VLAs struggle across both embodiments.

Figure 10: Posttraining Results . WAMs enable stronger post-training results across three tasks, indicating that environment generalization of DreamZero is retained after post-training.

Figure 11: Cross-Embodiment Transfer. We explore robot-to-robot (YAM $\rightarrow$ AgiBot) and human-to-robot embodiment transfer to unseen tasks.

Method	Task Progress
DreamZero	38.3% $\pm$ 7.6%
DreamZero + Human2Robot Transfer	54.3% $\pm$ 10.4%
DreamZero + Robot2Robot Transfer	55.4% $\pm$ 9.5%

Table 2: Cross-Embodiment Transfer Results. Average task progress on unseen tasks (

\pm

standard error). Both transfer settings improve over baseline (result from Table 9 ) using only 10–20 minutes of video-only demonstration data.

Figure 12: Few-shot Embodiment Adaptation. We explore few-shot embodiment adaptation by post-training on 30 minutes of new embodiment play data and evaluating on pick-and-place variants requiring strong language following.

Method	Denoising steps	Task Progress	Inference speed	$\times$ Speed up
DreamZero	4	83% $\pm$ 6.1%	350ms	1
DreamZero	1	52% $\pm$ 10.2%	150ms	2.33 $\times$
DreamZero -Flash	1	74% $\pm$ 10.1%	150ms	2.33 $\times$

Table 3: DreamZero -Flash Evaluation. Task progress on table bussing with varying denoising steps (

\pm

standard error). DreamZero -Flash recovers most of the 4-step performance using only 1 denoising step.

	Architecture	Model Size	Data	Task Progress
Q1. Data Diversity
	DreamZero (AR)	14B	Repetitive	33% $\pm$ 4.2%
	DreamZero (AR)	14B	Diverse	50% $\pm$ 6.3%
Q2. Model Scale
	DreamZero (AR)	5B	Diverse	21% $\pm$ 4.2%
	DreamZero (AR)	14B	Diverse	50% $\pm$ 6.3%

Table 4: Model and Data Ablations. Task progress on PnP Easy tasks (

\pm

standard error). AR = autoregressive, BD = bidirectional. All models trained with 50K steps and batch size 32.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 13: Bidirectional vs. Autoregressive WAMs. When the sampling point falls mid-task (T=20), bidirectional WAMs must subsample video to align with the language caption, distorting native FPS and degrading video-action alignment. Autoregressive WAMs avoid this trade-off by conditioning on video context, preserving both language-video correspondence and native frame rate.

Figure 14: Attention startegy of DreamZero . (a) QKV Self-Attention mask for DreamZero training. Y-axis shows the Query (Q) and X-axis shows the Key/Value (KV). Given conditioning frames (C0, C1, C2), we train the model to predict the velocities of next frames (Z1, Z2, Z3) and actions (Y1, Y2, Y3). (b) During inference, we compute the KV-cache of conditional frames and concatenate them to predict the action and frames. For example, Y3 (action) is able to attend to C0, C1, and C2, taking into account previous visual observations as history to predict the current actions during both training and inference. Note that the C0, C1, and C2 during inference is replaced with the GT observations.

Figure 19

Figure 15: Data Collection Environments. We collect teleoperation data across 22 diverse real-world environments, including offices, laboratories, restaurants, supermarkets, coffee shops, warehouses, homes, hotels, and retail stores. This diversity enables DreamZero to generalize to unseen environments without task-specific fine-tuning.

Category	#	Task	Image	Instruction	Image	Instruction	Image	Instruction	Image	Instruction
	1	PnP Fruit		The left arm picks up the Banana on the table and places it in the Blue Plate.		The left arm picks up the lime on the table and places it on the light green plate.		The left arm picks up the peach on the table and places it on the baking pan.		The left arm picks up the green pear on the table and places it on the blue checkered bowl.
PnP Easy	2	Taking out Fruit		The left arm picks up the Mango from the plastic bag and places it into the Wooden Basket		The left arm picks up the yellow pear from the plastic bag and places it onto the blue tray.		The left arm picks up the purple grapes from the plastic bag and places it into the brown basket.		The left arm picks up the watermelon from the plastic bag and places it onto the Blue Plate.
	3	Wipe the Mess		The left arm uses a sponge to wipe the coffee spill off the table.		The left arm used a black cloth to wipe the white powder off the table.		The left arm uses a napkin to wipe the creamer spill off the table.		The left arm used a paper towel to wipe the water off the table.
	4	PnP Fork/Spoon		The left arm picks up the Pink Fork from the table and places it onto the blue plate.		The left arm picks up the orange fork from the table and places it into the glass cup		The left arm picks up the light blue fork from the table and places it onto the orange plate		The left arm picks up the green fork from the table and places it into the blue plate.
PnP Hard	5	Put Pen in Holder		The left arm picks up the Red Marker pen from the table and placed it into the pen holder.		The left arm picked up the black marker from the table and placed it into the pen holder.		The left arm picked up the white marker pen from the table and placed it into the pen holder.		The left arm picked up the mechanical pencil from the table and placed it into the pen holder.
	6	Put Cup on Coaster		The left arm picks up the clear cup from the table and places it on the grey coaster.		The left arm picks up the plastic cup from the table and places it on the blue coaster.		The left arm picks up the plastic cup from the table and places it on the white coaster.		The left arm picks up the pink cup from the table and places it on the gray coaster.

Appendix

Table 5: Seen Tasks Evaluation Setup for AgiBot G1

#	Task	Image	Instruction	Image	Instruction	Image	Instruction	Image	Instruction
1	Untie Shoelaces		The robot coordinates both arms to simultaneously grasp the two loops of the shoelace. It then moves both arms outward in synchronized, opposing directions until the shoelace is fully untied.		The robot reaches its left arm to grasp the blue box and hold it steady. It then reaches its right arm toward the blue ribbon and moves it to untie the knot.		The robot coordinates both arms to simultaneously grasp the two loops of the knot of the box. It then moves both arms outward in synchronized, opposing directions until the knot of the box is fully untied.		The robot coordinates both arms to simultaneously grasp the two loops of the knot of the package. It then moves both arms outward in synchronized, opposing directions until the knot of the package is fully untied.
2	Remove /Put Hat		The robot reaches its right arm to gasp the crown and lifts it to remove it from the mannequin’s head.		The robot reaches its left arm to grasp the hat and lifts it to remove it from the mannequin’s head.		The robot reaches its left arm to grasp the hat and lifts it to remove it from the mannequin’s head.		The robot reaches its left arm to grasp the hat and lifts it to remove it from the mannequin’s head.
3	Draw Circle		The robot reaches its left arm to pick up the red marker and the left arm moves the marker to draw a circle on the book.		The robot reaches its right arm to pick up the marker and the right arm draws a circle on the whiteboard with the marker.		The robot reaches its left arm to pick up the black marker and the left arm moves the marker to draw a circle on the paper		The robot reaches its right arm to pick up the marker and the right arm moves the marker to draw a llne on the whiteboard.
4	Take out Straw		The left arm holds the cup on the table. Then the right arm pulls the straw out of the cup.		The left arm holds the cup on the table. Then the right arm pulls the straw out of the cup.		The left arm holds the cup on the table. Then the right arm pulls the straw out of the cup.		The right arm holds the cup on the table.Then the left arm pulls the straw out of the cup.
5	Cube Stacking		The robot reaches its right arm to pick up the green cube, moves it over the red cube, and releases it to stack. It then reaches its right arm to pick up the yellow cube, moves it over the stack, and releases it onto the green cube to finish the task.		The robot reaches its right arm to pick up the red cube, moves it over the colorful cube, and releases it to stack. It then reaches its right arm to pick up the green cube, moves it over the red cube, and releases it to finish the tower.		The robot reaches its right arm to pick up the white cube, moves it over the green cube, and releases it to stack. It then reaches its left arm to pick up the blue cube, moves it over the stack, and releases it onto the white cube to finish the task.		The robot reaches its right arm to pick up the red cube, moves it over the blue cube, and releases it to begin the stack. It then reaches its left arm to pick up the orange cube, moves it over the stack, and releases it onto the red cube to complete the three-tier structure.
6	Painting		The left arm grabs the brush. Then left arm paints with the brush on the notebook		The right arm grabs the brush. Then right arm paints with the brush on the paper		The left arm grabs the brush. Then left arm paints with the brush on the notebook		The right arm grabs the brush. Then right arm paints with the brush on the notebook

Appendix

Table 6: Unseen Tasks Evaluation Setup for AgiBot G1

Figure 16: Illustration of generated and executed pair. We illustrate the generated video and action execution pair. These two examples show scenarios where the video prediction failed and the robot followed the failed plan. DreamZero failed to generate video of the Agibot G1 robot drawing in the whiteboard and failed to generate video of the Franka robot opening the oven first.

World Action Models

World Action Models connect predictions of how scenes change under action to executable policies for robots, vehicles, and other embodied agents.

See all 210 papers

May 12, 2026

Reinforcing VLAs in Task-Agnostic World Models

Post-training Vision-Language-Action (VLA) models via reinforcement learning (RL) in learned world models has emerged as an effective strategy to adapt to new tasks without costly real-world interactions. However, while using imagined trajectories reduces the sample complexity of policy training, existing methods still heavily rely on task-specific data to fine-tune both the world and reward models, fundamentally limiting their scalability to unseen tasks. To overcome this, we argue that world and reward models should capture transferable physical priors that enable zero-shot inference. We propose RAW-Dream (Reinforcing VLAs in task-Agnostic World Dreams), a new paradigm that completely disentangles world model learning from downstream task dependencies. RAW-Dream utilizes a world model pre-trained on diverse task-free behaviors for predicting future rollouts, and an off-the-shelf Vision-Language Model (VLM) for reward generation. Because both components are task-agnostic, VLAs can be readily finetuned for any new task entirely within this zero-shot imagination. Furthermore, to mitigate world model hallucinations, we introduce a dual-noise verification mechanism to filter out unreliable rollouts. Extensive experiments across simulation and real-world settings demonstrate consistent performance gains, proving that generalized physical priors can effectively substitute for costly task-dependent data, offering a highly scalable roadmap for VLA adaptation.

Yucen Wang, Rui Yu, Fengming Zhang +5

Nanjing University · Interns at Microsoft · University of Illinois at Urbana-Champaign +4

World Model Learning World Models Vision-Language-Action Models VLMs for Robotics RL for VLMs

Ask PDF

Jul 30, 2026

World Action Planner: Generalizable Decision-Making with Action-Conditioned World Models

Building generalizable agents for diverse applications remains a fundamental challenge. While imitation learning-based policies succeed in specific training environments, they often fail to generalize to novel scenes and tasks. In this work, we propose World Action Planner, a robot planning system that leverages the reasoning capabilities of Vision-Language Models (VLMs) and the physical grounding of a multi-task pose-image conditioned world model. Our system enables an agent to propose initial action plans and iteratively refine them via optimization and search, reasoning over imagined world model rollouts. We demonstrate that our approach achieves superior performance across compositional tasks, new layouts, and zero-shot generalization scenarios, significantly outperforming state-of-the-art end-to-end policy models such as VLAs and WAMs. Project website at worldactionplanner.github.io

Xiangcheng Zhang, Yilun Du

Harvard University

World Models for Robotics World Model-Based Planning Action-Conditioned World Models Robot Policy Generalization World Models for VLA Models

Ask PDF

Mar 11, 2026

World2Act: Latent Action Post-Training from World Model Dynamics

World Models (WMs) offer a promising mechanism for post-training Vision-Language-Action (VLA) policies by providing dynamics priors that improve generalization under task and scene variation. However, most WM-based post-training methods rely on pixel-space supervision, making policies sensitive to visual artifacts introduced by imperfect WM rollouts. We present World2Act, a latent-space post-training framework that transfers WM dynamics to the VLA policy without pixel-space supervision. World2Act operates in two stages: 1) it induces a shared video-action latent space by contrastively aligning WM-dynamics latents with action embeddings, and 2) it post-trains the VLA by guiding policy action representations toward WM-imagined dynamics rather than decoded pixels. Built on GR00T-N1.6, World2Act delivers absolute success-rate gains of up to +2.5% on simulation benchmarks (RoboCasa, LIBERO, Bridge-SIMPLER) and +6.7% on a real robot over finetuned VLA baselines. Notably, it outperforms pixel-space WM supervision by up to +6.0%, including on LIBERO where pixel supervision degrades the baseline, suggesting that latent WM dynamics offer a more stable WM-based post-training alternative to pixel-space transfer.

An Dinh Vuong, Tuan Van Vo, Abdullah Sohail +6

MBZUAI

Vision-Language-Action Models World Models for VLA Models VLMs for Robotics Latent Action Learning

Ask PDF

Abstract

World Action Models

Explore similar work

Reinforcing VLAs in Task-Agnostic World Models

World Action Planner: Generalizable Decision-Making with Action-Conditioned World Models

World2Act: Latent Action Post-Training from World Model Dynamics