We introduce Hydra-0, a generalist world model conditioned on action flow, which represents robot actions as pixel motion. This shared visual interface enables generalist world modeling and control by learning action consequences across embodiments, tasks, environments, and video-generation backbones. Our best configuration achieves 90.4% lower robot-motion error and 60.2% lower object-motion error than our action-conditioned baseline, while supporting zero-shot composition and data-efficient adaptation. On the RoboLab benchmark, Hydra-0 achieves a Pearson correlation of r=0.96 between replayed and reference success rates. Finally, we uncover an emergent inverse mode of this interface: a world action model that predicts compatible robot motion from desired object flow transferred from a human demonstration. A trained action head maps the resulting latent features to executable actions without requiring task-specific expert robot demonstrations. Together, these results demonstrate the potential of action flow as a shared control interface connecting heterogeneous training data, open-loop policy evaluation, and robot control.
Figures & tables
Figure 1: Overview of Hydra-0. During offline training (top) , we recover action flow from interaction videos by tracking both the embodiment and the manipulated object , as detailed in Section 2.1 . The future video is encoded to construct the flow-matching training target. During online deployment (bottom) , Isaac Lab executes a candidate command sequence a0:H−1 through the robot controller and physics simulation, producing robot link transforms over the prediction horizon. Using these transforms, we propagate visible robot-surface points and project them through the calibrated camera following Equation 1 , producing embodiment action flow that conditions the causal autoregressive model to predict the command sequence’s real-world consequence.
Figure 2: Action Flow Construction and Sampling. Dataset videos without geometric information are tracked densely and grounded into embodiment and manipulated-object masks. During training, we sample an input condition from four conditioning modes: Embodiment selects acting-body tracks as the primary action condition; Object supplies task or desired-motion flow; All provides a grounding fallback; and None provides conditioning dropout.
Figure 3: Action Flow as Visual Conditioning. Action-flow trajectories propagate first-frame latent features through time to form the motion-aware visual condition Cmotion . This condition is concatenated with the noisy video latent Zt before DiT patch embedding.
Windows
Dataset
Embodiment
Episodes
before
after
Hours
License
DROID [ 26 ]
single arm
70,339
494,680
223,075
313.7
CC-BY-4.0
ABC-130k [ 3 ]
bimanual arms
54,961
1,060,456
1,048,681
1,474.7
Apache-2.0
MolmoAct2 [ 14 ]
bimanual arms
7,723
129,589
126,335
177.7
Apache-2.0
EgoDex [ 23 ]
human hands
41,888
89,380
89,380
125.7
CC-BY-NC-ND-4.0
Deform360 [ 30 ]
handheld grippers
1,714
87,224
60,315
84.8
MIT
Table 1: Multi-Embodiment Training Corpus. Episodes and windows are counted on the annotated camera streams after resampling to 480p at 16 fps; a window spans 81 consecutive frames ( ≈ 5 s), and windows are non-overlapping within a camera stream. Source-specific filtering removes static windows, frozen-gripper windows, and DROID episodes with contentless language annotations (see text).
Figure 4: Qualitative Evaluation on XVLA-Soft-Fold and Deform360. Each dataset-specific panel shows six synchronized snapshots over the full five-second horizon. From top to bottom, the rows show Wan-Move , fine-tuned Cosmos 2.5 , the action-flow conditioning input, our clean generated output, and the matched ground truth. Additional results appear in the appendix.
Figure 5: Wrist-Camera Egomotion Prediction. The first and last synchronized snapshots show frames 0 and 16 of the 17-frame generated clip, corresponding to 0.0 and 1.0 seconds of physical time.
Figure 6: Data Efficiency on IWS Tasks. Each task reports LPIPS, object-flow endpoint error (EPE), and FVD versus the percentage of task-specific training data. For all metrics, lower is better. Ours (PT) starts from the original Wan2.2 weights, whereas Ours (MT) starts from our multi-embodiment mid-trained checkpoint. Wan-Move and ATI are evaluated zero-shot.
Stage
s / clip ↓
FPS ↑
Speedup ↑
Bidirectional teacher
20.92
3.87
1.0 ×
Autoregressive teacher
12.48
6.49
1.68 ×
Few-step student
1.31
61.98
16.0×
Table 3: Inference Speed Across Optimization Stages. Measurements use a single 80-GB H100 GPU with bfloat16 and a batch size of one to generate 81 frames at 480×832 from a single conditioning frame. Guidance is disabled, and VAE decoding time is excluded. We report the median over five timed clips following three warm-up clips. Both teacher models use 50 sampling steps, whereas the student uses four. Speedup is calculated relative to the bidirectional teacher.
Figure 7: Open-Loop Policy Evaluation with Hydra-0. Each point aggregates 10 trials; dashed lines show least-squares fits. We evaluate π0 [ 6 ] , π0.5 [ 25 ] , GR00T N1.7 [ 33 ] , Cosmos-3 Edge, and Cosmos-3 Nano [ 35 ] in RoboLab. Legend abbreviations: GR00T denotes GR00T N1.7; C-3 Edge and C-3 Nano denote Cosmos-3 Edge and Cosmos-3 Nano.
Figure 8: Real-World Open-Loop Policy Evaluation. Qualitative cloth-folding rollouts show the recorded reference trajectory and the corresponding open-loop prediction conditioned on the recorded robot motion.
Figure 9: Object-Flow-Conditioned Real-Robot Execution. We illustrate the method on a flexible-pipe-bending task. The columns show six matched time steps. From a held-out human demonstration (row 1), we extract the motion of the manipulated object (row 2) in the form of desired object flow. Conditioned on that flow alone, Hydra-0 generates compatible robot motion in row 3. A learned action-output head converts the latent rollout into executable robot actions, yielding the successful execution shown in row 4.
Figure 10: Additional Qualitative Evaluation on DROID. The full-size Action Flow row provides the conditioning input, while the Ours row shows the corresponding clean generated rollout without a flow overlay. All rows show the same matched validation sample at the same six synchronized timestamps.
World models let robots imagine possible futures, but exploiting this capability for real-time planning is bottlenecked by a representation misalignment: generative models and planners operate on decoupled manifolds, requiring computationally expensive decoding of every candidate back to the high-dimensional observation space for evaluation. In this paper, we present Hydra, a discrete World Action Model that tackles this by establishing a unified latent manifold over visual states, physical poses, and control actions. By compressing this manifold through modality-specific Vector-Quantized bottlenecks, Hydra yields discrete vocabularies of kinodynamic intents and visual states. This enables Discrete Latent Planning (DLP), where candidates are sampled directly from the shared manifold and ranked by a Kinematic-Perceptual Cost within the discrete latent space. To bridge discrete planning with the continuous commands required for physical actuation, Hydra pairs DLP with conditional Flow Matching to map selected intents to smooth execution trajectories. Evaluated on two physical robotic platforms, Hydra outperforms state-of-the-art navigation world models in goal-directed planning, while matching or exceeding the closed-loop execution capabilities of leading reactive navigation policies.
Mohammad Nazeri, Alexandyr Card, Samira Huber +6
George Mason University · Kiel University · Ludwig Maximilian University Munich +1
Learning controllable world dynamics from heterogeneous interaction videos is challenging because robot embodiments use incompatible action spaces, while human videos lack native control labels. We present RynnWorld-Latent, a cross-embodiment video world model that uses the 3D-grounded latent actions introduced by RynnLAM as a shared conditioning interface. This representation enables joint pretraining on human interaction videos, robot demonstrations, and simulation rollouts without requiring a common native action space. Built on a pretrained video transformer, RynnWorld-Latent combines global action tokens with an identity-initialized local pathway that aligns actions with future video latents. We pretrain on 23 datasets comprising approximately 553.5M frames and adapt the model to target robots through post-training. The framework supports two data-generation routes: digital teleoperation uses target robot commands to generate action-video demonstrations, while latent-action synthesis transfers source motion to an edited initial frame and pairs the resulting rollout with retargeted pseudo robot actions. Experiments show improved video prediction metrics on public benchmarks and two real-robot platforms, alongside qualitative action following when different action sequences share the same initial observation. Synthetic mid-training improves policy success rates on LIBERO and LIBERO-Plus, and both generation routes improve real-world policy success rates over real-demonstration-only training in the evaluated settings. These results support universal latent actions as a scalable interface for learning controllable visual dynamics across heterogeneous embodiments.
Haoyu Zhao, Yinzhou Tang, Zixuan Wang +11
DAMO Academy, Alibaba Group · Hong Kong Embodied AI Lab · CUHK +1
State-of-the-art Vision-Language-Action (VLA) models excel at semantic generalization but struggle to generalize to unseen physical motions in novel environments. We introduce DreamZero, a World Action Model (WAM) built upon a pretrained video diffusion backbone. Unlike VLAs, WAMs learn physical dynamics by predicting future world states and actions, using video as a dense representation of how the world evolves. By jointly modeling video and action, DreamZero learns diverse skills effectively from heterogeneous robot data without relying on repetitive demonstrations. This results in over 2x improvement in generalization to new tasks and environments compared to state-of-the-art VLAs in real robot experiments. Crucially, through model and system optimizations, we enable a 14B autoregressive video diffusion model to perform real-time closed-loop control at 7Hz. Finally, we demonstrate two forms of cross-embodiment transfer: video-only demonstrations from other robots or humans yield a relative improvement of over 42% on unseen task performance with just 10-20 minutes of data. More surprisingly, DreamZero enables few-shot embodiment adaptation, transferring to a new embodiment with only 30 minutes of play data while retaining zero-shot generalization.