Hydra-0: Action Flow for Generalist World Modeling and Control

Hongyu Li; Bowen Wen; Xinghao Zhu; Yixuan Wang; Yilun Du; Yunzhu Li; George Konidaris; Stan Birchfield

cs.ROAug 18, 2026

Hydra-0: Action Flow for Generalist World Modeling and Control

Authors: Hongyu Li, Bowen Wen, Xinghao Zhu, Yixuan Wang, Yilun Du, Yunzhu Li, George Konidaris, Stan Birchfield,

Organizations: NVIDIA · Brown University · Columbia University · Harvard University

Abstract

We introduce Hydra-0, a generalist world model conditioned on action flow, which represents robot actions as pixel motion. This shared visual interface enables generalist world modeling and control by learning action consequences across embodiments, tasks, environments, and video-generation backbones. Our best configuration achieves 90.4% lower robot-motion error and 60.2% lower object-motion error than our action-conditioned baseline, while supporting zero-shot composition and data-efficient adaptation. On the RoboLab benchmark, Hydra-0 achieves a Pearson correlation of r=0.96 between replayed and reference success rates. Finally, we uncover an emergent inverse mode of this interface: a world action model that predicts compatible robot motion from desired object flow transferred from a human demonstration. A trained action head maps the resulting latent features to executable actions without requiring task-specific expert robot demonstrations. Together, these results demonstrate the potential of action flow as a shared control interface connecting heterogeneous training data, open-loop policy evaluation, and robot control.

Figures & tables

Figure 1: Overview of Hydra-0. During offline training (top) , we recover action flow from interaction videos by tracking both the embodiment and the manipulated object , as detailed in Section 2.1 . The future video is encoded to construct the flow-matching training target. During online deployment (bottom) , Isaac Lab executes a candidate command sequence $\mathbf{a}_{0:H-1}$ through the robot controller and physics simulation, producing robot link transforms over the prediction horizon. Using these transforms, we propagate visible robot-surface points and project them through the calibrated camera following Equation 1 , producing embodiment action flow that conditions the causal autoregressive model to predict the command sequence’s real-world consequence.

Figure 2: Action Flow Construction and Sampling. Dataset videos without geometric information are tracked densely and grounded into embodiment and manipulated-object masks. During training, we sample an input condition from four conditioning modes: Embodiment selects acting-body tracks as the primary action condition; Object supplies task or desired-motion flow; All provides a grounding fallback; and None provides conditioning dropout.

Figure 3: Action Flow as Visual Conditioning. Action-flow trajectories propagate first-frame latent features through time to form the motion-aware visual condition $C_{\mathrm{motion}}$ . This condition is concatenated with the noisy video latent $Z_{t}$ before DiT patch embedding.

			Windows
Dataset	Embodiment	Episodes	before	after	Hours	License
DROID [ 26 ]	single arm	70,339	494,680	223,075	313.7	CC-BY-4.0
ABC-130k [ 3 ]	bimanual arms	54,961	1,060,456	1,048,681	1,474.7	Apache-2.0
MolmoAct2 [ 14 ]	bimanual arms	7,723	129,589	126,335	177.7	Apache-2.0
EgoDex [ 23 ]	human hands	41,888	89,380	89,380	125.7	CC-BY-NC-ND-4.0
Deform360 [ 30 ]	handheld grippers	1,714	87,224	60,315	84.8	MIT

Table 1: Multi-Embodiment Training Corpus. Episodes and windows are counted on the annotated camera streams after resampling to 480p at 16 fps; a window spans 81 consecutive frames (

\approx

5 s), and windows are non-overlapping within a camera stream. Source-specific filtering removes static windows, frozen-gripper windows, and DROID episodes with contentless language annotations (see text).

Panel of Figure 4 — Figure 4: Qualitative Evaluation on XVLA-Soft-Fold and Deform360. Each dataset-specific panel shows six synchronized snapshots over the full five-second horizon. From top to bottom, the rows show Wan-Move , fine-tuned Cosmos 2.5 , the action-flow conditioning input, our clean generated output, and the matched ground truth. Additional results appear in the appendix.

Figure 5: Wrist-Camera Egomotion Prediction. The first and last synchronized snapshots show frames 0 and 16 of the 17-frame generated clip, corresponding to 0.0 and 1.0 seconds of physical time.

Figure 6: Data Efficiency on IWS Tasks. Each task reports LPIPS, object-flow endpoint error (EPE), and FVD versus the percentage of task-specific training data. For all metrics, lower is better. Ours (PT) starts from the original Wan2.2 weights, whereas Ours (MT) starts from our multi-embodiment mid-trained checkpoint. Wan-Move and ATI are evaluated zero-shot.

Stage	s / clip $\downarrow$	FPS $\uparrow$	Speedup $\uparrow$
Bidirectional teacher	20.92	3.87	1.0 $\times$
Autoregressive teacher	12.48	6.49	1.68 $\times$
Few-step student	$\mathbf{1.31}$	$\mathbf{61.98}$	$\mathbf{16.0\times}$

Table 3: Inference Speed Across Optimization Stages. Measurements use a single 80-GB H100 GPU with bfloat16 and a batch size of one to generate 81 frames at

480\times 832

from a single conditioning frame. Guidance is disabled, and VAE decoding time is excluded. We report the median over five timed clips following three warm-up clips. Both teacher models use 50 sampling steps, whereas the student uses four. Speedup is calculated relative to the bidirectional teacher.

Figure 7: Open-Loop Policy Evaluation with Hydra-0. Each point aggregates 10 trials; dashed lines show least-squares fits. We evaluate

\pi_{0}

[ 6 ] ,

\pi_{0.5}

[ 25 ] , GR00T N1.7 [ 33 ] , Cosmos-3 Edge, and Cosmos-3 Nano [ 35 ] in RoboLab. Legend abbreviations: GR00T denotes GR00T N1.7; C-3 Edge and C-3 Nano denote Cosmos-3 Edge and Cosmos-3 Nano.

Figure 8: Real-World Open-Loop Policy Evaluation. Qualitative cloth-folding rollouts show the recorded reference trajectory and the corresponding open-loop prediction conditioned on the recorded robot motion.

Figure 9: Object-Flow-Conditioned Real-Robot Execution. We illustrate the method on a flexible-pipe-bending task. The columns show six matched time steps. From a held-out human demonstration (row 1), we extract the motion of the manipulated object (row 2) in the form of desired object flow. Conditioned on that flow alone, Hydra-0 generates compatible robot motion in row 3. A learned action-output head converts the latent rollout into executable robot actions, yielding the successful execution shown in row 4.

Figure 10: Additional Qualitative Evaluation on DROID. The full-size Action Flow row provides the conditioning input, while the Ours row shows the corresponding clean generated rollout without a flow overlay. All rows show the same matched validation sample at the same six synchronized timestamps.

World Models for Robotics

World models for robotics predict how scenes and robot states change under actions to support planning, control, and learning.

See all 384 papers

Hydra: A Navigation World Action Model with Discrete Latent Planning and Continuous Flow-Matching Execution
Aug 29, 2026Mohammad Nazeri, Alexandyr Card, Samira Huber +6Kinodynamic Motion Planning World Models for Robotics
George Mason University · Kiel University · Ludwig Maximilian University Munich +1
RynnWorld-Latent: A Cross-Embodiment World Model Conditioned on Universal Latent Actions
Oct 27, 2026Haoyu Zhao, Yinzhou Tang, Zixuan Wang +11Embodied World Models World Models
DAMO Academy, Alibaba Group · Hong Kong Embodied AI Lab · CUHK +1
World Action Models are Zero-shot Policies
Feb 17, 2026Seonghyeon Ye, Yunhao Ge, Kaiyuan Zheng +33World Action Models Zero-Shot Robotic Manipulation
NVIDIA

Aug 29, 2026J/K move · Enter open · S save

Hydra: A Navigation World Action Model with Discrete Latent Planning and Continuous Flow-Matching Execution

Mohammad Nazeri, Alexandyr Card, Samira Huber, Anuj Pokhrel+5

George Mason University · Kiel University · Ludwig Maximilian University Munich+1

Kinodynamic Motion Planning World Models for Robotics Robot Navigation World Model-Based Planning Action-Conditioned World Models Model-Based Planning+1

Open paper AskPDF arXiv

World models let robots imagine possible futures, but exploiting this capability for real-time planning is bottlenecked by a representation misalignment: generative models and planners operate on decoupled manifolds, requiring computationally expensive decoding of every candidate back to the high-dimensional observation space for evaluation. In this paper, we present Hydra, a discrete World Action Model that tackles this by establishing a unified latent manifold over visual states, physical poses, and control actions. By compressing this manifold through modality-specific Vector-Quantized bottlenecks, Hydra yields discrete vocabularies of kinodynamic intents and visual states. This enables Discrete Latent Planning (DLP), where candidates are sampled directly from the shared manifold and ranked by a Kinematic-Perceptual Cost within the discrete latent space. To bridge discrete planning with the continuous commands required for physical actuation, Hydra pairs DLP with conditional Flow Matching to map selected intents to smooth execution trajectories. Evaluated on two physical robotic platforms, Hydra outperforms state-of-the-art navigation world models in goal-directed planning, while matching or exceeding the closed-loop execution capabilities of leading reactive navigation policies.

Abstract

World Models for Robotics

Explore similar work

Hydra: A Navigation World Action Model with Discrete Latent Planning and Continuous Flow-Matching Execution

RynnWorld-Latent: A Cross-Embodiment World Model Conditioned on Universal Latent Actions

World Action Models are Zero-shot Policies