DreamX: Video Diffusion Model is a Universal Latent Action Learner

Jisoo Kim; Chaehyun Kim; Junsu Kim; Jiho Park; Seongchan Kim; Eunbeen Hong; Soowon Son; Fu-En Yang

cs.CVNov 3, 2026

DreamX: Video Diffusion Model is a Universal Latent Action Learner

Authors: Jisoo Kim, Chaehyun Kim, Junsu Kim, Jiho Park, Seongchan Kim, Eunbeen Hong, Soowon Son, Fu-En Yang,

Organizations: KAIST · NVIDIA

Abstract

Learning a universal action representation from video requires modeling dynamics that are consistent across different executions of the same action yet invariant to changes in embodiment and appearance. Existing latent action models infer actions from visual transitions, but their representations often entangle the underlying action with irrelevant visual variations, limiting transfer across embodiments. We introduce DreamX, which repurposes a pretrained video diffusion model as a universal latent action learner. Our approach extracts compact latent actions from the video model's rich spatiotemporal features, compressing each latent video frame by $\boldsymbol{300\times}$ ( $\boldsymbol{1{,}200 \rightarrow 4}$ tokens). In addition, by leveraging the video model's generative prior, these latent actions capture action-relevant dynamics disentangled from visual appearance. The resulting latent space serves as a shared action interface across embodiments (human and robot), environments, and action parameterizations. On LARY-Bench, DreamX achieves state-of-the-art performance among latent action models. Moreover, actions encoded in one setting remain effective in another: the latent space supports any-to-any video transfer and zero-shot action transfer across diverse embodiments and environments. Furthermore, this shared interface also enables policy learning in a compact latent action space, reducing the computational cost of video-diffusion-based policies.

Figures & tables

Figure 1: DreamX learns a shared action interface from egocentric human videos and robot videos. Compact, embodiment-agnostic latent actions extracted from a pretrained video diffusion model capture motion consistently across changes in embodiment and scene appearance. The same representation enables transferring motion from a source video to a target video, decoding human motion into executable robot actions without paired human–robot training data, and training a lightweight latent policy that jointly predicts latent actions and robot actions.

Figure 2: DreamX architecture and attention masks. (a) Learnable queries read out the pretrained video backbone’s spatiotemporal representations to encode latent actions. A decoder initialized from the same pretrained backbone reconstructs future video from the latent actions and the initial frame, with auxiliary action supervision when labels are available. (b) Encoder and decoder attention masks control information flow among video, query, and action tokens. In the decoder, video and action tokens read the latent-action queries but do not attend to one another.

Figure 3: Downstream applications of DreamX. (a) Motion transfer. Source latent actions condition the DreamX decoder to generate video from a target initial frame or low-level actions from a target robot state, without paired source–target training data. (b) DreamX policy. A variational bottleneck compresses latent actions into compact codes while preserving the token layout. Conditioned on the current observation and instruction, the policy generates these codes alongside low-level actions through a separate DiT action head.

	Action regression $\downarrow$					Action classification $\uparrow$
Method	CALVIN	VLABench	RoboCOIN	AgiBotWorld-Beta	Avg.	Atomic robot	Composite human	Composite robot	Avg.
Latent actions from 2 frames
LAPA	0.96	0.95	0.96	1.00	0.97	22.27	14.61	23.64	20.17
UniVLA	0.82	0.74	0.94	0.97	0.87	18.62	19.08	18.56	18.75
villa-X	0.86	0.72	0.94	0.97	0.87	15.00	17.80	29.90	20.90
LAPA-MAGVIT2	0.59	0.36	0.80	0.83	0.65	33.12	59.70	29.53	40.78

Table 1: LARYBench action regression and classification. Regression reports normalized action MSE and classification reports accuracy. Bold and underline denote the best and second-best results.

Figure 4: Qualitative Any-to-Any video transfer. Each example shows a source motion and the corresponding transferred video generated from a target-domain initial observation. These results demonstrate that the learned latent action transfers motion across substantial changes in embodiment and scene appearance. Blue dots indicate the end-effector trajectory throughout the video frames.

Figure 5: Zero-shot human-to-robot video transfer on the H&R benchmark. DreamX preserves the source motion and target robot appearance, while the baselines exhibit visual degradation or human-hand leakage.

Figure 6: Qualitative human-to-robot action transfer. Given a human demonstration, DreamX extracts a latent action and decodes it into low-level robot actions without paired human–robot training data. We show diverse transferred behaviors spanning writing, single-arm pushing, and bimanual manipulation. Across substantial embodiment differences, the robot reproduces the motion structure of the human demonstration.

Figure 8: Query-to-video attention. We visualize attention maps from latent queries to video tokens during encoding. Across human and robot demonstrations, latent queries attend to dynamic regions, such as moving hands and grippers, capturing the underlying actions in each video.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Pretraining data composition. DreamX learns from six datasets spanning human and robot demonstrations. Sector areas represent video duration, with human datasets highlighted in warm colors.

Panel of Figure 11 — Figure 11: Real-world drawer-closing setup. Initial observations for the three task configurations, all conditioned on close the drawer . When both drawers are open, the robot must close the upper drawer before the lower drawer.

	Upper		Lower		Both
Method	Reach	Push	Reach	Push	Reach	Push
Mimic-Video	80	20	100	0	100	0
DreamX	100	80	100	100	100	100

Appendix

Table 5: Reaching and pushing completion rates (%) for real-world drawer closing. Each subgoal contributes equally to the SR reported in the main table.

Method	Upper	Lower	Both	SR $\uparrow$
DreamX	100/100	100/100	100/100	100

Appendix

Table 6: Policy learning with human-only demonstrations of the upper-only configuration. Task columns report approach/push completion rates (%); SR denotes subgoal success rate (%), averaged equally over reaching and pushing and across the three configurations.

Policy	$\geq 1$	$\geq 2$	$\geq 3$	$\geq 4$	$\geq 5$	Avg. Len. $\uparrow$
DreamX policy w/o latent actions	83.4	61.7	42.5	29.8	20.4	2.378
DreamX policy	85.7	67.0	49.5	36.4	25.7	2.643

Appendix

Table 7: CALVIN policy ablation with and without latent-action conditioning. Columns

\geq k

report success rates (%) for completing at least

k

consecutive tasks; Avg. Len. is the average completed sequence length.

Figure 12: Directional structure across robot embodiments. Colors denote motion directions and marker shapes denote robot types.

Figure 14: Additional qualitative Any-to-Any video transfer results. Each example shows a source motion and the corresponding transferred sequence generated from a target-domain initial frame.

Figure 15: Additional zero-shot human-to-robot video transfer results on the HumanAndRobot benchmark.

Figure 16: Additional zero-shot human-to-robot video transfer results on the HumanAndRobot benchmark (continued).

Figure 17: Additional zero-shot human-to-robot video transfer results on the HumanAndRobot benchmark (continued).

Figure 18: Additional qualitative human-to-robot action transfer results. DreamX transfers human demonstrations into executable robot actions while preserving task-specific trajectory structure and temporal ordering across writing and single-arm manipulation examples.

Figure 19: Real-world drawer-closing rollouts. Six frames are sampled at equal fractions of each rollout’s duration.

Panel of Figure 20 — Figure 20: Query-to-video attention across time. Human demonstrations (left) and robot demonstrations (right), each shown with the input frame beside its attention overlay. Rows follow temporal order and highlight attention around the acting embodiment and objects.

Latent Action Learning

Latent action learning discovers compact representations of behavior from observations or trajectories to support control, planning, and transfer.

See all 115 papers

World Action Models are Zero-shot Policies
Feb 17, 2026Seonghyeon Ye, Yunhao Ge, Kaiyuan Zheng +33World Action Models Zero-Shot Robotic Manipulation
NVIDIA
RynnWorld-Latent: A Cross-Embodiment World Model Conditioned on Universal Latent Actions
Oct 27, 2026Haoyu Zhao, Yinzhou Tang, Zixuan Wang +11Embodied World Models World Models
DAMO Academy, Alibaba Group · Hong Kong Embodied AI Lab · CUHK +1
FlexLAM: Resolving the Bottleneck Trade-off in Latent Action Learning
Jun 17, 2026Takanori Yoshimoto, Yang Hu, Naruya Kondo +1Latent Action Learning
University of Tsukuba · The University of Tokyo

Feb 17, 2026J/K move · Enter open · S save

World Action Models are Zero-shot Policies

Seonghyeon Ye, Yunhao Ge, Kaiyuan Zheng, Shenyuan Gao+32

NVIDIA

World Action Models Zero-Shot Robotic Manipulation Robot Learning from Video Cross-Embodiment Robot Learning Video Diffusion Models Efficient Inference for World Action Models

Open paper AskPDF arXiv

State-of-the-art Vision-Language-Action (VLA) models excel at semantic generalization but struggle to generalize to unseen physical motions in novel environments. We introduce DreamZero, a World Action Model (WAM) built upon a pretrained video diffusion backbone. Unlike VLAs, WAMs learn physical dynamics by predicting future world states and actions, using video as a dense representation of how the world evolves. By jointly modeling video and action, DreamZero learns diverse skills effectively from heterogeneous robot data without relying on repetitive demonstrations. This results in over 2x improvement in generalization to new tasks and environments compared to state-of-the-art VLAs in real robot experiments. Crucially, through model and system optimizations, we enable a 14B autoregressive video diffusion model to perform real-time closed-loop control at 7Hz. Finally, we demonstrate two forms of cross-embodiment transfer: video-only demonstrations from other robots or humans yield a relative improvement of over 42% on unseen task performance with just 10-20 minutes of data. More surprisingly, DreamZero enables few-shot embodiment adaptation, transferring to a new embodiment with only 30 minutes of play data while retaining zero-shot generalization.

Abstract

Latent Action Learning

Explore similar work

World Action Models are Zero-shot Policies

RynnWorld-Latent: A Cross-Embodiment World Model Conditioned on Universal Latent Actions

FlexLAM: Resolving the Bottleneck Trade-off in Latent Action Learning