Learning a universal action representation from video requires modeling dynamics that are consistent across different executions of the same action yet invariant to changes in embodiment and appearance. Existing latent action models infer actions from visual transitions, but their representations often entangle the underlying action with irrelevant visual variations, limiting transfer across embodiments. We introduce DreamX, which repurposes a pretrained video diffusion model as a universal latent action learner. Our approach extracts compact latent actions from the video model's rich spatiotemporal features, compressing each latent video frame by 300× (1,200→4 tokens). In addition, by leveraging the video model's generative prior, these latent actions capture action-relevant dynamics disentangled from visual appearance. The resulting latent space serves as a shared action interface across embodiments (human and robot), environments, and action parameterizations. On LARY-Bench, DreamX achieves state-of-the-art performance among latent action models. Moreover, actions encoded in one setting remain effective in another: the latent space supports any-to-any video transfer and zero-shot action transfer across diverse embodiments and environments. Furthermore, this shared interface also enables policy learning in a compact latent action space, reducing the computational cost of video-diffusion-based policies.
Figures & tables
Figure 1: DreamX learns a shared action interface from egocentric human videos and robot videos. Compact, embodiment-agnostic latent actions extracted from a pretrained video diffusion model capture motion consistently across changes in embodiment and scene appearance. The same representation enables transferring motion from a source video to a target video, decoding human motion into executable robot actions without paired human–robot training data, and training a lightweight latent policy that jointly predicts latent actions and robot actions.
Figure 2: DreamX architecture and attention masks. (a) Learnable queries read out the pretrained video backbone’s spatiotemporal representations to encode latent actions. A decoder initialized from the same pretrained backbone reconstructs future video from the latent actions and the initial frame, with auxiliary action supervision when labels are available. (b) Encoder and decoder attention masks control information flow among video, query, and action tokens. In the decoder, video and action tokens read the latent-action queries but do not attend to one another.
Figure 3: Downstream applications of DreamX. (a) Motion transfer. Source latent actions condition the DreamX decoder to generate video from a target initial frame or low-level actions from a target robot state, without paired source–target training data. (b) DreamX policy. A variational bottleneck compresses latent actions into compact codes while preserving the token layout. Conditioned on the current observation and instruction, the policy generates these codes alongside low-level actions through a separate DiT action head.
Action regression ↓
Action classification ↑
Method
CALVIN
VLABench
RoboCOIN
AgiBotWorld-Beta
Avg.
Atomic robot
Composite human
Composite robot
Avg.
Latent actions from 2 frames
LAPA
0.96
0.95
0.96
1.00
0.97
22.27
14.61
23.64
20.17
UniVLA
0.82
0.74
0.94
0.97
0.87
18.62
19.08
18.56
18.75
villa-X
0.86
0.72
0.94
0.97
0.87
15.00
17.80
29.90
20.90
LAPA-MAGVIT2
0.59
0.36
0.80
0.83
0.65
33.12
59.70
29.53
40.78
Table 1: LARYBench action regression and classification. Regression reports normalized action MSE and classification reports accuracy. Bold and underline denote the best and second-best results.
Figure 4: Qualitative Any-to-Any video transfer. Each example shows a source motion and the corresponding transferred video generated from a target-domain initial observation. These results demonstrate that the learned latent action transfers motion across substantial changes in embodiment and scene appearance. Blue dots indicate the end-effector trajectory throughout the video frames.
Figure 5: Zero-shot human-to-robot video transfer on the H&R benchmark. DreamX preserves the source motion and target robot appearance, while the baselines exhibit visual degradation or human-hand leakage.
Figure 6: Qualitative human-to-robot action transfer. Given a human demonstration, DreamX extracts a latent action and decodes it into low-level robot actions without paired human–robot training data. We show diverse transferred behaviors spanning writing, single-arm pushing, and bimanual manipulation. Across substantial embodiment differences, the robot reproduces the motion structure of the human demonstration.
Figure 8
Figure 8: Query-to-video attention. We visualize attention maps from latent queries to video tokens during encoding. Across human and robot demonstrations, latent queries attend to dynamic regions, such as moving hands and grippers, capturing the underlying actions in each video.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Pretraining data composition. DreamX learns from six datasets spanning human and robot demonstrations. Sector areas represent video duration, with human datasets highlighted in warm colors.
Figure 10: BimanualYAM setup.
Figure 11: Real-world drawer-closing setup. Initial observations for the three task configurations, all conditioned on close the drawer . When both drawers are open, the robot must close the upper drawer before the lower drawer.
Upper
Lower
Both
Method
Reach
Push
Reach
Push
Reach
Push
Mimic-Video
80
20
100
0
100
0
DreamX
100
80
100
100
100
100
Appendix
Table 5: Reaching and pushing completion rates (%) for real-world drawer closing. Each subgoal contributes equally to the SR reported in the main table.
Method
Upper
Lower
Both
SR ↑
DreamX
100/100
100/100
100/100
100
Appendix
Table 6: Policy learning with human-only demonstrations of the upper-only configuration. Task columns report approach/push completion rates (%); SR denotes subgoal success rate (%), averaged equally over reaching and pushing and across the three configurations.
Policy
≥1
≥2
≥3
≥4
≥5
Avg. Len. ↑
DreamX policy w/o latent actions
83.4
61.7
42.5
29.8
20.4
2.378
DreamX policy
85.7
67.0
49.5
36.4
25.7
2.643
Appendix
Table 7: CALVIN policy ablation with and without latent-action conditioning. Columns ≥k report success rates (%) for completing at least k consecutive tasks; Avg. Len. is the average completed sequence length.
Figure 12: Directional structure across robot embodiments. Colors denote motion directions and marker shapes denote robot types.
Figure 14: Additional qualitative Any-to-Any video transfer results. Each example shows a source motion and the corresponding transferred sequence generated from a target-domain initial frame.
Figure 15: Additional zero-shot human-to-robot video transfer results on the HumanAndRobot benchmark.
Figure 16: Additional zero-shot human-to-robot video transfer results on the HumanAndRobot benchmark (continued).
Figure 17: Additional zero-shot human-to-robot video transfer results on the HumanAndRobot benchmark (continued).
Figure 18: Additional qualitative human-to-robot action transfer results. DreamX transfers human demonstrations into executable robot actions while preserving task-specific trajectory structure and temporal ordering across writing and single-arm manipulation examples.
Figure 19: Real-world drawer-closing rollouts. Six frames are sampled at equal fractions of each rollout’s duration.
Figure 20: Query-to-video attention across time. Human demonstrations (left) and robot demonstrations (right), each shown with the input frame beside its attention overlay. Rows follow temporal order and highlight attention around the acting embodiment and objects.
State-of-the-art Vision-Language-Action (VLA) models excel at semantic generalization but struggle to generalize to unseen physical motions in novel environments. We introduce DreamZero, a World Action Model (WAM) built upon a pretrained video diffusion backbone. Unlike VLAs, WAMs learn physical dynamics by predicting future world states and actions, using video as a dense representation of how the world evolves. By jointly modeling video and action, DreamZero learns diverse skills effectively from heterogeneous robot data without relying on repetitive demonstrations. This results in over 2x improvement in generalization to new tasks and environments compared to state-of-the-art VLAs in real robot experiments. Crucially, through model and system optimizations, we enable a 14B autoregressive video diffusion model to perform real-time closed-loop control at 7Hz. Finally, we demonstrate two forms of cross-embodiment transfer: video-only demonstrations from other robots or humans yield a relative improvement of over 42% on unseen task performance with just 10-20 minutes of data. More surprisingly, DreamZero enables few-shot embodiment adaptation, transferring to a new embodiment with only 30 minutes of play data while retaining zero-shot generalization.
Learning controllable world dynamics from heterogeneous interaction videos is challenging because robot embodiments use incompatible action spaces, while human videos lack native control labels. We present RynnWorld-Latent, a cross-embodiment video world model that uses the 3D-grounded latent actions introduced by RynnLAM as a shared conditioning interface. This representation enables joint pretraining on human interaction videos, robot demonstrations, and simulation rollouts without requiring a common native action space. Built on a pretrained video transformer, RynnWorld-Latent combines global action tokens with an identity-initialized local pathway that aligns actions with future video latents. We pretrain on 23 datasets comprising approximately 553.5M frames and adapt the model to target robots through post-training. The framework supports two data-generation routes: digital teleoperation uses target robot commands to generate action-video demonstrations, while latent-action synthesis transfers source motion to an edited initial frame and pairs the resulting rollout with retargeted pseudo robot actions. Experiments show improved video prediction metrics on public benchmarks and two real-robot platforms, alongside qualitative action following when different action sequences share the same initial observation. Synthetic mid-training improves policy success rates on LIBERO and LIBERO-Plus, and both generation routes improve real-world policy success rates over real-demonstration-only training in the evaluated settings. These results support universal latent actions as a scalable interface for learning controllable visual dynamics across heterogeneous embodiments.
Haoyu Zhao, Yinzhou Tang, Zixuan Wang +11
DAMO Academy, Alibaba Group · Hong Kong Embodied AI Lab · CUHK +1
Latent actions provide a compact interface between action-free video and downstream decision-making, yet existing Latent Action Models (LAMs) force every transition through a fixed-capacity bottleneck. We identify a bottleneck trade-off: overly tight codes can discard transition cues needed for action alignment, while overly loose codes preserve additional transition variation that must be resolved when alignment labels are scarce or narrowly distributed. FlexLAM replaces this fixed capacity with variable-length latent actions trained by nested dropout, yielding prefix-valid codes that capture compact transition structure first and add detail only when needed, without new architectures or losses. A single FlexLAM matches or surpasses separately trained fixed-capacity LAMs at every evaluated token budget under standard scarce-label supervision and under a low-return single-task alignment stress test, indicating that FlexLAM is not merely adjustable at inference time but learns a better latent-action interface at the same token budgets. The same model supports inference-time token-budget adjustment without retraining, and FlexLAM improves Ego4D transition reconstruction. These results suggest that variable-length latent actions are an architecture-free, drop-in upgrade to the fixed-capacity bottleneck in latent action models, latent-action world models, and video-pretrained action interfaces.