Learning controllable world dynamics from heterogeneous interaction videos is challenging because robot embodiments use incompatible action spaces, while human videos lack native control labels. We present RynnWorld-Latent, a cross-embodiment video world model that uses the 3D-grounded latent actions introduced by RynnLAM as a shared conditioning interface. This representation enables joint pretraining on human interaction videos, robot demonstrations, and simulation rollouts without requiring a common native action space. Built on a pretrained video transformer, RynnWorld-Latent combines global action tokens with an identity-initialized local pathway that aligns actions with future video latents. We pretrain on 23 datasets comprising approximately 553.5M frames and adapt the model to target robots through post-training. The framework supports two data-generation routes: digital teleoperation uses target robot commands to generate action-video demonstrations, while latent-action synthesis transfers source motion to an edited initial frame and pairs the resulting rollout with retargeted pseudo robot actions. Experiments show improved video prediction metrics on public benchmarks and two real-robot platforms, alongside qualitative action following when different action sequences share the same initial observation. Synthetic mid-training improves policy success rates on LIBERO and LIBERO-Plus, and both generation routes improve real-world policy success rates over real-demonstration-only training in the evaluated settings. These results support universal latent actions as a scalable interface for learning controllable visual dynamics across heterogeneous embodiments.
Figures & tables
Figure 1: Pretraining corpus overview. The corpus contains 23 datasets, approximately 6,190 hours, and 2.57M episodes. The figure summarizes the distribution of human, real-robot, and simulation video, dataset contributions by duration, and object and action vocabulary.
Figure 2: Overview of the world-model data engine. Top: Scalable data editing. RynnLAM extracts latent actions from a source human video, while its first frame is edited to depict the target robot. Conditioned on the edited frame and the original latent actions, RynnWorld-Latent generates robot video rollouts. The generated observations are paired with pseudo-actions obtained through motion retargeting and inverse kinematics. Bottom: Digital teleoperation. Human control signals are retargeted to robot actions, which condition RynnWorld-Latent together with a reference scene. Generated frames provide visual feedback to the operator, while the recorded actions and generated observations form synthetic demonstrations without physical robot execution.
Figure 3: RynnEdit. Two routes construct a robot-centric reference frame: explicit hand removal, retargeting, and robot rendering; or direct replacement of human hands with robotic counterparts using an image-editing model. The resulting first frame conditions subsequent world-model generation.
Figure 4: Standardized Experimental Platforms. Two dual-arm embodiments used for real-world evaluation: the stationary Marvin-WUJI (dual 7-DOF arms with 20-DOF dexterous hands) and the mobile Astribot-S1 (dual 7-DOF arms with parallel grippers on a wheeled base).
Model
In-lab Eval
EgoDex Eval
DreamDojo-HV Eval
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
Cosmos-Predict2.5 ( Ali et al., 2025 )
20.58
0.77
0.22
19.95
0.79
0.22
18.27
0.75
0.24
DreamDojo-2B ( Gao et al., 2026 )
21.11
0.77
0.22
20.41
0.78
0.23
18.81
0.75
0.24
DreamDojo-14B ( Gao et al., 2026 )
21.41
0.79
0.21
20.53
0.79
0.21
18.92
0.75
0.23
RynnWorld-Latent
21.42
0.82
0.20
20.89
0.81
0.19
19.22
0.79
0.21
Table 2: Results on the DreamDojo GR-1 evaluation sets ( Gao et al., 2026 ) . Baseline values are literature-reported measurements after robot post-training.
Method
Marvin-WUJI Eval
Astribot-S1 Eval
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
Cosmos3-Edge ( Agarwal et al., 2026 )
20.03
0.78
0.31
13.99
0.42
0.54
Cosmos3-Nano ( Agarwal et al., 2026 )
20.09
0.80
0.26
14.22
0.53
0.45
RynnWorld-Teleop ( Zhao et al., 2026c )
20.89
0.76
0.19
N/A
N/A
N/A
RynnWorld-Latent
21.86
0.80
0.12
15.96
0.60
0.34
Ablation Study
Table 3: Results and ablation study on our real-robot evaluation sets. ‘N/A’ denotes baseline implementations without support for Astribot-S1 mobile-base motion. PSNR, SSIM, and LPIPS are interpreted jointly as measures of video prediction fidelity.
Figure 5: Action following under a shared initial observation. Each comparison uses latent-action sequences extracted from two different source videos. Both sequences are paired with the same initial frame to generate separate rollouts. Blue and green arrows associate source clips with their corresponding generated rollouts. The generated motion qualitatively follows the respective source sequence despite the shared initial scene.
Source Dataset
Synthetic Frames
EPIC-KITCHENS ( Damen et al., 2020 )
2.5M
EgoDex ( Hoque et al., 2025 )
80.8M
Something-Something ( Goyal et al., 2017 )
1.5M
RynnWorld-Latent-80M (total)
84.8M
Table 5: Synthetic corpus by source dataset. The RynnWorld-Latent data engine processes these human interaction videos into robot-action-labeled synthetic episodes. After filtering, the total corpus contains approximately 800 hours.
Method
LIBERO
LIBERO-Plus
π0 ( Black et al., 2024a )
94.4
53.6
π0 + RynnWorld-Latent-80M
96.1
79.4
π0.5 ( Intelligence et al., 2025 )
96.9
84.4
π0.5 + RynnWorld-Latent-80M
97.8
84.8
Table 6: Policy success rates on LIBERO ( Liu et al., 2023 ) and LIBERO-Plus. Values are percentages. Each synthetic-data variant uses RynnWorld-Latent-80M for mid-training before downstream evaluation.
Figure 6: Scalable data editing. Given a source video, we edit its first frame to replace the embodiment or environment, then condition RynnWorld-Latent on the edited frame and the source latent-action sequence to synthesize a robotized rollout.
Figure 7: Interactive digital teleoperation. Operator control signals are retargeted to target-robot actions and condition RynnWorld-Latent together with a reference scene; the recorded actions and generated frames form synthetic demonstrations without physical robot execution.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Marvin-WUJI demonstration collection. An operator controls dual robot arms and dexterous hands using motion trackers and data gloves.
Task
Description
Episodes
Plate-Placement
Take a plate from a shelf and place it beside the oven.
500
Steak-Cooking
Flip a steak in a pan.
800
Plate-Wiping
Take a dirty plate from a shelf, wipe it with a cloth, and return it.
500
Table-Cleaning
Collect scattered items into a box and place the box on a shelf.
850
Steak-Serving
Carry a steak plate to the dining area and place it on a placemat.
600
Plate-Retrieval
Collect a dirty plate and cutlery and return them to their designated positions.
600
Appendix
Table 7: Collected real-world demonstrations. The corpus is shared with RynnVLA-Latent. Counts refer to physically collected episodes, not synthetic data.
We introduce Qwen-RobotWorld, a language-conditioned video world model for embodied intelligence. With natural language as a unified action interface, it predicts physically grounded future visual trajectories from current observations across robotic manipulation, autonomous driving, indoor navigation, and human-to-robot transfer. This unified formulation provides three promising application directions: synthetic data generation for policy training augmentation, scalable virtual environments for policy evaluation, and language-guided planning signals for downstream robot control. This is achieved through a three-part design: a) Double-Stream MMDiT with MLLM Action Encoding, where a 60-layer double-stream diffusion transformer couples frozen Qwen2.5-VL semantics with video-VAE latents through layer-wise joint attention; b) Embodied World Knowledge (EWK), an 8.6M video-text corpus (200M+ frames) with action-language mapping over 20+ embodiments and 500+ action categories; and c) General+Expert Progressive Curriculum, a two-stage training strategy that first learns general visual priors and then injects embodied specialization under a shared language interface. Extensive results show strong competitiveness: ranks 1st overall on EWMBench and DreamGen Bench, outperforms all open-source models on WorldModelBench and PBench. Additional zero-shot analyses on RoboTwin-IF benchmark further support robust generalization and multi-view consistency.
Robot world models simulate how environments evolve in response to robot actions. Latent action models enable learning dynamics from videos by encoding visual transitions as proxy actions. However, reconstruction-driven latent actions mix robot motion with object responses and other scene changes, complicating their correspondence with robot controls. We propose SkeLAM, a skeleton-guided latent action model for controllable robot world modeling. SkeLAM decomposes latent actions into a kinematic component grounded by robot skeleton supervision and a residual component retaining complementary visual dynamics. Cross-modal swap reconstruction encourages RGB-derived and skeleton-derived kinematic representations to be interchangeable during video reconstruction. The world model incorporates both components through separate conditioning pathways, while branch dropout reduces its dependence on future-derived residual information. A lightweight adapter maps robot actions to the kinematic space, and subsequent world-model fine-tuning adapts generation to these predicted conditions. At inference, the model generates future observations from the current observation and robot actions. On DROID, SkeLAM reduces end-effector ADE by 19.5% and LPIPS by 11.1% compared with DreamDojo. Motion probing supports kinematic specialization, while qualitative results demonstrate zero-shot human-to-robot motion transfer in generated rollouts. Policy evaluation shows agreement between predicted success rates and those measured in real robot executions.
The diversity of robot embodiments and action spaces makes it challenging to build robot world models that generalize across different embodiments. We introduce the Latent Action-Conditioned Robot World Model (LAC-WM), which operates within a learned unified latent action space shared across diverse embodiments. This unified action space improves the world model's performance when adapted to previously unseen robot embodiments. We compare LAC-WM with an Explicit Action-Conditioned World Model (EAC-WM), which conditions on explicit motion labels. Our results show that explicit action conditioning leads to disjoint action representations across embodiments, limiting downstream performance when adapting to new robots. We evaluate both models on dexterous manipulation tasks and a modified LIBERO benchmark. LAC-WM improves downstream performance over EAC-WM by up to 46.7% on dexterous manipulation and 11.7% on LIBERO. Crucially, the unified latent action space allows LAC-WM's downstream performance to scale positively with the number of embodiments used during pretraining. In contrast, the disjoint action space in EAC-WM leads to decreased performance as the number of pretraining embodiments increases. These results highlight the importance of a unified action space for efficient cross-embodiment learning, addressing a key challenge in robotics. Project website: https://lacwm.github.io/
Huang Huang, Sriram Yenamandra, Arjun Majumdar +9
Department of Computer Science, Stanford University, US · Work partially done while at Meta FAIR Robotics.