Learning controllable world dynamics from heterogeneous interaction videos is challenging because robot embodiments use incompatible action spaces, while human videos lack native control labels. We present RynnWorld-Latent, a cross-embodiment video world model that uses the 3D-grounded latent actions introduced by RynnLAM as a shared conditioning interface. This representation enables joint pretraining on human interaction videos, robot demonstrations, and simulation rollouts without requiring a common native action space. Built on a pretrained video transformer, RynnWorld-Latent combines global action tokens with an identity-initialized local pathway that aligns actions with future video latents. We pretrain on 23 datasets comprising approximately 553.5M frames and adapt the model to target robots through post-training. The framework supports two data-generation routes: digital teleoperation uses target robot commands to generate action-video demonstrations, while latent-action synthesis transfers source motion to an edited initial frame and pairs the resulting rollout with retargeted pseudo robot actions. Experiments show improved video prediction metrics on public benchmarks and two real-robot platforms, alongside qualitative action following when different action sequences share the same initial observation. Synthetic mid-training improves policy success rates on LIBERO and LIBERO-Plus, and both generation routes improve real-world policy success rates over real-demonstration-only training in the evaluated settings. These results support universal latent actions as a scalable interface for learning controllable visual dynamics across heterogeneous embodiments.
Figures & tables
Figure 1: Pretraining corpus overview. The corpus contains 23 datasets, approximately 6,190 hours, and 2.57M episodes. The figure summarizes the distribution of human, real-robot, and simulation video, dataset contributions by duration, and object and action vocabulary.
Figure 2: Overview of the world-model data engine. Top: Scalable data editing. RynnLAM extracts latent actions from a source human video, while its first frame is edited to depict the target robot. Conditioned on the edited frame and the original latent actions, RynnWorld-Latent generates robot video rollouts. The generated observations are paired with pseudo-actions obtained through motion retargeting and inverse kinematics. Bottom: Digital teleoperation. Human control signals are retargeted to robot actions, which condition RynnWorld-Latent together with a reference scene. Generated frames provide visual feedback to the operator, while the recorded actions and generated observations form synthetic demonstrations without physical robot execution.
Figure 3: RynnEdit. Two routes construct a robot-centric reference frame: explicit hand removal, retargeting, and robot rendering; or direct replacement of human hands with robotic counterparts using an image-editing model. The resulting first frame conditions subsequent world-model generation.
Figure 4: Standardized Experimental Platforms. Two dual-arm embodiments used for real-world evaluation: the stationary Marvin-WUJI (dual 7-DOF arms with 20-DOF dexterous hands) and the mobile Astribot-S1 (dual 7-DOF arms with parallel grippers on a wheeled base).
Model
In-lab Eval
EgoDex Eval
DreamDojo-HV Eval
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
Cosmos-Predict2.5 ( Ali et al., 2025 )
20.58
0.77
0.22
19.95
0.79
0.22
18.27
0.75
0.24
DreamDojo-2B ( Gao et al., 2026 )
21.11
0.77
0.22
20.41
0.78
0.23
18.81
0.75
0.24
DreamDojo-14B ( Gao et al., 2026 )
21.41
0.79
0.21
20.53
0.79
0.21
18.92
0.75
0.23
RynnWorld-Latent
21.42
0.82
0.20
20.89
0.81
0.19
19.22
0.79
0.21
Table 2: Results on the DreamDojo GR-1 evaluation sets ( Gao et al., 2026 ) . Baseline values are literature-reported measurements after robot post-training.
Method
Marvin-WUJI Eval
Astribot-S1 Eval
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
Cosmos3-Edge ( Agarwal et al., 2026 )
20.03
0.78
0.31
13.99
0.42
0.54
Cosmos3-Nano ( Agarwal et al., 2026 )
20.09
0.80
0.26
14.22
0.53
0.45
RynnWorld-Teleop ( Zhao et al., 2026c )
20.89
0.76
0.19
N/A
N/A
N/A
RynnWorld-Latent
21.86
0.80
0.12
15.96
0.60
0.34
Ablation Study
Table 3: Results and ablation study on our real-robot evaluation sets. ‘N/A’ denotes baseline implementations without support for Astribot-S1 mobile-base motion. PSNR, SSIM, and LPIPS are interpreted jointly as measures of video prediction fidelity.
Figure 5: Action following under a shared initial observation. Each comparison uses latent-action sequences extracted from two different source videos. Both sequences are paired with the same initial frame to generate separate rollouts. Blue and green arrows associate source clips with their corresponding generated rollouts. The generated motion qualitatively follows the respective source sequence despite the shared initial scene.
Source Dataset
Synthetic Frames
EPIC-KITCHENS ( Damen et al., 2020 )
2.5M
EgoDex ( Hoque et al., 2025 )
80.8M
Something-Something ( Goyal et al., 2017 )
1.5M
RynnWorld-Latent-80M (total)
84.8M
Table 5: Synthetic corpus by source dataset. The RynnWorld-Latent data engine processes these human interaction videos into robot-action-labeled synthetic episodes. After filtering, the total corpus contains approximately 800 hours.
Method
LIBERO
LIBERO-Plus
π0 ( Black et al., 2024a )
94.4
53.6
π0 + RynnWorld-Latent-80M
96.1
79.4
π0.5 ( Intelligence et al., 2025 )
96.9
84.4
π0.5 + RynnWorld-Latent-80M
97.8
84.8
Table 6: Policy success rates on LIBERO ( Liu et al., 2023 ) and LIBERO-Plus. Values are percentages. Each synthetic-data variant uses RynnWorld-Latent-80M for mid-training before downstream evaluation.
Figure 6: Scalable data editing. Given a source video, we edit its first frame to replace the embodiment or environment, then condition RynnWorld-Latent on the edited frame and the source latent-action sequence to synthesize a robotized rollout.
Figure 7: Interactive digital teleoperation. Operator control signals are retargeted to target-robot actions and condition RynnWorld-Latent together with a reference scene; the recorded actions and generated frames form synthetic demonstrations without physical robot execution.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Marvin-WUJI demonstration collection. An operator controls dual robot arms and dexterous hands using motion trackers and data gloves.
Task
Description
Episodes
Plate-Placement
Take a plate from a shelf and place it beside the oven.
500
Steak-Cooking
Flip a steak in a pan.
800
Plate-Wiping
Take a dirty plate from a shelf, wipe it with a cloth, and return it.
500
Table-Cleaning
Collect scattered items into a box and place the box on a shelf.
850
Steak-Serving
Carry a steak plate to the dining area and place it on a placemat.
600
Plate-Retrieval
Collect a dirty plate and cutlery and return them to their designated positions.
600
Appendix
Table 7: Collected real-world demonstrations. The corpus is shared with RynnVLA-Latent. Counts refer to physically collected episodes, not synthetic data.