Render the Robot, Generate the World: One Video World Model Across Embodiments
Organizations: Johns Hopkins University · Carnegie Mellon University · Toyota Research Institute
Abstract
Predicting the consequences of an agent's actions requires modeling both the body's motion and the environment's response. Most video world models attempt to learn both from numerical actions, even though geometry and kinematics already specify the intended motion. We introduce Doppel, an embodiment-agnostic video world model conditioned on camera-aligned renderings of future body configurations and numerical action trajectories. By explicitly providing the intended motion, Doppel can focus on predicting object motion and interaction-driven changes in the environment. Rendered motion also provides a common visual interface across different embodiments and action spaces. To train on robot datasets without camera extrinsics, we recover camera poses and align rendered trajectories with recorded observations. We jointly train a single shared video backbone and conditioning adapter on eight datasets spanning eight robot and human embodiments, from single- and bimanual robot arms to a dexterous humanoid and human hands and bodies, using about 600 hours of video without embodiment labels. Across three robot-arm datasets, Doppel improves PSNR over action-only baselines by 1.00 dB on held-out trajectories and 1.20 dB under out-of-distribution evaluation, while predicting more consistent object motion. Training on all eight embodiments (Doppel-8) improves robot predictions over the three-robot model (Doppel-3) on four of six evaluation sets. Project page: https://doppelwm.github.io/.
Figures & tables
| Dataset | Embodiment | Train episodes | Train clips | Val. clips | Test clips | Test distribution shift |
|---|---|---|---|---|---|---|
| Droid | Franka Panda | 16726 | 83511 | 200 | 200 | Unseen lab |
| Bridge | WidowX | 12387 | 61935 | 200 | 200 | Unseen kitchen |
| RoboMind | UR5e + 2F-85 | 14801 | 74005 | 200 | 200 | Unseen objects |
| ABC | Bimanual YAM | 9115 | 274523 | 200 | 200 | Unseen tasks |
| T-Rex | Vega-1 + Sharpa hands | 4784 | 58120 | 200 | 200 | Unseen backgrounds |
| Ego-Exo4D | Human body (SMPL-H) | 2041 | 213292 | 200 | 200 | Unseen tasks |
| Validation: unseen trajectories | OOD test | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Dataset | Method | Input | Trained on | PSNR | SSIM | LPIPS | SCS | PSNR | SSIM | LPIPS | SCS |
| Droid | Action-only | Actions | This dataset | 22.888 | 0.841 | 0.078 | 0.927 | 20.069 | 0.779 | 0.131 | 0.873 |
| OSCAR (fine-tuned) | Skeleton | This dataset | 20.427 | 0.751 | 0.135 | 0.903 | 18.619 | 0.705 | 0.174 | 0.854 | |
| Doppel | Render + actions | This dataset | 24.357 | 0.864 | 0.065 | 0.940 | 21.276 | 0.802 | 0.113 | 0.902 | |
| Doppel-3 | Render + actions | 3 robot datasets | 24.297 | 0.862 | 0.066 | 0.940 | 21.250 | 0.803 | 0.111 | 0.899 | |
| Doppel-8 | Render + actions | All 8 datasets | 24.819 | 0.870 | 0.062 | 0.943 | 21.460 | 0.806 | 0.108 | 0.903 | |
| Validation: unseen trajectories | Test: held-out shift | |||||||
|---|---|---|---|---|---|---|---|---|
| Dataset | PSNR | SSIM | LPIPS | SCS | PSNR | SSIM | LPIPS | SCS |
| ABC | 24.653 | 0.850 | 0.096 | 0.920 | 25.217 | 0.861 | 0.089 | 0.927 |
| T-Rex | 28.327 | 0.871 | 0.050 | 0.961 | 28.346 | 0.867 | 0.051 | 0.956 |
| Ego-Exo4D | 27.935 | 0.895 | 0.045 | 0.966 | 27.611 | 0.892 | 0.044 | 0.965 |
| GigaHands | 25.622 | 0.885 | 0.053 | 0.941 | 25.784 | 0.897 | 0.049 | 0.944 |
| OakInk2 | 29.014 | 0.918 | 0.036 | 0.970 | 29.090 | 0.921 | 0.033 | 0.970 |
Appendix figures & tables26 assets
Supplementary material from the paper’s appendix.
Appendix
| Dataset | Train | Val. | Test | Total |
|---|---|---|---|---|
| Droid | 98.9 | 5.4 | 11.6 | 115.9 |
| Bridge | 26.4 | 1.4 | 3.2 | 31.1 |
| RoboMind | 62.1 | 3.2 | 6.8 | 72.1 |
| ABC | 242.6 | 12.7 | 31.4 | 286.7 |
| T-Rex | 44.2 | 2.4 | 3.9 | 50.5 |
| Ego-Exo4D | 88.3 | 5.1 | 10.4 | 103.8 |
| Released OSCAR | OSCAR (Fine-tuned) | |
|---|---|---|
| Initialization | Cosmos-Predict2.5 [ 43 ] , 2B | Released OSCAR checkpoint |
| Conditioning | 2D skeleton + text | 2D skeleton + empty text prompt |
| Training data | 180657 episodes: 94830 robot episodes (RH20T [ 14 ] , InternData-A1 [ 53 ] , DROID [ 32 ] 21904, AgiBot-Beta [ 1 ] , AIROA-MoMa [ 51 ] ) and 85827 human episodes (EgoDex [ 26 ] , EPIC-Kitchens [ 10 ] ) | Our training split (Table 1 ), using 81-frame windows at the selected start positions, with overlapping windows permitted |
| Clip length | 81 frames, native frame rate | 81 frames, native frame rate |
| Resolution | ||
| Optimizer | AdamW | AdamW, , weight decay |
| Backbone | Wan 2.1 I2V 14B (16.4B parameters). We train all 40 transformer blocks together with the output and timestep projections, yielding 16.3B trainable backbone parameters. |
| Adapters | Render–action adapter: 30.5M parameters; action-only adapter: 4.2M parameters. |
| Data | One dataset per model: see Table 1 . |
| Input | 17 frames at . Contiguous windows use s temporal spacing, while full-span windows cover the complete demonstration. The text prompt is always empty. |
| Optimizer | AdamW [ 39 ] with weight decay and gradient clipping at . Training uses bf16, fp32 master weights, FSDP, and activation checkpointing. |
| Learning rates | for the backbone and for the conditioning adapter. |
| Schedule | Cosine decay with 500 warm-up steps and a minimum learning-rate ratio of . |
| Data | Joint training on Droid , RoboMind , and Bridge . Datasets are sampled with equal probability, and clips are sampled uniformly within each dataset. |
| Hardware | 8 H100-80GB GPUs on one node, with one FSDP rank per GPU. |
| Batch size | 8 examples per GPU, giving a global batch size of 64. |
| Learning rates | Backbone: ; adapter: . |
| Schedule | Cosine decay with 500 warm-up steps and a minimum learning-rate ratio of . |
| EMA | Decay of , updated every 10 steps after a 500-step warm-up. |
| Data | Joint training on Droid , RoboMind , and Bridge . Datasets are sampled with equal probability, and clips are sampled uniformly within each dataset. |
| Hardware | 32 H100-80GB GPUs across four nodes, with one FSDP rank per GPU. |
| Batch size | 8 examples per GPU, giving a global batch size of 256. |
| Learning rates | Backbone: ; adapter: . |
| Schedule | Cosine decay with 250 warm-up steps and a minimum learning-rate ratio of . |
| Weight EMA | Decay of , updated every 10 steps after a 250-step warm-up. |
| Data | Joint training on the eight datasets of Table 1 (803,602 training clips, 593 hours of training video, Table 4 ). Datasets are sampled in proportion to their training clips, and clips uniformly within each dataset. |
| Actions | Every action vector is zero-padded to 159 dimensions and passed to one shared action encoder; there is no embodiment label or embodiment-specific parameter. |
| Hardware | 32 GPUs across four nodes, one FSDP rank per GPU: H100-80GB. |
| Dataset | Split | Doppel-3 | Doppel-8 | PSNR | Better | arm | moving obj. |
|---|---|---|---|---|---|---|---|
| Droid | Val. | 24.30 | 24.82 | 92% | |||
| OOD | 21.25 | 21.46 | 73% | ||||
| Bridge | Val. | 26.55 | 26.54 | 57% | |||
| OOD | 22.47 | 22.24 | 38% | ||||
| RoboMind | Val. | 31.07 | 31.30 | 75% | |||
| OOD | 30.44 | 30.66 | 72% |
| Doppel-3 | PSNR ( Doppel-8 Doppel-3 ) | ||||
|---|---|---|---|---|---|
| Dataset | Split | passes | passes | 51 passes | final |
| Droid | Val. | 24.8 | |||
| OOD | 24.8 | ||||
| Bridge | Val. | 33.4 | |||
| OOD | 33.4 | ||||
| RoboMind | Val. | 27.9 | |||
| Validation: unseen trajectories | OOD test | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Dataset | Method | Input | Trained on | PSNR | SSIM | LPIPS | SCS | PSNR | SSIM | LPIPS | SCS |
| Droid | Action-only | Actions | This dataset | 22.847 | 0.841 | 0.079 | 0.927 | 20.047 | 0.780 | 0.131 | 0.873 |
| Doppel | Render + actions | This dataset | 24.322 | 0.864 | 0.065 | 0.940 | 21.278 | 0.802 | 0.113 | 0.903 | |
| Doppel-3 | Render + actions | 3 robot datasets | 24.297 | 0.862 | 0.066 | 0.940 | 21.250 | 0.803 | 0.111 | 0.899 | |
| Doppel-8 | Render + actions | All 8 datasets | 24.749 | 0.869 | 0.063 | 0.943 | 21.421 | 0.805 | 0.109 | 0.901 | |
| Bridge | Action-only | Actions | This dataset | 25.625 | 0.864 | 0.054 | 0.941 | 20.665 | 0.756 | 0.129 | 0.869 |
| Validation: unseen trajectories | OOD test | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Dataset | Method | Input | Trained on | ||||||||
| Droid | Action-only | Actions | This dataset | 18.74 | 27.26 | 0.808 | 0.839 | 17.64 | 24.67 | 0.712 | 0.746 |
| OSCAR (fine-tuned) | Skeleton | This dataset | 16.54 | 22.92 | 0.778 | 0.801 | 16.05 | 22.08 | 0.712 | 0.728 | |
| Doppel | Render + actions | This dataset | 20.88 | 28.63 | 0.843 | 0.867 | 19.33 | 26.50 | 0.773 | 0.805 | |
| Doppel-3 | Render + actions | 3 robot datasets | 20.89 | 28.56 | 0.846 | 0.869 | 19.23 | 26.41 | 0.777 | 0.800 | |
| Doppel-8 | Render + actions | All 8 datasets | 21.59 | 29.25 | 0.853 | 0.876 | 19.57 | 26.54 | 0.795 | 0.805 | |
| Droid | Bridge | RoboMind | |||||
|---|---|---|---|---|---|---|---|
| Method | Draws/dataset (M) | Val | OOD | Val | OOD | Val | OOD |
| Action-only expert | 19.194 | 18.058 | 22.136 | 21.119 | 25.199 | 25.848 | |
| Doppel-3 | 20.815 | 19.505 | 24.253 | 22.314 | 28.560 | 28.634 | |
| Action-only expert | 19.583 | 18.110 | 23.374 | 21.279 | 26.449 | 27.016 | |
| Doppel-3 | 22.768 | 20.545 | 24.935 | 22.907 | 29.386 | 29.589 | |
| Action-only expert | 20.963 | 18.982 | 24.805 | 20.552 | 27.501 | 28.039 | |
| Droid | Bridge | RoboMind | |||||
|---|---|---|---|---|---|---|---|
| Method | Draws/dataset (M) | Val | OOD | Val | OOD | Val | OOD |
| Action-only expert | 0.868 | 0.826 | 0.901 | 0.875 | 0.897 | 0.899 | |
| Doppel-3 | 0.914 | 0.876 | 0.927 | 0.917 | 0.927 | 0.926 | |
| Action-only expert | 0.877 | 0.826 | 0.916 | 0.883 | 0.911 | 0.909 | |
| Doppel-3 | 0.926 | 0.888 | 0.936 | 0.915 | 0.933 | 0.931 | |
| Action-only expert | 0.905 | 0.852 | 0.933 | 0.868 | 0.919 | 0.919 | |
| Droid | Bridge | RoboMind | |||||
|---|---|---|---|---|---|---|---|
| Method | Total draws (M) | Val | OOD | Val | OOD | Val | OOD |
| Action-only (multi) | 19.126 | 18.029 | 21.703 | 20.762 | 24.520 | 24.958 | |
| Doppel-3 (batch 64) | 22.070 | 20.251 | 24.098 | 22.781 | 28.504 | 29.082 | |
| Action-only (multi) | 20.381 | 18.762 | 23.196 | 21.722 | 26.575 | 26.965 | |
| Doppel-3 (batch 64) | 22.832 | 20.697 | 24.968 | 23.156 | 29.382 | 29.670 | |
| Action-only (multi) | 20.959 | 19.036 | 24.070 | 21.873 | 27.564 | 28.200 | |
| Droid | Bridge | RoboMind | |||||
|---|---|---|---|---|---|---|---|
| Method | Total draws (M) | Val | OOD | Val | OOD | Val | OOD |
| Action-only (multi) | 0.864 | 0.824 | 0.892 | 0.859 | 0.886 | 0.886 | |
| Doppel-3 (batch 64) | 0.919 | 0.883 | 0.927 | 0.914 | 0.927 | 0.932 | |
| Action-only (multi) | 0.893 | 0.839 | 0.914 | 0.886 | 0.910 | 0.910 | |
| Doppel-3 (batch 64) | 0.927 | 0.892 | 0.936 | 0.919 | 0.930 | 0.934 | |
| Action-only (multi) | 0.901 | 0.846 | 0.925 | 0.890 | 0.920 | 0.916 | |
World Models for Robotics
World models for robotics predict how scenes and robot states change under actions to support planning, control, and learning.
See all 426 papers