Predicting the consequences of an agent's actions requires modeling both the body's motion and the environment's response. Most video world models attempt to learn both from numerical actions, even though geometry and kinematics already specify the intended motion. We introduce Doppel, an embodiment-agnostic video world model conditioned on camera-aligned renderings of future body configurations and numerical action trajectories. By explicitly providing the intended motion, Doppel can focus on predicting object motion and interaction-driven changes in the environment. Rendered motion also provides a common visual interface across different embodiments and action spaces. To train on robot datasets without camera extrinsics, we recover camera poses and align rendered trajectories with recorded observations. We jointly train a single shared video backbone and conditioning adapter on eight datasets spanning eight robot and human embodiments, from single- and bimanual robot arms to a dexterous humanoid and human hands and bodies, using about 600 hours of video without embodiment labels. Across three robot-arm datasets, Doppel improves PSNR over action-only baselines by 1.00 dB on held-out trajectories and 1.20 dB under out-of-distribution evaluation, while predicting more consistent object motion. Training on all eight embodiments (Doppel-8) improves robot predictions over the three-robot model (Doppel-3) on four of six evaluation sets. Project page: https://doppelwm.github.io/.
Figures & tables
Figure 1: Doppel is a surprisingly simple approach for incorporating future embodiment renderings and improving robot-object interactions in action-conditioned video world models.
Figure 2: Doppel Architecture. We encode camera-aligned robot renderings with a frozen VAE and fuse their features with numerical actions. We use the resulting representation to modulate the video backbone, which predicts future frames from the initial observation.
Figure 3: We show predictions with Doppel on Franka, WidowX, and UR5e arms in out-of-domain scenes with unseen objects (top to bottom). Each row shows the initial image, future robot renderings used as conditioning, and our predicted frames. We use the same model across all three embodiments. Our model preserves embodiment-consistent motion while predicting plausible environment responses under substantial distribution shift.
Figure 4: We show predictions with Doppel-8 on held-out test clips from the five added datasets: a bimanual YAM robot (ABC), a humanoid with dexterous hands (T-Rex), a human body (Ego-Exo4D), human hands (GigaHands), and a human upper body (OakInk2), from top to bottom. Each row shows the initial image, future embodiment renderings used as conditioning, and our predicted frames. We use the same model across all five embodiments, with no embodiment label.
Figure 5: Qualitative comparison across trajectory and OOD generalization settings. Compared to the baseline we observe that Doppel generates much plausible robot-object interactions as intended by the provided action trajectory. Additional qualitative results are available on our project website (Appendix A.1 ) and a trajectory generalization illustration in Fig. 21 .
Dataset
Embodiment
Train episodes
Train clips
Val. clips
Test clips
Test distribution shift
Droid
Franka Panda
16726
83511
200
200
Unseen lab
Bridge
WidowX
12387
61935
200
200
Unseen kitchen
RoboMind
UR5e + 2F-85
14801
74005
200
200
Unseen objects
ABC
Bimanual YAM
9115
274523
200
200
Unseen tasks
T-Rex
Vega-1 + Sharpa hands
4784
58120
200
200
Unseen backgrounds
Ego-Exo4D
Human body (SMPL-H)
2041
213292
200
200
Unseen tasks
Table 1: Training corpus. Validation clips come from held-out trajectories in seen settings; test clips hold out the listed shift.
Validation: unseen trajectories
OOD test
Dataset
Method
Input
Trained on
PSNR ↑
SSIM ↑
LPIPS ↓
SCS ↑
PSNR ↑
SSIM ↑
LPIPS ↓
SCS ↑
Droid
Action-only
Actions
This dataset
22.888
0.841
0.078
0.927
20.069
0.779
0.131
0.873
OSCAR (fine-tuned)
Skeleton
This dataset
20.427
0.751
0.135
0.903
18.619
0.705
0.174
0.854
Doppel
Render + actions
This dataset
24.357
0.864
0.065
0.940
21.276
0.802
0.113
0.902
Doppel-3
Render + actions
3 robot datasets
24.297
0.862
0.066
0.940
21.250
0.803
0.111
0.899
Doppel-8
Render + actions
All 8 datasets
24.819
0.870
0.062
0.943
21.460
0.806
0.108
0.903
Table 2: Prediction quality on the three robot datasets. Input is what the model is conditioned on besides the first frame; Trained on is its training data. All our models are evaluated at their final checkpoint and saw a similar amount of each robot dataset during training.
Validation: unseen trajectories
Test: held-out shift
Dataset
PSNR ↑
SSIM ↑
LPIPS ↓
SCS ↑
PSNR ↑
SSIM ↑
LPIPS ↓
SCS ↑
ABC
24.653
0.850
0.096
0.920
25.217
0.861
0.089
0.927
T-Rex
28.327
0.871
0.050
0.961
28.346
0.867
0.051
0.956
Ego-Exo4D
27.935
0.895
0.045
0.966
27.611
0.892
0.044
0.965
GigaHands
25.622
0.885
0.053
0.941
25.784
0.897
0.049
0.944
OakInk2
29.014
0.918
0.036
0.970
29.090
0.921
0.033
0.970
Table 3: Doppel-8 , evaluated on the five datasets added beyond the three robot-arm datasets: a bimanual robot (ABC), a humanoid with dexterous hands (T-Rex), human bodies (Ego-Exo4D), human hands (GigaHands), and human upper bodies (OakInk2). These are the same weights as the last row of each block in Table 2 . Validation holds out trajectories; the test split holds out the shift listed in Table 1 .
Figure 6: Prediction quality by image region (robot and moving objects) on the three robot datasets, for every model at its final checkpoint, on the same clips as Table 2 . Values are in Appendix Table 13 .
Figure 7: Training efficiency of Doppel versus single-dataset action-only experts. Doppel achieves higher validation and OOD prediction quality at comparable training draws.
Figure 8: Long-horizon rollouts of Doppel-8 on held-out test episodes from OakInk2 (top) and Ego-Exo4D (bottom). Each episode is predicted for 25.6 s from its first frame alone by chaining 3.2 s windows on the model’s own predictions. Each row shows the initial image, the embodiment renderings given as conditioning, and our predictions at 8.6, 17.0, and 25.6 s.
Figure 9: Training efficiency of Doppel-3 versus a three-robot action-only baseline. Doppel achieves higher validation and OOD prediction quality throughout 3.36M training draws.
Appendix figures & tables26 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 10: Alignment between the observed and rendered robot trajectories. The top row shows frames from the real RGB trajectory, and the bottom row shows the corresponding target-view robot renderings. The renderings express the commanded embodiment motion in the same image coordinates as the observations, providing the dense visual component of Doppel .
Figure 11: Alignment between the observed and rendered trajectories for the five added embodiments: a bimanual YAM robot (ABC), a humanoid with dexterous hands (T-Rex), a human body (Ego-Exo4D), human hands (GigaHands), and a human upper body (OakInk2). For each dataset, the top row shows the real RGB frames and the bottom row the corresponding camera-aligned renderings.
Dataset
Train
Val.
Test
Total
Droid
98.9
5.4
11.6
115.9
Bridge
26.4
1.4
3.2
31.1
RoboMind
62.1
3.2
6.8
72.1
ABC
242.6
12.7
31.4
286.7
T-Rex
44.2
2.4
3.9
50.5
Ego-Exo4D
88.3
5.1
10.4
103.8
Appendix
Table 4: Hours of video per dataset and split, from the full length of each source episode; each episode counts once, and Ego-Exo4D is counted per take (training uses up to two exocentric cameras per take, 176.3 h of camera video).
Our training split (Table 1 ), using 81-frame windows at the selected start positions, with overlapping windows permitted
Clip length
81 frames, native frame rate
81 frames, native frame rate
Resolution
640×480
432×240
Optimizer
AdamW
AdamW, β=(0.9,0.99) , weight decay 10−3
Appendix
Table 5: OSCAR configurations used in our experiments. The fine-tuned models are initialized from the released OSCAR checkpoint [ 64 ] .
Figure 12: Representative examples from Droid with corresponding camera-aligned robot renderings.
Figure 13: Representative examples from Bridge with corresponding camera-aligned robot renderings.
Figure 14: Representative examples from RoboMind with corresponding camera-aligned robot renderings.
Figure 15: Representative examples from ABC (bimanual YAM) with corresponding camera-aligned renderings.
Figure 16: Representative examples from T-Rex (humanoid with dexterous hands) with corresponding camera-aligned renderings.
Figure 17: Representative examples from Ego-Exo4D (human body) with corresponding camera-aligned renderings.
Figure 18: Representative examples from GigaHands (human hands) with corresponding camera-aligned renderings.
Figure 19: Representative examples from OakInk2 (human upper body) with corresponding camera-aligned renderings.
Backbone
Wan 2.1 I2V 14B (16.4B parameters). We train all 40 transformer blocks together with the output and timestep projections, yielding 16.3B trainable backbone parameters.
17 frames at 240×432 . Contiguous windows use 0.2 s temporal spacing, while full-span windows cover the complete demonstration. The text prompt is always empty.
Optimizer
AdamW [ 39 ] with weight decay 10−3 and gradient clipping at 1.0 . Training uses bf16, fp32 master weights, FSDP, and activation checkpointing.
Learning rates
5×10−6 for the backbone and 2×10−5 for the conditioning adapter.
Schedule
Cosine decay with 500 warm-up steps and a minimum learning-rate ratio of 0.05 .
Appendix
Table 6: Training configuration for the single-embodiment models.
Data
Joint training on Droid , RoboMind , and Bridge . Datasets are sampled with equal probability, and clips are sampled uniformly within each dataset.
Hardware
8 H100-80GB GPUs on one node, with one FSDP rank per GPU.
Batch size
8 examples per GPU, giving a global batch size of 64.
Learning rates
Backbone: 5×10−6 ; adapter: 2×10−5 .
Schedule
Cosine decay with 500 warm-up steps and a minimum learning-rate ratio of 0.05 .
EMA
Decay of 0.9975 , updated every 10 steps after a 500-step warm-up.
Appendix
Table 7: Training configuration for the single-node multi-embodiment model. Settings not listed are identical to Table 6 .
Data
Joint training on Droid , RoboMind , and Bridge . Datasets are sampled with equal probability, and clips are sampled uniformly within each dataset.
Hardware
32 H100-80GB GPUs across four nodes, with one FSDP rank per GPU.
Batch size
8 examples per GPU, giving a global batch size of 256.
Learning rates
Backbone: 1×10−5 ; adapter: 4×10−5 .
Schedule
Cosine decay with 250 warm-up steps and a minimum learning-rate ratio of 0.05 .
Weight EMA
Decay of 0.995 , updated every 10 steps after a 250-step warm-up.
Appendix
Table 8: Training configuration for the four-node multi-embodiment render–action model. Settings not listed are identical to Table 6 .
Data
Joint training on the eight datasets of Table 1 (803,602 training clips, 593 hours of training video, Table 4 ). Datasets are sampled in proportion to their training clips, and clips uniformly within each dataset.
Actions
Every action vector is zero-padded to 159 dimensions and passed to one shared action encoder; there is no embodiment label or embodiment-specific parameter.
Hardware
32 GPUs across four nodes, one FSDP rank per GPU: H100-80GB.
Appendix
Table 9: Training configuration for Doppel-8 . Settings not listed are identical to Table 8 .
Figure 20: Overlays of camera calibration results.
Figure 21: Additional trajectory-generalization example on Droid . We compare Doppel with the action-only baseline on a held-out trajectory from a familiar environment.
Dataset
Split
Doppel-3
Doppel-8
Δ PSNR
Better
Δ arm
Δ moving obj.
Droid
Val.
24.30
24.82
+0.52±0.03
92%
+0.70
+0.69
OOD
21.25
21.46
+0.21±0.03
73%
+0.34
+0.13
Bridge
Val.
26.55
26.54
−0.01±0.06
57%
+0.09
+0.12
OOD
22.47
22.24
−0.24±0.05
38%
−0.01
−0.08
RoboMind
Val.
31.07
31.30
+0.24±0.04
75%
+0.42
+0.14
OOD
30.44
30.66
+0.22±0.04
72%
+0.34
+0.16
Appendix
Table 10: Doppel-3 versus Doppel-8 , paired on the same 200 clips per set (final checkpoints). Δ is Doppel-8 minus Doppel-3 with its standard error; Better is the share of clips where Doppel-8 has higher PSNR; arm and moving-object Δ use the region metrics of Appendix A.8 .
Doppel-3
Δ PSNR ( Doppel-8 − Doppel-3 )
Dataset
Split
passes
≈30 passes
51 passes
final
Droid
Val.
24.8
+0.70
+0.45
+0.52
OOD
24.8
+0.51
+0.17
+0.21
Bridge
Val.
33.4
+0.25
+0.13
−0.01
OOD
33.4
−0.10
−0.05
−0.24
RoboMind
Val.
27.9
+0.52
+0.21
+0.24
Appendix
Table 11: PSNR of Doppel-8 minus Doppel-3 , on the same 200 clips per set, at three points in training.
Validation: unseen trajectories
OOD test
Dataset
Method
Input
Trained on
PSNR ↑
SSIM ↑
LPIPS ↓
SCS ↑
PSNR ↑
SSIM ↑
LPIPS ↓
SCS ↑
Droid
Action-only
Actions
This dataset
22.847
0.841
0.079
0.927
20.047
0.780
0.131
0.873
Doppel
Render + actions
This dataset
24.322
0.864
0.065
0.940
21.278
0.802
0.113
0.903
Doppel-3
Render + actions
3 robot datasets
24.297
0.862
0.066
0.940
21.250
0.803
0.111
0.899
Doppel-8
Render + actions
All 8 datasets
24.749
0.869
0.063
0.943
21.421
0.805
0.109
0.901
Bridge
Action-only
Actions
This dataset
25.625
0.864
0.054
0.941
20.665
0.756
0.129
0.869
Appendix
Table 12: Prediction quality on the three robot datasets at matched training exposure: every model is evaluated at its saved checkpoint closest to 51 passes over each robot dataset.
Validation: unseen trajectories
OOD test
Dataset
Method
Input
Trained on
PSNRarm
PSNRobj
SCSarm
SCSobj
PSNRarm
PSNRobj
SCSarm
SCSobj
Droid
Action-only
Actions
This dataset
18.74
27.26
0.808
0.839
17.64
24.67
0.712
0.746
OSCAR (fine-tuned)
Skeleton
This dataset
16.54
22.92
0.778
0.801
16.05
22.08
0.712
0.728
Doppel
Render + actions
This dataset
20.88
28.63
0.843
0.867
19.33
26.50
0.773
0.805
Doppel-3
Render + actions
3 robot datasets
20.89
28.56
0.846
0.869
19.23
26.41
0.777
0.800
Doppel-8
Render + actions
All 8 datasets
21.59
29.25
0.853
0.876
19.57
26.54
0.795
0.805
Appendix
Table 13: Prediction quality by image region on the three robot datasets, for the same models, checkpoints and clips as Table 2 . Arm is the robot; Obj. is the objects that move during the clip (non-robot regions segmented in the first frame with SAM 2 and tracked through the clip). PSNR is computed inside each region and averaged over the 16 predicted frames; SCS is structural consistency of the same regions.
Droid
Bridge
RoboMind
Method
Draws/dataset (M)
Val
OOD
Val
OOD
Val
OOD
Action-only expert
∼0.5
19.194
18.058
22.136
21.119
25.199
25.848
Doppel-3
∼0.5
20.815
19.505
24.253
22.314
28.560
28.634
Action-only expert
∼1.0
19.583
18.110
23.374
21.279
26.449
27.016
Doppel-3
∼1.0
22.768
20.545
24.935
22.907
29.386
29.589
Action-only expert
∼2.0
20.963
18.982
24.805
20.552
27.501
28.039
Appendix
Table 14: PSNR at every checkpoint in the expert-versus-shared-model scaling comparison. Action-only experts are trained separately on each dataset; Doppel is trained jointly on all three.
Droid
Bridge
RoboMind
Method
Draws/dataset (M)
Val
OOD
Val
OOD
Val
OOD
Action-only expert
∼0.5
0.868
0.826
0.901
0.875
0.897
0.899
Doppel-3
∼0.5
0.914
0.876
0.927
0.917
0.927
0.926
Action-only expert
∼1.0
0.877
0.826
0.916
0.883
0.911
0.909
Doppel-3
∼1.0
0.926
0.888
0.936
0.915
0.933
0.931
Action-only expert
∼2.0
0.905
0.852
0.933
0.868
0.919
0.919
Appendix
Table 15: SCS at every checkpoint in the expert-versus-shared-model scaling comparison. Action-only experts are trained separately on each dataset; Doppel is trained jointly on all three.
Droid
Bridge
RoboMind
Method
Total draws (M)
Val
OOD
Val
OOD
Val
OOD
Action-only (multi)
∼1.2
19.126
18.029
21.703
20.762
24.520
24.958
Doppel-3 (batch 64)
∼1.2
22.070
20.251
24.098
22.781
28.504
29.082
Action-only (multi)
∼2.4
20.381
18.762
23.196
21.722
26.575
26.965
Doppel-3 (batch 64)
∼2.4
22.832
20.697
24.968
23.156
29.382
29.670
Action-only (multi)
∼3.4
20.959
19.036
24.070
21.873
27.564
28.200
Appendix
Table 16: PSNR at every checkpoint in the matched shared-model scaling comparison. Both models are trained jointly on all three datasets.
Droid
Bridge
RoboMind
Method
Total draws (M)
Val
OOD
Val
OOD
Val
OOD
Action-only (multi)
∼1.2
0.864
0.824
0.892
0.859
0.886
0.886
Doppel-3 (batch 64)
∼1.2
0.919
0.883
0.927
0.914
0.927
0.932
Action-only (multi)
∼2.4
0.893
0.839
0.914
0.886
0.910
0.910
Doppel-3 (batch 64)
∼2.4
0.927
0.892
0.936
0.919
0.930
0.934
Action-only (multi)
∼3.4
0.901
0.846
0.925
0.890
0.920
0.916
Appendix
Table 17: SCS at every checkpoint in the matched shared-model scaling comparison. Both models are trained jointly on all three datasets.
We present DreamTrue, a multi-view, cross-embodiment robot world model for action-faithful and physically plausible video prediction. Training such a model on existing robot datasets faces two obstacles: imprecise calibration can impair action following, while limited coverage of unsuccessful interactions can bias predictions toward successful outcomes. To improve action following across embodiments, we render action trajectories into image-space conditions and introduce offline geometric calibration to align these conditions with the target videos. To broaden interaction coverage, we introduce counterfactual post-training, modifying recorded action trajectories and generating future videos under a wider range of actions and contact configurations. To provide feedback on these predictions without paired ground-truth futures, we construct a human-annotated video dataset covering robot, object, and interaction defects and use it to train an embodied video reward model. Its scores guide reinforcement-learning post-training toward more physically plausible interaction outcomes. On AgiBot, DreamTrue attains state-of-the-art action following, while reducing the human-assessed interaction defect rate from from 48.12% to 6.25%. Notably, our model ranks first in the world model track of the AgiBot World Challenge 2026. The project page can be found at https://brave-eai.github.io/DreamTrue.
Junyan Li, Ruizhi Li, Yu Liu +6
NLPR, Institute of Automation, Chinese Academy of Sciences (CASIA) · Amap, Alibaba Group
Action-conditioned video world models predict future observations from an initial observation and an action signal. We propose robot-factored world models (RoFacto), which move two robot-specific factors outside the world model to separate action realization from scene-response prediction. First, action realization: each command sequence is rolled through the robot's own controller and kinematics without scene contact into a deployment-available nominal trajectory that does not require future realized states. Second, robot rendering: this nominal trajectory is rendered through the robot URDF, factoring the robot's geometry, kinematics, and appearance out of the model and into explicit rendered robot geometry. Paired end-effector and scene depth provide geometric cues for contact and occlusion, while camera-aware static RGB context and rendered robot geometry form a shared visual interface across viewpoints and embodiments through which the model learns scene responses. Our experiments show that the rendered interface outperforms vector-conditioned baselines, supports transfer to unseen robot embodiments, and generates robot manipulation videos from retargeted human demonstrations.
We introduce Qwen-RobotWorld, a language-conditioned video world model for embodied intelligence. With natural language as a unified action interface, it predicts physically grounded future visual trajectories from current observations across robotic manipulation, autonomous driving, indoor navigation, and human-to-robot transfer. This unified formulation provides three promising application directions: synthetic data generation for policy training augmentation, scalable virtual environments for policy evaluation, and language-guided planning signals for downstream robot control. This is achieved through a three-part design: a) Double-Stream MMDiT with MLLM Action Encoding, where a 60-layer double-stream diffusion transformer couples frozen Qwen2.5-VL semantics with video-VAE latents through layer-wise joint attention; b) Embodied World Knowledge (EWK), an 8.6M video-text corpus (200M+ frames) with action-language mapping over 20+ embodiments and 500+ action categories; and c) General+Expert Progressive Curriculum, a two-stage training strategy that first learns general visual priors and then injects embodied specialization under a shared language interface. Extensive results show strong competitiveness: ranks 1st overall on EWMBench and DreamGen Bench, outperforms all open-source models on WorldModelBench and PBench. Additional zero-shot analyses on RoboTwin-IF benchmark further support robust generalization and multi-view consistency.