Organizations: State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · Beijing Innovation Center of Humanoid Robotics · New York University · Nanyang Technological University · University of Electronic Science and Technology of China · The Chinese University of Hong Kong
Action-conditioned world models increasingly serve as learned simulators for robotic policy improvement, yet their reliability rests on an unverified assumption: generated futures faithfully reflect the consequences of arbitrary valid actions. Systematic evaluation against action-specific ground truth remains limited beyond expert demonstrations. To address this gap, we introduce WorldEcho, which evaluates action following over a broader action distribution through visual integrity and SE(3) end-effector trajectory alignment. Our diagnosis shows that expert-trained models perform better on demonstrated actions but struggle with off-expert trajectories, producing motion inconsistent with the commanded actions or visually invalid rollouts. To mitigate these failures, we propose WorldSync, which broadens action coverage, grounds intermediate video representations in action-induced robot dynamics through an Action-Forcing Expert, and aligns predicted and ground-truth changes under action interventions. Experiments show that WorldSync reduces integrity-gated error on RoboTwin and, when used for iterative policy improvement, yields higher success rates in simulation and on real robots.
Figure 2: Off-expert failures. Left to right: input and actions, ground-truth final frame, visual collapse, and plausible but action-inconsistent motion.
Figure 3: WorldEcho evaluation across five action-query categories, combining visual integrity with SE(3) end-effector trajectory alignment against matched simulator references.
Figure 4: WorldSync overview: action coverage, the pretrained video backbone, and training-only supervision from the Action-Forcing Expert (AFE) and Intervention-Effect (IE) objectives.
Figure 5: Off-expert action following. (a) Gated error with equally weighted off-expert categories within tasks. Error bars show 95% confidence intervals from task-level resampling. (b) Expert-to-off-expert changes in raw NDTW and visual failure rate. (c) Action PCA with 95% highest-density regions (off-expert: union across categories).
WorldEcho metrics
Video generation metrics †
Model
Emb. PT
Training regime
Gated error ↓
Raw NDTW ↓
Visual pass (%) ↑
PSNR ↑
SSIM ↑
LPIPS ↓
FVD ↓
CtrlWorld
✓
Expert
0.0379
0.0266
95.83
19.41
0.8466
0.1137
113.58
CtrlWorld
✓
Expanded
0.0293
0.0210
96.97
20.83
0.8665
0.0915
69.58
Cosmos-Predict2.5
✓
Expert
0.0551
0.0190
87.09
21.10
0.8750
0.0848
95.69
Cosmos-Predict2.5
✓
Expanded
0.0445
0.0127
88.80
22.95
0.8952
0.0658
51.89
Cosmos3
✓
Expert
0.0989
0.0572
82.51
14.45
0.6846
0.1833
122.58
Table 1: Main WorldEcho comparison. Expert and Expanded denote Expert Demonstrations and Expanded Action Coverage. Bold marks the best scores. Underlining marks second-best WorldEcho scores. Emb. PT marks embodied-data pretraining before task-specific fine-tuning, excluding generic-only video pretraining (Appendix B ). † Video metrics use reconstructed references, not verified as pixel-identical to the original RGB videos. Baseline video and WorldEcho scores use separate prediction batches at the same checkpoint. PSNR, SSIM, and LPIPS average over frames, clips, then tasks. FVD pools all clips.
Figure 6: Policy improvement in simulation and four real-robot tasks. Each real-robot success rate uses 25 trials. Init. denotes the initial policy, and R1/R2 denote improvement rounds.
Variant
Gated ↓
Raw ↓
Pass (%) ↑
PSNR ↑
SSIM ↑
LPIPS ↓
FVD ↓
Expert Base
0.0352
0.0314
98.57
21.23
0.8585
0.1154
175.09
Base
0.0264
0.0204
97.86
20.66
0.8591
0.1087
141.10
+ IE
0.0214
0.0176
98.57
21.05
0.8665
0.0963
125.92
+ AFE
0.0232
0.0172
97.86
21.30
0.8683
0.0962
118.81
IE + AFE
0.0193
0.0138
97.86
22.33
0.8821
0.0830
105.83
Table 2: Data and component ablation on ten RoboTwin tasks at 40k training steps. Bold denotes the best reported value, including ties. Video metrics use separate generations at the same checkpoint. PSNR, SSIM, and LPIPS average over frames, clips, then tasks. FVD pools all video pairs.
Appendix figures & tables37 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: TianJi hardware and teleoperation setup. Numbered markers match the installed devices to the equipment list; the lower panel identifies the hand-motion and end-effector tracking devices.
Figure 8: i7 Pro hardware and teleoperation setup. Numbered markers link the installed cameras and grippers to the equipment list; the left side shows the PICO teleoperation device.
Figure 9: Franka FR3 hardware and camera setup. Numbered markers link the camera locations to the equipment list.
Figure 10: Cup Stacking from a third-person viewpoint. Read left to right; timestamps mark selected frames from one rollout.
Figure 11: Cup Stacking through the robot’s cameras. Columns show the head, left wrist, and right wrist; rows advance in time, with synchronized views within each row. This rollout is separate from Figure 10 ; timestamps are relative to each recording.
Figure 12: Box Packing from a third-person viewpoint. Read left to right; timestamps mark selected frames from one rollout.
Figure 13: Box Packing through the robot’s cameras. Columns show the head, left wrist, and right wrist; rows advance in time, with synchronized views within each row. This rollout is separate from Figure 12 ; timestamps are relative to each recording.
Figure 14: Bread Placement from a third-person viewpoint. Read left to right; timestamps mark selected frames from one rollout.
Figure 15: Bread Placement through the robot’s cameras. Columns show the left external, right external, and wrist views; rows advance in time, with corresponding views within each row. This rollout is separate from Figure 14 ; timestamps are relative to each recording.
Figure 16: Pen Collection from a third-person viewpoint. Read left to right; timestamps mark selected frames from one rollout.
Figure 17: Pen Collection through the robot’s cameras. Columns show the left external, right external, and wrist views; rows advance in time, with corresponding views within each row. This rollout is separate from Figure 16 ; timestamps are relative to each recording.
Dataset
Episodes
Steps
Mean length (steps)
Hours
AgiBot-World-Alpha
32,836
55,255,078
1,682.76
511.62
DROID
92,917
26,959,694
290.15
499.25
Total
125,753
82,214,772
653.78
1,010.87
Appendix
Table 3: Robotic pretraining data after filtering. Steps count recorded timesteps, including episode endpoints, rather than optimizer updates or sampled clips; synchronized camera views are counted once. Hours give the corresponding video duration. Mean length is the number of recorded steps per episode, with the total-row mean computed over all retained episodes.
Setting
Initial
Round 1
Round 2
CtrlWorld + Qwen-OFT
59
46
14
Appendix
Table 4: Reproduction of the VLAW framework. Policy success rates (%) on rotate_qrcode , measured over 100 episodes at each stage. Both CtrlWorld and Qwen-OFT are updated each round.
Figure 18: Action mismatch under identical conditioning actions. World-model prediction (top) and simulator replay (bottom) on dump_bin_bigbin , using the same actions from matched initial conditions. The prediction depicts lifting and pouring, whereas the simulator replay leaves the cup and its contents in place. The arrow indicates temporal order; rightmost panels magnify the boxed final-frame regions.
Setting
Configuration
Improvement rounds
2
Online collection
30 rollouts, including successes and failures
Imagined rollouts
3,000=50 expert starts ×60
Action representation
20-dimensional actions, 32 actions per chunk
Generation horizon
7 chunks (224 actions), 20 denoising steps per chunk
World-model update
1,000 updates, global batch 16, learning rate 10−5
Appendix
Table 5: Shared simulation protocol. Counts are per condition and per round unless stated otherwise. Sampling ratios do not denote unique rollout counts.
Rollout counts : N0→N1→N2
Retention (%)
Method
Round
Initial N0
After S1 N1
After S2 N2
Stage 1 N1/N0
Stage 2 N2/N1
Overall N2/N0
WorldSync
1
3,000
1,114
553
37.1
49.6
18.4
WorldSync
2
3,000
1,088
365
36.3
33.5
12.2
CtrlWorld
1
3,000
962
452
32.1
47.0
15.1
CtrlWorld
2
3,000
909
323
30.3
35.5
10.8
Appendix
Table 6: Sequential filtering counts and retention rates. Stage 1 applies task-progress filtering with sR(z)>0.8 , followed by action-consistency filtering with qk(z)≤η in Stage 2. Stage-wise rates use the preceding stage’s count, whereas overall rates use all 3,000 generated rollouts.
Method
Initial (%)
Round 1 (%)
Round 2 (%)
Gain (pp)
WorldSync
52
57
65
+13
CtrlWorld
52
54
57
+5
Appendix
Table 7: Policy success over two improvement rounds. Success rates use the same ordered set of 100 evaluation seeds. Gains are percentage-point changes from each condition’s measured initial rate.
Mechanism
Representative work
Alignment
Injection site
Fusion
Numerical-based
Additive
NanoWM ( 2026a )
Frame-aligned
Patch features
Element-wise addition
AdaLN Fusion
Cosmos-Predict2.5 ( 2025 )
Frame-aligned
Native AdaLN path
Native-condition fusion
Action AdaLN
IRASim ( 2025 )
Frame-aligned
Per-block attention/FFN
Low-rank shift/scale/gate
Cross-Attention
CtrlWorld ( 2026b )
Frame-aligned, local
Native attention
Gated action attention
Shared-KV Attention
Motus ( 2025 )
Global sequence
Native self-attention
Shared video/action K/V; video Q
Appendix
Table 8: Action-conditioning mechanisms and representative works. All variants use the same video diffusion backbone.
Figure 19: Overview of the evaluated action-conditioning mechanisms. Numerical-based methods use feature addition, modulation, attention, or latent frames, whereas map-based methods use rendered action maps. Blue, orange, green, purple, and gray denote video features, numerical actions, action maps, fusion operations, and DiT blocks, respectively.
Figure 20: RGB action-map representation. (a) Rendered action maps at the current frame and four future steps. (b) Corresponding overlays on the RGB observations. Each column shows the head, left-wrist, and right-wrist views; colored markers encode the projected end-effector poses.
Mechanism
Raw NDTW ↓
Gated ↓
Pass (%) ↑
Params (M)
FLOPs (T)
Additive
0.0320
0.0403
96.43
568.07
0.04
AdaLN Fusion
0.0330
0.0389
97.86
568.07
4.93
Action AdaLN
0.0137
0.0176
98.57
572.01
0.09
Cross-Attention
0.0302
0.0354
97.86
566.49
0.82
Shared-KV Attention
0.0332
0.0370
98.57
566.49
1.27
Latent Frame
0.0239
0.0297
97.86
0
69.39
Appendix
Table 9: Comparison at the last training checkpoint. Params and FLOPs denote additional trainable parameters and inference compute relative to unconditioned video generation. Best performance is bold.
Figure 21: Performance across training checkpoints under the WorldEcho evaluation protocol. The panels report raw NDTW, integrity-gated error, and visual pass rate for all nine mechanisms. Raw and gated errors are task-macro averages; pass rate counts samples. Solid and dashed lines denote numerical-based and map-based mechanisms, respectively. Latent Frame uses equal-weight aggregation of its component results.
Figure 22: Performance–efficiency trade-offs at the common 40k checkpoint under the WorldEcho evaluation protocol. (a) Visual pass rate versus raw NDTW. (b) Integrity-gated error versus additional inference compute on a logarithmic scale. (c) Integrity-gated error versus additional parameters. Arrows indicate preferred directions, and the star marks Action AdaLN (Ours).
Figure 23: AFE architecture. (a) Training-only state supervision from intermediate video features. (b) Query updates within an AFE block. (c) Temporal alignment between future-step queries and video features. AFE updates the backbone through loss gradients and is removed at inference.
Layers ℓi
Latent times j
State target
Gated error ↓
Raw NDTW ↓
Visual pass ↑
Baseline
–
–
0.0717
0.0208
82.14
⌊29i/7+1/2⌋
0≤j≤8
sh
0.0816
0.0210
78.57
8+2i
0≤j≤8
sh
0.0784
0.0197
79.29
8+2i
⌈h/4⌉
sh
0.0702
0.0191
82.14
8+2i
⌈h/4⌉
sh−sh−1
0.0714
0.0159
80.71
8+2i
⌈h/4⌉
sh−s0
0.0624
0.0151
83.57
Appendix
Table 10: AFE design ablation at 40k training steps with IE disabled. State targets are shown before preprocessing. Cumulative-change supervision achieves the lowest Gated error and Raw NDTW.
Figure 24: Selected Baseline and AFE video predictions at 5k training steps. Images show final frames and hand details; curves report position errors averaged over both arms in millimeters. Endpoint labels show final-frame errors, and brackets mark the Baseline–AFE differences. Cases illustrate improvements rather than average performance.
Training data
Gated error ↓
Raw NDTW ↓
Visual pass (%) ↑
Expert Demonstrations
0.0352
0.0314
98.57
+ Feasible-Space Sampling
0.0336
0.0282
97.86
+ Local Perturbation
0.0330
0.0270
97.86
+ Cross-State Replay
0.0300
0.0242
97.86
+ Policy Rollout
0.0264
0.0204
97.86
Appendix
Table 11: Incremental data-composition ablation at 40k with IE and AFE disabled. Each row retains all data families in the preceding row and adds the named family. The final row includes all four off-expert families and constitutes Expanded Action Coverage. Bold denotes the best value in each metric column.
Figure 25: Results across query categories. Panels show gated error, raw NDTW, and visual pass rate. E/X denote Expert Demonstrations/Expanded Action Coverage. DA, CSR, LP, PR, and FSS denote Demonstrated Action, Cross-State Replay, Local Perturbation, Policy Rollout, and Feasible-Space Sampling. Cells give point estimates, and both error panels share an absolute color scale.
Figure 26: Gated error by query category. Each panel compares one baseline under both training regimes with the two WorldSync variants, repeated as common references. All axes share a 0–0.14 m-equivalent scale, with lower error closer to the center. Category abbreviations follow Figure 25 .
Figure 27: Three-metric profiles of expert-trained baselines and WorldSync with additional robotic pretraining and Expanded Action Coverage. Error axes start at zero, while visual pass spans 60–100%. Lower error lies closer to the center and higher pass rate farther out. Abbreviations follow Figure 25 .
Figure 28: Metric distributions of generated clips. Panels (a–e) show the five metrics; case labels correspond to Figure 29 , and dashed lines mark thresholds. Panel (f) reports each check’s Pass/Fail percentages (blue/orange), with overlapping failures allowed. Percentages use the full evaluation set, including clips omitted from motion scoring.
Figure 29: Examples corresponding to the case labels in Figure 28 . Scores are computed on full videos. Colored boxes locate the enlarged regions; C2 enlarges the left and right EEFs above and below, respectively. In (e), solid blue and dashed orange paths are estimated from ground-truth (GT) and generated videos on shared 3D axes; dots mark starts.
Figure 30: Automatic measurements and human judgments. (a) Raw NDTW versus median human ratings for 40 clips with five or six ratings each. Points retain exact coordinates, and marker area scales with the number of coincident clips. The interval is a task-stratified bootstrap interval for Spearman’s ρ . (b) Joint human and automatic pass/fail decisions for 15 clips per visual metric. Human labels use a strict majority of five or six ratings. Solid segments indicate agreement, and hatched segments indicate disagreement. Segment labels give clip counts, and percentages give total agreement.
Visual metric
Accuracy (%)
Pass recall (%)
Failure recall (%)
Cohen’s κ
Rater α
Image quality
80.0
100.0
50.0
0.545
0.815
Motion smoothness
80.0
90.0
60.0
0.526
0.605
EEF visibility
93.3
100.0
75.0
0.815
0.651
Arm integrity
86.7
90.0
80.0
0.700
0.888
Appendix
Table 12: Visual classification and inter-rater agreement. Each metric uses 15 clips with five or six ratings per clip. Pass recall is the fraction of human-passed clips accepted automatically. Failure recall is the fraction of human-failed clips rejected automatically. Cohen’s κ compares automatic decisions with majority labels, while nominal Krippendorff’s α compares individual raters.
Figure 31: Qualitative comparisons of WorldSync with CtrlWorld, Cosmos-Predict2.5, and Cosmos3. Rows show ground truth, WorldSync, and the baseline for each task. The last column enlarges the marked final-frame region.
Figure 32: Qualitative comparisons of WorldSync with DreamDojo, Motus, and LingBotVA. Rows show ground truth, WorldSync, and the baseline for each task. The last column enlarges the marked final-frame region.
Figure 33: WorldEcho metrics for the six cases in Figures 31 and 32 . (a) Scores and gate decisions. (b,c) Raw NDTW and integrity-gated error. Image quality and motion smoothness pass at ≥0.33 and ≥0.45 , respectively. EEF visibility passes when either tracking or text-prompted detection meets its requirement; arm integrity must equal 1. The displayed EEF fraction comes from tracking when that branch passes and from text detection otherwise; the fractions do not share a universal 0.5 cutoff. The overall gate requires all four checks; gated error equals raw NDTW when passed and 0.3 otherwise.
World models are increasingly used as policy-in-the-loop imagination environments, where reliable rollouts require fine-grained controllability with respect to low-level robot actions. A key obstacle to scaling such models in robotics is that actions are not a universal language in pixel space: changes in visual environment, camera view, robot placement, or embodiment alter how the same numerical action manifests visually, leading to conflicting supervision under mixed training and brittle generalization at deployment. We introduce SyncWorld, an action-conditioned world model that serves as a zero-shot simulator across unseen environments without any additional training. SyncWorld leverages a visual calibration episode---paired frames and actions that showcase all the controllable degrees of freedom---to specify the setup-specific Action--Visual Mapping in context. Training with visual calibration contexts teaches the model to interpret actions through visual evidence and to leverage interaction history when explicit calibration is unavailable. Experiments show that SyncWorld can accurately simulate action outcomes in previously unseen settings, and that its capability of simulating rollouts enables test-time policy improvement without training.
Action-conditioned world models are increasingly used as scalable simulators for robot learning, yet current evaluations provide limited evidence that their predictions are reliable under the actions they condition on. Existing benchmarks largely emphasize visual fidelity, leaving unclear whether predicted futures are physically plausible, faithful to commanded actions, and calibrated to failure when actions should not succeed. We introduce MiraBench, a hierarchical benchmark that defines action-conditioned reliability as a core evaluation target for robotic world models. MiraBench decomposes this target into three progressively demanding levels: Physics Adherence, which evaluates reference-free physical consistency; Action-Following Fidelity, which measures whether predictions respect task-relevant action inputs; and Optimism Bias Detection, which probes the tendency to predict successful outcomes under failure-inducing actions. To support this evaluation, we curate a human-annotated corpus with over 16,000 judgments across tasks, failure categories, and leading world models. We evaluate 12 representative model configurations spanning vector-conditioned robotic world models, text-conditioned generative world models, open-weight systems, closed-source systems, and multiple model scales. Across this broad model landscape, MiraBench reveals three central findings: visual fidelity is a poor proxy for action fidelity; increasing model scale does not reliably improve action following; and optimism bias is pervasive across current systems. By shifting evaluation from appearance to action-conditioned reliability, MiraBench provides a diagnostic foundation for assessing and improving robotic world models as faithful simulators.
Tianzhuo Yang, Zihan Shen, Zirui Mi +7
Institute for Artificial Intelligence, Peking University · 2Physis Lab
Action-conditioned video prediction models (often referred to as world models) have shown strong potential for robotics applications, but existing approaches are often slow and struggle to capture physically consistent interactions over long horizons, limiting their usefulness for scalable robot policy training and evaluation. We present Interactive World Simulator, a framework for building interactive world models from a moderate-sized robot interaction dataset. Our approach leverages consistency models for both image decoding and latent-space dynamics prediction, enabling fast and stable simulation of physical interactions. In our experiments, the learned world models produce interaction-consistent pixel-level predictions and support stable long-horizon interactions for more than 10 minutes at 15 FPS on a single RTX 4090 GPU. Our framework enables scalable demonstration collection solely within the world models to train state-of-the-art imitation policies. Through extensive real-world evaluation across diverse tasks involving rigid objects, deformable objects, object piles, and their interactions, we find that policies trained on world-model-generated data perform comparably to those trained on the same amount of real-world data. Additionally, we evaluate policies both within the world models and in the real world across diverse tasks, and observe a strong correlation between simulated and real-world performance. Together, these results establish the Interactive World Simulator as a stable and physically consistent surrogate for scalable robotic data generation and faithful, reproducible policy evaluation.
Yixuan Wang, Rhythm Syed, Fangyu Wu +7
Columbia University · Toyota Research Institute · Amazon +1