Do Robotic World Models Really Follow Actions? Diagnosing and Aligning Action-Conditioned Generation for Policy Learning

Sixiang Chen; Jiaming Liu; Jixian Wu; Yichen Guo; Tinghao Wang; QR Li; Siyuan Qian; Peterson Co

cs.ROAug 25, 2026

Do Robotic World Models Really Follow Actions? Diagnosing and Aligning Action-Conditioned Generation for Policy Learning

Authors: Sixiang Chen, Jiaming Liu, Jixian Wu, Yichen Guo, Tinghao Wang, QR Li, Siyuan Qian, Peterson Co,

Organizations: State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · Beijing Innovation Center of Humanoid Robotics · New York University · Nanyang Technological University ·

Abstract

Action-conditioned world models increasingly serve as learned simulators for robotic policy improvement, yet their reliability rests on an unverified assumption: generated futures faithfully reflect the consequences of arbitrary valid actions. Systematic evaluation against action-specific ground truth remains limited beyond expert demonstrations. To address this gap, we introduce WorldEcho, which evaluates action following over a broader action distribution through visual integrity and SE(3) end-effector trajectory alignment. Our diagnosis shows that expert-trained models perform better on demonstrated actions but struggle with off-expert trajectories, producing motion inconsistent with the commanded actions or visually invalid rollouts. To mitigate these failures, we propose WorldSync, which broadens action coverage, grounds intermediate video representations in action-induced robot dynamics through an Action-Forcing Expert, and aligns predicted and ground-truth changes under action interventions. Experiments show that WorldSync reduces integrity-gated error on RoboTwin and, when used for iterative policy improvement, yields higher success rates in simulation and on real robots.

Figures & tables

Figure 1: Diagnose–improve–validate pipeline. WorldEcho evaluates action following, WorldSync aligns action-conditioned generation, and simulation and real-robot experiments assess downstream policy improvement.

Figure 2: Off-expert failures. Left to right: input and actions, ground-truth final frame, visual collapse, and plausible but action-inconsistent motion.

Figure 3: WorldEcho evaluation across five action-query categories, combining visual integrity with $\mathrm{SE}(3)$ end-effector trajectory alignment against matched simulator references.

Figure 4: WorldSync overview: action coverage, the pretrained video backbone, and training-only supervision from the Action-Forcing Expert (AFE) and Intervention-Effect (IE) objectives.

Figure 5: Off-expert action following. (a) Gated error with equally weighted off-expert categories within tasks. Error bars show 95% confidence intervals from task-level resampling. (b) Expert-to-off-expert changes in raw NDTW and visual failure rate. (c) Action PCA with 95% highest-density regions (off-expert: union across categories).

			WorldEcho metrics			Video generation metrics $\dagger$
Model	Emb. PT	Training regime	Gated error $\downarrow$	Raw NDTW $\downarrow$	Visual pass (%) $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FVD $\downarrow$
CtrlWorld	$\checkmark$	Expert	0.0379	0.0266	95.83	19.41	0.8466	0.1137	113.58
CtrlWorld	$\checkmark$	Expanded	0.0293	0.0210	96.97	20.83	0.8665	0.0915	69.58
Cosmos-Predict2.5	$\checkmark$	Expert	0.0551	0.0190	87.09	21.10	0.8750	0.0848	95.69
Cosmos-Predict2.5	$\checkmark$	Expanded	0.0445	0.0127	88.80	22.95	0.8952	0.0658	51.89
Cosmos3	$\checkmark$	Expert	0.0989	0.0572	82.51	14.45	0.6846	0.1833	122.58

Table 1: Main WorldEcho comparison. Expert and Expanded denote Expert Demonstrations and Expanded Action Coverage. Bold marks the best scores. Underlining marks second-best WorldEcho scores. Emb. PT marks embodied-data pretraining before task-specific fine-tuning, excluding generic-only video pretraining (Appendix B ).

\dagger

Video metrics use reconstructed references, not verified as pixel-identical to the original RGB videos. Baseline video and WorldEcho scores use separate prediction batches at the same checkpoint. PSNR, SSIM, and LPIPS average over frames, clips, then tasks. FVD pools all clips.

Figure 6: Policy improvement in simulation and four real-robot tasks. Each real-robot success rate uses 25 trials. Init. denotes the initial policy, and R1/R2 denote improvement rounds.

Variant	Gated $\downarrow$	Raw $\downarrow$	Pass (%) $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FVD $\downarrow$
Expert Base	0.0352	0.0314	98.57	21.23	0.8585	0.1154	175.09
Base	0.0264	0.0204	97.86	20.66	0.8591	0.1087	141.10
+ IE	0.0214	0.0176	98.57	21.05	0.8665	0.0963	125.92
+ AFE	0.0232	0.0172	97.86	21.30	0.8683	0.0962	118.81
IE + AFE	0.0193	0.0138	97.86	22.33	0.8821	0.0830	105.83

Table 2: Data and component ablation on ten RoboTwin tasks at 40k training steps. Bold denotes the best reported value, including ties. Video metrics use separate generations at the same checkpoint. PSNR, SSIM, and LPIPS average over frames, clips, then tasks. FVD pools all video pairs.

Appendix figures & tables37 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: TianJi hardware and teleoperation setup. Numbered markers match the installed devices to the equipment list; the lower panel identifies the hand-motion and end-effector tracking devices.

Figure 8: i7 Pro hardware and teleoperation setup. Numbered markers link the installed cameras and grippers to the equipment list; the left side shows the PICO teleoperation device.

Figure 9: Franka FR3 hardware and camera setup. Numbered markers link the camera locations to the equipment list.

Figure 10: Cup Stacking from a third-person viewpoint. Read left to right; timestamps mark selected frames from one rollout.

Figure 11: Cup Stacking through the robot’s cameras. Columns show the head, left wrist, and right wrist; rows advance in time, with synchronized views within each row. This rollout is separate from Figure 10 ; timestamps are relative to each recording.

Figure 12: Box Packing from a third-person viewpoint. Read left to right; timestamps mark selected frames from one rollout.

Figure 13: Box Packing through the robot’s cameras. Columns show the head, left wrist, and right wrist; rows advance in time, with synchronized views within each row. This rollout is separate from Figure 12 ; timestamps are relative to each recording.

Figure 14: Bread Placement from a third-person viewpoint. Read left to right; timestamps mark selected frames from one rollout.

Figure 15: Bread Placement through the robot’s cameras. Columns show the left external, right external, and wrist views; rows advance in time, with corresponding views within each row. This rollout is separate from Figure 14 ; timestamps are relative to each recording.

Figure 16: Pen Collection from a third-person viewpoint. Read left to right; timestamps mark selected frames from one rollout.

Figure 17: Pen Collection through the robot’s cameras. Columns show the left external, right external, and wrist views; rows advance in time, with corresponding views within each row. This rollout is separate from Figure 16 ; timestamps are relative to each recording.

Dataset	Episodes	Steps	Mean length (steps)	Hours
AgiBot-World-Alpha	32,836	55,255,078	1,682.76	511.62
DROID	92,917	26,959,694	290.15	499.25
Total	125,753	82,214,772	653.78	1,010.87

Appendix

Table 3: Robotic pretraining data after filtering. Steps count recorded timesteps, including episode endpoints, rather than optimizer updates or sampled clips; synchronized camera views are counted once. Hours give the corresponding video duration. Mean length is the number of recorded steps per episode, with the total-row mean computed over all retained episodes.

Setting	Initial	Round 1	Round 2
CtrlWorld + Qwen-OFT	59	46	14

Appendix

Table 4: Reproduction of the VLAW framework. Policy success rates (%) on rotate_qrcode , measured over 100 episodes at each stage. Both CtrlWorld and Qwen-OFT are updated each round.

Figure 18: Action mismatch under identical conditioning actions. World-model prediction (top) and simulator replay (bottom) on dump_bin_bigbin , using the same actions from matched initial conditions. The prediction depicts lifting and pouring, whereas the simulator replay leaves the cup and its contents in place. The arrow indicates temporal order; rightmost panels magnify the boxed final-frame regions.

Setting	Configuration
Improvement rounds	2
Online collection	30 rollouts, including successes and failures
Imagined rollouts	$3{,}000=50$ expert starts $\times 60$
Action representation	20-dimensional actions, 32 actions per chunk
Generation horizon	7 chunks (224 actions), 20 denoising steps per chunk
World-model update	1,000 updates, global batch 16, learning rate $10^{-5}$

Appendix

Table 5: Shared simulation protocol. Counts are per condition and per round unless stated otherwise. Sampling ratios do not denote unique rollout counts.

		Rollout counts : $N_{0}\rightarrow N_{1}\rightarrow N_{2}$			Retention (%)
Method	Round	Initial $N_{0}$	After S1 $N_{1}$	After S2 $N_{2}$	Stage 1 $N_{1}/N_{0}$	Stage 2 $N_{2}/N_{1}$	Overall $N_{2}/N_{0}$
WorldSync	1	3,000	1,114	553	37.1	49.6	18.4
WorldSync	2	3,000	1,088	365	36.3	33.5	12.2
CtrlWorld	1	3,000	962	452	32.1	47.0	15.1
CtrlWorld	2	3,000	909	323	30.3	35.5	10.8

Appendix

Table 6: Sequential filtering counts and retention rates. Stage 1 applies task-progress filtering with

s_{\mathcal{R}}(z)>0.8

, followed by action-consistency filtering with

q_{k}(z)\leq\eta

in Stage 2. Stage-wise rates use the preceding stage’s count, whereas overall rates use all

3{,}000

generated rollouts.

Method	Initial (%)	Round 1 (%)	Round 2 (%)	Gain (pp)
WorldSync	52	57	65	+13
CtrlWorld	52	54	57	+5

Appendix

Table 7: Policy success over two improvement rounds. Success rates use the same ordered set of 100 evaluation seeds. Gains are percentage-point changes from each condition’s measured initial rate.

Mechanism	Representative work	Alignment	Injection site	Fusion
Numerical-based
Additive	NanoWM ( 2026a )	Frame-aligned	Patch features	Element-wise addition
AdaLN Fusion	Cosmos-Predict2.5 ( 2025 )	Frame-aligned	Native AdaLN path	Native-condition fusion
Action AdaLN	IRASim ( 2025 )	Frame-aligned	Per-block attention/FFN	Low-rank shift/scale/gate
Cross-Attention	CtrlWorld ( 2026b )	Frame-aligned, local	Native attention	Gated action attention
Shared-KV Attention	Motus ( 2025 )	Global sequence	Native self-attention	Shared video/action K/V; video Q

Appendix

Table 8: Action-conditioning mechanisms and representative works. All variants use the same video diffusion backbone.

Figure 19: Overview of the evaluated action-conditioning mechanisms. Numerical-based methods use feature addition, modulation, attention, or latent frames, whereas map-based methods use rendered action maps. Blue, orange, green, purple, and gray denote video features, numerical actions, action maps, fusion operations, and DiT blocks, respectively.

Figure 20: RGB action-map representation. (a) Rendered action maps at the current frame and four future steps. (b) Corresponding overlays on the RGB observations. Each column shows the head, left-wrist, and right-wrist views; colored markers encode the projected end-effector poses.

Mechanism	Raw NDTW $\downarrow$	Gated $\downarrow$	Pass (%) $\uparrow$	Params (M)	FLOPs (T)
Additive	0.0320	0.0403	96.43	568.07	0.04
AdaLN Fusion	0.0330	0.0389	97.86	568.07	4.93
Action AdaLN	0.0137	0.0176	98.57	572.01	0.09
Cross-Attention	0.0302	0.0354	97.86	566.49	0.82
Shared-KV Attention	0.0332	0.0370	98.57	566.49	1.27
Latent Frame	0.0239	0.0297	97.86	0	69.39

Appendix

Table 9: Comparison at the last training checkpoint. Params and FLOPs denote additional trainable parameters and inference compute relative to unconditioned video generation. Best performance is bold.

Figure 21: Performance across training checkpoints under the WorldEcho evaluation protocol. The panels report raw NDTW, integrity-gated error, and visual pass rate for all nine mechanisms. Raw and gated errors are task-macro averages; pass rate counts samples. Solid and dashed lines denote numerical-based and map-based mechanisms, respectively. Latent Frame uses equal-weight aggregation of its component results.

Figure 22: Performance–efficiency trade-offs at the common 40k checkpoint under the WorldEcho evaluation protocol. (a) Visual pass rate versus raw NDTW. (b) Integrity-gated error versus additional inference compute on a logarithmic scale. (c) Integrity-gated error versus additional parameters. Arrows indicate preferred directions, and the star marks Action AdaLN (Ours).

Figure 23: AFE architecture. (a) Training-only state supervision from intermediate video features. (b) Query updates within an AFE block. (c) Temporal alignment between future-step queries and video features. AFE updates the backbone through loss gradients and is removed at inference.

Layers $\ell_{i}$	Latent times $j$	State target	Gated error $\downarrow$	Raw NDTW $\downarrow$	Visual pass $\uparrow$
Baseline	–	–	0.0717	0.0208	82.14
$\lfloor 29i/7+1/2\rfloor$	$0\leq j\leq 8$	$s_{h}$	0.0816	0.0210	78.57
$8+2i$	$0\leq j\leq 8$	$s_{h}$	0.0784	0.0197	79.29
$8+2i$	$\lceil h/4\rceil$	$s_{h}$	0.0702	0.0191	82.14
$8+2i$	$\lceil h/4\rceil$	$s_{h}-s_{h-1}$	0.0714	0.0159	80.71
$8+2i$	$\lceil h/4\rceil$	$s_{h}-s_{0}$	0.0624	0.0151	83.57

Appendix

Table 10: AFE design ablation at 40k training steps with IE disabled. State targets are shown before preprocessing. Cumulative-change supervision achieves the lowest Gated error and Raw NDTW.

Figure 24: Selected Baseline and AFE video predictions at 5k training steps. Images show final frames and hand details; curves report position errors averaged over both arms in millimeters. Endpoint labels show final-frame errors, and brackets mark the Baseline–AFE differences. Cases illustrate improvements rather than average performance.

Training data	Gated error $\downarrow$	Raw NDTW $\downarrow$	Visual pass (%) $\uparrow$
Expert Demonstrations	0.0352	0.0314	98.57
+ Feasible-Space Sampling	0.0336	0.0282	97.86
+ Local Perturbation	0.0330	0.0270	97.86
+ Cross-State Replay	0.0300	0.0242	97.86
+ Policy Rollout	0.0264	0.0204	97.86

Appendix

Table 11: Incremental data-composition ablation at 40k with IE and AFE disabled. Each row retains all data families in the preceding row and adds the named family. The final row includes all four off-expert families and constitutes Expanded Action Coverage. Bold denotes the best value in each metric column.

Figure 25: Results across query categories. Panels show gated error, raw NDTW, and visual pass rate. E/X denote Expert Demonstrations/Expanded Action Coverage. DA, CSR, LP, PR, and FSS denote Demonstrated Action, Cross-State Replay, Local Perturbation, Policy Rollout, and Feasible-Space Sampling. Cells give point estimates, and both error panels share an absolute color scale.

Figure 26: Gated error by query category. Each panel compares one baseline under both training regimes with the two WorldSync variants, repeated as common references. All axes share a 0–0.14 m-equivalent scale, with lower error closer to the center. Category abbreviations follow Figure 25 .

Figure 27: Three-metric profiles of expert-trained baselines and WorldSync with additional robotic pretraining and Expanded Action Coverage. Error axes start at zero, while visual pass spans 60–100%. Lower error lies closer to the center and higher pass rate farther out. Abbreviations follow Figure 25 .

Figure 28: Metric distributions of generated clips. Panels (a–e) show the five metrics; case labels correspond to Figure 29 , and dashed lines mark thresholds. Panel (f) reports each check’s Pass/Fail percentages (blue/orange), with overlapping failures allowed. Percentages use the full evaluation set, including clips omitted from motion scoring.

Figure 29: Examples corresponding to the case labels in Figure 28 . Scores are computed on full videos. Colored boxes locate the enlarged regions; C2 enlarges the left and right EEFs above and below, respectively. In (e), solid blue and dashed orange paths are estimated from ground-truth (GT) and generated videos on shared 3D axes; dots mark starts.

Figure 30: Automatic measurements and human judgments. (a) Raw NDTW versus median human ratings for 40 clips with five or six ratings each. Points retain exact coordinates, and marker area scales with the number of coincident clips. The interval is a task-stratified bootstrap interval for Spearman’s

\rho

. (b) Joint human and automatic pass/fail decisions for 15 clips per visual metric. Human labels use a strict majority of five or six ratings. Solid segments indicate agreement, and hatched segments indicate disagreement. Segment labels give clip counts, and percentages give total agreement.

Visual metric	Accuracy (%)	Pass recall (%)	Failure recall (%)	Cohen’s $\kappa$	Rater $\alpha$
Image quality	80.0	100.0	50.0	0.545	0.815
Motion smoothness	80.0	90.0	60.0	0.526	0.605
EEF visibility	93.3	100.0	75.0	0.815	0.651
Arm integrity	86.7	90.0	80.0	0.700	0.888

Appendix

Table 12: Visual classification and inter-rater agreement. Each metric uses 15 clips with five or six ratings per clip. Pass recall is the fraction of human-passed clips accepted automatically. Failure recall is the fraction of human-failed clips rejected automatically. Cohen’s

\kappa

compares automatic decisions with majority labels, while nominal Krippendorff’s

\alpha

compares individual raters.

Figure 31: Qualitative comparisons of WorldSync with CtrlWorld, Cosmos-Predict2.5, and Cosmos3. Rows show ground truth, WorldSync, and the baseline for each task. The last column enlarges the marked final-frame region.

Figure 32: Qualitative comparisons of WorldSync with DreamDojo, Motus, and LingBotVA. Rows show ground truth, WorldSync, and the baseline for each task. The last column enlarges the marked final-frame region.

Figure 33: WorldEcho metrics for the six cases in Figures 31 and 32 . (a) Scores and gate decisions. (b,c) Raw NDTW and integrity-gated error. Image quality and motion smoothness pass at

\geq 0.33

and

\geq 0.45

, respectively. EEF visibility passes when either tracking or text-prompted detection meets its requirement; arm integrity must equal 1. The displayed EEF fraction comes from tracking when that branch passes and from text detection otherwise; the fractions do not share a universal 0.5 cutoff. The overall gate requires all four checks; gated error equals raw NDTW when passed and

0.3

otherwise.

Action-Conditioned World Models

Action-conditioned world models predict how environments change under actions to support planning, control, and simulation.

See all 341 papers

SyncWorld: Visual Calibration Enables World Models as Zero-Shot Simulators
Sep 8, 2026Yuncong Yang, Zhengtao Han, Furkan Ozyurt +6World Model Learning World Models for Robotics
UMass Amherst · UC Berkeley · NYU +1
MiraBench: Evaluating Action-Conditioned Reliability in Robotic World Models
May 28, 2026Tianzhuo Yang, Zihan Shen, Zirui Mi +7World Models for Robotics Action-Conditioned World Models
Institute for Artiﬁcial Intelligence, Peking University · 2Physis Lab
Interactive World Simulator for Robot Policy Training and Evaluation
Mar 9, 2026Yixuan Wang, Rhythm Syed, Fangyu Wu +7Interactive World Models World Model Learning
Columbia University · Toyota Research Institute · Amazon +1

Sep 8, 2026J/K move · Enter open · S save

SyncWorld: Visual Calibration Enables World Models as Zero-Shot Simulators

Yuncong Yang, Zhengtao Han, Furkan Ozyurt, Zeyuan Yang+5

UMass Amherst · UC Berkeley · NYU+1

World Model Learning World Models for Robotics Action-Conditioned World Models

Open paper AskPDF arXiv

World models are increasingly used as policy-in-the-loop imagination environments, where reliable rollouts require fine-grained controllability with respect to low-level robot actions. A key obstacle to scaling such models in robotics is that actions are not a universal language in pixel space: changes in visual environment, camera view, robot placement, or embodiment alter how the same numerical action manifests visually, leading to conflicting supervision under mixed training and brittle generalization at deployment. We introduce SyncWorld, an action-conditioned world model that serves as a zero-shot simulator across unseen environments without any additional training. SyncWorld leverages a visual calibration episode---paired frames and actions that showcase all the controllable degrees of freedom---to specify the setup-specific Action--Visual Mapping in context. Training with visual calibration contexts teaches the model to interpret actions through visual evidence and to leverage interaction history when explicit calibration is unavailable. Experiments show that SyncWorld can accurately simulate action outcomes in previously unseen settings, and that its capability of simulating rollouts enables test-time policy improvement without training.

Abstract

Action-Conditioned World Models

Explore similar work

SyncWorld: Visual Calibration Enables World Models as Zero-Shot Simulators

MiraBench: Evaluating Action-Conditioned Reliability in Robotic World Models

Interactive World Simulator for Robot Policy Training and Evaluation