WorldOracle: Benchmarking Action-Conditioned World Models against Real-World Executions

Haoyi Duan; Liliang Chen; Nan Wang; Yang Zhang; Yuhang Lu; Xiaotong Chen; Wei-Cheng Tseng; Shenyuan Gao

cs.ROOct 13, 2026

WorldOracle: Benchmarking Action-Conditioned World Models against Real-World Executions

Authors: Haoyi Duan, Liliang Chen, Nan Wang, Yang Zhang, Yuhang Lu, Xiaotong Chen, Wei-Cheng Tseng, Shenyuan Gao,

Organizations: AgiBot · Princeton University · NVIDIA · University of Toronto ·

Abstract

We introduce WorldOracle, a benchmark specifically designed to evaluate action-conditioned world models (ACWMs) against paired real-world executions. Given the same initial observation and low-level action sequence, WorldOracle asks whether a predicted rollout reproduces what happens in the real world, including physical consequences and task-state transitions. We evaluate models along four complementary dimensions: physical state correctness, interaction spatial alignment, key interaction area fidelity, and multi-view action coherence. To support this evaluation, we curate real-world training and evaluation suites that cover different interaction dynamics and embodiments. The evaluation suite contains long, multi-segment episodes with both successful and failed executions, enabling evaluation across varied action outcomes and behavior distributions. Experiments on seven representative ACWMs reveal distinct model strengths and failure modes. WorldOracle provides a systematic foundation for measuring and advancing action-conditioned world modeling for manipulation. We release the datasets, evaluation code, and leaderboard: https://worldoracle-benchmark.github.io/.

Figures & tables

Figure 1: Overview of WorldOracle. (a) The human-operated trajectories are collected through robot teleoperation or direct human-hand execution. (b) The policy-rollout trajectories are collected by running a policy in closed loop on a real robot. (c) For both distributions, the recorded action trajectory is replayed open loop by the ACWM, yielding a generated rollout. (d) The predicted rollout is compared with the paired real execution along four dimensions: physical state correctness, interaction spatial alignment, key interaction area fidelity, and multi-view action coherence.

Benchmark	# Samples	Action.	Real.	Fail.	Long.	Multi-Embod.	Multi-view.	Policy Dist.
EWMBench [ 36 ]	100	✗	✓	✗	✗	✗ (AgiBot G1)	✗	✗
PAI-Bench [ 40 ]	2808	✗	✓	✗	✗	✓ (AgiBot G1, WidowX, Human hand)	✗	✗
WoW-World-Eval [ 11 ]	609	✗	✓	✗	✗	✓ (Franka, UR5e, AgileX, humanoid)	✗	✗
RBench [ 9 ]	650	✗	✓	✗	✗	✓ (single-, dual-arm,humanoid, quadruped)	✗	✗
WorldArena [ 25 ]	500	✓	✗	✗	✗	✗ (RoboTwin)	✗	✓
EZSBench [ 6 ]	196	✗	✓	✗	✗	✓ (real/synthetic robot morphologies)	✗	✗

Table 1: Comparison of Embodied World-Model Benchmarks. # Samples : number of samples for evaluation. Action. : evaluation of models conditioned on action sequences. Real. : real-world data. Fail. : data with failed executions. Long. : long-duration execution traces with multiple interaction segments. Multi-Embod. : evaluation across multiple embodiments. Multi-view. : evaluation across multiple camera views. Policy Dist. : action trajectories generated by autonomous policy rollouts.

Figure 2: WorldOracle annotation pipeline. (a) Execution segmentation: each synchronized multi-view episode is divided into interaction segments $\mathcal{S}_{i,n}$ , and the final frame of each segment is the outcome keyframe $K_{i,n}$ capturing the observable physical outcome. (b) Physical criteria annotation: a VLM generates physical-state criteria for each outcome keyframe. (c) Interaction region annotation: manipulated objects, robot arms, and human hands are annotated frame-wise across camera views.

Figure 3: Qualitative examples. Each row compares a real-world reference with high-, intermediate-, and low-scoring predictions. Physical state correctness checks whether the prediction satisfies the criteria. The bad case predicts states of both towels wrong. Interaction spatial alignment. Points indicate matched visual features and lines show their displacement to reference. Point color denotes matching confidence from green (high) to red (low), while shorter lines indicate better spatial alignment. Key interaction area fidelity. Within the interaction regions, the predicted entities of the bad example exhibit degraded appearance and structural distortion. Multi-view action coherence. In the bad example, the predicted robot arm is poorly aligned in the synchronized camera views.

Model	Regime	Physical Correctness	Spatial Alignment	Key Area Fidelity	Multi-View Coherence	WorldOracle
Model	Regime	Physical Correctness	Spatial Alignment	Key Area Fidelity	Multi-View Coherence	-SV	-MV
IRASim [ 41 ]	30s	$\underline{55.64}\pm 3.16$	$53.96\pm 2.10$	$38.00\pm 1.26$	–	$49.20\pm 1.31$	–
IRASim [ 41 ]	Long	$\underline{42.06}\pm 2.03$	$\underline{37.60}\pm 1.72$	$\underline{25.47}\pm 1.00$	–	$\underline{35.04}\pm 0.95$	–
Ctrl-World [ 14 ]	30s	$51.15\pm 3.47$	$57.08\pm 2.40$	$40.53\pm 1.51$	$\underline{65.57}\pm 2.05$	$49.59\pm 1.47$	$\underline{53.58}\pm 1.19$
Ctrl-World [ 14 ]	Long	$38.31\pm 2.96$	$29.44\pm 2.34$	$23.45\pm 1.23$	$\underline{39.31}\pm 1.88$	$30.40\pm 1.32$	$\underline{32.62}\pm 1.10$
DreamDojo [ 12 ]	30s	$53.32\pm 3.63$	$53.96\pm 1.71$	$\underline{44.39}\pm 1.31$	–	$50.56\pm 1.41$	–

Table 2: Results on AgiBot G2. The 30s and Long rows evaluate the first 30 seconds and the complete episode. All metrics are reported on a

[0,100]

scale; higher is better.

Model	Physical Correctness	Spatial Alignment	Key Area Fidelity	Multi-View Coherence	WorldOracle
Model	Physical Correctness	Spatial Alignment	Key Area Fidelity	Multi-View Coherence	-SV	-MV
IRASim [ 41 ]	$\mathbf{62.18}\pm 7.04$	$51.62\pm 3.74$	$32.30\pm 2.19$	–	$\underline{48.70}\pm 2.67$	–
Ctrl-World [ 14 ]	$27.22\pm 6.52$	$42.95\pm 3.81$	$33.16\pm 2.52$	$\underline{62.91}\pm 3.69$	$34.44\pm 2.69$	$\underline{41.56}\pm 2.20$
DreamDojo [ 12 ]	$44.44\pm 7.86$	$52.88\pm 3.53$	$\underline{36.57}\pm 2.52$	–	$44.63\pm 2.97$	–
Cosmos 3 [ 1 ]	$34.29\pm 6.78$	$\underline{59.81}\pm 3.02$	$34.31\pm 2.81$	–	$42.80\pm 2.60$	–
GE-Sim 2.0 [ 24 ]	$\underline{57.03}\pm 6.89$	$\mathbf{78.70}\pm 1.84$	$\mathbf{58.33}\pm 1.39$	$\mathbf{96.64}\pm 0.35$	$\mathbf{64.69}\pm 2.37$	$\mathbf{72.67}\pm 1.79$

Table 3: Results on the AgiBot G2 policy-rollout distribution. Each rollout is evaluated over its complete duration. All scores are reported on a

[0,100]

scale, and higher is better.

Model	Regime	Physical Correctness	Spatial Alignment	Key Area Fidelity	Multi-View Coherence	WorldOracle
Model	Regime	Physical Correctness	Spatial Alignment	Key Area Fidelity	Multi-View Coherence	-SV	-MV
IRASim [ 41 ]	30s	$65.23\pm 4.69$	$51.52\pm 1.96$	$39.67\pm 1.13$	–	$52.14\pm 1.74$	–
IRASim [ 41 ]	Long	$\underline{53.39}\pm 2.75$	$\underline{38.71}\pm 1.60$	$\underline{30.35}\pm 0.93$	–	$\underline{40.82}\pm 1.11$	–
Ctrl-World [ 14 ]	30s	$56.39\pm 4.88$	$53.43\pm 2.70$	$35.93\pm 1.58$	$\underline{41.87}\pm 2.15$	$48.59\pm 1.91$	$\underline{46.91}\pm 1.53$
Ctrl-World [ 14 ]	Long	$39.03\pm 3.02$	$20.71\pm 2.04$	$18.97\pm 1.35$	$\underline{15.52}\pm 1.34$	$26.24\pm 1.28$	$\underline{23.56}\pm 1.02$
DreamDojo [ 12 ]	30s	$59.14\pm 4.31$	$56.35\pm 2.78$	$\underline{45.84}\pm 1.65$	–	$53.78\pm 1.79$	–

Table 4: Results on YAM. The 30s and Long rows report short-duration and full-episode evaluation, respectively. All metrics are reported on a

[0,100]

scale, and higher is better.

Model	Regime	Physical	Spatial	Key Area
Model	Regime	Correctness	Alignment	Fidelity	WorldOracle-SV
JointControlVideo [ 37 ]	30s	$\mathbf{55.47}\pm 4.24$	$\mathbf{57.24}\pm 2.46$	$20.50\pm 1.40$	$\mathbf{44.40}\pm 1.69$
JointControlVideo [ 37 ]	Long	$\mathbf{56.33}\pm 3.02$	$\mathbf{23.79}\pm 1.66$	$8.10\pm 0.67$	$\mathbf{29.41}\pm 1.18$
Cosmos 3 [ 1 ]	30s	$47.46\pm 3.88$	$33.70\pm 2.33$	$\underline{22.21}\pm 1.06$	$34.46\pm 1.58$
Cosmos 3 [ 1 ]	Long	$30.23\pm 2.22$	$9.43\pm 0.87$	$\underline{11.44}\pm 0.72$	$17.03\pm 0.82$
OSCAR [ 32 ]	30s	$\underline{53.18}\pm 3.83$	$\underline{47.48}\pm 2.34$	$\mathbf{27.46}\pm 1.08$	$\underline{42.71}\pm 1.55$

Table 5: Results on Human Hand. The 30s and Long rows report short-duration and full-episode evaluation, respectively. All metrics are reported on a

[0,100]

scale, and higher is better.

Figure 4: Task-wise interaction-dynamics analysis on human-operated AgiBot G2 split. Results are evaluated over the first 30 seconds on physical state correctness, interaction spatial alignment, key interaction area fidelity, and WorldOracle-SV. In each heatmap, Average denotes the arithmetic mean across tasks, while Worst denotes the lowest task score.

Figure 5: Task-wise interaction-dynamics analysis on AgiBot G2 policy-rollout split. We report results on physical state correctness, interaction spatial alignment, key interaction area fidelity, and WorldOracle-SV. In each heatmap, Average denotes the arithmetic mean across tasks, while Worst denotes the lowest task score.

Figure 6: Task-wise interaction-dynamics analysis on human-operated YAM split. Results are evaluated over the first 30 seconds on physical state correctness, interaction spatial alignment, key interaction area fidelity, and WorldOracle-SV. In each heatmap, Average denotes the arithmetic mean across tasks, while Worst denotes the lowest task score.

Figure 7: Task-wise interaction-dynamics analysis for human-hand split. Results are evaluated over the first 30 seconds on physical state correctness, interaction spatial alignment, key interaction area fidelity, and WorldOracle-SV. In each heatmap, Average denotes the arithmetic mean across tasks, while Worst denotes the lowest task score.

Figure 8: Task-wise interaction-dynamics analysis for multi-view action coherence. Results are evaluated over the first 30 seconds. In each heatmap, Average denotes the arithmetic mean across tasks, while Worst denotes the lowest task score.

Figure 9: Entity-centric error decomposition. interaction spatial alignment and key interaction area fidelity are decomposed into scores computed over the object and robot-arm/hand masks, separately. In the figure, Spatial denotes interaction spatial alignment, while Key Area denotes key interaction area fidelity. Each cell reports the score for the corresponding metric and mask category.

Figure 10: Performance across cumulative rollout horizons on the human-operated AgiBot G2 distribution. Each point reports the corresponding metric evaluated over the cumulative interval

[0,T)

, for horizons

T

ranging from 10 to 120 seconds. Higher scores indicate better performance.

Figure 11: Performance across cumulative rollout horizons on the human-operated YAM distribution. Each point reports the corresponding metric evaluated over the cumulative interval

[0,T)

, for horizons

T

ranging from 10 to 120 seconds. Higher scores indicate better performance.

Figure 12: Performance across cumulative rollout horizons on the human-hand distribution. Each point reports the corresponding metric evaluated over the cumulative interval

[0,T)

, for horizons

T

ranging from 10 to 120 seconds. Higher scores indicate better performance.

Figure 13: Alignment between WorldOracle metrics and human judgments. Each panel reports the fraction of human choices that agree with the preference induced by the corresponding metric, grouped by metric score gap. Shaded regions denote

95\%

Wilson confidence intervals, and dashed horizontal lines indicate overall agreement. The reported Spearman coefficient

\rho

measures the pair-level association between the signed metric-score difference and the human preference rate.

Figure 14: Complementary failure modes captured by physical state correctness and interaction spatial alignment. Each panel compares the real and predicted images at the annotated outcome frame. Physical is the criterion-level physical state correctness score at this frame, while Spatial is the interaction spatial alignment score computed over a four-second local window centered on it. In (a), the prediction approximately follows the robot motion but incorrectly leaves the green drawer open when the real execution closes all drawers. In (b), the model predicts that the yellow towel is grasped, satisfying the stated physical criterion, but renders the arm and towel along a trajectory that differs from the real execution.

Figure 15: Controlled case study of key interaction area fidelity. Starting from clean real interaction frames, we apply Gaussian blur either within the object and robot-arm masks ( $\sigma\in\{2,4,8\}$ ) or only outside this region ( $\sigma=8$ ). Blue and orange contours denote the object and robot-arm masks, respectively. Each panel reports the object score, arm score, and their aggregate key interaction area fidelity score. Increasing mask-local blur produces a monotonic decrease in key interaction area fidelity for both an object-centric Pick Fruit example and a contact-centric Pour Water example, whereas applying the strongest blur only to the background leaves the score unchanged.

Figure 16: Qualitative examples of the outcome-swap intervention. Although both frames come from the same episode, the preceding keyframe contradicts the newly introduced physical-state criterion. Each example reports all four metric scores.

Figure 17: Cross-view desynchronization across increasing delays. We progressively delay only the left auxiliary view while retaining the top-head observation and right view at time $t$ . All four scores are shown for each condition. In this example, multi-view action coherence decreases monotonically from $100.0$ to $94.8$ , $83.7$ , $82.4$ , and $81.3$ as the delay increases, whereas the three top-head metrics remain unchanged.

Figure 18: Nuisance robustness and sensitivity. Left panels show the signed mean score change relative to the default setting; right panels show model-ranking Spearman correlation. Appearance perturbations leave geometry-centered metrics nearly unchanged while key interaction area fidelity appropriately responds to visible degradation. Mask morphology changes absolute scores more strongly but generally preserves rankings. Temporal subsampling produces negligible score changes and identical rankings. The dashed line denotes

\rho=0.9

Figure 19: Hyperparameter sensitivity of Spatial Alignment and Key-Area Fidelity. We vary one setting at a time from $0.5\theta_{0}$ to $1.5\theta_{0}$ , where $\theta_{0}$ is the default. The left panels show agreement with the default ranking of seven ACWMs; the right panels show the average score change.

Figure 20: Duration statistics of public robot-manipulation datasets. Blue rows denote short, atomic datasets, whereas orange rows denote long, compositional datasets. Each row is annotated with its available timing statistic: an episode-level P50–P90 interval, or a reported majority range. Simulated datasets are marked with “sim.” explicitly. The dashed line indicates our selected 30-second short-horizon boundary.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Episodes	FPS	Mean $\pm$ SD (s)	Range (s)	Hours
Pick fruit	426	30	$60.19\pm 1.62$	$56.27$ – $67.10$	7.12
Hang towel	391	30	$60.57\pm 1.73$	$54.70$ – $67.30$	6.58
Pour water	523	30	$61.82\pm 3.05$	$51.83$ – $64.94$	8.98
Pull drawer	439	30	$60.70\pm 1.35$	$55.30$ – $66.77$	7.40
Total	1,779	30	$\mathbf{60.88\pm 2.21}$	$\mathbf{51.83}$ – $\mathbf{67.30}$	30.08

Appendix

Table S1: AgiBot G2 training-set statistics. Duration is reported as mean

\pm

standard deviation and observed minimum–maximum in seconds.

Embodiment	Action source	Task distribution	Episodes	FPS	Mean $\pm$ SD (s)	Range (s)	Hours
AgiBot G2	Human-operated	Pick (25), Hang (25), Pour (25), Pull (25)	100	30	$118.70\pm 3.88$	$108.27$ – $130.47$	3.30
AgiBot G2	Policy-rollout	Pick (20), Hang (20), Pour (20)	60	30	$22.59\pm 13.01$	$4.93$ – $60.00$	0.38
YAM	Human-operated	Pick (25), Hang (25), Pull (25)	75	30	$122.90\pm 4.80$	$110.00$ – $138.77$	2.56
Human hand	Human-operated	Pick (25), Hang (25), Pour (25), Pull (25)	100	15	$125.35\pm 2.28$	$121.40$ – $134.53$	3.48
Total	–	Pick (95), Hang (95), Pour (70), Pull (75)	335	–	$\mathbf{104.41\pm 38.85}$	$\mathbf{4.93}$ – $\mathbf{138.77}$	9.72

Appendix

Table S2: Evaluation-set statistics. The evaluation suite contains 335 episodes from human-operated and policy-rollout distributions. Duration is reported as mean

\pm

standard deviation and observed minimum–maximum in seconds.

Model	Pretrained Embodiment(s)	Step	Batch	Optimizer	Peak LR	Weight Decay	Precision
IRASim	WidowX (Bridge)	100K	32	AdamW	$1{\times}10^{-4}$	0	FP32
Ctrl-World	Franka Panda	250K	32	AdamW	$1{\times}10^{-5}$	$1{\times}10^{-2}$	BF16
DreamDojo	Human, Unitree G1, Fourier GR-1, AgiBot, YAM	100K	16	Fused AdamW	$5{\times}10^{-6}$	$1{\times}10^{-1}$	BF16
Cosmos 3	AgiBot, Franka Panda, Google Robot, WidowX, UMI, UR	200K	8	AdamW	$2{\times}10^{-5}$	$5{\times}10^{-2}$	BF16
GE-Sim 2.0	AgiBot G1	23K	64	8-bit AdamW	$1{\times}10^{-4}$	$5{\times}10^{-5}$	BF16
$\pi_{0.5}$ policy	Franka Panda	51K	256	AdamW	$2.5{\times}10^{-5}$	$1{\times}10^{-10}$	BF16

Appendix

Table S3 : Training configurations for the evaluated ACWMs and policy. Pretrained embodiment(s) denotes the robot or embodiment domains covered by the official checkpoint used for initialization. Batch denotes the effective global batch size across eight NVIDIA H100 GPUs. The reported step corresponds to the checkpoint used for evaluation.

Model	Size	Action Conditioning	Generated Frames	History Frames	Output	Resolution
IRASim	0.7B	Relative EEF/gripper; frame-level AdaLN	15	1	Video	$320{\times}256$
Ctrl-World	1.5B	Absolute EEF/gripper; frame-wise cross-attention	5	6 (sparse)	Multi-view video	$384{\times}256$
DreamDojo	2B	Block-relative robot actions; 4-action chunks via AdaLN	12	1	Video	$640{\times}480$
Cosmos 3	16B	Relative EEF/gripper/hand pose; action-token conditioning	16	1	Video	$832{\times}480$
GE-Sim 2.0	2B	Absolute EEF/gripper; channel-concat trajectory + ray maps	25	4 (sparse)	Multi-view video, proprioception, reward	$512{\times}384$
JointControlVideo	14B	Absolute hand pose; channel-concat depth-aware heatmaps	80	1	Video	$832{\times}480$

Appendix

Table S4: Architectural and interface configurations. Generated Frames denotes the number of future frames produced per model invocation, while History Frames denotes the number of previous RGB frames used as visual context. The Output column indicates whether the model natively generates synchronized multi-view observations.

World Models for Robotics

World models for robotics predict how scenes and robot states change under actions to support planning, control, and learning.

See all 384 papers

MiraBench: Evaluating Action-Conditioned Reliability in Robotic World Models
May 28, 2026Tianzhuo Yang, Zihan Shen, Zirui Mi +7World Models for Robotics Action-Conditioned World Models
Institute for Artiﬁcial Intelligence, Peking University · 2Physis Lab
WorldSimProbe: Diagnosing Simulator Faithfulness in Action-Conditioned World Models for Embodied Manipulation
Aug 10, 2026Peterson Co, Sicheng Hu, Chunxuan Jiao +17Robotics Simulation Action-Conditioned World Models
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University
WMEval: Evaluating Interactive World Models Grounded on Physical Simulations
Oct 24, 2026Lanxiang Hu, Yixin Huang, Abhilash Shankarampeta +13World Model Evaluation Interactive World Models
UC San Diego · Unity

May 28, 2026J/K move · Enter open · S save

MiraBench: Evaluating Action-Conditioned Reliability in Robotic World Models

Tianzhuo Yang, Zihan Shen, Zirui Mi, Zhaoyi Zhang+6

Institute for Artiﬁcial Intelligence, Peking University · 2Physis Lab

World Models for Robotics Action-Conditioned World Models World Model Evaluation

Open paper AskPDF arXiv

Action-conditioned world models are increasingly used as scalable simulators for robot learning, yet current evaluations provide limited evidence that their predictions are reliable under the actions they condition on. Existing benchmarks largely emphasize visual fidelity, leaving unclear whether predicted futures are physically plausible, faithful to commanded actions, and calibrated to failure when actions should not succeed. We introduce MiraBench, a hierarchical benchmark that defines action-conditioned reliability as a core evaluation target for robotic world models. MiraBench decomposes this target into three progressively demanding levels: Physics Adherence, which evaluates reference-free physical consistency; Action-Following Fidelity, which measures whether predictions respect task-relevant action inputs; and Optimism Bias Detection, which probes the tendency to predict successful outcomes under failure-inducing actions. To support this evaluation, we curate a human-annotated corpus with over 16,000 judgments across tasks, failure categories, and leading world models. We evaluate 12 representative model configurations spanning vector-conditioned robotic world models, text-conditioned generative world models, open-weight systems, closed-source systems, and multiple model scales. Across this broad model landscape, MiraBench reveals three central findings: visual fidelity is a poor proxy for action fidelity; increasing model scale does not reliably improve action following; and optimism bias is pervasive across current systems. By shifting evaluation from appearance to action-conditioned reliability, MiraBench provides a diagnostic foundation for assessing and improving robotic world models as faithful simulators.

Abstract

World Models for Robotics

Explore similar work

MiraBench: Evaluating Action-Conditioned Reliability in Robotic World Models

WorldSimProbe: Diagnosing Simulator Faithfulness in Action-Conditioned World Models for Embodied Manipulation

WMEval: Evaluating Interactive World Models Grounded on Physical Simulations