WorldFailBench: Do World Models Understand Policy Failures?

Zheng Zhang; Xiaoyang Tang; Shuaiqi Cheng; Jiakai Huang; Zhongbo Wu; Ziyi Zhang; Yuan Si; Zeyu Liu

cs.ROOct 24, 2026

WorldFailBench: Do World Models Understand Policy Failures?

Authors: Zheng Zhang, Xiaoyang Tang, Shuaiqi Cheng, Jiakai Huang, Zhongbo Wu, Ziyi Zhang, Yuan Si, Zeyu Liu,

Abstract

Robot policy evaluation informs deployment decisions and policy improvement, but physical testing and simulator preparation require substantial effort. World models offer a complementary route through closed-loop interaction in generated environments. Recent benchmarks have demonstrated their potential as policy evaluators through outcome-based evaluation (i.e., assessing task success or failure). However, such evaluation does not establish whether world models faithfully simulate how policies fail. Misrepresenting failures can lead to incorrect diagnoses of policy weaknesses and make policies appear ready for deployment even when small scene changes cause them to fail. We therefore ask: do world models faithfully simulate how and at which stages policies fail, and under what conditions they transition from success to failure? To answer this question, we introduce WorldFailBench with three evaluation tasks. Failure reason tests whether models reproduce the observable errors policies make. Failure stage assesses whether failures concentrate at the same stages of full-task execution. Failure boundary examines whether models reproduce the conditions under which policies transition from success to failure under controlled scene changes in simulation. The benchmark covers manipulation tasks in the LIBERO and RoboTwin simulation environments and real-world manipulation tasks performed by the G1 humanoid robot. Experiments across seven world models reveal substantial gaps in their understanding of policy failures, providing guidance for developing more reliable world models for policy evaluation.

Figures & tables

Figure 1: Failure discrepancies in cloth collection. (A) Failure reason: dropping the cloth versus grasping the basket. (B) Failure stage: reference placement versus model transport. (C) Failure boundary: missed transitions under basket displacement.

Benchmark	WM loop	Reason	Stage-level	Boundary	Sim.	Real	Single	Dual	Humanoid
	Protocol	Failure understanding			Environment		Embodiment
WorldArena [ 15 ]	$\checkmark$	$\times$	$\times$	$\times$	$\checkmark$	$\times$	$\times$	$\checkmark$	$\times$
WMBench [ 16 ]	$\checkmark$	$\times$	$\checkmark^{\dagger}$	$\times$	$\checkmark$	$\checkmark$	—	$\checkmark$	—
WorldArena 2.0 [ 23 ]	$\checkmark$	$\times$	$\times$	$\times$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\times$
EWMBench [ 24 ]	$\times$	$\triangle$	$\triangle$	$\times$	$\times$	$\checkmark$	—	$\checkmark$	—
WorldSimBench (robot) [ 25 ]	$\times$	$\times$	$\triangle$	$\times$	$\checkmark$	$\times$	$\checkmark$	$\times$	$\times$

Table 1: Comparison of benchmark designs.

\checkmark

: dedicated evaluation;

\triangle

: partial or indirect;

\times

: not evaluated; —: unverified.

\dagger

: subtask outcome agreement.

Figure 2: Overview of WorldFailBench: comparing reference and world-model rollouts through failure reason, failure stage, and failure boundary.

Platform	Reference environment	Tasks	Policies / task	Rollouts / task–policy	Rollouts / task	Total rollouts
LIBERO	Sim., single arm	10	3	50	150	1,500
RoboTwin	Sim., bimanual	20	5	50	250	5,000
G1	Real, humanoid	2	4	50	200	400

Table 2: Benchmark composition and reference rollout allocation for failure-reason and failure-stage evaluation. Rollouts per task sum over that task’s policies.

Figure 3: Benchmark data distribution: (a) reference failure counts by type, with

n

total reference failures per domain; (b) episode counts grouped by the number of named task stages; (c) boundary-pair counts by perturbation type. Real World denotes the real-world tasks on G1.

Evaluator	$\pi_{0.5}$	$\pi_{0.5}$ -data-50	$\pi_{0.5}$ -data-20	LingBot	FastWAM	Mean
Ref–Ref	$0.059$	$0.049$	$0.068$	$0.057$	$0.072$	$0.061$
A2World	$0.212$	$0.203$	$0.217$	$0.207$	$0.216$	$0.211$
BWM	$0.267$	$0.230$	$0.176$	$0.262$	$0.244$	$0.236$
Cosmos	$0.422$	$0.272$	$0.295$	$0.381$	$0.304$	$0.335$
Ctrl	$\mathbf{0.153}$	$0.190$	$0.141$	$0.229$	$\mathbf{0.142}$	$0.171$
DW05	$0.155$	$\mathbf{0.166}$	$\mathbf{0.136}$	$\mathbf{0.164}$	$0.174$	$\mathbf{0.159}$

Table 3: Type Frequency Error (TFE) on RoboTwin (

\downarrow

). Policy columns average tasks; Mean weights the five policies equally. Bold marks the best world-model values.

Figure 4: RoboTwin mean within-task Pearson correlations for (a) success rate, (b) Task Planning, and (c) Action Execution. Ref–Ref is the reference-repeatability baseline; each correlation uses five policies.

Evaluator	$\pi_{0.5}$	$\pi_{0.5}$ -data-50	$\pi_{0.5}$ -data-20	LingBot	FastWAM	Mean
Ref–Ref	$0.079$	$0.104$	$0.098$	$0.097$	$0.093$	$0.094$
A2World	$0.691$	$0.880$	$0.959$	$0.613$	$0.772$	$0.783$
BWM	$0.682$	$0.751$	$0.808$	$0.751$	$0.766$	$0.751$
Cosmos	$0.861$	$0.760$	$0.871$	$0.853$	$0.740$	$0.817$
Ctrl	$0.525$	$\mathbf{0.611}$	$\mathbf{0.679}$	$0.658$	$0.691$	$\mathbf{0.633}$
DW05	$\mathbf{0.521}$	$0.730$	$0.818$	$0.543$	$\mathbf{0.663}$	$0.655$

Table 4: Stage Frequency Error (SFE) on RoboTwin (

\downarrow

). Policy columns average tasks; Mean averages policies within each task, then tasks. Frequencies use all valid paired episodes, including successes. Bold marks the best world-model values.

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Task / capability	Instruction	Native success criterion	Limit
beat_block_hammer Tool use	Pick up the hammer and strike the block.	Hammer contacts the block, with planar functional-point errors below 0.02 m per axis.	400
blocks_ranking_rgb Color-based arrangement	Arrange the blocks in red–green–blue order.	Correct left-to-right order and adjacent spacing; both grippers open.	1200
blocks_ranking_size Size-based arrangement	Arrange the blocks from large to small.	Correct left-to-right size order and adjacent spacing; both grippers open.	1200
handover_block Bimanual transfer	Transfer the block between arms and place it at the target.	Block reaches the target within position tolerances; right gripper open.	800
hanging_mug Reorientation and hanging	Reorient the mug and hang it on the rack.	Mug functional point aligns with the rack target and satisfies the height threshold; right gripper open.	900
move_pillbottle_pad Precise placement	Place the pill bottle on the pad.	Bottle satisfies planar alignment and resting-height tolerances; both grippers open.	400

Appendix

Table 7: RoboTwin tasks for failure-reason and failure-stage evaluation under demo_clean . Instructions are abbreviated; policies receive the original episode-specific text. Success conditions summarize native checks. Limits count waypoint commands.

Task	Stages
beat_block_hammer	1. Reach; 2. Grasp; 3. Transfer; 4. Strike
blocks_ranking_rgb	1. Reach; 2. Grasp red; 3. Transfer right; 4. Place; 5. Grasp green; 6. Transfer center; 7. Grasp blue; 8. Transfer left
blocks_ranking_size	1. Reach; 2. Grasp small; 3. Transfer right; 4. Place; 5. Grasp medium; 6. Transfer center; 7. Grasp large; 8. Transfer left
handover_block	1. Arm 1: Reach; 2. Arm 1: Grasp; 3. Arm 1: Transfer; 4. Arm 2: Reach; 5. Arm 2: Grasp; 6. Arm 1: Release; 7. Arm 2: Transfer; 8. Arm 2: Place
hanging_mug	1. Arm 1: Reach; 2. Arm 1: Grasp; 3. Arm 1: Place; 4. Arm 2: Reach; 5. Arm 2: Grasp; 6. Arm 2: Transfer; 7. Arm 2: Place
move_pillbottle_pad	1. Reach; 2. Grasp; 3. Transfer; 4. Place

Appendix

Table 8: Task-specific RoboTwin stage categories. Repeated Reach and Place actions share labels in blocks_ranking_rgb and blocks_ranking_size ; other repeated actions retain separate sequence positions. Slash-separated parallel actions form one stage.

Task / capability	Instruction	Native success criterion	Limit
task_00 Two-object placement	Put both the alphabet soup and the tomato sauce in the basket.	Both objects are inside the basket.	520
task_01 Two-object placement	Put both the cream cheese box and the butter in the basket.	Both objects are inside the basket.	520
task_02 State change and placement	Turn on the stove and put the moka pot on it.	Stove is on; moka pot is on its cooking region.	520
task_03 Placement and drawer closure	Put the black bowl in the bottom drawer of the cabinet and close it.	Bowl is inside the bottom drawer; drawer is closed.	520
task_04 Object–target association	Put the white mug on the left plate and put the yellow and white mug on the right plate.	Each mug is on its designated plate.	520
task_05 Compartment placement	Pick up the book and place it in the back compartment of the caddy.	Book is inside the caddy’s rear compartment.	520

Appendix

Table 9: Complete LIBERO-10 task set. All listed goal predicates must hold simultaneously. Limits count policy actions and exclude the ten settling steps.

Task	Ordered stages
task_00	1. Grasp the alphabet soup.; 2. Move the alphabet soup to the basket.; 3. Release the alphabet soup and retreat the arm.; 4. Grasp the tomato sauce.; 5. Move the tomato sauce to the basket.; 6. Release the tomato sauce and retreat the arm.
task_01	1. Grasp the cream cheese box.; 2. Move the cream cheese box to the basket.; 3. Release the cream cheese box and retreat the arm.; 4. Grasp the butter.; 5. Move the butter to the basket.; 6. Release the butter and retreat the arm.
task_02	1. Turn on the stove.; 2. Grasp the moka pot.; 3. Move the moka pot to the stove.; 4. Place the moka pot on the stove, release it, and retreat the arm.
task_03	1. Open the bottom drawer.; 2. Grasp the black bowl.; 3. Move the black bowl into the bottom drawer.; 4. Release the bowl and retreat the arm.; 5. Close the bottom drawer and retreat the arm.
task_04	1. Grasp the white mug.; 2. Move the white mug to the left plate.; 3. Release the white mug and retreat the arm.; 4. Grasp the yellow-and-white mug.; 5. Move the yellow-and-white mug to the right plate.; 6. Release the mug and retreat the arm.
task_05	1. Grasp the book.; 2. Move the book to the back compartment of the caddy.; 3. Release the book and retreat the arm.

Appendix

Table 10: Ordered stages for all LIBERO-10 tasks.

Figure 7: Task scenes, instructions, and stage decompositions for two real-world tasks on G1: (a) sweeping trash and (b) collecting clothes. The robot uses LinkerHand dexterous hands; red annotations highlight relevant objects and alignment.

Platform	Environment and policy observations	Policy action space	Policy execution settings
LIBERO	Initial states reconstructed with MuJoCo 3.3.2 and robosuite 1.4.0; 256 $\times$ 256 fixed front-view RGB; 8D robot state; task instruction	7D action: end-effector translation and rotation increments plus a gripper command	The evaluated $\pi_{0.5}$ variants predict 10 actions and execute the first 8 at 20 Hz, replanning every 0.4 s
RoboTwin	demo_clean scenes and seen instructions; head-camera RGB; 14D robot state; task instruction	14D absolute targets: six joint targets and one gripper target per arm	30 Hz action timeline; $\pi_{0.5}$ : prediction horizon 10; LingBot-VLA-v2: reference action chunk 50; FastWAM: reference replanning interval 24 actions and 10 sampling steps
G1	256 $\times$ 448 front-view RGB; 26D robot state; task instruction	26D action	Kairos; four policy versions per task

Appendix

Table 11: Environment and policy interfaces. Prediction horizon denotes the number of actions returned by a policy call; executed prefix or replanning interval denotes the portion advanced before another call. Action-timeline frequencies and iterative sampling steps are separate quantities. Reference commands are executed through the native platform interfaces described below.

Model	Applied inference configuration
BWM (R)	Image input: Head-camera view. Sampling: 50 steps; classifier-free guidance (CFG) scale 1. Temporal and action interface: 21-frame window including 9 history frames. Forward kinematics converts joint targets into bimanual end-effector poses and gripper commands.
Ctrl-World	Image input: 192 $\times$ 320. Sampling: 25 steps; guidance 1. Temporal and action interface: 6 history frames and 5 output frames; 5 Hz. Actions use training-set first- and ninety-ninth-percentile normalization statistics.
Cosmos-Predict-2.5	Image input: L: 256 $\times$ 256; R/K: 256 $\times$ 320. Sampling: R: 35 steps; L/K: 25 steps; guidance 0. Temporal and action interface: 1 conditioning frame and 12 future frames; 5 Hz. Platform-specific 7D/14D/26D action conditioning. The consumed prediction prefix follows the actual input action length.
DreamDojo	Image input: 480 $\times$ 640. Sampling: 25 steps; guidance 0. Temporal and action interface: 1 conditioning frame and 12 future frames; 5 Hz. Actions map to a 384D conditioning interface. In R, each block of four actions subtracts its corresponding anchor.
DW05	Image input: L: letterboxed to 384 $\times$ 320. Sampling: R: 20 steps; L/K: 25 steps. Temporal and action interface: 1 conditioning frame and 8 future frames. L: 5 Hz; R/K: 7.5 Hz. L/K actions are padded to 32D.
A2World	Image input: 256 $\times$ 256. Sampling: 20 steps; guidance 0. Temporal and action interface: 1 conditioning frame and 20 future frames. L: 20 Hz; R/K: 5 Hz. The L adapter maps 7D policy actions to the original simulation model’s 14D conditioning interface.

Appendix

Table 12: Inference configurations used with the official implementations. R, L, and K denote RoboTwin, LIBERO, and G1. Each row groups the settings for one evaluated system; platform-specific values are marked explicitly. History/conditioning frames, future frames, and sampling iterations have distinct meanings. Guidance values follow the corresponding sampler’s convention.

Model	Official implementation	Initialization source
BWM	Implementation	BWM checkpoint
Ctrl-World	Implementation	DROID-trained checkpoint
Cosmos-Predict-2.5	Implementation	2B action-conditioned checkpoint
DreamDojo	Implementation	2B pretrained checkpoint
DW05	OpenDW implementation	Base checkpoint ; RoboTwin-adapted checkpoint
A2World	Implementation	Pretrained and LIBERO-adapted checkpoints

Appendix

Table 13: Official implementations and released checkpoint sources for platform-specific fine-tuning.

Figure 8: Examples of task-planning and action-execution failures. (a) The robot arranges blocks in a row instead of the instructed tower, or attempts to place soap without first opening the drawer. (b) A released bottle tips and rolls away, or placing the third bowl disrupts the stack. Frames proceed from left to right; red boxes mark the relevant objects and interactions. These cases illustrate the two interpretable failure categories.

Intervention	Search range	Pair step	Selected endpoint range
Planar position ( $x$ or $y$ )	$-40$ to $40$ cm	4 cm	$-36$ to $28$ cm
Yaw about the vertical axis	$-120^{\circ}$ to $120^{\circ}$	$20^{\circ}$	$-120^{\circ}$ to $120^{\circ}$
Linear size multiplier	0.4–2.0	0.2	0.4–1.8

Appendix

Table 14: RoboTwin boundary search ranges, grid steps, and selected endpoint ranges. Position and yaw are offsets from the initial scene; size is a linear scale multiplier. LIBERO shares the search ranges and uses 2 cm position steps, a

10^{\circ}

yaw step, and 0.1 scale-factor steps.

Figure 9: Scene interventions for boundary construction. SR denotes success rate; $\Delta y$ and $\Delta\mathrm{yaw}$ are offsets from the base scene, with scale factors relative to the original object size. The conditions differ by (a) cup translation along world $+y$ , from $+20$ to $+24$ cm (success: 100% to 0%); (b) tray rotation about world $z$ , from $-60^{\circ}$ to $-80^{\circ}$ (100% to 0%); or (c) uniform roller scaling, from $0.6\times$ to $0.4\times$ (100% to 0%).

Evaluator	Task Planning	Action Execution	Overall Mean
Ref–Ref	$0.796$	$0.831$	$0.814$
A2World	$0.353$	$-0.086$	$0.134$
BWM	$-0.121$	$0.143$	$0.011$
Cosmos	$0.008$	$0.038$	$0.023$
Ctrl	$0.177$	$0.204$	$0.191$
DW05	$\mathbf{0.456}$	$\mathbf{0.348}$	$\mathbf{0.402}$

Appendix

Table 15: Type-specific Policy Correlation (TPC) on RoboTwin (

\uparrow

). Type columns average defined within-task Pearson correlations across five policies. Overall Mean averages defined type correlations within tasks, then tasks equally. Bold marks the best world-model values.

Figure 10: Stage-specific failure frequencies for blocks_ranking_size across five policies. Each panel shows reference execution and seven world models; cells report the percentage of all valid episodes with a failure at each stage. The stage key and color scale are shared.

Figure 11: Responses across 60 frozen boundary pairs. (a–c) Reference and model success-rate changes, labeled $\Delta^{\mathrm{GT}}$ and $\Delta^{\mathrm{WM}}$ , respectively. Each column uses the endpoint order fixed during discovery; reference changes come from independent re-estimation. (d–f) Setting-weighted shares of preserved, flat, and reversed responses. Model changes within $\pm 2$ percentage points are flat; nonflat changes are preserved or reversed according to agreement with the independently estimated reference direction. The 95.2% underestimation rate is descriptive: the 420 model–pair comparisons across seven models share boundary pairs and some endpoints and are not independent.

Evaluator	$\pi_{0.5}$	$\pi_{0.5}$ -data-50	$\pi_{0.5}$ -data-20	Mean
Ref–Ref	$0.032$	$0.056$	$0.048$	$0.045$
A2World	$0.300$	$0.264$	$\mathbf{0.248}$	$\mathbf{0.271}$
Cosmos	$0.322$	$0.288$	$0.254$	$0.288$
Ctrl	$\mathbf{0.290}$	$\mathbf{0.257}$	$0.282$	$0.276$
DW05	$0.352$	$0.293$	$0.284$	$0.310$
DreamDojo	$0.421$	$0.384$	$0.338$	$0.381$

Appendix

Table 16: Type Frequency Error (TFE) on LIBERO (

\downarrow

). Policy columns average tasks; Mean weights the three policies equally. Bold marks the best world-model values.

Evaluator	TPC
Ref–Ref	$0.780$
A2World	$-0.062$
Cosmos	$\mathbf{0.242}$
Ctrl	$-0.189$
DW05	$0.043$
DreamDojo	$0.108$

Appendix

Table 17: LIBERO TPC across three policies (

\uparrow

). Bold marks the best world-model values.

	SFE ( $\downarrow$ )				SPC ( $\uparrow$ )
Evaluator	$\pi_{0.5}$	$\pi_{0.5}$ -data-50	$\pi_{0.5}$ -data-20	Mean
Ref–Ref	$0.064$	$0.124$	$0.140$	$0.109$	$0.680$
A2World	$0.808$	$0.796$	$\mathbf{0.768}$	$0.791$	$-0.271$
Cosmos	$0.836$	$0.832$	$0.824$	$0.831$	$-0.173$
Ctrl	$\mathbf{0.692}$	$\mathbf{0.682}$	$0.844$	$\mathbf{0.739}$	$-0.021$
DW05	$0.932$	$0.862$	$0.928$	$0.907$	$-0.235$

Appendix

Table 18: Stage Frequency Error (SFE,

\downarrow

) and Stage-specific Policy Correlation (SPC,

\uparrow

) on LIBERO. SFE policy columns average tasks; Mean averages policies within tasks, then tasks. Frequencies use all valid paired episodes. SPC averages defined within-task Pearson correlations over stages, then tasks. Bold marks the best world-model values.

Figure 12: Stage-specific failure frequencies for LIBERO task_00 across three policies. Each panel compares reference execution with six world models; cells give the percentage of all valid episodes with a failure at each stage. The stage key and color scale are shared. The data-100 policy is the full-data $\pi_{0.5}$ used in the text and tables.

Evaluator	Position		Orientation		Size
	OA $\uparrow$	GE $\downarrow$	OA $\uparrow$	GE $\downarrow$	OA $\uparrow$	GE $\downarrow$
Ref–Ref	0.950	0.035	0.850	0.080	0.900	0.065
A2World	0.067	0.517	0.150	0.584	0.000	0.584
Cosmos-Predict-2.5	0.067	0.522	0.075	0.534	0.000	0.556
Ctrl-World	0.000	0.528	0.175	0.514	0.077	0.543
DreamDojo	0.033	0.522	0.000	0.543	0.000	0.530

Appendix

Table 19: Failure-boundary results on LIBERO (OA

\uparrow

, GE

\downarrow

). Ref–Ref reports reference repeatability. Bold marks the best world-model value in each column, including ties.

Figure 13: Orientation case for

\pi_{0.5}

-data-50 on the LIBERO task of placing a book in a caddy. The reference curve shows sampled book yaw perturbations; world-model curves connect the highlighted

40^{\circ}

and

50^{\circ}

endpoints. Cosmos-P2.5 denotes Cosmos-Predict-2.5.

	Failure type: TFE			Failure stage: SFE
World model	Sweep	Clothes	Mean	Sweep	Clothes	Mean
Ctrl	0.0971	0.1195	0.1083	0.8284	0.4042	0.6163
DreamDojo	0.1528	0.1715	0.1621	0.8340	0.5800	0.7070
Cosmos	0.1796	0.2929	0.2362	0.7143	0.9544	0.8343
DW05	0.1429	0.2171	0.1800	0.6436	0.7269	0.6852
A2World	0.0603	0.1258	0.0930	0.2759	0.3975	0.3367

Appendix

Table 20: Failure-diagnostic errors for real-world tasks on G1 (

\downarrow

). TFE averages two failure-type errors; SFE sums four stage errors. Task scores average four policy versions; Mean weights the two tasks equally. Bold marks column minima.

	Sweeping trash		Collecting clothes
Model	TPC ( $\uparrow$ )	SPC ( $\uparrow$ )	TPC ( $\uparrow$ )	SPC ( $\uparrow$ )
Ctrl	0.5016	-0.1270	0.6812	0.3463
DreamDojo	-0.4996	0.1648	0.3714	0.4824
Cosmos	-0.6619	-0.2548	0.1005	0.2258
DW05	0.1046	-0.0251	0.0569	0.1479
A2World	0.5118	0.4667	-0.2089	0.0545

Appendix

Table 21: Task-specific Pearson agreement across four policy versions for real-world tasks on G1. TPC and SPC average defined type and stage correlations, respectively. Bold marks column maxima.

Policy	Sweeping trash	Collecting clothes
V1	30.61	28.00
V2	50.00	59.18
V3	65.31	78.00
V4	28.00	56.25

Appendix

Table 22: Reference success rates (%) for real-world tasks on G1, computed over executions with judged outcomes. Each task has four policy versions; bold marks the highest rate within each task.

Figure 14: Qualitative failure comparisons between reference and world-model rollouts in simulation. (a) Reference and Ctrl-World both manipulate the red block directly instead of using the hammer. (b) The blocks-ranking-color example shows different annotated failure types: the reference arranges the blue block on top of the red one in instead of in a row, while the model fails to acquire the red block. (c) The reference fails during placement of the third bowl, while A2World fails during acquisition of the first. Frames proceed from left to right, with red boxes marking the displayed evidence.

Figure 15: Reference and world-model rollouts for real-world tasks on G1, illustrating (a) preserved failure, (b) a failure-type mismatch, and (c) a failure-stage mismatch. Frames proceed from left to right; instructions, labels, and visual evidence appear in each panel.

World Model Evaluation

World Model Evaluation measures whether learned simulators predict consistent, physically plausible, action-responsive futures that help agents plan and act.

See all 212 papers

GigaWorld-1: A Roadmap to Build World Models for Robot Policy Evaluation
Jul 2, 2026GigaWorld Team, Angyuan Ma, Boyuan Wang +24Robot Foundation Models Action-Conditioned World Models
GigaAI · Tsinghua University
World Models Dream of Success: Diagnosing and Repairing Failure Insensitivity in Robot World Models
Oct 6, 2026Jiuyi Xu, Xiao Hu, Meida Chen +3World Models for Robotics Action-Conditioned World Models
Colorado School of Mines · Northeastern University · Institute for Creative Technologies, University of Southern California +1
MiraBench: Evaluating Action-Conditioned Reliability in Robotic World Models
May 28, 2026Tianzhuo Yang, Zihan Shen, Zirui Mi +7World Models for Robotics Action-Conditioned World Models
Institute for Artiﬁcial Intelligence, Peking University · 2Physis Lab

Jul 2, 2026J/K move · Enter open · S save

GigaWorld-1: A Roadmap to Build World Models for Robot Policy Evaluation

GigaWorld Team, Angyuan Ma, Boyuan Wang, Bohan Li+23

GigaAI · Tsinghua University

Robot Foundation Models Action-Conditioned World Models Robotic Policy Evaluation Robot Manipulation Benchmarks World Model Evaluation Video World Models

Open paper AskPDF arXiv

Evaluating embodied robot foundation models remains a critical bottleneck; unlike large language models efficiently assessed via digital benchmarks, robotic policies require slow, costly real-world rollouts limited by hardware and human supervision, which has driven interest in world models as surrogate policy evaluators, yet the key properties that make a world model reliable for policy assessment remain poorly understood. This work presents a systematic study of world models for robotic policy evaluation and introduces WMBench, a benchmark constructed from real-robot teleoperation data and matched policy rollouts covering diverse manipulation tasks to enable controlled comparisons across model families, action encodings, rollout horizons, and evaluation metrics. Using WMBench, we analyze 7 video world models, 4 action representation schemes, and over 324,000 simulated policy rollouts paired with real robot executions, further enriching our analysis with large-scale community submissions from the CVPR 2026 GigaBrain Challenge, curated synthetic trajectories, and a training videos spanning more than 12,000 hours. Our experiments deliver three core insights: evaluator quality is dominated by long-horizon, action-faithful rollout consistency rather than short-term visual realism; pretraining gains stem not only from data scale but from balancing general world knowledge with robot-specific controllability; and architectural choices including action encoding, memory design, and evaluator-focused post-training strongly determine alignment with real-world robot behavior. Drawing on these results, we derive a practical design roadmap and realize it in GigaWorld-1, a world model specially optimized for policy evaluation, and we fully release our code, models, datasets, and toolkits to advance scalable evaluation research for embodied foundation models.

Abstract

World Model Evaluation

Explore similar work

GigaWorld-1: A Roadmap to Build World Models for Robot Policy Evaluation

World Models Dream of Success: Diagnosing and Repairing Failure Insensitivity in Robot World Models

MiraBench: Evaluating Action-Conditioned Reliability in Robotic World Models