cs.ROAug 30, 2026

Do World Models Make Better Robots? A Survey of Evaluation Benchmarks for Predictive Embodied Intelligence

Authors: Gaytri Jena, Kapil Wanaskar, Vinija Jain, Aman Chadha, Vasu Sharma, Amitava Das

Organizations: UC Berkeley, USA · San Jose State University, USA · Meta, USA · Apple, USA · 

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Robot learning now advances along two tracks that rarely meet. On one side, direct Vision-Language-Action (VLA) policies map observations to actions and are scored by closed-loop task success. On the other, predictive and generative world models forecast future observations and are scored by open-loop prediction or generation quality. A natural question sits between them: does world modelling earn a measurable, closed-loop advantage over a direct policy, and for which robotic capabilities? We argue that the field cannot yet answer this question, and that the reason is a gap in how it is measured, not in the models themselves. World-model benchmarks score prediction without ever executing it, while task-success suites host a single policy and never build a world-model versus VLA contrast. This survey maps the evaluation landscape around that gap. We catalogue 160 web-verified benchmarks spanning 2017 to 2026 and organise them by evaluation mode, robotic capability, and model family into four lanes: policy suites, embodied agents, world model evaluation, and prediction-to-action bridges. Across the corpus, 138 of 160 benchmarks are model-agnostic and only 11 (7%) build an explicit VLA-versus-world-model contrast; counterfactual capability is almost entirely unmeasured, and only four benchmarks turn prediction into executed action. We contribute an operational taxonomy, a coverage comparison against the eight closest surveys (ours is the only one to cross capability with model family), an evaluation loop that isolates the advantage of prediction, and an actionable protocol of four advantage-aware metrics anchored on named testbeds. The organising claim is not that world models help or do not help, but that answering the question requires benchmarks built to ask it.

Figures & tables

Figure 1. Predict or Act? The survey’s organising question, posed with real benchmarks. Closed-loop (left): policy and embodied suites score task success by running a policy in the environment. Open-loop (right): world-model and video-generation benchmarks score prediction or generation quality with no action. The centre axis is the question neither pole answers, whether prediction earns a closed-loop advantage, built explicitly by only 11 of 160 benchmarks (Table 10 ). Every tile is a representative figure from the benchmark’s paper and links to it; per-tile provenance is in Table 13 . Images © their respective authors, reproduced for scholarly review. A two-pole collage of benchmark figures. A black title bar reads ``Predict or Act?''. The left band, CLOSED-LOOP, shows nine policy and embodied benchmark figures; the right band, OPEN-LOOP, shows nine world-model and video-generation benchmark figures; a centre spine contrasts task success (closed loop) with prediction quality (open loop) and notes that only 11 of 160 build the contrast.
SurveyVenueYrα\alpha modeβ\beta cap.γ\gamma familyδ\delta metricε\varepsilon catalog
Evaluation of Embodied AI ( Hou et al., 2026a )Authorea2026~~✘✘✔
VLA Datasets & Bench. ( Wang et al., 2026 )arXiv2026✘✘✘✘✔
Benchmark Construction ( Lai et al., 2026 )arXiv2026✘~✘✘✔
RL Reproducibility ( Lynnerup et al., 2020 )CoRL2020✘✘✘✘✘
World Models for Robots ( Hou et al., 2026b )arXiv2026~✘~~~
Embodied World Models ( Shang et al., 2026b )preprint2026~✘~✘~
Table 1. Topical coverage of the closest benchmark/evaluation surveys versus ours. ✔ covered as an organising axis, ~ partial or mentioned in passing, ✘ absent. Axes: α\alpha evaluation mode (open- vs closed-loop as the primary axis); β\beta capability cuts (hidden-state, counterfactual, long-horizon, social); γ\gamma capability ×\times model-family cross-tab (direct VLA vs latent / action-conditioned / generative world models under one protocol); δ\delta behavioural advantage-of-prediction metric ( assessed, not adopted ; e.g. ACTION-ATLAS’s World Advantage Score); ε\varepsilon comprehensive benchmark catalogue. This survey is the only work that covers γ\gamma , and δ\delta is otherwise reached only partially , and only by a position paper ( Yu et al., 2026 ) , never by a survey. Venues: CoRL, IEEE TETCI; the remaining rows are arXiv or unrefereed preprints (Authorea, Preprints.org).
Figure 2. The closest benchmark/evaluation surveys as a timeline. Two predate the recent wave, an IEEE TETCI survey of embodied-AI simulators and tasks ( Duan et al., 2022 ) and a CoRL survey of real-robot RL reproducibility ( Lynnerup et al., 2020 ) ( amber = refereed venue); the rest are 2026 preprints on embodied evaluation, VLA data, and world models. None organises the robot-evaluation landscape by evaluation-mode ×\times capability ×\times model-family; the sole work reaching the advantage-of-prediction axis is a position paper ( Yu et al., 2026 ) . Ours ( ⋆\star ) fills that cell; per-axis coverage is in Table 1 . A horizontal timeline of eight related surveys. Two amber refereed markers sit at 2019 (RL reproducibility, CoRL) and 2021 (embodied-AI simulators, IEEE TETCI); after a time break, a dense cluster of 2026 preprint markers covers evaluation of embodied AI, VLA datasets, world models for robots, embodied world models, benchmark construction, and a world-model-evaluation position paper. A red star marks the present survey at the end.
Figure 3. Corpus construction in the PRISMA 2020 style ( Page et al., 2021 ) . The left column is the web-search stream behind the landscape corpus (Table 10 ): five thematic sweeps, de-duplicated by name and arXiv ID, screened to drop datasets and models, and metadata-verified by parallel audit agents. The right column is the venue-targeted and snowball search behind the closest surveys (Table 1 ). Both streams feed the included set. Candidates whose metadata could not be verified were discarded at harvest; their count was not logged, which we state rather than estimate. A PRISMA 2020 style flow diagram with two identification columns. The left column (structured web search) shows candidate benchmark records from five thematic sweeps, de-duplicated by name and arXiv ID, screened to remove seven datasets and models, and 160 metadata-verified records forming the landscape corpus (155 with an arXiv ID, 5 without). The right column (survey search) yields 8 closest surveys including one position paper. Both columns feed an included box of 160 benchmarks plus 8 surveys and one methodology citation, totalling 164 references.
ConstructCounts (a benchmark has it if and only if…)Does not count
Open-loop eval ( open )it scores prediction or generation quality on fixed inputs without executing the prediction in a feedback loop: video fidelity (FVD), physics or rollout consistency, or frame-level accuracyany protocol where an action is executed and the environment responds; offline scoring of a policy’s success
Closed-loop eval ( closed )success is measured by running a policy in the environment with state feedback and scoring task outcome: success rate, sub-goal completion, or reward under interactionscoring a generated video or a static prediction; a metric computed without stepping the environment
Bridge ( bridge )a prediction (imagined rollout, video, or latent plan) is converted into executed actions and scored on the resulting task successa benchmark that stops at prediction quality; one that never turns a prediction into a control signal
Native VLA-vs-WM contrast ( γ\gamma )the benchmark itself runs both a direct Vision-Language-Action policy and a world-model policy under one protocol and reports the head-to-heada model-agnostic suite that merely hosts either family; a suite evaluated with only one family in the original paper
Advantage-of-prediction ( δ\delta )a metric quantifies the closed-loop gain of a predictive policy over a matched direct-VLA baseline on the same tasks (assessed, not adopted)absolute success with no matched baseline; open-loop quality scores; a ranking that never isolates the predictive component
Table 2. Operational definitions used to place each benchmark on the evaluation axes (§ 2 ). They resolve the ambiguous boundary cases: hosting a policy is not the same as building a VLA-vs-world-model contrast ( γ\gamma ), and an absolute success number is not an advantage-of-prediction measurement ( δ\delta ) without a matched baseline. open open-loop, closed closed-loop, bridge bridge.
Figure 4. The catalogued corpus at a glance. The survey maps 160 web-verified benchmarks across 4 evaluation lanes, spanning 2017–2026 and totalling 164 references; only 11 build an explicit Vision-Language-Action vs world-model contrast. The bars give benchmarks per capability area, with kindred clusters merged (dexterity into manipulation, vision-language navigation into navigation, theory-of-mind into hidden-state); the twelve areas sum to the 160 works of Table 10 . Bars are coloured by the dominant lane of each area (teal policy, blue embodied, purple world-model). An overview figure. Six stat callouts read: 160 benchmarks, 164 references, 4 evaluation lanes, 12 capability areas, 11 explicit contrasts, span 2017 to 2026. Below, a horizontal bar chart gives benchmarks per capability area: long-horizon and planning 26, manipulation and dexterity 22, generation quality 19, navigation 16, physical reasoning 13, social and multi-agent 12, hidden-state and theory-of-mind 9, counterfactual 9, autonomous driving 9, generalization 7, locomotion 6, other areas 12.
Figure 5. Profile of the 160 catalogue benchmarks (Table 10 ), re-counted directly from the catalogue. (a) publication year, showing the 2023–2024 surge; (b) evaluation lane (embodied navigation dominates, then policy suites); (c) evaluation mode (closed-loop task success dominates, open-loop prediction is second, only four bridges reach action); (d) whether the benchmark builds a native Vision-Language-Action vs world-model contrast, the survey’s organising question: 138 of 160 are model-agnostic , and only 11 (7%) build the contrast explicitly. Counts are the exact column tallies of Table 10 . Four bar charts of the 160 benchmarks. (a) Benchmarks per year rise from 2 in 2017 to 31 in 2023 and 36 in 2024, then 20 in 2025. (b) Evaluation lane: policy 37, embodied 85, world-model evaluation 34, bridge 4. (c) Evaluation mode: closed 113, open 43, bridge 4. (d) VLA-vs-world-model contrast: model-agnostic 138, partial 11, explicit 11, with a note that only 11 (7 percent) build a contrast.
Figure 6. The world-model-evaluation wave is recent. Per-year counts of closed-loop task-success benchmarks (green) and open-loop world-model / video-generation benchmarks (magenta) in the catalogue. Closed-loop suites dominate through 2024; in 2025 open-loop evaluation overtakes them (13 vs 6) as the world-model-generation literature arrives. The four prediction-to-action bridges are newer still, all appearing in 2024 or later. Years are the benchmarks’ first-release years (2026 is partial and omitted); counts are the tallies of Table 10 . A grouped bar chart of benchmarks by evaluation mode per year, 2019 to 2025. Closed-loop (green): 10, 16, 19, 8, 24, 24, 6. Open-loop (magenta): 3, 1, 4, 3, 7, 11, 13. Closed-loop dominates until 2024, then open-loop overtakes it in 2025.
BenchmarkYearVenueEvalCapabilityVLA vs WM
§3.1 Policy / manipulation suites closed
CALVIN ( Mees et al., 2021 )2021RA-Lclosedlong-horizon language–
LIBERO ( Liu et al., 2023c )2023NeurIPS D&Bclosedshort-horizon + transfer–
Meta-World ( Yu et al., 2019 )2020CoRLclosedmulti-task / meta-RL–
ManiSkill2 ( Gu et al., 2023 )2023ICLRclosedgeneralizable skills–
SIMPLER ( Li et al., 2024a )2024CoRLclosedgeneralization (real vs sim)–
Table 3. Detailed comparison of a representative 31 of the 160 benchmarks in the landscape (Table 10 ), grouped by evaluation lane. Eval : open open-loop prediction/generation quality, closed closed-loop task success, bridge prediction-to-action bridge. VLA vs WM : ✔ native direct-policy vs world-model contrast, ~ partial, – model-agnostic. Venues are web-verified; “arXiv” marks preprint-only works.
DimensionLIBERO ( Liu et al., 2023c )Habitat 3.0 ( Puig et al., 2023 )WorldModelBench ( Li et al., 2025 )World-in-World ( Zhang et al., 2025 )
 policy suiteembodied navworld-model evalbridge
What it scorestask successsocial-nav successvideo-pred. qualityclosed-loop WM success
Evaluation modeclosedclosedopenbridge
Executes actions in a loop?✔✔✘✔
Native VLA-vs-WM contrast?✘✘✘~
Capability stressedshort-horizon + transfersocial / multi-agentphysical realismplan executability
Table 4. Capability matrix, one representative benchmark per evaluation lane. No lane’s exemplar builds a native VLA-vs-world-model contrast: policy and embodied suites are model-agnostic (they host any policy), world-model suites score generation quality with no action, and the bridge exemplar reaches closed-loop success but reports only a partial contrast (“visual quality is not task success”). Counterfactual probing is absent throughout. ✔ yes, ~ partial, ✘ no, – n/a; open open-loop, closed closed-loop, bridge bridge.
Figure 8. Benchmarks across the surveyed corpus. Representative subject figures from the catalogue, grouped by evaluation lane rather than embodiment: closed-loop task-success suites, open-loop world-model and video-generation evaluation, and the prediction-to-action bridges . Each panel is a live hyperlink to the source paper on arXiv and is annotated with its arXiv id (per-panel figure provenance in Table 11 ); the figure is vector with photographs embedded at full resolution, so it can be zoomed without pixelation. Only figures that depict the benchmark’s task or scene are shown; results and leaderboard figures appear in Figure 9 . Benchmarks with no locatable subject figure are omitted, not fabricated. Images © their respective authors, reproduced for scholarly review. A gallery of benchmark subject figures in three lane bands. The top band, closed-loop task-success suites, shows ten policy and embodied benchmark scenes (robot arms and simulated households). The middle band, open-loop world-model and video-generation evaluation, shows ten generated-video or physical-scene samples. The bottom band, prediction-to-action bridges, shows four world-model benchmark scenes. Each panel is labelled with the benchmark name and its arXiv id.
BenchmarkYearEvalWhat it probesVLA vs WM
Hidden-state / memory / occlusion3/4
LIBERO-Mem ( Chung et al., 2025 )2025closedoccluded object state carried across time✔
TEACh ( Padmakumar et al., 2021 )2021closedpartner belief and history via dialog–
ComPhy ( Chen et al., 2022b )2022openhidden mass and charge inferred from collisions✔
Bongard-HOI ( Jiang et al., 2022b )2022openfew-shot hidden-concept induction~
Theory-of-mind3/4
Table 5. Capability deep-dive : the benchmarks that probe each of the β\beta capability cuts, and whether each builds a native VLA-vs-world-model contrast. Native contrasts cluster in the hidden-state, theory-of-mind and counterfactual reasoning cuts (LIBERO-Mem, ComPhy, AGENT, BIB, ACRE, CoPhy, CausalWorld); the long-horizon and social task-success cuts are largely model-agnostic (ALFWorld’s abstract-to-grounded transfer is the lone partial). Eval : open / closed . VLA vs WM : ✔ native contrast, ~ partial, – model-agnostic; each cut’s header tallies the fraction of its benchmarks that build any (native or partial) contrast.
BenchmarkYearModeWhat it scoresSignalPhysical
Generation quality (multi-dimensional)
EvalCrafter ( Liu et al., 2023a )2023Gen17-metric text-to-video quality suiteAuto✘
FETV ( Liu et al., 2023b )2023Genfine-grained text-to-video qualityAuto✘
VBench ( Huang et al., 2023 )2023Gen16 disentangled generation dimensionsAuto✘
StoryBench ( Bugliarello et al., 2023 )2023Gencontinuous story-visualisation qualityAuto✘
DEVIL ( Liao et al., 2024 )2024Gencontent-dynamics quality of T2VAuto✘
Table 6. World-model-evaluation deep-dive : the 34 benchmarks of the world-model-evaluation lane, by sub-type – the P3 counterpart of the skill-building branch deep-dive. Mode : Gen evaluates generated video, QA video question-answering, Pred physical prediction. Signal : Auto computed/learned metric, Acc accuracy vs ground truth, VOE violation-of-expectation, Human human study. Physical : targets physical-law understanding, ✔ core, ~ adjacent, ✘ general. Generation-quality suites dominate and are physics-agnostic; only the physical-reasoning and physical-commonsense sub-types test physical law, and every lane benchmark scores open-loop, never in a closed loop.
Figure 9. Results reported across the surveyed corpus. Representative results figures from the catalogue, grouped by type of evidence: leaderboard and model-comparison charts , per-dimension radar and multi-metric plots, and distributions and composition . Each panel is a live hyperlink to the source paper on arXiv and is annotated with its arXiv id (per-panel provenance in Table 12 ); the figure is vector with plots embedded at full resolution. Only results/leaderboard/metric figures are shown; task and scene figures appear in Figure 8 . As a benchmark survey the reported evidence is dominated by rankings, radars and distributions rather than training curves, and is sparser than the subject figures, so this gallery is smaller; benchmarks that publish no results figure are omitted, not fabricated. A benchmark may appear more than once when it reports several evidence types. Images © their respective authors, reproduced for scholarly review. A gallery of benchmark results figures in three evidence-type bands. The top band, leaderboard and model-comparison charts, shows four ranking bar charts and win-rate tables. The middle band, per-dimension radar and multi-metric, shows three radar plots over evaluation dimensions. The bottom band, distributions and composition, shows a prompt-length histogram, a meta-type pie chart, and a prompt word cloud. Each panel is labelled with the benchmark name and its arXiv id.
Figure 10. The evaluation loop that isolates the advantage of prediction. A benchmark that can answer “does world modelling help a robot act” must run a direct Vision-Language-Action policy and a world-model policy (predict then plan) under one closed loop, then report the closed-loop gain ( δ\delta ) sliced by capability ( β\beta ). Today’s suites break this loop in one of two ways: model-agnostic policy suites host a single policy and never build the contrast, while world-model suites score the prediction open-loop and never execute it (§ 4 ). A left-to-right diagram. A task/scene forks into two branches, a direct VLA policy and a world model that predicts then plans; both feed an environment closed loop with state feedback, producing task success, which feeds an advantage-of-prediction measurement sliced per capability. An annotation notes that model-agnostic suites run only one branch and world-model suites never close the loop, so the contrast is missing.
Sense of “world model”What it isExamplesAct. cond.LoopMetricVLA-cmp.
As a representation (what the model is)
latent WM (LWM)latent-dynamics predictor zt+1=f(zt,at)z_{t+1}=f(z_{t},a_{t})DreamerV3, TD-MPC2✔✔~~
action-cond. WM (WAM)action-conditioned future-frame predictoriVideoGPT, GR-2✔~~~
generative WM (WFM)text / image-to-video generatorSora, Cosmos, Genie~✘✔✘
As a role (what the model is used for)
neural simulatorlearned environment for policy training / evalUniSim✔✔~~
Table 7. The term “world model” spans two orthogonal axes the literature routinely conflates: a representation axis (what the model is: latent-state dynamics, action-conditioned video, or generative video, a soft boundary since action-conditioned generators straddle WAM and WFM) and a role axis (what it is used for: a neural simulator, or the object a benchmark scores). Columns record whether a sense is act ion- cond itioned, is used in a closed loop , carries a standard evaluation metric , and is readily VLA-comp arable under one protocol ( ✔ yes, ~ partial, ✘ no, – n/a). No single sense is action-conditioned, closed-loop, and VLA-comparable at once, which is why a benchmark that contrasts a world-model policy with a VLA policy per capability (§ 4 ) is still missing. Evaluation benchmarks score the world model as an object ; they are not world models acting as judges.
 Policy suites ( Liu et al., 2023c )Embodied nav / social ( Puig et al., 2023 )World-model eval ( Li et al., 2025 )Bridges ( Zhang et al., 2025 )
Loop closure✔ closed-loop; steps the environment ( Mees et al., 2021 )✔ closed-loop; long-horizon rollouts ( Li et al., 2024b )✘ open-loop; scores generation, no action ( Motamed et al., 2025 )~ converts a prediction into an executed action ( Qin et al., 2024 )
Task horizon~ short reactive skills; long by chaining ( Liu et al., 2023c )✔ long-horizon household and dialog ( Padmakumar et al., 2021 )~ short clips; temporal but not task-level ( Hu et al., 2025 )~ a single executed rollout ( Jiang et al., 2026 )
Capability breadth~ manipulation + transfer; hidden-state rare ( Pumacay et al., 2024 )✔ navigation, social, some hidden-state ( Deitke et al., 2020 )~ physical / temporal; counterfactual barely ( Chen et al., 2025 )✘ narrow; executability only ( Shang et al., 2026a )
Model-family contrast✘ model-agnostic host of any policy ( Atreya et al., 2025 )~ memory-vs-Markovian in one suite ( Chung et al., 2025 )✘ world-model-only scores ( Li et al., 2025 )~ WM as policy substrate; no VLA pole ( Zhang et al., 2025 )
Reproducibility~ sim; some real-vs-sim transfer ( Li et al., 2024a )~ sim-heavy; a sim-to-real gap ( Deitke et al., 2020 )✔ fixed offline sets; deterministic scoring ( Zheng et al., 2025 )✘ new, still-unstandardised protocols ( Shang et al., 2026a )
Structural blind spothosts a world-model policy but never builds the VLA contrastcapability cuts exist, but no per-capability family ablationvisual quality is not task success; no behavioural utilityreaches action, but omits the per-capability VLA head-to-head
Table 8. Evaluation-lane comparison matrix : the four benchmark lanes on six evaluation axes (loop closure, task horizon, capability breadth, model-family contrast, reproducibility, and characteristic blind spot). Marks grade each lane on the axis ( ✔ favourable, ~ mixed, ✘ weak), with an anchoring benchmark per cell. No lane closes the loop, slices capability, and builds a native VLA-vs-world-model contrast at once; the bridge lane comes closest yet still omits the per-capability VLA head-to-head, which is the empty cell this survey names (§ 4 ).
MetricDefinitionInstantiation (testbed)Axis stressed
Advantage-of-prediction curveclosed-loop success gain of a predictive policy over a matched direct-VLA baseline, reported per capability as a curve, not a single endpointa capability-sliced protocol over LIBERO ( Liu et al., 2023c ) and CALVIN ( Mees et al., 2021 ) that runs a world-model policy and a VLA policy under one harnessγ\gamma family, δ\delta metric
Counterfactual accuracytask success on episodes that require inference over unobserved or hypothetical dynamics (occlusion, object permanence, “what if”)WorldPrediction ( Chen et al., 2025 ) plus occlusion / memory splits in the style of LIBERO-Mem ( Chung et al., 2025 )β\beta capability
Prediction-to-action fidelity gapthe difference between a world model’s open-loop generation quality and its closed-loop task success (“visual quality is not task success”)bridge protocols such as WorldSimBench ( Qin et al., 2024 ) and World-in-World ( Zhang et al., 2025 )α\alpha mode
Per-capability contrast coveragethe fraction of capabilities for which a benchmark actually runs a VLA-vs-world-model head-to-head under a single protocolaudits of the landscape (Table 10 ) against a declared contrast schemaγ\gamma family, ε\varepsilon catalog
Table 9. An actionable protocol for the future agenda : four measurable quantities, their definitions, and concrete testbeds. The first anchors the advantage-of-prediction question ( δ\delta ) on LIBERO / CALVIN-style suites; the second targets the near-unmeasured counterfactual capability; the third quantifies the prediction-to-action gap the bridges expose; the fourth audits per-capability contrast coverage across the landscape. Each is reported as a distribution or curve, never a single headline number.
Figure 11. An actionable evaluation protocol for the future agenda (Table 9 ). The panels are schematic : they show the axes and the target shape of each measurement, not measured results. (a) a world-model policy should beat a matched direct-VLA baseline in closed-loop success, and the per-capability gap is the advantage of prediction ( δ\delta ), on LIBERO / CALVIN-style suites run with both families; (b) counterfactual accuracy is near chance today and should rise, on WorldPrediction and occlusion splits; (c) a world model’s open-loop quality far exceeds its closed-loop task success, and that gap must be reported, on bridge suites; (d) the fraction of capabilities with a native VLA-vs-world-model contrast is far below target across the landscape. Four schematic panels showing target measurement shapes, not real data. (a) Two curves over capability, a world-model policy above a dashed direct-VLA baseline, with the gap shaded as the advantage of prediction. (b) A short today bar near a chance line versus a tall target bar for counterfactual accuracy. (c) A tall open-loop quality bar and a short closed-loop success bar with a red gap arrow. (d) Four short per-capability contrast-coverage bars well below a dashed target line.
Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

BenchmarkYearCapabilityEvalVLA vs WM
Policy / manipulation suites (37)
Manipulation
ARNOLD ( Gong et al., 2023 )2023language-grounded continuous-state 3D manipulationclosed–
BiGym ( Chernyadev et al., 2024 )2024mobile bimanual manipulationclosed–
FMB ( Luo et al., 2024 )2024real-world functional multi-step assembly manipulationclosed–
Franka Kitchen ( Gupta et al., 2019 )2019multi-task kitchen manipulationclosed–
Appendix
Table 10. The robot-evaluation benchmark landscape: 160 web-verified benchmarks, compared on four axes. Grouped into four evaluation lanes and sub-grouped by capability. Columns: Year ; Capability (what the benchmark stresses); Eval (evaluation mode: open-loop prediction/generation quality, closed-loop task success, or a prediction-to-action bridge); and VLA vs WM (does the benchmark build a native direct-policy vs world-model contrast: ✔ explicit, ~ partial, – model-agnostic). Every entry is a real, individually citeable benchmark.
BenchmarkarXivPanel source (subject scene)
Closed-loop task-success suites
ALFRED1912.01734sim task scene (rep. figure, pp.1–3)
BEHAVIOR-1K2403.09227household sim scene (rep. figure, pp.1–3)
CALVIN2112.03227tabletop manipulation scene (rep. figure, pp.1–3)
GemBench2410.01345manipulation scene (rep. figure, pp.1–3)
ManiSkill22302.04659multi-task sim scenes (rep. figure, pp.1–3)
Appendix
Table 11. Per-panel provenance for Figure 8 . Every panel is a real subject/scene figure cropped from the benchmark’s own arXiv paper and hyperlinked to its source. Exact figure numbers are given where verified from the paper (e.g. Physion Fig. 1); otherwise the teaser-region basis is noted (“rep. figure, pp.1–3” for tiles reused from the collage pool; “embedded raster, pp.1–8” for the nine gap tiles hand-picked from per-paper candidate contact sheets after a subject/results/neither classification pass). No figure number is invented. Images © their respective authors, reproduced for scholarly review.
BenchmarkarXivPanel source (results plot)
Leaderboard & model-comparison charts
World-in-World2510.18135task-success / gen-quality leaderboard (embedded raster)
EvalCrafter2310.11440overall model-ranking bar chart (embedded raster)
VideoScore2406.15252annotation score-ratio grouped bars (embedded raster)
WorldModelBench2502.20694VBench-vs-Ours win-rate comparison (embedded raster)
Per-dimension radar & multi-metric
Appendix
Table 12. Per-panel provenance for Figure 9 . Every panel is a real results/leaderboard/metric plot cropped from the benchmark’s own arXiv paper and hyperlinked to its source. Exact figure numbers are given where verified from the paper (e.g. WorldArena Fig. 1); otherwise the extraction basis is noted (“embedded raster”). Plots are contain-fitted, not cropped-to-fill, so no bars, axes or radar spokes are cut off. No figure number is invented. A benchmark appears more than once when it reports several evidence types. Images © their respective authors, reproduced for scholarly review.
BenchmarkarXivSourceBenchmarkarXivSource
Closed-loop (task-success suites)Open-loop (world-model & video eval)
LIBERO2306.03310Fig. 1 (p.2)WorldModelBench2502.20694rep. figure (pp.1–3)
CALVIN2112.03227rep. figure (pp.1–3)Physics-IQ2501.09038rep. figure (pp.1–3)
Meta-World1910.10897rep. figure (pp.1–3)WorldScore2504.00983rep. figure (pp.1–3)
ManiSkill22302.04659rep. figure (pp.1–3)EWMBench2505.09694rep. figure (pp.1–3)
RoboCasa2406.02523rep. figure (pp.1–3)EVA-Bench2410.15461rep. figure (pp.1–3)
Appendix
Table 13. Per-tile provenance for Figure 1 . Every tile is a representative figure cropped from the benchmark’s own arXiv paper (page/figure noted; “rep. figure” marks the paper’s teaser-region figure on pp.1–3), reproduced for scholarly review and hyperlinked to its source. Images © their respective authors.
More in this topic

World Model Evaluation

World Model Evaluation measures whether learned simulators predict consistent, physically plausible, action-responsive futures that help agents plan and act.

See all 212 papers

Explore similar work

CardsList
Apr 30, 2026

World Model for Robot Learning: A Comprehensive Survey

World models, which are predictive representations of how environments evolve under actions, have become a central component of robot learning. They support policy learning, planning, simulation, evaluation, data generation, and have advanced rapidly with the rise of foundation models and large-scale video generation. However, the literature remains fragmented across architectures, functional roles, and embodied application domains. To address this gap, we present a comprehensive review of world models from a robot-learning perspective. We examine how world models are coupled with robot policies, how they serve as learned simulators for reinforcement learning and evaluation, and how robotic video world models have progressed from imagination-based generation to controllable, structured, and foundation-scale formulations. We further connect these ideas to navigation and autonomous driving, and summarize representative datasets, benchmarks, and evaluation protocols. Overall, this survey systematically reviews the rapidly growing literature on world models for robot learning, clarifies key paradigms and applications, and highlights major challenges and future directions for predictive modeling in embodied agents. To facilitate continued access to newly emerging works, benchmarks, and resources, we will maintain and regularly update the accompanying GitHub repository alongside this survey.

Bohan Hou, Gen Li, Jindou Jia +15

Nanyang Technological University · University of California, Berkeley · Stanford University +6

World Model LearningWorld Models for RoboticsEmbodied World ModelsRobot LearningVideo World Models
AskPDF
Jul 2, 2026

GigaWorld-1: A Roadmap to Build World Models for Robot Policy Evaluation

Evaluating embodied robot foundation models remains a critical bottleneck; unlike large language models efficiently assessed via digital benchmarks, robotic policies require slow, costly real-world rollouts limited by hardware and human supervision, which has driven interest in world models as surrogate policy evaluators, yet the key properties that make a world model reliable for policy assessment remain poorly understood. This work presents a systematic study of world models for robotic policy evaluation and introduces WMBench, a benchmark constructed from real-robot teleoperation data and matched policy rollouts covering diverse manipulation tasks to enable controlled comparisons across model families, action encodings, rollout horizons, and evaluation metrics. Using WMBench, we analyze 7 video world models, 4 action representation schemes, and over 324,000 simulated policy rollouts paired with real robot executions, further enriching our analysis with large-scale community submissions from the CVPR 2026 GigaBrain Challenge, curated synthetic trajectories, and a training videos spanning more than 12,000 hours. Our experiments deliver three core insights: evaluator quality is dominated by long-horizon, action-faithful rollout consistency rather than short-term visual realism; pretraining gains stem not only from data scale but from balancing general world knowledge with robot-specific controllability; and architectural choices including action encoding, memory design, and evaluator-focused post-training strongly determine alignment with real-world robot behavior. Drawing on these results, we derive a practical design roadmap and realize it in GigaWorld-1, a world model specially optimized for policy evaluation, and we fully release our code, models, datasets, and toolkits to advance scalable evaluation research for embodied foundation models.

GigaWorld Team, Angyuan Ma, Boyuan Wang +24

GigaAI · Tsinghua University

Robot Foundation ModelsAction-Conditioned World ModelsRobotic Policy EvaluationRobot Manipulation BenchmarksWorld Model EvaluationVideo World Models
AskPDF
Jul 1, 2026

From World Models to World Action Models: A Concise Tutorial for Robotics

World models are increasingly used in embodied intelligence and generative simulation, yet their scope remains ambiguous across communities. This tutorial presents a design-space view of world models as action-conditioned predictive models that estimate the future evolution of task-relevant observations or states. We categorize existing methods into observation-space and state-space world models, comparing their trade-offs in visual fidelity, spatial structure, physical interpretability, and control usability. We further introduce world action models, which connect predicted futures with executable robot actions, and summarize four representative paradigms: imagine-then-execute, video-feature-conditioned action prediction, joint video-action modeling, and auxiliary video prediction for policy learning. The goal of this tutorial is to clarify the conceptual scope of world (action) models and provide a structured taxonomy for embodied prediction and control.

Xiaoxiong Zhang, Xiong Zeng, Wei Zhang

School of Automation and Intelligent Manufacturing, Southern University of Science and Technology, Shenzhen, China · LimX Dynamics, Shenzhen, China

World Model LearningWorld Models for RoboticsAction-Conditioned World ModelsWorld Action ModelsEmbodied World Models
AskPDF