cs.ROAug 30, 2026
SaveDo World Models Make Better Robots? A Survey of Evaluation Benchmarks for Predictive Embodied Intelligence
Organizations: UC Berkeley, USA · San Jose State University, USA · Meta, USA · Apple, USA ·
Abstract
Robot learning now advances along two tracks that rarely meet. On one side, direct Vision-Language-Action (VLA) policies map observations to actions and are scored by closed-loop task success. On the other, predictive and generative world models forecast future observations and are scored by open-loop prediction or generation quality. A natural question sits between them: does world modelling earn a measurable, closed-loop advantage over a direct policy, and for which robotic capabilities? We argue that the field cannot yet answer this question, and that the reason is a gap in how it is measured, not in the models themselves. World-model benchmarks score prediction without ever executing it, while task-success suites host a single policy and never build a world-model versus VLA contrast. This survey maps the evaluation landscape around that gap. We catalogue 160 web-verified benchmarks spanning 2017 to 2026 and organise them by evaluation mode, robotic capability, and model family into four lanes: policy suites, embodied agents, world model evaluation, and prediction-to-action bridges. Across the corpus, 138 of 160 benchmarks are model-agnostic and only 11 (7%) build an explicit VLA-versus-world-model contrast; counterfactual capability is almost entirely unmeasured, and only four benchmarks turn prediction into executed action. We contribute an operational taxonomy, a coverage comparison against the eight closest surveys (ours is the only one to cross capability with model family), an evaluation loop that isolates the advantage of prediction, and an actionable protocol of four advantage-aware metrics anchored on named testbeds. The organising claim is not that world models help or do not help, but that answering the question requires benchmarks built to ask it.
Figures & tables
Figure 1. Predict or Act? The survey’s organising question, posed with real benchmarks. Closed-loop (left): policy and embodied suites score task success by running a policy in the environment. Open-loop (right): world-model and video-generation benchmarks score prediction or generation quality with no action. The centre axis is the question neither pole answers, whether prediction earns a closed-loop advantage, built explicitly by only 11 of 160 benchmarks (Table 10 ). Every tile is a representative figure from the benchmark’s paper and links to it; per-tile provenance is in Table 13 . Images © their respective authors, reproduced for scholarly review. A two-pole collage of benchmark figures. A black title bar reads ``Predict or Act?''. The left band, CLOSED-LOOP, shows nine policy and embodied benchmark figures; the right band, OPEN-LOOP, shows nine world-model and video-generation benchmark figures; a centre spine contrasts task success (closed loop) with prediction quality (open loop) and notes that only 11 of 160 build the contrast.
| Survey | Venue | Yr | mode | cap. | family | metric | catalog |
|---|---|---|---|---|---|---|---|
| Evaluation of Embodied AI ( Hou et al., 2026a ) | Authorea | 2026 | ~ | ~ | ✘ | ✘ | ✔ |
| VLA Datasets & Bench. ( Wang et al., 2026 ) | arXiv | 2026 | ✘ | ✘ | ✘ | ✘ | ✔ |
| Benchmark Construction ( Lai et al., 2026 ) | arXiv | 2026 | ✘ | ~ | ✘ | ✘ | ✔ |
| RL Reproducibility ( Lynnerup et al., 2020 ) | CoRL | 2020 | ✘ | ✘ | ✘ | ✘ | ✘ |
| World Models for Robots ( Hou et al., 2026b ) | arXiv | 2026 | ~ | ✘ | ~ | ~ | ~ |
| Embodied World Models ( Shang et al., 2026b ) | preprint | 2026 | ~ | ✘ | ~ | ✘ | ~ |
Table 1. Topical coverage of the closest benchmark/evaluation surveys versus ours. ✔ covered as an organising axis, ~ partial or mentioned in passing, ✘ absent. Axes: evaluation mode (open- vs closed-loop as the primary axis); capability cuts (hidden-state, counterfactual, long-horizon, social); capability model-family cross-tab (direct VLA vs latent / action-conditioned / generative world models under one protocol); behavioural advantage-of-prediction metric ( assessed, not adopted ; e.g. ACTION-ATLAS’s World Advantage Score); comprehensive benchmark catalogue. This survey is the only work that covers , and is otherwise reached only partially , and only by a position paper ( Yu et al., 2026 ) , never by a survey. Venues: CoRL, IEEE TETCI; the remaining rows are arXiv or unrefereed preprints (Authorea, Preprints.org).
Figure 2. The closest benchmark/evaluation surveys as a timeline. Two predate the recent wave, an IEEE TETCI survey of embodied-AI simulators and tasks ( Duan et al., 2022 ) and a CoRL survey of real-robot RL reproducibility ( Lynnerup et al., 2020 ) ( amber = refereed venue); the rest are 2026 preprints on embodied evaluation, VLA data, and world models. None organises the robot-evaluation landscape by evaluation-mode capability model-family; the sole work reaching the advantage-of-prediction axis is a position paper ( Yu et al., 2026 ) . Ours ( ) fills that cell; per-axis coverage is in Table 1 . A horizontal timeline of eight related surveys. Two amber refereed markers sit at 2019 (RL reproducibility, CoRL) and 2021 (embodied-AI simulators, IEEE TETCI); after a time break, a dense cluster of 2026 preprint markers covers evaluation of embodied AI, VLA datasets, world models for robots, embodied world models, benchmark construction, and a world-model-evaluation position paper. A red star marks the present survey at the end.
Figure 3. Corpus construction in the PRISMA 2020 style ( Page et al., 2021 ) . The left column is the web-search stream behind the landscape corpus (Table 10 ): five thematic sweeps, de-duplicated by name and arXiv ID, screened to drop datasets and models, and metadata-verified by parallel audit agents. The right column is the venue-targeted and snowball search behind the closest surveys (Table 1 ). Both streams feed the included set. Candidates whose metadata could not be verified were discarded at harvest; their count was not logged, which we state rather than estimate. A PRISMA 2020 style flow diagram with two identification columns. The left column (structured web search) shows candidate benchmark records from five thematic sweeps, de-duplicated by name and arXiv ID, screened to remove seven datasets and models, and 160 metadata-verified records forming the landscape corpus (155 with an arXiv ID, 5 without). The right column (survey search) yields 8 closest surveys including one position paper. Both columns feed an included box of 160 benchmarks plus 8 surveys and one methodology citation, totalling 164 references.
| Construct | Counts (a benchmark has it if and only if…) | Does not count |
|---|---|---|
| Open-loop eval ( open ) | it scores prediction or generation quality on fixed inputs without executing the prediction in a feedback loop: video fidelity (FVD), physics or rollout consistency, or frame-level accuracy | any protocol where an action is executed and the environment responds; offline scoring of a policy’s success |
| Closed-loop eval ( closed ) | success is measured by running a policy in the environment with state feedback and scoring task outcome: success rate, sub-goal completion, or reward under interaction | scoring a generated video or a static prediction; a metric computed without stepping the environment |
| Bridge ( bridge ) | a prediction (imagined rollout, video, or latent plan) is converted into executed actions and scored on the resulting task success | a benchmark that stops at prediction quality; one that never turns a prediction into a control signal |
| Native VLA-vs-WM contrast ( ) | the benchmark itself runs both a direct Vision-Language-Action policy and a world-model policy under one protocol and reports the head-to-head | a model-agnostic suite that merely hosts either family; a suite evaluated with only one family in the original paper |
| Advantage-of-prediction ( ) | a metric quantifies the closed-loop gain of a predictive policy over a matched direct-VLA baseline on the same tasks (assessed, not adopted) | absolute success with no matched baseline; open-loop quality scores; a ranking that never isolates the predictive component |
Table 2. Operational definitions used to place each benchmark on the evaluation axes (§ 2 ). They resolve the ambiguous boundary cases: hosting a policy is not the same as building a VLA-vs-world-model contrast ( ), and an absolute success number is not an advantage-of-prediction measurement ( ) without a matched baseline. open open-loop, closed closed-loop, bridge bridge.
Figure 4. The catalogued corpus at a glance. The survey maps 160 web-verified benchmarks across 4 evaluation lanes, spanning 2017–2026 and totalling 164 references; only 11 build an explicit Vision-Language-Action vs world-model contrast. The bars give benchmarks per capability area, with kindred clusters merged (dexterity into manipulation, vision-language navigation into navigation, theory-of-mind into hidden-state); the twelve areas sum to the 160 works of Table 10 . Bars are coloured by the dominant lane of each area (teal policy, blue embodied, purple world-model). An overview figure. Six stat callouts read: 160 benchmarks, 164 references, 4 evaluation lanes, 12 capability areas, 11 explicit contrasts, span 2017 to 2026. Below, a horizontal bar chart gives benchmarks per capability area: long-horizon and planning 26, manipulation and dexterity 22, generation quality 19, navigation 16, physical reasoning 13, social and multi-agent 12, hidden-state and theory-of-mind 9, counterfactual 9, autonomous driving 9, generalization 7, locomotion 6, other areas 12.
Figure 5. Profile of the 160 catalogue benchmarks (Table 10 ), re-counted directly from the catalogue. (a) publication year, showing the 2023–2024 surge; (b) evaluation lane (embodied navigation dominates, then policy suites); (c) evaluation mode (closed-loop task success dominates, open-loop prediction is second, only four bridges reach action); (d) whether the benchmark builds a native Vision-Language-Action vs world-model contrast, the survey’s organising question: 138 of 160 are model-agnostic , and only 11 (7%) build the contrast explicitly. Counts are the exact column tallies of Table 10 . Four bar charts of the 160 benchmarks. (a) Benchmarks per year rise from 2 in 2017 to 31 in 2023 and 36 in 2024, then 20 in 2025. (b) Evaluation lane: policy 37, embodied 85, world-model evaluation 34, bridge 4. (c) Evaluation mode: closed 113, open 43, bridge 4. (d) VLA-vs-world-model contrast: model-agnostic 138, partial 11, explicit 11, with a note that only 11 (7 percent) build a contrast.
Figure 6. The world-model-evaluation wave is recent. Per-year counts of closed-loop task-success benchmarks (green) and open-loop world-model / video-generation benchmarks (magenta) in the catalogue. Closed-loop suites dominate through 2024; in 2025 open-loop evaluation overtakes them (13 vs 6) as the world-model-generation literature arrives. The four prediction-to-action bridges are newer still, all appearing in 2024 or later. Years are the benchmarks’ first-release years (2026 is partial and omitted); counts are the tallies of Table 10 . A grouped bar chart of benchmarks by evaluation mode per year, 2019 to 2025. Closed-loop (green): 10, 16, 19, 8, 24, 24, 6. Open-loop (magenta): 3, 1, 4, 3, 7, 11, 13. Closed-loop dominates until 2024, then open-loop overtakes it in 2025.
| Benchmark | Year | Venue | Eval | Capability | VLA vs WM |
|---|---|---|---|---|---|
| §3.1 Policy / manipulation suites closed | |||||
| CALVIN ( Mees et al., 2021 ) | 2021 | RA-L | closed | long-horizon language | – |
| LIBERO ( Liu et al., 2023c ) | 2023 | NeurIPS D&B | closed | short-horizon + transfer | – |
| Meta-World ( Yu et al., 2019 ) | 2020 | CoRL | closed | multi-task / meta-RL | – |
| ManiSkill2 ( Gu et al., 2023 ) | 2023 | ICLR | closed | generalizable skills | – |
| SIMPLER ( Li et al., 2024a ) | 2024 | CoRL | closed | generalization (real vs sim) | – |
Table 3. Detailed comparison of a representative 31 of the 160 benchmarks in the landscape (Table 10 ), grouped by evaluation lane. Eval : open open-loop prediction/generation quality, closed closed-loop task success, bridge prediction-to-action bridge. VLA vs WM : ✔ native direct-policy vs world-model contrast, ~ partial, – model-agnostic. Venues are web-verified; “arXiv” marks preprint-only works.
| Dimension | LIBERO ( Liu et al., 2023c ) | Habitat 3.0 ( Puig et al., 2023 ) | WorldModelBench ( Li et al., 2025 ) | World-in-World ( Zhang et al., 2025 ) |
|---|---|---|---|---|
| policy suite | embodied nav | world-model eval | bridge | |
| What it scores | task success | social-nav success | video-pred. quality | closed-loop WM success |
| Evaluation mode | closed | closed | open | bridge |
| Executes actions in a loop? | ✔ | ✔ | ✘ | ✔ |
| Native VLA-vs-WM contrast? | ✘ | ✘ | ✘ | ~ |
| Capability stressed | short-horizon + transfer | social / multi-agent | physical realism | plan executability |
Table 4. Capability matrix, one representative benchmark per evaluation lane. No lane’s exemplar builds a native VLA-vs-world-model contrast: policy and embodied suites are model-agnostic (they host any policy), world-model suites score generation quality with no action, and the bridge exemplar reaches closed-loop success but reports only a partial contrast (“visual quality is not task success”). Counterfactual probing is absent throughout. ✔ yes, ~ partial, ✘ no, – n/a; open open-loop, closed closed-loop, bridge bridge.
Figure 8. Benchmarks across the surveyed corpus. Representative subject figures from the catalogue, grouped by evaluation lane rather than embodiment: closed-loop task-success suites, open-loop world-model and video-generation evaluation, and the prediction-to-action bridges . Each panel is a live hyperlink to the source paper on arXiv and is annotated with its arXiv id (per-panel figure provenance in Table 11 ); the figure is vector with photographs embedded at full resolution, so it can be zoomed without pixelation. Only figures that depict the benchmark’s task or scene are shown; results and leaderboard figures appear in Figure 9 . Benchmarks with no locatable subject figure are omitted, not fabricated. Images © their respective authors, reproduced for scholarly review. A gallery of benchmark subject figures in three lane bands. The top band, closed-loop task-success suites, shows ten policy and embodied benchmark scenes (robot arms and simulated households). The middle band, open-loop world-model and video-generation evaluation, shows ten generated-video or physical-scene samples. The bottom band, prediction-to-action bridges, shows four world-model benchmark scenes. Each panel is labelled with the benchmark name and its arXiv id.
| Benchmark | Year | Eval | What it probes | VLA vs WM |
|---|---|---|---|---|
| Hidden-state / memory / occlusion | 3/4 | |||
| LIBERO-Mem ( Chung et al., 2025 ) | 2025 | closed | occluded object state carried across time | ✔ |
| TEACh ( Padmakumar et al., 2021 ) | 2021 | closed | partner belief and history via dialog | – |
| ComPhy ( Chen et al., 2022b ) | 2022 | open | hidden mass and charge inferred from collisions | ✔ |
| Bongard-HOI ( Jiang et al., 2022b ) | 2022 | open | few-shot hidden-concept induction | ~ |
| Theory-of-mind | 3/4 | |||
Table 5. Capability deep-dive : the benchmarks that probe each of the capability cuts, and whether each builds a native VLA-vs-world-model contrast. Native contrasts cluster in the hidden-state, theory-of-mind and counterfactual reasoning cuts (LIBERO-Mem, ComPhy, AGENT, BIB, ACRE, CoPhy, CausalWorld); the long-horizon and social task-success cuts are largely model-agnostic (ALFWorld’s abstract-to-grounded transfer is the lone partial). Eval : open / closed . VLA vs WM : ✔ native contrast, ~ partial, – model-agnostic; each cut’s header tallies the fraction of its benchmarks that build any (native or partial) contrast.
| Benchmark | Year | Mode | What it scores | Signal | Physical |
|---|---|---|---|---|---|
| Generation quality (multi-dimensional) | |||||
| EvalCrafter ( Liu et al., 2023a ) | 2023 | Gen | 17-metric text-to-video quality suite | Auto | ✘ |
| FETV ( Liu et al., 2023b ) | 2023 | Gen | fine-grained text-to-video quality | Auto | ✘ |
| VBench ( Huang et al., 2023 ) | 2023 | Gen | 16 disentangled generation dimensions | Auto | ✘ |
| StoryBench ( Bugliarello et al., 2023 ) | 2023 | Gen | continuous story-visualisation quality | Auto | ✘ |
| DEVIL ( Liao et al., 2024 ) | 2024 | Gen | content-dynamics quality of T2V | Auto | ✘ |
Table 6. World-model-evaluation deep-dive : the 34 benchmarks of the world-model-evaluation lane, by sub-type – the P3 counterpart of the skill-building branch deep-dive. Mode : Gen evaluates generated video, QA video question-answering, Pred physical prediction. Signal : Auto computed/learned metric, Acc accuracy vs ground truth, VOE violation-of-expectation, Human human study. Physical : targets physical-law understanding, ✔ core, ~ adjacent, ✘ general. Generation-quality suites dominate and are physics-agnostic; only the physical-reasoning and physical-commonsense sub-types test physical law, and every lane benchmark scores open-loop, never in a closed loop.
Figure 9. Results reported across the surveyed corpus. Representative results figures from the catalogue, grouped by type of evidence: leaderboard and model-comparison charts , per-dimension radar and multi-metric plots, and distributions and composition . Each panel is a live hyperlink to the source paper on arXiv and is annotated with its arXiv id (per-panel provenance in Table 12 ); the figure is vector with plots embedded at full resolution. Only results/leaderboard/metric figures are shown; task and scene figures appear in Figure 8 . As a benchmark survey the reported evidence is dominated by rankings, radars and distributions rather than training curves, and is sparser than the subject figures, so this gallery is smaller; benchmarks that publish no results figure are omitted, not fabricated. A benchmark may appear more than once when it reports several evidence types. Images © their respective authors, reproduced for scholarly review. A gallery of benchmark results figures in three evidence-type bands. The top band, leaderboard and model-comparison charts, shows four ranking bar charts and win-rate tables. The middle band, per-dimension radar and multi-metric, shows three radar plots over evaluation dimensions. The bottom band, distributions and composition, shows a prompt-length histogram, a meta-type pie chart, and a prompt word cloud. Each panel is labelled with the benchmark name and its arXiv id.
Figure 10. The evaluation loop that isolates the advantage of prediction. A benchmark that can answer “does world modelling help a robot act” must run a direct Vision-Language-Action policy and a world-model policy (predict then plan) under one closed loop, then report the closed-loop gain ( ) sliced by capability ( ). Today’s suites break this loop in one of two ways: model-agnostic policy suites host a single policy and never build the contrast, while world-model suites score the prediction open-loop and never execute it (§ 4 ). A left-to-right diagram. A task/scene forks into two branches, a direct VLA policy and a world model that predicts then plans; both feed an environment closed loop with state feedback, producing task success, which feeds an advantage-of-prediction measurement sliced per capability. An annotation notes that model-agnostic suites run only one branch and world-model suites never close the loop, so the contrast is missing.
| Sense of “world model” | What it is | Examples | Act. cond. | Loop | Metric | VLA-cmp. |
|---|---|---|---|---|---|---|
| As a representation (what the model is) | ||||||
| latent WM (LWM) | latent-dynamics predictor | DreamerV3, TD-MPC2 | ✔ | ✔ | ~ | ~ |
| action-cond. WM (WAM) | action-conditioned future-frame predictor | iVideoGPT, GR-2 | ✔ | ~ | ~ | ~ |
| generative WM (WFM) | text / image-to-video generator | Sora, Cosmos, Genie | ~ | ✘ | ✔ | ✘ |
| As a role (what the model is used for) | ||||||
| neural simulator | learned environment for policy training / eval | UniSim | ✔ | ✔ | ~ | ~ |
Table 7. The term “world model” spans two orthogonal axes the literature routinely conflates: a representation axis (what the model is: latent-state dynamics, action-conditioned video, or generative video, a soft boundary since action-conditioned generators straddle WAM and WFM) and a role axis (what it is used for: a neural simulator, or the object a benchmark scores). Columns record whether a sense is act ion- cond itioned, is used in a closed loop , carries a standard evaluation metric , and is readily VLA-comp arable under one protocol ( ✔ yes, ~ partial, ✘ no, – n/a). No single sense is action-conditioned, closed-loop, and VLA-comparable at once, which is why a benchmark that contrasts a world-model policy with a VLA policy per capability (§ 4 ) is still missing. Evaluation benchmarks score the world model as an object ; they are not world models acting as judges.
| Policy suites ( Liu et al., 2023c ) | Embodied nav / social ( Puig et al., 2023 ) | World-model eval ( Li et al., 2025 ) | Bridges ( Zhang et al., 2025 ) | |
|---|---|---|---|---|
| Loop closure | ✔ closed-loop; steps the environment ( Mees et al., 2021 ) | ✔ closed-loop; long-horizon rollouts ( Li et al., 2024b ) | ✘ open-loop; scores generation, no action ( Motamed et al., 2025 ) | ~ converts a prediction into an executed action ( Qin et al., 2024 ) |
| Task horizon | ~ short reactive skills; long by chaining ( Liu et al., 2023c ) | ✔ long-horizon household and dialog ( Padmakumar et al., 2021 ) | ~ short clips; temporal but not task-level ( Hu et al., 2025 ) | ~ a single executed rollout ( Jiang et al., 2026 ) |
| Capability breadth | ~ manipulation + transfer; hidden-state rare ( Pumacay et al., 2024 ) | ✔ navigation, social, some hidden-state ( Deitke et al., 2020 ) | ~ physical / temporal; counterfactual barely ( Chen et al., 2025 ) | ✘ narrow; executability only ( Shang et al., 2026a ) |
| Model-family contrast | ✘ model-agnostic host of any policy ( Atreya et al., 2025 ) | ~ memory-vs-Markovian in one suite ( Chung et al., 2025 ) | ✘ world-model-only scores ( Li et al., 2025 ) | ~ WM as policy substrate; no VLA pole ( Zhang et al., 2025 ) |
| Reproducibility | ~ sim; some real-vs-sim transfer ( Li et al., 2024a ) | ~ sim-heavy; a sim-to-real gap ( Deitke et al., 2020 ) | ✔ fixed offline sets; deterministic scoring ( Zheng et al., 2025 ) | ✘ new, still-unstandardised protocols ( Shang et al., 2026a ) |
| Structural blind spot | hosts a world-model policy but never builds the VLA contrast | capability cuts exist, but no per-capability family ablation | visual quality is not task success; no behavioural utility | reaches action, but omits the per-capability VLA head-to-head |
Table 8. Evaluation-lane comparison matrix : the four benchmark lanes on six evaluation axes (loop closure, task horizon, capability breadth, model-family contrast, reproducibility, and characteristic blind spot). Marks grade each lane on the axis ( ✔ favourable, ~ mixed, ✘ weak), with an anchoring benchmark per cell. No lane closes the loop, slices capability, and builds a native VLA-vs-world-model contrast at once; the bridge lane comes closest yet still omits the per-capability VLA head-to-head, which is the empty cell this survey names (§ 4 ).
| Metric | Definition | Instantiation (testbed) | Axis stressed |
|---|---|---|---|
| Advantage-of-prediction curve | closed-loop success gain of a predictive policy over a matched direct-VLA baseline, reported per capability as a curve, not a single endpoint | a capability-sliced protocol over LIBERO ( Liu et al., 2023c ) and CALVIN ( Mees et al., 2021 ) that runs a world-model policy and a VLA policy under one harness | family, metric |
| Counterfactual accuracy | task success on episodes that require inference over unobserved or hypothetical dynamics (occlusion, object permanence, “what if”) | WorldPrediction ( Chen et al., 2025 ) plus occlusion / memory splits in the style of LIBERO-Mem ( Chung et al., 2025 ) | capability |
| Prediction-to-action fidelity gap | the difference between a world model’s open-loop generation quality and its closed-loop task success (“visual quality is not task success”) | bridge protocols such as WorldSimBench ( Qin et al., 2024 ) and World-in-World ( Zhang et al., 2025 ) | mode |
| Per-capability contrast coverage | the fraction of capabilities for which a benchmark actually runs a VLA-vs-world-model head-to-head under a single protocol | audits of the landscape (Table 10 ) against a declared contrast schema | family, catalog |
Table 9. An actionable protocol for the future agenda : four measurable quantities, their definitions, and concrete testbeds. The first anchors the advantage-of-prediction question ( ) on LIBERO / CALVIN-style suites; the second targets the near-unmeasured counterfactual capability; the third quantifies the prediction-to-action gap the bridges expose; the fourth audits per-capability contrast coverage across the landscape. Each is reported as a distribution or curve, never a single headline number.
Figure 11. An actionable evaluation protocol for the future agenda (Table 9 ). The panels are schematic : they show the axes and the target shape of each measurement, not measured results. (a) a world-model policy should beat a matched direct-VLA baseline in closed-loop success, and the per-capability gap is the advantage of prediction ( ), on LIBERO / CALVIN-style suites run with both families; (b) counterfactual accuracy is near chance today and should rise, on WorldPrediction and occlusion splits; (c) a world model’s open-loop quality far exceeds its closed-loop task success, and that gap must be reported, on bridge suites; (d) the fraction of capabilities with a native VLA-vs-world-model contrast is far below target across the landscape. Four schematic panels showing target measurement shapes, not real data. (a) Two curves over capability, a world-model policy above a dashed direct-VLA baseline, with the gap shaded as the advantage of prediction. (b) A short today bar near a chance line versus a tall target bar for counterfactual accuracy. (c) A tall open-loop quality bar and a short closed-loop success bar with a red gap arrow. (d) Four short per-capability contrast-coverage bars well below a dashed target line.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Benchmark | Year | Capability | Eval | VLA vs WM |
|---|---|---|---|---|
| Policy / manipulation suites (37) | ||||
| Manipulation | ||||
| ARNOLD ( Gong et al., 2023 ) | 2023 | language-grounded continuous-state 3D manipulation | closed | – |
| BiGym ( Chernyadev et al., 2024 ) | 2024 | mobile bimanual manipulation | closed | – |
| FMB ( Luo et al., 2024 ) | 2024 | real-world functional multi-step assembly manipulation | closed | – |
| Franka Kitchen ( Gupta et al., 2019 ) | 2019 | multi-task kitchen manipulation | closed | – |
Table 10. The robot-evaluation benchmark landscape: 160 web-verified benchmarks, compared on four axes. Grouped into four evaluation lanes and sub-grouped by capability. Columns: Year ; Capability (what the benchmark stresses); Eval (evaluation mode: open-loop prediction/generation quality, closed-loop task success, or a prediction-to-action bridge); and VLA vs WM (does the benchmark build a native direct-policy vs world-model contrast: ✔ explicit, ~ partial, – model-agnostic). Every entry is a real, individually citeable benchmark.
| Benchmark | arXiv | Panel source (subject scene) |
| Closed-loop task-success suites | ||
| ALFRED | 1912.01734 | sim task scene (rep. figure, pp.1–3) |
| BEHAVIOR-1K | 2403.09227 | household sim scene (rep. figure, pp.1–3) |
| CALVIN | 2112.03227 | tabletop manipulation scene (rep. figure, pp.1–3) |
| GemBench | 2410.01345 | manipulation scene (rep. figure, pp.1–3) |
| ManiSkill2 | 2302.04659 | multi-task sim scenes (rep. figure, pp.1–3) |
Table 11. Per-panel provenance for Figure 8 . Every panel is a real subject/scene figure cropped from the benchmark’s own arXiv paper and hyperlinked to its source. Exact figure numbers are given where verified from the paper (e.g. Physion Fig. 1); otherwise the teaser-region basis is noted (“rep. figure, pp.1–3” for tiles reused from the collage pool; “embedded raster, pp.1–8” for the nine gap tiles hand-picked from per-paper candidate contact sheets after a subject/results/neither classification pass). No figure number is invented. Images © their respective authors, reproduced for scholarly review.
| Benchmark | arXiv | Panel source (results plot) |
| Leaderboard & model-comparison charts | ||
| World-in-World | 2510.18135 | task-success / gen-quality leaderboard (embedded raster) |
| EvalCrafter | 2310.11440 | overall model-ranking bar chart (embedded raster) |
| VideoScore | 2406.15252 | annotation score-ratio grouped bars (embedded raster) |
| WorldModelBench | 2502.20694 | VBench-vs-Ours win-rate comparison (embedded raster) |
| Per-dimension radar & multi-metric | ||
Table 12. Per-panel provenance for Figure 9 . Every panel is a real results/leaderboard/metric plot cropped from the benchmark’s own arXiv paper and hyperlinked to its source. Exact figure numbers are given where verified from the paper (e.g. WorldArena Fig. 1); otherwise the extraction basis is noted (“embedded raster”). Plots are contain-fitted, not cropped-to-fill, so no bars, axes or radar spokes are cut off. No figure number is invented. A benchmark appears more than once when it reports several evidence types. Images © their respective authors, reproduced for scholarly review.
| Benchmark | arXiv | Source | Benchmark | arXiv | Source |
|---|---|---|---|---|---|
| Closed-loop (task-success suites) | Open-loop (world-model & video eval) | ||||
| LIBERO | 2306.03310 | Fig. 1 (p.2) | WorldModelBench | 2502.20694 | rep. figure (pp.1–3) |
| CALVIN | 2112.03227 | rep. figure (pp.1–3) | Physics-IQ | 2501.09038 | rep. figure (pp.1–3) |
| Meta-World | 1910.10897 | rep. figure (pp.1–3) | WorldScore | 2504.00983 | rep. figure (pp.1–3) |
| ManiSkill2 | 2302.04659 | rep. figure (pp.1–3) | EWMBench | 2505.09694 | rep. figure (pp.1–3) |
| RoboCasa | 2406.02523 | rep. figure (pp.1–3) | EVA-Bench | 2410.15461 | rep. figure (pp.1–3) |
Table 13. Per-tile provenance for Figure 1 . Every tile is a representative figure cropped from the benchmark’s own arXiv paper (page/figure noted; “rep. figure” marks the paper’s teaser-region figure on pp.1–3), reproduced for scholarly review and hyperlinked to its source. Images © their respective authors.
More in this topic
World Model Evaluation
World Model Evaluation measures whether learned simulators predict consistent, physically plausible, action-responsive futures that help agents plan and act.
See all 212 papers