cs.CVOct 24, 2026

WMEval: Evaluating Interactive World Models Grounded on Physical Simulations

Authors: Lanxiang Hu, Yixin Huang, Abhilash Shankarampeta, Zilin Dai, Yuyuan Wu, Kun Qian, Junze Liu, Arvind Srinivasan, 

Organizations: UC San Diego · Unity

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Interactive world models hold immense potential for high-stakes applications in robotics, game engines, and immersive digital twins, yet existing benchmarks primarily focus on perceptual similarity rather than the physical-causal coherence essential for real-world utility. We introduce WMEval, a comprehensive benchmark that evaluates interactive world models against Unity-based physical simulations. By leveraging Unity as a high-fidelity ground-truth oracle, we shift evaluation beyond perceptual visual assessment toward the objective state-based verification required for downstream applications. Spanning 9 diverse scenes, WMEval assesses action controllability, scene and character consistency, prompt adherence, kinematics, fluid mechanics, physical causality, and lighting. By executing matched action sequences in both the simulator and the world model, WMEval enables evaluation at both the perceptual level and the physical-causal level, using simulator-derived states as reference. Experiments on state-of-the-art models reveal significant room for improvement in dimensions that remain hidden by standard perceptual metrics but are vital for real-world deployments. Our code is available at: https://github.com/hao-ai-lab/WMEval.

Figures & tables

Figure 1: Overview of WMEval. A shared action sequence {ai}i=1T\{a_{i}\}_{i=1}^{T} , sampled from predefined or rule-based synthetic trajectories over actor and camera controls, is dispatched to two parallel flows: a Unity physical simulator yielding reference observations o^i\hat{o}_{i} and simulator-derived states s^i\hat{s}_{i} , and a world model under evaluation producing generated observations oio_{i} , from which state-like quantities sis_{i} are recovered via CV-based estimators.
Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 2: Visual Gallery of the WMEval Benchmark Environments. The nine Unity scenes are architected to evaluate world models across six core dimensions defined in Section 3 . (a–c) Causal & Kinematic Interactions: BallRoller, BeamDomino, and BilliardGame assess action controllability, multi-body collision dynamics, and multi-stage causal chains. (d, e, i) Spatial & Identity Consistency: ChessBoard, MirrorRollingBall, and WanderingGarden test scene geometry preservation, object permanence, and character identity. (f–h) Specialized Continuum & Environmental Physics: LiquidContainer (fluid mechanics), RacingCar (non-linear vehicle kinematics), and SunRisesAndSets (temporal lighting/shadow kinematics) isolate complex physical phenomena.
SceneAct.Obs.CameraPrimary Evaluation Dimensions
BallRoller2C48Fixed-offset followControllability, cLPIPS, Consistency
BeamDomino2C92Fixed-offset followPhysical Causality, Controllability, cLPIPS
BilliardGame10C + 2D13Fixed overheadKinematics & Mechanics, cLPIPS
ChessBoard2C + 2D8Third-person followScene & Character Consistency
MirrorRollingBall2C297Fixed-offset followConsistency, Kinematics
LiquidContainer6C24Static sideFluid Mechanics
Appendix
Table 10: Summary of the nine WMEval Unity scenes. Act. = action-space dimensionality (C = continuous, D = discrete branches); Obs. = observation-vector length; Camera = camera regime used during evaluation.
Figure 3: BallRoller Scene Views.
Figure 4: BeamDomino Scene Views.
Figure 5: BilliardGame Scene Views.
Figure 6: ChessBoard Scene Views.
Figure 7: MirrorRollingBall Scene Views.
Figure 8: LiquidContainer Scene Views.
Figure 9: RacingCar Scene Views.
Figure 10: SunRisesAndSets Scene Views. (a–c) The environment across a full diurnal cycle, illustrating dynamic shadow kinematics and global illumination changes. (d) A bird’s-eye environmental layout showing the static objects used to evaluate shadow projection accuracy.
Figure 11: WanderingGarden Scene Views.
ModelVLM judgeHuman
Unity Reference0.9170.9170.9040.904
Project Genie0.8900.8900.7800.780
LingBot-World (Cam)0.8900.8900.8430.843
Matrix-Game 3.00.7320.7320.5050.505
HY-WorldPlay0.7780.7780.5220.522
Spearman ρ\rho ( ↑\uparrow )0.900.90
Appendix
Table 11: VLM judge vs. blinded human ratings on prompt-scene consistency. Scores are averaged over the six view ×\times scene cells; Human is the mean of three raters. ρ\rho and τ\tau are the Spearman and Kendall rank correlations between the model rankings induced by the judge and by the human ratings, computed from unrounded scores.
Table 12: Evaluator calibration and validity. Each learned evaluator is checked against simulator-derived quantities or human annotations.
Table 13: Finite-sample robustness. We test whether conclusions from five generations remain stable under three-run subsampling and statistical testing.
Figure 12: impulse_decay (brief vertical jump pulse, then brief lateral strafe pulse, idle otherwise; isolates an underdamped slosh response). Columns: pre-jump, post-jump ring-down, post-strafe lateral slosh, and late decay.
Figure 13: LiquidContainer : chirp_strafe (square-wave strafe sweep 0.3→3.00.3{\to}3.0 Hz; the only profile that excites the bottle’s natural slosh modes). Columns sample the low / mid-low / mid-high / high frequency regimes of the sweep. Unity shows the cola surface visibly raised against the bottle wall by the high-frequency excitation.
CategoryMetricRacing Car (3P)Billiard Game (1P)
KinematicsDTW ↓\downarrow0.940.940.010.01
FDE ↓\downarrow1.731.730.010.01
Mechanical LawsRobust Error (%) ↓\downarrow5.55.54.94.9
Appendix
Table 16: Unity reference for Table 4 : the simulator video scored against itself, giving the error floor of the trajectory and mechanics measurement.
Figure 14: BeamDomino : Stage-aligned keyframes on the canonical action trajectory.
Figure 15: BeamDomino : Stage-aligned keyframes on the direct_strike action trajectory.
Figure 16: BeamDomino : Stage-aligned keyframes on the glancing action trajectory.
Figure 17: Cross-model snapshots on SunRisesAndSets at four orbit fractions (one representative seed per model). Rows top-to-bottom: Project Genie , LingBot-World , Matrix-Game 2.0 , Matrix-Game 3.0 . Columns left-to-right: 0%0\% , 33%33\% , 66%66\% , 99%99\% of orbit completion.
Figure 18: Worst single-step Shadow Shape Drift events per model. Each triplet shows three consecutive frames covering the largest morphological discontinuity in the rollout. The Hu-moment shape descriptor is invariant to translation, rotation, and scale, so a smooth perspective sweep does not register; only genuine shape morphing does.
Figure 19: First scene drop-out per model, with ±8\pm 8 -frame context. tt is the first frame at which the anchor mask falls below the area threshold used for Scene Retention Rate.
More in this topic

World Model Evaluation

World Model Evaluation measures whether learned simulators predict consistent, physically plausible, action-responsive futures that help agents plan and act.

See all 212 papers

Explore similar work

CardsList
May 25, 2026

WBench: A Comprehensive Multi-turn Benchmark for Interactive Video World Model Evaluation

Interactive world models are advancing rapidly, yet existing benchmarks cover only part of the required competencies, leaving no unified standard for systematic evaluation. To fill this gap, we introduce WBench, a comprehensive multi-turn benchmark for interactive world model evaluation along five dimensions, namely video quality, setting adherence, interaction adherence, consistency, and physics compliance. WBench contains 289 test cases and 1,058 interaction turns, where each case specifies a world setting and a multi-turn interaction sequence, covering diverse scenes, styles, subjects, and both first- and third-person perspectives, together with four interaction types, including navigation, subject action, event editing, and perspective switching. For navigation, WBench unifies text, 6-DoF pose, and discrete-action control, enabling evaluation of models with different native input interfaces. Evaluation uses 22 automatic sub-metrics that combine specialist vision models with large multimodal models, and all metrics are validated against human judgments. Across 20 state-of-the-art models, we find that no single model performs strongly across all dimensions. We provide detailed diagnostic insights into the characteristic strengths, weaknesses, and open challenges of each model. Code and data are available at https://github.com/meituan-longcat/WBench.

Kaining Ying, Hengrui Hu, Siyu Ren +6

Fudan University · Meituan Longcat Team

World Model EvaluationInteractive World ModelsVideo World Models
AskPDF
Jun 30, 2026

WorldRoamBench: An Open-World Benchmark for Long-Horizon Stability of Interactive World Models

Despite rapid progress in interactive world models (IWMs), existing benchmarks evaluate action following only at trajectory level and ignore memory and interaction physics. We introduce WorldRoamBench, an open-world benchmark for long-horizon stability across four dimensions, each with tailored innovations: (i) Action: per-frame action metric bypassing cross-model semantic scale disparity and exposing failures hidden by trajectory; (ii) Vision: segment-based drift metric capturing non-monotonic mid-sequence collapse missed by start-vs-end comparisons; (iii) Physics: controllability-gated evaluation over mechanics, optics, and 3D consistency, scoring plausibility under faithful action execution; (iv) Memory: action-decoupled protocol evaluating scene memory via transition-localized 3D point-cloud reconstruction and subject memory via tracking-plus-VLM reasoning. The benchmark comprises 600+ test cases across Nature, Urban, and Indoor scenes in first/third-person views with WASD 10-60s continuous interaction. Evaluating 10+ open/closed-source models reveals none reliably satisfies all dimensions; even the best achieves only moderate scores. Advances on WorldRoamBench are steps toward IWMs that are stable, physically grounded, memory-faithful, and deployable in real-world applications.

Ting-Bing Xu, Jiacheng Sui, Zhe Gao +12

AMAP CV Lab, Alibaba Group · Nanjing University · Tsinghua University +1

World Model EvaluationInteractive World Models
AskPDF
Aug 13, 2026

PlayWorld: Benchmarking World Models with Agent Players over Long-Horizon Objectives

Video world models simulate future states conditioned on current observations and user actions. Recent systems have demonstrated impressive video consistency and action controllability over long sequences. However, fairly comparing these interactive models remains challenging. In practice, a human player typically evaluates a world model by pursuing long-horizon objectives through interaction. For example, a user may turn around 360 degrees to see whether the environment remains consistent, or walk into the water and inspect whether realistic water ripples are generated. The action sequence required to achieve the same objective may vary substantially between models, making fixed action-conditioned evaluation unsuitable for cross-model comparison. To address this, we employ multi-modal Agent Players to interact with world models toward specified long-horizon objectives. Building on this paradigm, we introduce PlayWorld, a benchmark providing 171 scenarios, each with a specified objective. To evaluate performance thoroughly, we assess models along four core dimensions: geometry consistency, interaction fidelity, out-of-sight evolution, and insight evolution. In addition, we incorporate basic ability metrics for video quality and controllability. Experiments across nine state-of-the-art world models reveal that current models remain unreliable on long-horizon interactive objectives, particularly in maintaining spatial consistency and persistent state evolution. Code and data are available at https://github.com/kxding/PlayWorld.

Kaixin Ding, Xi Chen, Minghong Cai +9

The University of Hong Kong · Kling Team, Kuaishou Technology · The Chinese University of Hong Kong +1

Long-Horizon Agent EvaluationWorld Model EvaluationInteractive World ModelsVideo World Models
AskPDF