Organizations: School of Computer Science and Technology, UCAS · The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, CASIA · AMAP, Alibaba Group · School of Computer Science and Engineering, Beihang University · State Key Laboratory of Networking and Switching Technology, BUPT
Video--based world models have emerged along two dominant paradigms: video generation and 3D reconstruction. However, existing evaluation benchmarks either focus narrowly on visual fidelity and text--video alignment for generative models, or rely on static 3D reconstruction metrics that fundamentally neglect temporal dynamics. We argue that the future of world modeling lies in 4D generation, which jointly models spatial structure and temporal evolution. In this paradigm, the core capability is interactive response: the ability to faithfully reflect how interaction actions drive state transitions across space and time. Yet no existing benchmark systematically evaluates this critical dimension. To address this gap, we propose Omni--WorldBench, a comprehensive benchmark specifically designed to evaluate the interactive response capabilities of world models in 4D settings. Omni--WorldBench comprises two key components: Omni--WorldSuite, a systematic prompt suite spanning diverse interaction levels and scene types; and Omni--Metrics, an agent-based evaluation framework that quantifies world modeling capabilities by measuring the causal impact of interaction actions on both final outcomes and intermediate state evolution trajectories. We conduct extensive evaluations of 18 representative world models across multiple paradigms. Our analysis reveals critical limitations of current world models in interactive response, providing actionable insights for future research. Omni-WorldBench will be publicly released to foster progress in interactive 4D world modeling.
Figures & tables
Figure 1: Overview of Omni-WorldBench. Left: Omni-WorldSuite defines three levels of interactions, each specified by an initial frame and a prompt. Right: Omni-Metrics comprises an evaluation pipeline that measures interaction effect fidelity, generated video quality, camera-object controllability, and spatiotemporal causal coherence, and then employs an MLLM to adaptively fuse these signals into the final AgenticScore.
Figure 2: Omni-WorldSuite Construction Pipeline and Analysis. (a) Dataset-grounded prompt generation. Prompts are generated from open-source data using first-frame and camera-motion cues, refined through VLM captioning, and finally verified by human annotators. (b) Concept-driven prompt generation. Prompts are derived from interaction prototypes using LLM/VLM-based generation and human curation, together with generated or edited first frames. (c) Suite taxonomy across indoor scenes, including diffusion (Diff.), sliding, and building-related (Buil.) scenarios; outdoor scenes, including natural, projectile motion (Proj.), and urban scenarios (Urban); and task-oriented settings, including robotics (Robot), autonomous driving (Driv.), and gaming (Game). (d) Coverage comparison by prompt modality and capability axes. Abbr.: Traj (camera trajectory); AD (autonomous driving), EAI (embodied AI); PP (physical principles); LCC (loop-closure consistency); Cau. (causality); CS (common sense); Inter. (Interaction).
Figure 3: Omni-WorldSuite examples across three interaction levels. Left: Examples from the General Scene domain. Right: Examples from the Task-Oriented Scene domain, including optics/camera trajectories, game, physics/common sense, autonomous driving, and embodied AI. Each example pairs a first-frame grounding (and trajectory, when applicable) with an action prompt, with red boxes indicating interaction-relevant entities.
Figure 4: Statistics of Omni-WorldSuite. (a) Overall Distributions; (b–g) Distributions of core principles; (h) prompt counts by interaction level; (i–k) word clouds of objects, actions, and scenes. NM (Newtonian Mechanics), FM (Fluid Mechanics), MP (Material Properties), WO (Waves and Optics), MC (Momentum and Collision), TP (Thermodynamics and Phase Transition), EC (Energy Conversion and Conservation); SEK (Scene/Event Knowledge), OFK (Object Function Knowledge), HAK (Human Action Knowledge); C2B (Condition-to-Behavior), A2M (Action-to-Motion), C2O (Collision-to-Outcome); TFS (Tracking / Follow Shot), OAS (Orbit / Arc Shot), HHS (Handheld / Shaky); ART (Axial Round-Trip Motion), ODC (Optical / Dynamic Consistency Closure), SCC (Spiral / Composite Closure), CDR (Curved / Diagonal Return Motion), PCP (Planar Closed-Path Motion), ORC (Orbital / Rotational Closure), UNC (Uncategorized); MKC (Mechanical / Kinematic Constraints), CSS (Contact & Support Stability), OAC (Occlusion & Accessibility Constraints), CL (Containment & Leakage), GFSC (Geometric Fit & Size Compatibility), DMC (Deformation & Material Constraints).
Figure 5: Non-camera-controlled Interaction Comparison. Qualitative comparison of generated videos from different models under the same prompt and first-frame condition. Representative frames illustrate differences in interaction effect fidelity, motion dynamics, and scene coherence during the throwing action.
Figure 6: Camera-Controlled Interaction Comparison. Qualitative Comparison of Generated Videos from Different Models under the Same Prompt, First-Frame, and Camera Trajectory Condition.
World Model Evaluation measures whether learned simulators predict consistent, physically plausible, action-responsive futures that help agents plan and act.
Interactive world models are advancing rapidly, yet existing benchmarks cover only part of the required competencies, leaving no unified standard for systematic evaluation. To fill this gap, we introduce WBench, a comprehensive multi-turn benchmark for interactive world model evaluation along five dimensions, namely video quality, setting adherence, interaction adherence, consistency, and physics compliance. WBench contains 289 test cases and 1,058 interaction turns, where each case specifies a world setting and a multi-turn interaction sequence, covering diverse scenes, styles, subjects, and both first- and third-person perspectives, together with four interaction types, including navigation, subject action, event editing, and perspective switching. For navigation, WBench unifies text, 6-DoF pose, and discrete-action control, enabling evaluation of models with different native input interfaces. Evaluation uses 22 automatic sub-metrics that combine specialist vision models with large multimodal models, and all metrics are validated against human judgments. Across 20 state-of-the-art models, we find that no single model performs strongly across all dimensions. We provide detailed diagnostic insights into the characteristic strengths, weaknesses, and open challenges of each model. Code and data are available at https://github.com/meituan-longcat/WBench.
Evaluating world models requires assessing both the quality of the worlds they generate and their consistency and responsiveness under exploration, interaction, and modification. We introduce HappyWorld-Bench, a comprehensive benchmark that evaluates whether generated worlds remain reliable as agents interact with them. Our design is built on a hierarchical capability framework of six world capabilities (W1-W6), from generative construction to unified world modeling, instantiated across three independent evaluation tracks: video world models, spatial world models, and embodied world models. HappyWorld-Bench comprises 1,138 video prompts, 300 spatial scenes, and 254 embodied test cases. Across all three tracks, we build and operate HappyWorld-Arena to organize human A/B comparisons and derive model-level Elo ratings, which complement newly designed automated metrics that capture behavioral correctness. We evaluate 14 video world models, 9 spatial systems, and 8 embodied candidates under this unified framework. Results reveal remaining reliability gaps across all three tracks: video models exhibit reduced consistency during extended rollouts and revisits, spatial models achieve at best 70.14% placement accuracy and 73.33% edit execution, and embodied models struggle to preserve state across multi-step actions and respond precisely to altered action conditions and physical rules. These findings highlight the need to evaluate world models not only by visual quality, but also by state consistency and the correctness of their responses to actions and interventions.
Zhiqi Bai, Junai Cai, Yixin Chen +33
Alibaba Token Hub, Alibaba Group · State Key Laboratory of General Artificial Intelligence, BIGAI · Tsinghua University +2
Interactive video generation models such as Genie, YUME, HY-World, and Matrix-Game are advancing rapidly, yet every model is evaluated on its own benchmark with private scenes and trajectories, making fair cross-model comparison impossible. Existing public benchmarks offer useful metrics such as trajectory error, aesthetic scores, and VLM-based judgments, but none supplies the standardized test conditions -- identical scenes, identical action sequences, and a unified control interface -- needed to make those metrics comparable across models with heterogeneous inputs. We introduce WorldMark, the first benchmark that provides such a common playing field for interactive Image-to-Video world models. WorldMark contributes: (1) a unified action-mapping layer that translates a shared WASD-style action vocabulary into each model's native control format, enabling apples-to-apples comparison across six major models on identical scenes and trajectories; (2) a hierarchical test suite of 500 evaluation cases covering first- and third-person viewpoints, photorealistic and stylized scenes, and three difficulty tiers from Easy to Hard spanning 20-60s; and (3) a modular evaluation toolkit for Visual Quality, Control Alignment, and World Consistency, designed so that researchers can reuse our standardized inputs while plugging in their own metrics as the field evolves. We will release all data, evaluation code, and model outputs to facilitate future research. Beyond offline metrics, we launch World Model Arena (warena.ai), an online platform where anyone can pit leading world models against each other in side-by-side battles and watch the live leaderboard.
Xiaojie Xu, Zhengyuan Lin, Kang He +5
Alaya Studio, Shanda AI Research Tokyo · The University of Tokyo · Shanghai Innovation Institute