While the ability of video generation models to accurately depict scientific phenomena has received growing attention, determining whether these models genuinely understand the underlying principles remains a critical challenge. Existing benchmarks typically evaluate video samples in isolation, making it difficult to expose models that simply fit superficial visual priors. In scientific experiments, a small change in one condition produces a markedly different outcome even when the rest of the setup remains fixed. Capitalizing on this sensitivity, we introduce SciWorldBench, a benchmark comprising paired scientific experiments. Each pair varies a single condition in an otherwise fixed setup, yielding two mutually exclusive expected outcomes. Each generation prompt specifies the condition but omits the expected outcome. We evaluate ten text-and-image-to-video (TI2V) models using an independent vision-language model (VLM) that assesses each video without access to the condition text or the label identifying the correct outcome. A pair is scored as correct only when both videos depict their respective scientifically correct outcomes. Across all ten models, we find that the two videos in an experimental pair frequently depict the same outcome even though the conditions require different results. This recurring failure suggests that current video world models struggle to apply the underlying scientific principles reliably during generation. We will publicly release the benchmark dataset and open-source the evaluation code.
Figures & tables
Figure 2: Example pairs from SciWorldBench. Each pair varies one condition while holding the setup fixed; the conditions are listed below the images. Top: both sides use the same first frame, and the condition is specified only in the prompt. Bottom: the condition is visible in the first frame.
Figure 3: Categories of SciWorldBench. The left panel illustrates the eight categories with representative objects and phenomena. The bars on the right show how the 222 experimental pairs are distributed across them.
Table 1: Results on SciWorldBench (%). PPR and SOA are computed over the 444 videos of each model, and PA over the 222 pairs. CR is computed over the decided pairs, in which both videos pass the prerequisite checks and each shows exactly one outcome. We assess model performance across four complementary metrics and rank models separately on each. In each column, the best and second-best results are shaded light blue and light yellow.
Model
PA ↑
SOA ↑
CR ↓
MiniMax-H3
7.2
39.9
84.3
+ outcome stated
33.3 (+26.1)
59.5 (+19.6)
40.2 ( − 44.1)
Cosmos3-Super
9.0
37.8
79.3
+ outcome stated
27.5 (+18.5)
56.5 (+18.7)
50.4 ( − 28.9)
Cosmos3-Nano
5.4
30.4
82.1
+ outcome stated
19.4 (+14.0)
47.5 (+17.1)
58.6 ( − 23.5)
Table 3: Results (%) without and with explicit outcome guidance. For each model, the first row reports the original setting and the second includes the outcome in the prompt. Colored values in parentheses give percentage-point changes from the first row (green for increases and red for decreases). CR is computed over decided pairs separately for each setting.
Category
Pairs
SOA ↑
PA ↑
CR ↓
Mechanics
27
30.2
8.1
71.3
Fluids
33
37.4
10.6
73.9
Thermodynamics
32
40.8
10.0
76.5
Optics
26
45.6
18.1
65.8
Electromagnetism
29
35.2
7.6
80.6
Chemistry
34
32.4
6.2
83.6
Table 6: Results by category (%), with metrics pooled across all ten models. The best and second-best results in each column are shaded light blue and light yellow, respectively.
We introduce Sci-VBench, a comprehensive benchmark for evaluating knowledge- and reasoning-intensive video generation across scientific domains. It contains 1,253 expert-annotated examples spanning 60 subjects across four core disciplines: Natural Science, Healthcare, Humanities & Social Sciences, and Engineering. Each example requires models to generate temporally rich videos that demand scientific reasoning and knowledge-grounded synthesis, going beyond surface-level visual plausibility. We further establish a rubric-based evaluation protocol. Our analysis shows that, under this protocol, both non-expert human evaluators and MLLM-as-Judge systems can achieve relatively high agreement with expert judgments, supporting reproducible evaluation at scale. We benchmark 16 frontier proprietary and open-source models and find that, while automatic perceptual-quality scores cluster tightly across systems, performance on Prompt Grounding and Scientific and Causal Correctness varies substantially, with a pronounced proprietary-open-source gap. These findings show that advances in visual realism have not yet translated into reliable modeling of scientific and causal dynamics.
Commercial video generation systems such as Seedance2.0 and Veo3.1 have rapidly improved, strengthening the view that video generators may be evolving into "world simulators." Yet the community still lacks a benchmark that directly tests whether a model can reason about how an observed world should evolve over time. We introduce WorldReasonBench, which reframes video generation evaluation as world-state prediction: given an initial state and an action, can a model generate a future video whose state evolution remains physically, socially, logically, and informationally consistent? WorldReasonBench contains 436 curated test cases with structured ground-truth QA annotations spanning four reasoning dimensions and 22 subcategories. We evaluate generated videos with a human-aligned two-part methodology: Process-aware Reasoning Verification uses structured QA and reasoning-phase diagnostics to detect temporal and causal failures, while Multi-dimensional Quality Assessment scores reasoning quality, temporal consistency, and visual aesthetics for ranking and reward modeling. We further introduce WorldRewardBench, a preference benchmark with approximately 6K expert-annotated pairs over 1.4K videos, supporting pair-wise and point-wise reward-model evaluation. Across modern video generators, our results expose a persistent gap between visual plausibility and world reasoning: videos can look convincing while failing dynamics, causality, or information preservation. We will release our benchmarks and evaluation toolkit to support community research on genuinely world-aware video generation at https://github.com/UniX-AI-Lab/WorldReasonBench/.
Keming Wu, Yijing Cui, Wenhan Xue +11
Tsinghua University · Nanyang Technological University · Hong Kong University of Science and Technology (Guangzhou) +2
Recent studies suggest that video generation models can exhibit certain forms of zero-shot visual reasoning through generated frames. Yet reliable evaluation remains challenging: benchmarks should adopt inputs aligned with the visual priors of current video models, require valid evolving processes rather than only plausible final states, and calibrate task difficulty to remain challenging yet partly feasible. To this end, we introduce VGI-bench, containing 27 tasks and 810 instances, organized by a two-level taxonomy of task domains and skill tags for fine-grained evaluation of visual reasoning capabilities of video generation models. Our evaluations show that current generative systems can solve a subset of visually grounded reasoning tasks, but remain far from reliable, with even the strongest model, Seedance 2.0, achieving only 51.0% under our evaluation criteria. Our analysis further explore the output failure modes, input condition sensitivity, performance transfer boundary from synthetic fine-tuning, and internal denoising perspective revealing limited self-correction, where later steps mainly refine early hypotheses rather than correct reasoning errors. We hope VGI-bench will help stimulate the development of next-generation video generation models. Website: https://hexuan21.github.io/VGI-Bench/
Xuan He, Cong Wei, Yuhao Cheng +20
University of Illinois Urbana Champaign · University of Waterloo · Vector Institute +6