cs.CVNov 16, 2026

SciWorldBench: Do Video World Models Really Understand Scientific Experiments?

Authors: Xue Yang, Haonan Cai, Xinyu Hang, Zhimeng Huang, Xuanyi Liu, Chuanmin Jia, Siwei Ma

Organizations: Peking University · Work done while visiting Nanyang Technological University, Singapore.

Abstract

While the ability of video generation models to accurately depict scientific phenomena has received growing attention, determining whether these models genuinely understand the underlying principles remains a critical challenge. Existing benchmarks typically evaluate video samples in isolation, making it difficult to expose models that simply fit superficial visual priors. In scientific experiments, a small change in one condition produces a markedly different outcome even when the rest of the setup remains fixed. Capitalizing on this sensitivity, we introduce SciWorldBench, a benchmark comprising paired scientific experiments. Each pair varies a single condition in an otherwise fixed setup, yielding two mutually exclusive expected outcomes. Each generation prompt specifies the condition but omits the expected outcome. We evaluate ten text-and-image-to-video (TI2V) models using an independent vision-language model (VLM) that assesses each video without access to the condition text or the label identifying the correct outcome. A pair is scored as correct only when both videos depict their respective scientifically correct outcomes. Across all ten models, we find that the two videos in an experimental pair frequently depict the same outcome even though the conditions require different results. This recurring failure suggests that current video world models struggle to apply the underlying scientific principles reliably during generation. We will publicly release the benchmark dataset and open-source the evaluation code.

Figures & tables

Explore similar work

CardsList
  1. Sci-VBench: Evaluating Knowledge- and Reasoning-Intensive Video Generation in Science Domains

    Aug 10, 2026Diandian Zhang, Tingyu Song, Lin Fu +2VLM EvaluationVideo Generation

  2. WorldReasonBench: Human-Aligned Stress Testing of Video Generators as Future World-State Predictors

    May 11, 2026Keming Wu, Yijing Cui, Wenhan Xue +11World ModelsVideo Reasoning

  3. VGI-Bench: Probing Visual Intelligence in Video Generation Models

    Aug 20, 2026Xuan He, Cong Wei, Yuhao Cheng +20Video Generation Evaluation