Out of Sight, Out of Mind? Evaluating State Evolution in Video World Models

Ziqi Ma; Mengzhan Liufu; Georgia Gkioxari

cs.CVMar 13, 2026

Out of Sight, Out of Mind? Evaluating State Evolution in Video World Models

Authors: Ziqi Ma, Mengzhan Liufu, Georgia Gkioxari

Organizations: California Institute of Technology

Abstract

Evolutions in the world, such as water pouring or ice melting, happen regardless of being observed. Video world models generate "worlds" via 2D frame observations. Can these generated "worlds" evolve regardless of observation? To probe this question, we design a benchmark to evaluate whether video world models can decouple state evolution from observation. Our benchmark, STEVO-Bench, applies observation control to evolving processes via instructions of occluder insertion, turning off the light, or specifying camera "lookaway" trajectories. By evaluating video models with and without camera control for a diverse set of naturally-occurring evolutions, we expose their limitations in decoupling state evolution from observation. STEVO-Bench proposes an evaluation protocol to automatically detect and disentangle failure modes of video world models across key aspects of natural state evolution. Analysis of STEVO-Bench results provide new insight into potential data and architecture bias of present-day video world models. Project website: https://glab-caltech.github.io/STEVOBench/. Blog: https://ziqi-ma.github.io/blog/2026/outofsight/

Figures & tables

Figure 2: StEvo-Bench probes whether video-based world models can decouple state evolution from observation. StEvo-Bench , consisting of 225 unique tasks spanning 6 categories, uses (image, text prompt, camera control) tuples to prompt video-based world models to generate evolutions under interrupted observation. Observation interruption is done either using text prompt (such as adding occluders, turning off the light), or using camera control (turning the camera view away). The generated videos go through StEvo-Bench ’s automatic verifiers for evaluation.

	Evolution criteria			Control criteria
Category	Progress	Physics	Coherence	Observation	Action
Physics & Common sense [ 13 , 3 , 4 , 15 ]	✓	✓	✕	✕	✕
Memory & consistency [ 34 , 23 , 30 ]	✕	✕	✓	✓	✕
Instruction following [ 36 , 6 ]	✕	✕	✕	✕	✓
StEvo-Bench	✓	✓	✓	✓	✓

Table 1: Evaluating state evolution decoupled from observation requires multiple criteria. While prior benchmarks for video (world) models only touch on subsets of them,, StEvo-Bench comprehensively evaluates all aspects to expose and disentangle failure modes of world models when generating evolutions that are not fully observed.

Figure 3: Automatic verifier pipeline for StEvo-Bench . Five video understanding verifiers independently assess each generated video on one criteria. Observation Control and Action Control jointly determine Control Success; State Progress, Physics Plausibility, and Coherence jointly determine Evolution Success.

	Model	Success	Progress	Physics	Coherence
Video models	Veo 3	8.7	17.4	82.6	66.5
	Sora 2 Pro	8.1	13.1	85.5	69.7
	WAN 2.2	0.9	7.7	52	58.4
	HY-Video 1.5	0.9	4.1	42.1	59.1
	CogVideoX 1.5	0.5	1.4	68.5	67.1
Camera- controlled	Genie 3	0.0	2.9	15.2	27.3

Table 2: Evaluation metrics for video models (top) and camera-controlled models (bottom). Both video models and camera-controlled video models struggle to evolve state correctly when observation control is applied. Metrics are in percentage (%).

Figure 4: Representative failure modes of video models when observation of the evolution process is interrupted. The model is capable of simulating the process correctly when the scene is fully visible. However, when observation of the process is temporarily interrupted, the model either stops evolving state, as seen in the mattress deflation example, or fails to preserve object coherence, as seen in the sponge example. These videos are generated by Veo 3.

	Progress	Success
Observ. Full	84.6	46.2
Observ. Control	17.4	12.4

Table 3: The effect of observation control: state progress rate and task success rate drop significantly when observation control is applied. Metrics are averaged over Veo 3 and Sora 2 Pro.

Figure 5: Open-source camera-controlled video models assume the scene to be static as the camera turns, and fail to correctly evolve state, such as the ball dropping, wave advancing, or tablet dropping.

Figure 6: When camera-controlled models can successfully evolve states, the model tends to ignore camera control while the evolution is happening. Such behavior is seen both in Genie 3 and Lingbot.

Figure 7: Qualitative example of VMem. Due to the memory design, VMem can perfectly recall the initial frame, but fails to evolve state, and suffers from scene incoherence.

		Evolution criteria				Control criteria
	Metric	Progress	Physics	Coherence	Success	Observation	Action	Success
V-H	Acc	0.795	0.700	0.860	0.810	0.878	0.903	0.849
	ROC-AUC	0.644	0.672	0.864	0.713	0.800	0.918	0.785
	MRA	0.829	0.743	0.905	0.858	0.891	0.949	0.857
H-H	Acc	0.747	0.722	0.911	0.737	0.659	0.885	0.692
	ROC-AUC	0.623	0.645	0.629	0.602	0.668	0.782	0.708

Table 4: Agreement between the automatic verifiers and the human annotators, benchmarked against inter-human agreement across three metrics. Bold indicates the higher value per column. The V-H row shows agreement between the verifiers and human annotators, and the H-H shows agreement within the human annotators.

	Model	Observ.	Action
Video models	Veo 3	81.6	88.0
	Sora 2 Pro	69.4	80.4
	WAN 2.2	46.2	76.5
	HY-Video 1.5	31.2	81.0
	CogVideoX 1.5	22.2	75.5
Camera- controlled	Genie 3	84.7	78.6

Table 5: Success rate of observation and action control. Models are largely able to follow control of StEvo-Bench tasks.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Example tasks from StEvo-Bench . Each row shows two tasks from a unique category. Category from top to bottom: continuous evolution, kinematics, relational change, state transformation, causal change, and expected human & animal behavior.

Figure 9: Example video generation failures that violate verifiers on the “state evolution” axes. We show separate examples that violate the state progress verifier, physical plausibility verifier, and the coherence verifier.

Figure 10: Example video generation failures that violate verifiers on the “control” axes. We show separate examples that violate the observation control verifier and action control verifier.

Figure 11: Example verifier rationales. The verifier evaluates various aspects defined for each evaluation axis (when applicable), and provides an explanation of how the video violates the corresponding criteria.

Panel of Figure 12 — Figure 12: Mean agreement and standard deviation across three Verifier-Human (V-H) pairs and three Human-Human (H-H) pairs for each criterion. Agreement measured by Accuracy , ROC-AUC and Model Ranking Agreement (MRA) is shown in top, middle and bottom panel respectively.

Panel of Figure 13 — Figure 13: Agreement heatmap for each criterion. V-H agreement and H-H agreement are calculated using Accuracy. With 3 human annotators, both V-H and H-H agreement can only take discrete values from the set $\{0,1/3,2/3,1\}$ . Correlation between V-H agreement and H-H agreement is calculated using Spearman’s correlation coefficient.

Figure 14: Example videos where the verifier and human annotators disagree with each other. The top 3 disagreements are caused by ambiguity in the generated videos, and bottom 3 disagreements show verifier failures when they overlooked aspects of the videos or failed to follow the prompts.

Video World Models

Video world models generate and predict action-conditioned visual futures to simulate changing scenes for embodied agents.

See all 245 papers

Current World Models Lack a Persistent State Core
Jun 18, 2026Jinpeng Lu, Dexu Zhu, Haoyuan Shi +8Physical Consistency in Video Generation World Models
University of Science and Technology of China · Beijing Innovation Center of Humanoid Robotics (X-Humanoid) · NLPR, Institute of Automation, Chinese Academy of Sciences +3
LiveWorld: Simulating Out-of-Sight Dynamics in Generative Video World Models
Mar 7, 2026Zicheng Duan, Jiatong Xia, Zeyu Zhang +7Video World Models World Models
Adelaide University · The Australian National University · Monash University +2
World Observer: Joint Actor-Observer Generation for Persistent World Modeling
Oct 1, 2026Hyunwook Choi, Dahyun Chung, Hyunsung Kim +4Camera-Controlled Video Generation Multi-View Video Generation
KAIST AI

Jun 18, 2026J/K move · Enter open · S save

Current World Models Lack a Persistent State Core

Jinpeng Lu, Dexu Zhu, Haoyuan Shi, Linghan Cai+7

University of Science and Technology of China · Beijing Innovation Center of Humanoid Robotics (X-Humanoid) · NLPR, Institute of Automation, Chinese Academy of Sciences+3

Physical Consistency in Video Generation World Models World Model Evaluation Temporal Consistency in Video Generation Video World Models

Open paper AskPDF arXiv

World models are increasingly regarded as a decisive step toward artificial general intelligence, yet modeling the physical world demands more than rendering convincing frames on demand: it requires an internal world state that keeps evolving over time, decoupled from observation, so that objects endure and events run to their conclusions whether or not a camera is watching, much as the moon holds to its orbit when no one is looking. This requirement is a blind spot of existing benchmarks, which reward surface properties such as fidelity, motion, and camera controllability while never asking whether a generated world keeps evolving once it is unobserved. We introduce WRBench, the first systematic diagnostic benchmark that treats camera motion as an intervention on observability and resolves evaluation into a human-calibrated chain that asks whether the camera executes the requested interaction, whether the scene stays continuous and identifiable while in view, and whether a returning target remains consistent with the event that was set in motion. Across 9{,}600 videos from 23 models spanning four control paradigms, one finding proves stubborn: current systems maintain the observed world as a tracking shot, resuming a returning target in the state at which it was abandoned rather than advancing the event while it went unseen. Because this failure recurs across control paradigms, model families, and increments of scale, robust world-state evolution does not follow from cleaner imagery, tighter control, richer geometric priors, or sheer parameter count We therefore argue that the stability of the physical state kernel and the consistency of worldlines under viewpoint intervention should become first-class objectives of world-model design, so that a world model captures how the world will unfold rather than how the next frame appears.

Abstract

Video World Models

Explore similar work

Current World Models Lack a Persistent State Core

LiveWorld: Simulating Out-of-Sight Dynamics in Generative Video World Models

World Observer: Joint Actor-Observer Generation for Persistent World Modeling