The Flip Test: Video World Models Fail to Learn the World State
Authors: Mi Luo, Lijun Yu, Adams Wei Yu, Ang Cao, Baochen Sun, Ming-Hsuan Yang, Nan Hua, Yuan Hao, Alex Dimakis, Ya Xu, Ed H. Chi, Xuezhi Wang, Yinxiao Liu
Organizations: UT Austin · Google DeepMind · UC Berkeley
Video world models promise to predict the future of a scene from its present. Do current models keep this promise? We probe this question counterfactually. When a small change to the present flips the future, such as a box resting a few millimeters before or past a table edge, a model that tracks the state of the world must flip its prediction too. We build FlipTest, a three-level hierarchy of increasingly strict tests over 1,680 initial frames from 10 simulated and 15 real-world tasks. At the base, visual minimal pairs, two nearly identical frames whose true futures differ, test whether predictions change across the boundary; matched controls go further and test whether they change only where the future does; and dense sweeps, the strictest level, locate exactly where they change. All 11 video world models we evaluate fall short. The best averages 18.3% pair accuracy over real and simulated scenes, below the 25% of random guessing. Models flip as often on steps that leave the future unchanged as on steps that change it, and their response stays flat across the sweeps. On the same frames, a vision-language model is selective and traces a near S-shaped curve near the boundary, and a coding agent that measures the image comes close to the true boundary. This implies that the deciding information is in the frame, and video models just miss it. We therefore urge the community to rethink world modeling as pure video prediction. Next-frame prediction must do more than draw plausible pixels; it must also identify the causal states that control future dynamics.
Figures & tables
Figure 1: A small change to the initial frame flips the correct future. Two nearly identical initial frames sit on opposite sides of a physical boundary, so the correct outcome differs. A model passes only if its two predictions also differ.
Figure 2: FlipTest-Level 1 at a glance. It has 300 minimal pairs (600 initial frames) over 25 tasks (outer ring), grouped into six physical families (inner ring). Dark wedges are simulated tasks and light wedges are real tasks, with 150 pairs and 300 initial frames each. Each callout shows two example pairs from one family. Each pair has two opposite outcomes.
Figure 3: Video world models rarely pass FlipTest Level 1. We show pair accuracy (%) of 11 video models on real scenes, simulation, and their average. The models are sorted by their average. A frame counts as correct only if both judges (Gemini 3.8 Flash and Gemini 3.6 Flash) read the ground-truth outcome from its video. A pair is solved only if both frames are correct. The dashed line marks chance (25%).
Figure 4: Video models are not selectively sensitive in FlipTest Level 2. (a) A control group on the box task: four equally spaced states (offsets exaggerated for display), of which only the middle step crosses the boundary. A model that reads the state flips only at the crossing step, landing below the diagonal in (b) and right of 0 in (c) like the VLM; video models sit near the diagonal, flipping about as often when the outcome should stay as when it should change. (b) Flip rate at the crossing step vs. at the non-crossing steps of a control group. The shaded region marks SS >0 . (c) Selective sensitivity (crossing minus non-crossing flip rate) with 95% CIs over scenes and two judges. The VLM is Gemini 3.6 Flash answering directly from the same first frames. It flips on 61% of crossing steps and only 15% of non-crossing steps ( +0.46 , 95% CI [0.36,0.56] ), while video models score between −0.07 and +0.07 , each with a CI that covers 0.
Figure 5: Video models do not see the boundary in FlipTest Level 3. (a) A sweep of 30 states per scene across the boundary, densest near it; a model’s P(event) curve shows where it places the boundary and how sharply it switches (model curve illustrative). P(event) along the sweep is shown (b) pooled and (c) per task (4 scenes × 30 states). A model that reads the state should follow the oracle, which jumps from 0 to 1 at the boundary. While VLMs bend toward the oracle step at m=0 , video models remain flat.
Figure 6: Failure mode: one future for both frames. Four evenly spaced frames of the generated videos for two minimal pairs, (a) Cosmos3-Super on the simulated seesaw and (b) VPP2 on real ring toss, labeled with the outcome both judges read. Each model shows the same outcome for both frames of its pair, so one of the two videos is wrong (red border).
Figure 7: Making the state explicit closes the gap. (a) Predictors mapped onto the three stages of Visual De-animation ( Wu et al., 2017a ) , with every pipeline terminating in a binary outcome label scored against ground truth. (b) Pair accuracy on the 150 real pairs and frame accuracy on the 480 sweep frames, comparing the best video world model, the direct VLM, the Measure agent, and the Simulation agent against chance (dashed lines: 25% for pairs, 50% for frames). (c) Sweep accuracy as a function of distance from the boundary, pooled over four tasks, four scenes, and both boundary sides (32 frames per step; invalid agent runs count as wrong).
Figure 8: Told the outcome, every video model improves. Pair accuracy (%) with the neutral prompt (hollow, as in Fig. 3 ) and with the ground-truth outcome stated in the prompt (filled; teal for simulated pairs, blue for real pairs). A pair counts only when both videos show the told outcome, scored by Gemini 3.8 Flash with unclear rollouts counted as wrong.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Task
Transition variable δ
Threshold
∣δ∣ range
Outcome labels
Box at desk edge
COM distance past the edge
0
0.008–0.045 m
stay / fall
Ball through pipe
bore radius − ball radius
0
0.0025–0.014 m
blocked / pass
Ball through gap
slot width − ball diameter
0
0.008–0.018 m
stuck / pass
Domino
first-gap ratio d/h
1.0136
0.10–0.50
full_fall / break
Chain over edge
hang fraction f
1/3
0.03–0.13
rest / slide
Loop-the-loop
release height h
0.776 m
0.03–0.22 m
falls / completes
Appendix
Table 1: Simulated tasks (15 pairs each). The two members of a pair sit at ±∣δ∣ around the threshold and differ only in the transition variable. The 15 pairs of a task span the full ∣δ∣ range, from knife-edge to clear-cut cases.
Figure 9: Where the coding agents spend their time. (a) Share of wall-clock time per phase, pooled over the 780 runs of each agent; we assign each model step and its tool call to one phase from the call content. (b, c) Phase share over normalised run time (start to end of each run, averaged over runs). Both agents spend most of their time measuring the image, and the Simulation agent spends about a quarter of its run building and running the MuJoCo scene.
Figure 10: Simulation agent case studies. For each run, we show the input frame and the agent’s rebuilt MuJoCo scene at t=0 and at the end of the rollout, re-rendered offline from the submitted scene.xml , together with the key measurement, the pre-simulation judgement, the simulated outcome, and the ground truth. Top: the rollout corrects a wrong judgement (box) or confirms a correct one in multi-body scenes (domino, Newton’s cradle). Bottom: the errors come from the reconstruction, where the scene misses a support (propped book stack) or reads two lever arms closer than the measurement error as equal (seesaw).
Figure 15: Per-task pair accuracy (%) of the 11 video models. (a) The 10 simulated tasks (15 pairs each); (b) the 15 real tasks (10 pairs each). Rows are sorted as in Fig. 3 , the last row is the mean over models, and the all column shows the split-level pair accuracy. A frame counts as correct only if both judges (Gemini 3.8 Flash and Gemini 3.6 Flash) read the ground-truth outcome, as in Fig. 3 .
Figure 16: All 11 video models on one simulated pipe pair. A ball rests on the mouth of a vertical pipe. In A, the ball is slightly wider than the bore (GT: blocked), and in B it is slightly narrower (GT: pass). The top row shows the input frames and the simulator’s final frame, and each model tile is the last frame of that model’s video. A label shows the outcome read by the two judges, green when both read GT, red when both read the wrong outcome, and grey when the video is unclear or the judges disagree. The tag above each model gives the pair result.
Figure 17: All 11 video models on one simulated seesaw pair. Two equal cubes sit on a seesaw that is level at release, and only the distance of the red cube from the pivot differs. In A, the red cube is closer to the pivot than the blue cube (GT: right down), and in B it is farther away (GT: left down). Layout as in Fig. 16 .
Figure 18: All 11 video models on one real bowl-stack pair. A hand holds one cup above another cup. In A, the held cup is smaller and nests (GT: stacks), and in B it is larger and can only rest on the rim (GT: no stack). Real pairs have no ground-truth video, so the top row shows only the inputs. Layout as in Fig. 16 .
Figure 19: All 11 video models on one simulated box pair. A 0.2 m cube rests near the edge of a desk, with its center of mass 2.9 cm inside the edge in A (GT: stay) and 2.9 cm past the edge in B (GT: fall). No model solves the pair. Layout as in Fig. 16 .
Figure 20: All 11 video models on one simulated funnel-slide pair. A ball rolls down a slide whose wave-shaped walls narrow to a 7.5 cm throat. In A, the ball (8.7 cm) is wider than the throat (GT: stuck), and in B it is narrower (6.3 cm; GT: finished). Layout as in Fig. 16 .
Figure 21: All 11 video models on the real propped-stack pair of Fig. 1 . A hand pulls away the book that props a stack. In A, the stack stands upright on its own (GT: stays standing), and in B it leans on the prop (GT: collapses). Real pairs have no ground-truth video, so the top row shows only the inputs. Layout as in Fig. 16 .
Figure 22: All 11 video models on one real desk-edge pair. A book lies near the edge of a desk. In A (captured), most of the book rests on the desk (GT: stay), and in B (edited) the book shifts past the edge (GT: fall). Layout as in Fig. 16 .
Simulated (150 pairs)
Real (150 pairs)
Model
solved
same
swap.
uncl.
disagr.
solved
same
swap.
uncl.
disagr.
Gemini Omni Flash 1.1
8
100
4
11
27
47
86
4
2
11
MiniMax-H3
5
111
2
5
27
38
87
5
0
20
Wan2.2-I2V-A14B
5
85
5
17
38
37
82
8
1
22
VBVR-Wan2.2
8
95
3
9
35
34
84
5
2
25
Cosmos3-Super
8
80
6
10
46
33
71
7
10
29
Appendix
Table 3: Outcome of every pair, per model and split. For solved , both judges read the ground truth on both frames, and the pair accuracy of Fig. 3 is this count over 150. For same , both judges agree and the model shows the same outcome for both frames. For swap. , both opposite outcomes appear on the wrong frames, a category that also includes the few pairs with a different wrong outcome on a three-outcome task. For uncl. , both judges agree and at least one frame is unclear . For disagr. , the judges disagree on at least one frame. Shaded columns give the solved counts, with the best count per split in bold.
Video world models are largely regarded as predictive models of the physical world and are therefore expected to anticipate the consequences of observed events. However, evaluation has mainly focused on reference similarity, physical-law consistency, or judgment plausibility, estimating anticipation only indirectly. We address this directly: when a release or impact has just occurred but its consequence is withheld, can a world model anticipate what should happen next? We introduce an event-anchored evaluation based on 62 controlled real-world free-fall recordings and 124 clips spanning three object types, with fine-grained release and impact annotations and ground-truth trajectories. The protocol separates consequence production, temporal placement, and physical realization. Across six contemporary video generation and world models, Runway and Veo produce release and subsequent impact events at rates above 93% but often initiate them substantially late, whereas Cosmos-Predict-2.5 and MAGI-1 frequently preserve the pre-event state and produce little or no measurable consequence. Among measurable falls, plausible timing does not necessarily imply physically consistent motion. We further conduct a 15-participant, 20-condition human study in which participants describe the expected consequence from a single event-anchored frame and draw its trajectory. Human predictions favor the recorded future in aggregate while revealing genuine ambiguity among plausible continuations. Overall, physical foresight emerges as a sequence of distinct challenges: initiating a consequence, anchoring it in time, and realizing its motion.
Acting in a physical scene requires knowing its real later state, not a plausible one. Current evaluations often accept words or a realistic-looking image, so the predicted state is never checked against the true one. We introduce DynaPix (Dynamic Pixels), a benchmark that makes prediction checkable. Given a video clip that stops before a key event and a question about a later moment, a model must pick the true future image from close candidates or a large gallery. The scenes come from a physics simulator, so the correct image and its time are known exactly and the wrong options are deliberately similar. Models often succeed when a visible event marks the target moment, but are near chance when only elapsed time marks it. Gallery search is harder still, as the true image rarely ranks first. People handle the elapsed-time items well, so the difficulty lies with the models, not the questions. Training on scene accounts drawn from the simulator's true record, not a teacher's guess, repairs much of this but not the longer elapsed time case. DynaPix thus exposes a temporal-anchoring gap: models attach a prediction to an event far better than to time itself.
Thong Nguyen, Vinh-Hien Do, Quynh Vo +2
Centre for AI Research, VinUniversity · National University of Singapore
Video generation models are increasingly used as world simulators for tasks like driving and robotic manipulation. What matters in these settings is not whether a single video looks right, but whether the model's output changes when its input changes. We test this by giving a model two prompts describing the same scene with one physical detail varied, and checking whether the two videos diverge the way physics predicts. The wording difference between the prompts is small by design, since only one variable is changed, but the correct physical difference is not. A model that misses this can still produce two videos that each look plausible individually, and existing benchmarks score videos one at a time and cannot detect this failure. We introduce What-If World, 319 such prompt pairs built on real frames from nuScenes and DROID, organized by a taxonomy of six physical variables shared across driving and manipulation. Each pair is scored with APEO, a four-part rubric checking whether each video follows its prompt (Adherence), is physically consistent (Physics), preserves the shared scene (Environment), and ends in the correct difference (Outcome). Across nine state-of-the-art models, no system exceeds 52% on the paired score, and open-source models cluster near 28%. Every model tested fails on a large fraction of causal interventions, indicating substantial room before these models can reliably support action-conditioned simulation or model-based planning. Where models do score well, performance appears to track the visual prominence of the intervention rather than the tractability of its underlying physics. Some visually subtle interventions score as low as 14.2%, while visually pronounced ones reach 40.4%.