cs.CVMar 20, 2026
SavePhysion-Eval: Evaluating Physical Realism in Generated Video via Human Reasoning
Organizations: Physion Labs · Stanford University · MIT · Character AI ·
Abstract
Video generation models are increasingly used as world simulators for storytelling, simulation, and embodied AI. As these models advance, a key question arises: do generated videos obey the physical laws of the real world? Existing evaluations largely rely on automated metrics or coarse human judgments such as preferences or rubric-based checks. While useful for assessing perceptual quality, these methods provide limited insight into when and why generated dynamics violate real-world physical constraints. We introduce Physion-Eval, a large-scale benchmark of expert human reasoning for diagnosing physical realism failures in videos generated by five state-of-the-art models across egocentric and exocentric views, containing 10,990 expert reasoning traces spanning 22 fine-grained physical categories. Each generated video is derived from a corresponding real-world reference video depicting a clear physical process, and annotated with temporally localized glitches, structured failure categories, and natural-language explanations of the violated physical behavior. Using this dataset, we reveal a striking limitation of current video generation models: in physics-critical scenarios, 83.3% of exocentric and 93.5% of egocentric generated videos exhibit at least one human-identifiable physical glitch. We hope Physion-Eval will set a new standard for physical realism evaluation and guide the development of physics-grounded video generation. The benchmark is publicly available at https://huggingface.co/datasets/PhysionLabs/Physion-Eval.
Figures & tables
Figure 1: Physion-Eval Benchmark. (Left) The benchmark spans diverse physical phenomena across egocentric and exocentric views, evaluating videos generated by five state-of-the-art generation models. (Right, top) Physion-Eval provides 10,990 expert-annotated reasoning traces with timestamped glitch localization, structured failure categories, and natural-language explanations. (Right, bottom) Results reveal a large physical realism gap: 83.3% of exocentric and 93.5% of egocentric generated videos contain at least one human-identifiable physical glitch, motivating physics-grounded video generation and automated critics.
Figure 2: Examples of physical glitches in AI-generated videos from Physion-Eval. Each row shows a representative failure mode where generated dynamics violate basic physical principles. Frame sequences illustrate how these glitches emerge over time.
| Physical Category | Visually Observable Physics | Mapped EPIC-KITCHENS Action Verbs |
|---|---|---|
| Rigid-Body Interaction | Rigid motion, contact, articulation, gravity, momentum | take, put, hold, carry, move, lift, remove, drop, let-go, set, open, close, turn, turn-on, turn-off, unlock, lock, press, push, pull, switch, adjust, use, attach, detach, throw |
| Deformation & Fracture | Elastic deformation, cutting, separation | bend, flatten, wrap, unwrap, unroll, cut, break, crush, stab, divide, mark, score, sharpen |
| Soft Materials & Mixing | Cloth dynamics, viscoplastic flow | mix, stir, shake, knead, wear, dry, fold |
| Fluid & Granular Flow | Liquid flow, particle scattering and piling | pour, fill, empty, wash, water, soak, spray, sprinkle, season, grate |
| Thermal & Frictional Effects | Heating/cooling cues, friction, surface wear | cook, bake, unfreeze, scrub, brush, rub |
Table 1: Mapping from EPIC-KITCHENS [ 13 ] verb labels to visually observable physical categories used in the egocentric setting.
Figure 3: Two complementary human evaluation studies for assessing physical realism in generated videos. (a) Perceptual detection by ordinary viewers. Untrained viewers evaluate a blinded 1:1 mixture of real-world videos and outputs from five video generation models, judging whether each clip appears physically realistic. The evaluation metric measures how often generated videos are perceived as physically realistic relative to real videos. (b) Physion-Eval expert reasoning benchmark. Expert annotators follow a three-expert workflow to annotate generated videos, producing temporally localized failures, category labels, severity scores, and natural-language explanations. The final dataset contains 10,990 adjudicated reasoning annotations for diagnosing failure modes in video generation models.
Figure 4: Evaluation results of the untrained human study across video generation models under (a) exocentric and (b) egocentric settings. The radial plots (left) visualize Youden’s J statistic ( ) for each evaluator, while the tables (right) report the corresponding metrics , and . Across models, untrained human viewers consistently achieve higher J scores than current MLLM critics, indicating a stronger sensitivity to physical glitches in genereated videos, especially in the egocentric setting.
Figure 5: Example from Physion-Eval comparing expert human annotations and MLLM reasoning. In this example, human annotators correctly identify that the ice object sprays water without a visible cause and later increases in volume while melting, violating expected causal behavior and mass conservation. In contrast, Gemini 3.1 Pro hallucinates a non-existent shadow artifact, highlighting a substantial gap between human reasoning and current automated critics.
| Exocentric | Egocentric | |||||||
|---|---|---|---|---|---|---|---|---|
| Model | # Videos | Failure Rate ( ) | Density ( ) | Severity ( ) | # Videos | Failure Rate ( ) | Density ( ) | Severity ( ) |
| Kling 2.5 [ 28 ] | 1,738 | 73.8% | 1.15 1.08 | 2.69 1.80 | 416 | 96.4% | 1.42 1.06 | 3.05 1.58 |
| Veo3.1 Fast [ 19 ] | 1,696 | 79.4% | 1.32 1.11 | 3.01 1.74 | 402 | 97.5% | 1.69 1.12 | 3.37 1.56 |
| Sora 2 [ 52 ] | 1,587 | 79.2% | 1.21 0.94 | 2.88 1.67 | 763 | 96.6% | 1.23 0.93 | 2.81 1.69 |
| Hailuo 2.3 [ 37 ] | 1,719 | 93.1% | 1.42 0.90 | 3.61 1.62 | 423 | 92.0% | 1.92 1.36 | 3.86 0.88 |
| Wan 2.2 [ 55 ] | 1,751 | 90.3% | 1.32 0.88 | 3.33 1.39 | 449 | 83.5% | 1.56 1.36 | 3.49 0.75 |
Table 2: Physical glitch statistics across video generation models. Failure rate denotes the percentage of generated videos that contain at least one human-identified glitch. Glitch density denotes the average number of glitches per video. Glitch severity denotes the average glitch severity score reported by the expert annotators. We also report mean standard deviation for glitch density and severity.
Figure 6: Distribution of physical glitch categories across models for exocentric (left) and egocentric (right) viewpoints. Stacked bars show the percentage of videos exhibiting different types of physical failures or no issue. Temporal coherence breakdown and material/state inconsistency are the most common failure modes in the exocentric view, while temporal coherence breakdown and object permanence violations dominate in the egocentric view.
| Critic | Sampling Rate | Exocentric | Egocentric |
|---|---|---|---|
| Human (reference) | - | 24.9 | 58.5 |
| Gemini 3.0 Pro | 1 FPS | 13.7 | 3.7 |
| 5 FPS | 10.5 | 5.6 | |
| 10 FPS | 8.1 | 4.1 | |
| GPT 5.2 | 12 frames | 3.2 | 1.5 |
Table 3: Effect of temporal sampling on MLLM critic performance measured by J-statistic, % ( ) in exocentric and egocentric views.
| Critic | Thinking | Exocentric | Egocentric | ||
|---|---|---|---|---|---|
| Kling 2.5 | Veo 3.1 Fast | Kling 2.5 | Veo 3.1 Fast | ||
| Human (reference) | - | 24.9 | 35.5 | 48.4 | 58.5 |
| Claude Opus 4.5 | No | 4.1 | 0.0 | 2.5 | 2.6 |
| Yes | 4.6 (+0.5) | 2.0 (+2.0) | 4.3 (+1.8) | 4.1 (+1.5) | |
| OpenAI GPT 5.2 | No | 2.6 | 3.3 | 2.3 | 2.6 |
| Yes | 4.5 (+1.9) | 3.5 (+0.2) | 0.4 (-1.9) | 3.5 (+0.9) | |
Table 4: Critic performance with and without explicit reasoning when evaluating videos from Kling 2.5 (K) and Veo 3.1 Fast (V), measured by J-statistic, % ( ) in exocentric and egocentric views.
Figure 7: J values computed from untrained human evaluations for three video generation models in the exocentric setting, stratified by physical intensity and dynamics scores, denoted as and , respectively. Lower J statistic indicates stronger generator performance, corresponding to videos that are perceived as more physically realistic by ordinary human viewers. Pink , purple , and cyan bars correspond to physical intensity or dynamics scores 0, 1, and 2, respectively.
Figure 8: Comparison of J values obtained using different MLLM critics when evaluating videos generated by Veo3.1 Fast, stratified by three-level measures of physical intensity (left) and dynamics (right) in the exocentric setting. Higher J values indicate stronger critic ability to distinguish generated videos from real ones within a given physical regime. Zero J statistic values result in bars of zero height and are therefore not visible. G2.5 and G3 denote Gemini 2.5 and Gemini 3, respectively. Pink , purple , and cyan bars correspond to physical intensity or dynamics scores 0, 1, and 2, respectively.
Figure 9: Average physical realism judgments by untrained human observers across physics categories. Radar plots show the J statistic computed from judgments by ordinary human viewers when evaluating videos generated by five video generation models in the exocentric (left) and egocentric (right) settings. Each axis corresponds to a physical phenomenon category, with representative frames and descriptions shown around the plots. The results reveal substantial variation across physics categories and models.
Figure 10: Expert Annotation Interface for Physically Grounded Video Evaluation.
| Metadata | Intensity Score | Dynamics Score | ||
|---|---|---|---|---|
| PCC | 95% CI | PCC | 95% CI | |
| Value | 0.724 | [0.657, 0.779] | 0.738 | [0.673, 0.790] |
Table 5: Pearson correlation (PCC) and 95% confidence intervals between expert-annotated and Gemini 2.5 Pro–extracted physical metadata, with PCC indicating strong correlation.
Figure 11: The first two videos ( a and b ) show that the model generates visually plausible results for physical processes with low dynamics. In contrast, the latter three videos reveal unrealistic artifacts when the processes involve irreversible state changes and entropy-increasing dynamics ( c ) or abrupt and complex system-level transitions ( d and e ) .
Figure 12: Comparison of expert human and MLLM reasoning on physical realism.
More in this topic
Physical Consistency in Video Generation
Physical consistency in video generation studies how to make and evaluate videos whose geometry, motion, interactions, and state changes obey real-world laws.
See all 186 papers