Video foundation models now reach human-level accuracy on physical-reasoning benchmarks, yet such tasks require predicting unobserved physical outcomes. Do these models perform human-like forward simulation, or do they exploit statistical regularities in visible scenes? Accuracy alone cannot distinguish these strategies. We introduce a distributional evaluation framework that treats model seeds and human raters as populations, enabling comparison of consensus, uncertainty, and strategy. On the Physion benchmark, we evaluate three ViT-L architectures (V-JEPA2, VideoMAEv2, DINOv2). V-JEPA2 narrows the accuracy gap to ~1 percentage point (73.2% vs. 74.2%), yet model-human disagreement reaches 26.4%, far exceeding human-human disagreement (4.8%), with substantially lower agreement (kappa ~ 0.48 vs. 0.91). The divergence follows forward-simulation demands: models outperform humans on geometric reasoning (linking, +11.8 pp) but underperform on gravitational dynamics (rolling, -11.8 pp) and causal chains (dominoes, -10.5 pp). Strategy fingerprinting confirms all three architectures share non-human strategies while none aligns with humans. Attribution analysis suggests that unobservable outcome features, rather than visible scene properties, predict this divergence, consistent with models relying more on scene-level statistical regularities than on explicit forward simulation, a systematic divergence that accuracy alone cannot reveal. Code is available at https://github.com/fanhong-li/model-human-divergence.
Figures & tables
Figure 1: Overview. Top: Standard evaluation compares a single model’s aggregate accuracy against human mean accuracy, masking systematic differences. Our distributional evaluation compares model populations (3 architectures × 8 seeds) against the full human rater population ( ∼100 raters × 1200 trials). Bottom: Three key findings. (A) Agreement hierarchy: model–model agreement ( κMM=0.80 ) far exceeds model–human agreement ( κMH=0.48 ), which in turn falls well below human–human agreement ( κHH=0.91 ). (B) Released-split scenario-specific accuracy gaps include a model advantage on linking ( + 11.8 pp) and human advantages on rolling/sliding ( − 11.8 pp) and dominoes ( − 10.5 pp). (C) Models share systematic errors with each other (error overlap Jaccard = 0.41, strategy cosine = 0.70) but not with humans (Jaccard = 0.26, cosine = 0.30). All Jaccard values are averages across model pairs.
Model
Seeds
Test Acc.
Train Acc.
Disagr.
κMH
V-JEPA2
8
73.16±0.95
90.60±0.56
26.4
0.484
VideoMAEv2
8
67.65±0.72
85.41±0.47
30.0
0.397
DINOv2
8
66.90±0.73
91.96±0.38
30.2
0.399
Human
∼ 100
74.2
—
4.8
0.906 (HH)
Table 1: Cross-model comparison on Physion (all ViT-L, unified protocol, E2E attentive probes). V-JEPA2 significantly outperforms alternatives ( p<0.001 , McNemar). All models show >5× the disagreement rate of human–human baseline (4.8%).
Figure 2: Models are consistent but differently consistent than humans. (a) Per-scenario variability of human raters (gray violins, ∼100 raters) versus model seeds (colored dots, 8 seeds per model). Human rater distributions are 2 – 3× wider ( σ=5 – 8% ) than model seed distributions ( σ=1 – 3% ), yet model clusters are offset from the human median on most scenarios. (b) Fuzzy zone overlap: each model’s “fuzzy zone” (near-chance internal agreement) contains 14 – 17% of samples, but pairwise Jaccard overlap is only ∼0.12 , meaning models are uncertain about different samples.
Scenario
V-JEPA2
VideoMAEv2
DINOv2
Human
Δ
Collision
83.8±1.7
77.8±2.2
70.6±1.8
80.3
+3.5
Containment
80.8±2.6
73.4±1.9
72.2±2.1
76.0
+4.8
Towers
80.3±2.1
75.8±1.9
72.2±1.9
75.5
+4.8
Linking
75.6±1.8
72.7±1.4
60.7±3.4
63.8
+11.8
Roll/Slide
75.9±2.1
74.8±2.4
74.6±1.8
87.7
−11.8
Drop
69.7±2.8
66.4±1.0
66.2±2.5
74.0
−4.3
Table 2: Per-scenario accuracy (mean ± std across seeds) for three models and humans. Δ = V-JEPA2 − Human. Best model per scenario in bold .
Figure 3: Strategy and difficulty divergence. (a) Strategy fingerprint similarity matrix: cosine similarity between Ridge-regression strategy vectors (37 physical features → per-sample accuracy). All three models form a tight cluster (cosine =0.60 – 0.75 , p<0.001 ), but no model’s strategy is significantly similar to humans ( p>0.1 for all). (b) Difficulty alignment reversal: model accuracy declines less steeply than human accuracy as stimuli get harder. On the hardest human-defined quintile, V-JEPA2 scores +19.8 pp higher while remaining near chance ( r=0.38 ).
Figure 1: Extended variability comparison (4 models). Per-scenario variability of human raters (amber violins) versus model seeds (colored dots). RVM (purple) clusters as tightly as V-JEPA2 (blue) and tracks the same per-scenario pattern, with both offset from the human median.
Figure 2: Fuzzy zone overlap (4 models, UpSet plot). Left: set sizes (each model’s fuzzy zone count). Top: intersection sizes sorted by count. Dot matrix below indicates set membership. The four exclusive (single-model) regions dominate ( 56 – 120 samples each), while pairwise overlaps are small ( ≤24 ). Only 1 sample lies in the intersection of all four fuzzy zones, confirming that models are uncertain about largely different samples (avg. Jaccard =0.111 ).
Figure 3: Extended strategy fingerprint (4 models + human). Cosine similarity between Ridge-regression strategy vectors. RVM is most similar to V-JEPA2 ( 0.83 , p<0.001 ) and least similar to humans ( 0.18 , p=0.39 ). All four models form a tight cluster; no model signature aligns with the human signature.
Figure 4: Extended difficulty alignment (4 models). On the hardest quintile, all four models outperform humans. RVM shows a pattern nearly identical to V-JEPA2, reinforcing that these architecturally different models converge on the same difficulty profile.
Figure 5: Layer-wise probing accuracy for V-JEPA2 (ViT-L, 24 layers). Attentive probes (blue) peak at layer 16 ( 71.8% ) within a broad plateau (L15–L22, shaded), then drop − 3.0pp at the final layer. Linear probes (pink) peak later (layer 19) and are ∼10 pp lower throughout, consistent with the capacity gap expected from a 2K-parameter linear head.
Probe
Params
Test Acc.
Mean-pooled baselines
Linear
2K
61.50%
Attentive (depth-4)
2.4M
62.50%
Token-level probes
GRU-Pool
3.7M
66.00%
TemporalAttn
7.9M
67.33%
Appendix
Table 3: Token-level probes on V-JEPA2 full features [ 8×256 , 1024 ].
Scenario
With
Without
Δ
roll/slide
80.0
84.0
+ 4.0
drop
63.3
66.0
+ 2.7
towers
76.7
74.7
− 2.0
linking
74.7
72.7
− 2.0
clothiness
61.3
60.0
− 1.3
dominoes
59.3
56.0
− 3.3
Appendix
Table 4: Red-yellow marker ablation on V-JEPA2.
Figure 6: Physion++ extended analysis. (a) Per-property accuracy: V-JEPA2 dramatically outperforms humans on deformability ( + 35.9pp) but underperforms on friction ( − 1.6pp). (b) Model–human disagreement roughly doubles from Physion ( 27% at frame_step = 5; cf. 26.4% at frame_step = 5.75 in main text) to Physion++ ( 43 – 58% ), with friction showing the highest disagreement ( 57.8% ).
Figure 7: Collision subtask variability.
Figure 8: Containment subtask variability.
Figure 9: Towers subtask variability.
Figure 10: Linking subtask variability.
Figure 11: Rolling/Sliding subtask variability.
Figure 12: Drop subtask variability.
Figure 13: Clothiness subtask variability.
Figure 14: Dominoes subtask variability.
Model
ECE
High-conf. acc.
Overconf. errors
V-JEPA2
0.166
84.2%
3
VideoMAEv2
0.216
79.4%
16
DINOv2
0.179
75.5%
24
Appendix
Table 5: Calibration across models. V-JEPA2 is best-calibrated with fewest overconfident errors (model > 80% confident but wrong).
Model
Q1 (Easy)
Q2 (M-Blind)
Q3 (H-Hard)
Q4 (Hard)
A
V-JEPA2
704 (58.7%)
103 (8.6%)
324 (27.0%)
69 (5.8%)
0.318
VideoMAEv2
708 (59.0%)
99 (8.3%)
320 (26.7%)
73 (6.1%)
0.309
DINOv2
682 (56.8%)
125 (10.4%)
318 (26.5%)
75 (6.3%)
0.393
Appendix
Table 6: Entropy quadrant distribution across models ( n=1200 ). The asymmetry ratio A=∣ Q2 ∣/∣ Q3 ∣ quantifies whether model uncertainty misaligns with human uncertainty ( A=1 would indicate symmetric uncertainty).
Figure 15: Entropy scatter plot (V-JEPA2). Each point is a test sample; axes show model-seed entropy ( Hm ) and human-rater entropy ( Hh ). Dashed lines at τ=0.811 bits partition the space into four quadrants. Q3 (bottom-right, model confident but humans uncertain) is 3× larger than Q2 (top-left), reflecting systematic overconfidence on human-hard samples.
Target
Features
Ridge
RF
GB
R2 (regression targets)
∣pm−ph∣ (divergence)
A+B (obs.)
−0.77
−0.32
−0.46
∣pm−ph∣ (divergence)
C (unobs.)
+0.10
+0.23
+0.29
AUROC (classification targets)
Disagree binary
A+B (obs.)
0.57
0.63
0.61
Disagree binary
C (unobs.)
0.70
0.70
0.70
Appendix
Table 7: Attribution results: observable features (A+B, 66 dim) vs. unobservable features (C, 23 dim). R2 is cross-validated (negative = worse than constant predictor). AUROC is for binary targets.
37-dim name
Ly
89-dim equivalent
Mt
Layer A: Static/Scene (6 features)
n_distractors
A
A03_n_distractors
E
n_occluders
A
A04_n_occluders
E
n_physical_objects
A
A02_n_physical_objects
E
target_mass
A
A05_target_mass
E
target_friction
A
A07_target_dynamic_friction
P
Appendix
Table 8: Correspondence between 37-feature (strategy fingerprint) and 89-feature (attribution) sets. Match type: E = exact, P = partial (similar but not identical computation), — = no equivalent.
Figure 16: Per-subtask accuracy heatmap (45 subtasks × 5 agents). Rows grouped by scenario, sorted by V-JEPA2 accuracy. Color: red ( < 50%) → yellow (50%) → blue ( > 80%).
Figure 17: Qualitative high-confidence disagreement cases. Rows show representative stimuli where human raters and model seeds give high-confidence opposite contact predictions. From left to right, each row shows the start frame, the observation cutoff, the full outcome, and contact-response rates for humans, V-JEPA2, VideoMAEv2, and DINOv2. The first five rows are human-majority-correct/model-wrong cases; the last three rows are model-correct/human-majority-wrong cases.
Modern video generation models are increasingly hailed as emerging world models with an internalized grasp of physical law. Yet existing benchmarks largely evaluate physical plausibility only at the output level, without verifying whether the model arrives there through a faithful, law-grounded reasoning process. We introduce Apple-PI, the first benchmark that anchors video-model evaluation explicitly in physical laws. Apple-PI comprises three components. 1) Orchard: a dataset of 400 videos covering ten canonical tasks in classical mechanics. It separates single-law tasks for confounder-free diagnosis from multi-law tasks for probing generalization. 2) Benchmark Protocol: a three-stage protocol based on scientific reasoning, including Perception, Formulation, and Deduction. It uses chain-of-frames prompting on infographic-annotated first frames, treating the generated video as the model's visible reasoning trace. 3) Evaluation Suite: a hybrid evaluation suite that combines MLLM-based subjective scoring with physics-law-grounded objective measures. This enables stage-resolved diagnosis of not only whether a model fails, but where it fails. Benchmarking 11 models shows that current video models remain far from reliable law-grounded world simulators, with the best video model scoring only 0.473. Our stage-, pillar-, and source-resolved analyses further expose a Perception-to-Formulation-to-Deduction bottleneck, weak multi-law state transfer, and a persistent Sim-to-Real gap. These findings position Apple-PI as a diagnostic foundation for guiding future video models toward world models with law-grounded physical intelligence.
Runmao Yao, Kairui Hu, Yukang Cao +11
1S-Lab, Nanyang Technological University · 2The Chinese University of Hong Kong
Video generation models are increasingly used as world simulators for storytelling, simulation, and embodied AI. As these models advance, a key question arises: do generated videos obey the physical laws of the real world? Existing evaluations largely rely on automated metrics or coarse human judgments such as preferences or rubric-based checks. While useful for assessing perceptual quality, these methods provide limited insight into when and why generated dynamics violate real-world physical constraints. We introduce Physion-Eval, a large-scale benchmark of expert human reasoning for diagnosing physical realism failures in videos generated by five state-of-the-art models across egocentric and exocentric views, containing 10,990 expert reasoning traces spanning 22 fine-grained physical categories. Each generated video is derived from a corresponding real-world reference video depicting a clear physical process, and annotated with temporally localized glitches, structured failure categories, and natural-language explanations of the violated physical behavior. Using this dataset, we reveal a striking limitation of current video generation models: in physics-critical scenarios, 83.3% of exocentric and 93.5% of egocentric generated videos exhibit at least one human-identifiable physical glitch. We hope Physion-Eval will set a new standard for physical realism evaluation and guide the development of physics-grounded video generation. The benchmark is publicly available at https://huggingface.co/datasets/PhysionLabs/Physion-Eval.
Evaluating physical reasoning in video models is difficult because absolute motion measurements depend on frame rate, object scale, and camera calibration, all of which are often ambiguous or unavailable in generated video. We propose a different approach. When two objects in the same scene obey the same physical law, their motions must satisfy predictable relationships, and these relationships hold independent of calibration. We introduce Principia, a benchmark that evaluates Newtonian physics through relational consistency between paired objects. Principia spans eight phenomena - gravity, restitution, friction, rotational inertia, projectile motion, momentum, pendulum, and mass-spring oscillation - across translational, rotational, collisional, and oscillatory dynamics, using real-world scenes recorded under controlled protocols. We also introduce a calibration-independent consistency score that quantifies physical violation directly in image space. Across thousands of generations from six state-of-the-art video generators, no model exceeds 0.42 on Principia despite all scoring around 0.8 on VBench. Vision-language models are evaluated on their ability to detect relational physics violations, with the best model achieving only 67% accuracy and most performing near chance level.