Human-Level Accuracy, Non-Human Strategies: Revealing Model-Human Divergence in Video Physical Reasoning

Fanhong Li; Shurui Zheng; Zi Yin; Junbo Cui; Lei Ji; Jia Liu

cs.CVSep 19, 2026

Human-Level Accuracy, Non-Human Strategies: Revealing Model-Human Divergence in Video Physical Reasoning

Authors: Fanhong Li, Shurui Zheng, Zi Yin, Junbo Cui, Lei Ji, Jia Liu

Organizations: Tsinghua University, Beijing, China · ModelBest Inc., Beijing, China · Microsoft, Beijing, China

Abstract

Video foundation models now reach human-level accuracy on physical-reasoning benchmarks, yet such tasks require predicting unobserved physical outcomes. Do these models perform human-like forward simulation, or do they exploit statistical regularities in visible scenes? Accuracy alone cannot distinguish these strategies. We introduce a distributional evaluation framework that treats model seeds and human raters as populations, enabling comparison of consensus, uncertainty, and strategy. On the Physion benchmark, we evaluate three ViT-L architectures (V-JEPA2, VideoMAEv2, DINOv2). V-JEPA2 narrows the accuracy gap to ~1 percentage point (73.2% vs. 74.2%), yet model-human disagreement reaches 26.4%, far exceeding human-human disagreement (4.8%), with substantially lower agreement (kappa ~ 0.48 vs. 0.91). The divergence follows forward-simulation demands: models outperform humans on geometric reasoning (linking, +11.8 pp) but underperform on gravitational dynamics (rolling, -11.8 pp) and causal chains (dominoes, -10.5 pp). Strategy fingerprinting confirms all three architectures share non-human strategies while none aligns with humans. Attribution analysis suggests that unobservable outcome features, rather than visible scene properties, predict this divergence, consistent with models relying more on scene-level statistical regularities than on explicit forward simulation, a systematic divergence that accuracy alone cannot reveal. Code is available at https://github.com/fanhong-li/model-human-divergence.

Figures & tables

Panel of Figure 1 — Figure 1: Overview. Top: Standard evaluation compares a single model’s aggregate accuracy against human mean accuracy, masking systematic differences. Our distributional evaluation compares model populations (3 architectures $\times$ 8 seeds) against the full human rater population ( ${\sim}100$ raters $\times$ 1200 trials). Bottom: Three key findings. (A) Agreement hierarchy: model–model agreement ( $\kappa_{MM}\!=\!0.80$ ) far exceeds model–human agreement ( $\kappa_{MH}\!=\!0.48$ ), which in turn falls well below human–human agreement ( $\kappa_{HH}\!=\!0.91$ ). (B) Released-split scenario-specific accuracy gaps include a model advantage on linking ( $+$ 11.8 pp) and human advantages on rolling/sliding ( $-$ 11.8 pp) and dominoes ( $-$ 10.5 pp). (C) Models share systematic errors with each other (error overlap Jaccard $\,{=}\,$ 0.41, strategy cosine $\,{=}\,$ 0.70) but not with humans (Jaccard $\,{=}\,$ 0.26, cosine $\,{=}\,$ 0.30). All Jaccard values are averages across model pairs.

Model	Seeds	Test Acc.	Train Acc.	Disagr.	$\kappa_{\mathrm{MH}}$
V-JEPA2	8	$\mathbf{73.16{\pm}0.95}$	$90.60{\pm}0.56$	$26.4$	$0.484$
VideoMAEv2	8	$67.65{\pm}0.72$	$85.41{\pm}0.47$	$30.0$	$0.397$
DINOv2	8	$66.90{\pm}0.73$	$91.96{\pm}0.38$	$30.2$	$0.399$
Human	${\sim}$ 100	$74.2$	—	$4.8$	$0.906$ (HH)

Table 1: Cross-model comparison on Physion (all ViT-L, unified protocol, E2E attentive probes). V-JEPA2 significantly outperforms alternatives (

p<0.001

, McNemar). All models show

>

5{\times}

the disagreement rate of human–human baseline (4.8%).

Figure 2: Models are consistent but differently consistent than humans. (a) Per-scenario variability of human raters (gray violins,

{\sim}100

raters) versus model seeds (colored dots, 8 seeds per model). Human rater distributions are

2

–

3{\times}

wider (

\sigma\!=\!5

–

8\%

) than model seed distributions (

\sigma\!=\!1

–

3\%

), yet model clusters are offset from the human median on most scenarios. (b) Fuzzy zone overlap: each model’s “fuzzy zone” (near-chance internal agreement) contains

14

–

17\%

of samples, but pairwise Jaccard overlap is only

{\sim}0.12

, meaning models are uncertain about different samples.

Scenario	V-JEPA2	VideoMAEv2	DINOv2	Human	$\Delta$
Collision	$\mathbf{83.8{\pm}1.7}$	$77.8{\pm}2.2$	$70.6{\pm}1.8$	80.3	$+3.5$
Containment	$\mathbf{80.8{\pm}2.6}$	$73.4{\pm}1.9$	$72.2{\pm}2.1$	76.0	$+4.8$
Towers	$\mathbf{80.3{\pm}2.1}$	$75.8{\pm}1.9$	$72.2{\pm}1.9$	75.5	$+4.8$
Linking	$\mathbf{75.6{\pm}1.8}$	$72.7{\pm}1.4$	$60.7{\pm}3.4$	63.8	$+11.8$
Roll/Slide	$75.9{\pm}2.1$	$\mathbf{74.8{\pm}2.4}$	$74.6{\pm}1.8$	87.7	$-11.8$
Drop	$\mathbf{69.7{\pm}2.8}$	$66.4{\pm}1.0$	$66.2{\pm}2.5$	74.0	$-4.3$

Table 2: Per-scenario accuracy (mean

{\pm}

std across seeds) for three models and humans.

\Delta

= V-JEPA2

-

Human. Best model per scenario in bold .

Figure 3: Strategy and difficulty divergence. (a) Strategy fingerprint similarity matrix: cosine similarity between Ridge-regression strategy vectors (37 physical features $\to$ per-sample accuracy). All three models form a tight cluster (cosine $\,{=}\,0.60$ – $0.75$ , $p\!<\!0.001$ ), but no model’s strategy is significantly similar to humans ( $p\!>\!0.1$ for all). (b) Difficulty alignment reversal: model accuracy declines less steeply than human accuracy as stimuli get harder. On the hardest human-defined quintile, V-JEPA2 scores $+19.8$ pp higher while remaining near chance ( $r\!=\!0.38$ ).

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Seeds	Test Acc.	Disagr.	$\kappa_{\mathrm{MH}}$
V-JEPA2	10	$\mathbf{73.3{\pm}0.9}$	$26.9$	$0.464$
RVM	10	$73.1{\pm}1.1$	$25.2$	$0.491$
VideoMAEv2	10	$67.8{\pm}0.8$	$30.7$	$0.385$
DINOv2	10	$66.9{\pm}0.7$	$32.0$	$0.364$
Human †	${\sim}$ 100	$74.2$	$4.8$	$0.906$ (HH)

Appendix

Table 1: Supplementary 10-seed cross-model comparison including RVM (all ViT-L, unified protocol).

Scenario	V-JEPA2	RVM	VideoMAEv2	DINOv2	Human	$\Delta$
Collision	$\mathbf{83.8{\pm}1.6}$	$83.7{\pm}1.5$	$78.5{\pm}2.0$	$70.1{\pm}2.0$	80.3	$+3.5$
Containment	$\mathbf{81.4{\pm}2.6}$	$79.0{\pm}1.9$	$73.8{\pm}1.9$	$72.7{\pm}2.3$	76.0	$+5.4$
Towers	$\mathbf{80.5{\pm}2.4}$	$79.1{\pm}2.1$	$76.1{\pm}2.0$	$72.0{\pm}2.2$	75.5	$+5.0$
Linking	$\mathbf{75.8{\pm}1.6}$	$74.7{\pm}1.7$	$72.9{\pm}1.5$	$60.9{\pm}3.3$	63.8	$+12.0$
Roll/Slide	$75.6{\pm}3.3$	$\mathbf{75.5{\pm}2.0}$	$74.7{\pm}3.5$	$74.5{\pm}3.3$	87.7	$-12.1$
Drop	$\mathbf{68.9{\pm}2.9}$	$67.7{\pm}3.2$	$66.9{\pm}1.3$	$66.5{\pm}2.4$	74.0	$-5.1$

Appendix

Table 2: Extended per-scenario accuracy including RVM.

\Delta

= V-JEPA2

-

Human.

Figure 1: Extended variability comparison (4 models). Per-scenario variability of human raters (amber violins) versus model seeds (colored dots). RVM (purple) clusters as tightly as V-JEPA2 (blue) and tracks the same per-scenario pattern, with both offset from the human median.

Figure 2: Fuzzy zone overlap (4 models, UpSet plot). Left: set sizes (each model’s fuzzy zone count). Top: intersection sizes sorted by count. Dot matrix below indicates set membership. The four exclusive (single-model) regions dominate (

56

–

120

samples each), while pairwise overlaps are small (

\leq 24

). Only 1 sample lies in the intersection of all four fuzzy zones, confirming that models are uncertain about largely different samples (avg. Jaccard

\,{=}\,0.111

Figure 3: Extended strategy fingerprint (4 models + human). Cosine similarity between Ridge-regression strategy vectors. RVM is most similar to V-JEPA2 ( $0.83$ , $p\!<\!0.001$ ) and least similar to humans ( $0.18$ , $p\!=\!0.39$ ). All four models form a tight cluster; no model signature aligns with the human signature.

Figure 4: Extended difficulty alignment (4 models). On the hardest quintile, all four models outperform humans. RVM shows a pattern nearly identical to V-JEPA2, reinforcing that these architecturally different models converge on the same difficulty profile.

Figure 5: Layer-wise probing accuracy for V-JEPA2 (ViT-L, 24 layers). Attentive probes (blue) peak at layer 16 (

71.8\%

) within a broad plateau (L15–L22, shaded), then drop

-

3.0pp at the final layer. Linear probes (pink) peak later (layer 19) and are

{\sim}10

pp lower throughout, consistent with the capacity gap expected from a 2K-parameter linear head.

Mean-pooled baselines
Probe	Params	Test Acc.
Linear	2K	61.50%
Attentive (depth-4)	2.4M	62.50%
Token-level probes
GRU-Pool	3.7M	66.00%
TemporalAttn	7.9M	67.33%

Appendix

Table 3: Token-level probes on V-JEPA2 full features [

8{\times}256

1024

Scenario	With	Without	$\Delta$
roll/slide	80.0	84.0	$+$ 4.0
drop	63.3	66.0	$+$ 2.7
towers	76.7	74.7	$-$ 2.0
linking	74.7	72.7	$-$ 2.0
clothiness	61.3	60.0	$-$ 1.3
dominoes	59.3	56.0	$-$ 3.3

Appendix

Table 4: Red-yellow marker ablation on V-JEPA2.

Figure 6: Physion++ extended analysis. (a) Per-property accuracy: V-JEPA2 dramatically outperforms humans on deformability (

+

35.9pp) but underperforms on friction (

-

1.6pp). (b) Model–human disagreement roughly doubles from Physion (

27\%

at frame_step

\,{=}\,

5; cf.

26.4\%

at frame_step

\,{=}\,

5.75 in main text) to Physion++ (

43

–

58\%

), with friction showing the highest disagreement (

57.8\%

Figure 7: Collision subtask variability.

Figure 8: Containment subtask variability.

Figure 9: Towers subtask variability.

Figure 10: Linking subtask variability.

Figure 11: Rolling/Sliding subtask variability.

Figure 12: Drop subtask variability.

Figure 13: Clothiness subtask variability.

Figure 14: Dominoes subtask variability.

Model	ECE	High-conf. acc.	Overconf. errors
V-JEPA2	$\mathbf{0.166}$	84.2%	3
VideoMAEv2	0.216	79.4%	16
DINOv2	0.179	75.5%	24

Appendix

Table 5: Calibration across models. V-JEPA2 is best-calibrated with fewest overconfident errors (model

>

80% confident but wrong).

Model	Q1 (Easy)	Q2 (M-Blind)	Q3 (H-Hard)	Q4 (Hard)	$A$
V-JEPA2	704 (58.7%)	103 (8.6%)	324 (27.0%)	69 (5.8%)	0.318
VideoMAEv2	708 (59.0%)	99 (8.3%)	320 (26.7%)	73 (6.1%)	0.309
DINOv2	682 (56.8%)	125 (10.4%)	318 (26.5%)	75 (6.3%)	0.393

Appendix

Table 6: Entropy quadrant distribution across models (

n\!=\!1200

). The asymmetry ratio

A\!=\!|

|/|

|

quantifies whether model uncertainty misaligns with human uncertainty (

A\!=\!1

would indicate symmetric uncertainty).

Figure 15: Entropy scatter plot (V-JEPA2). Each point is a test sample; axes show model-seed entropy ( $H_{m}$ ) and human-rater entropy ( $H_{h}$ ). Dashed lines at $\tau\!=\!0.811$ bits partition the space into four quadrants. Q3 (bottom-right, model confident but humans uncertain) is $3{\times}$ larger than Q2 (top-left), reflecting systematic overconfidence on human-hard samples.

$R^{2}$ (regression targets)
Target	Features	Ridge	RF	GB
$\|p_{m}-p_{h}\|$ (divergence)	A+B (obs.)	$-0.77$	$-0.32$	$-0.46$
$\|p_{m}-p_{h}\|$ (divergence)	C (unobs.)	$+0.10$	$+0.23$	$\mathbf{+0.29}$
AUROC (classification targets)
Disagree binary	A+B (obs.)	$0.57$	$0.63$	$0.61$
Disagree binary	C (unobs.)	$\mathbf{0.70}$	$0.70$	$0.70$

Appendix

Table 7: Attribution results: observable features (A+B, 66 dim) vs. unobservable features (C, 23 dim).

R^{2}

is cross-validated (negative

=

worse than constant predictor). AUROC is for binary targets.

37-dim name	Ly	89-dim equivalent	Mt
Layer A: Static/Scene (6 features)
n_distractors	A	A03_n_distractors	E
n_occluders	A	A04_n_occluders	E
n_physical_objects	A	A02_n_physical_objects	E
target_mass	A	A05_target_mass	E
target_friction	A	A07_target_dynamic_friction	P

Appendix

Table 8: Correspondence between 37-feature (strategy fingerprint) and 89-feature (attribution) sets. Match type: E = exact, P = partial (similar but not identical computation), — = no equivalent.

Figure 16: Per-subtask accuracy heatmap (45 subtasks $\times$ 5 agents). Rows grouped by scenario, sorted by V-JEPA2 accuracy. Color: red ( $<$ 50%) $\to$ yellow (50%) $\to$ blue ( $>$ 80%).

Figure 17: Qualitative high-confidence disagreement cases. Rows show representative stimuli where human raters and model seeds give high-confidence opposite contact predictions. From left to right, each row shows the start frame, the observation cutoff, the full outcome, and contact-response rates for humans, V-JEPA2, VideoMAEv2, and DINOv2. The first five rows are human-majority-correct/model-wrong cases; the last three rows are model-correct/human-majority-wrong cases.

Physical Reasoning in Video

Physical reasoning in video studies how models infer, predict, and generate motion and interactions consistent with physical laws.

See all 72 papers

Apple- $π$ : Benchmarking Thinking with Video Towards Law-Grounded Physical Intelligence
Jul 17, 2026Runmao Yao, Kairui Hu, Yukang Cao +11Physical Consistency in Video Generation Physical Reasoning in Video
1S-Lab, Nanyang Technological University · 2The Chinese University of Hong Kong
Physion-Eval: Evaluating Physical Realism in Generated Video via Human Reasoning
Mar 20, 2026Qin Zhang, Peiyu Jing, Hong-Xing Yu +7Physical Consistency in Video Generation Video Generation Evaluation
Physion Labs · Stanford University · MIT +2
Principia: Relational Physics Tests for Video Models
Sep 3, 2026Varun Varma Thozhiyoor, Shivam Tripathi, Venkatesh Babu Radhakrishnan +1VLM Evaluation Physical Consistency in Video Generation
Indian Institute of Science · Johns Hopkins University

Jul 17, 2026J/K move · Enter open · S save

Apple- $π$ : Benchmarking Thinking with Video Towards Law-Grounded Physical Intelligence

Runmao Yao, Kairui Hu, Yukang Cao, Ruisi Wang+10

1S-Lab, Nanyang Technological University · 2The Chinese University of Hong Kong

Physical Consistency in Video Generation Physical Reasoning in Video Video World Models Video Generation Evaluation

Open paper AskPDF arXiv

Modern video generation models are increasingly hailed as emerging world models with an internalized grasp of physical law. Yet existing benchmarks largely evaluate physical plausibility only at the output level, without verifying whether the model arrives there through a faithful, law-grounded reasoning process. We introduce Apple-PI, the first benchmark that anchors video-model evaluation explicitly in physical laws. Apple-PI comprises three components. 1) Orchard: a dataset of 400 videos covering ten canonical tasks in classical mechanics. It separates single-law tasks for confounder-free diagnosis from multi-law tasks for probing generalization. 2) Benchmark Protocol: a three-stage protocol based on scientific reasoning, including Perception, Formulation, and Deduction. It uses chain-of-frames prompting on infographic-annotated first frames, treating the generated video as the model's visible reasoning trace. 3) Evaluation Suite: a hybrid evaluation suite that combines MLLM-based subjective scoring with physics-law-grounded objective measures. This enables stage-resolved diagnosis of not only whether a model fails, but where it fails. Benchmarking 11 models shows that current video models remain far from reliable law-grounded world simulators, with the best video model scoring only 0.473. Our stage-, pillar-, and source-resolved analyses further expose a Perception-to-Formulation-to-Deduction bottleneck, weak multi-law state transfer, and a persistent Sim-to-Real gap. These findings position Apple-PI as a diagnostic foundation for guiding future video models toward world models with law-grounded physical intelligence.

Abstract

Physical Reasoning in Video

Explore similar work

Apple-πππ: Benchmarking Thinking with Video Towards Law-Grounded Physical Intelligence

Physion-Eval: Evaluating Physical Realism in Generated Video via Human Reasoning

Principia: Relational Physics Tests for Video Models

Apple- $π$ : Benchmarking Thinking with Video Towards Law-Grounded Physical Intelligence