Video-HOCA: A Diagnostic Benchmark for Physical Anomaly Reasoning in Video-LLMs

Chang Liu; Yunfan Ye; Qingyang Zhou; Xichen Tan; Mengxuan Luo; Zhenyu Qiu; Wei Peng; Zhiping Cai

cs.CVFeb 23, 2026

Video-HOCA: A Diagnostic Benchmark for Physical Anomaly Reasoning in Video-LLMs

Authors: Chang Liu, Yunfan Ye, Qingyang Zhou, Xichen Tan, Mengxuan Luo, Zhenyu Qiu, Wei Peng, Zhiping Cai

Organizations: National University of Defense Technology, Changsha, China · Hunan University, Changsha, China

Abstract

We introduce Video-HOCA, a diagnostic benchmark for physical anomaly reasoning in videos. Video-HOCA uses an Ontological-Causal taxonomy to distinguish violations of an entity's own properties or capabilities from violations of physical relations among entities and the environment. It contains more than 1,400 generated and real-world videos and 3,470 question-answer pairs, with human verification of labels and reference answers. The benchmark evaluates four levels of reasoning: plausibility checking, anomaly attribution, fine-grained recognition, and open-ended physical explanation. Across 20 Instruct-mode Video-LLMs, we find that recognition outpaces explanation: Task I scores cluster at 75-88, while Task II macro-F1 stays mostly below 50. We also find that the Ontological-Causal gap depends on the task and model configuration, and that Thinking-mode gains are not explained by sampling or output budget alone. Annotation agreement, Task-IV human-judge and judge-judge checks, alternative metrics, and temporal/decoding controls validate the evaluation pipeline and bound the claims supported by the benchmark.

Figures & tables

Figure 1: Video-HOCA overview. The benchmark organizes physical anomalies into Ontological (entity-centered) and Causal (relation-centered) branches, totaling 9 L2 groups and 18 L3 tags (full hierarchy in Appendix Table 4 ); spatial and temporal cases span both generated anomaly videos and semantically matched real videos. The four main tasks move from plausibility checking to category attribution, fine-grained anomaly recognition, and open-ended physical explanation.

Figure 2: Construction pipeline. We use both text-source candidates, where existing benchmarks provide videos with textual annotations, and video-source candidates, where raw generated clips require frame-wise captioning, anomaly summarization, operational taxonomy attribution, schema checking, and human review. Real-world clips are filtered and semantically matched to anomaly clips so that Task I compares physically plausible and implausible videos under similar scene content.

Figure 3: Structured task design in Video-HOCA. The benchmark evaluates physical anomaly reasoning through four progressive tasks: (Task I) Plausibility Check, (Task II) Anomaly Category Attribution, (Task III) Fine-grained Anomaly Recognition, and (Task IV) Open-ended Physical Reasoning. The sequence probes model capability from coarse plausibility detection to fine-grained recognition and mechanism-level explanation.

Figure 4: Task-IV failure modes on a coffee-level anomaly. All models observe the same video, where coffee is continuously dispensed but the liquid level does not rise. Green marks the correct conservation-based mechanism, red marks wrong or misleading mechanisms, and yellow marks partially correct but imprecise event descriptions. The example shows that models may recognize visible activity while missing or misattributing the physical mechanism.

Model	Task I		Task II		Task III		Task IV		Task IV (D)		Overall
	ACC	F1	Onto.	Caus.	Onto.	Caus.	Onto.	Caus.	Onto.	Caus.
Human	–	–	–	–	96.0		81.4		84.9		–
Open-weight dense models ( $<$ 7B)
InternVL-3.5-2B	61.7	69.6	25.0	49.6	70.4	55.6	40.9	47.1	39.7	44.6	43.3
Qwen3-VL-2B	71.7	75.0	34.9	46.6	73.1	67.8	50.2	51.0	44.6	44.6	48.0
Open-weight dense models ( $\geq$ 7B)

Table 1: Main Instruct-mode evaluation results on Video-HOCA. Task I reports binary-plausibility ACC/F1 (%); Task II and Task III report macro-F1 (%) on Ontological–Causal subsets; Task IV reports rubric-based judge scores (0–100) decomposed into Ontological–Causal dimensions. Task IV (D) uses DeepSeek-V4-Flash as the judge. “Overall” first standardizes each main task score with z-scores computed over the main-table models, averages the four standardized scores, and linearly rescales the result as

50+10z

for readability. Bold indicates the best model result in each column, and underlining indicates the second best; the Human row is excluded from this marking. All rows use deterministic Instruct-mode decoding; Thinking-mode results are analyzed separately in § 4.4 .

Branch	Temporal type	Videos	Task III	Task IV
Ontological	Static	130	71.2	63.9
	Dynamic	124	68.7	63.6
Causal	Static	128	69.4	60.3
	Dynamic	248	59.9	53.0
Static gap (Onto. – Caus.)	–	–	+1.8	+3.6

Table 2: Static/dynamic analysis by anomaly branch. Scores are averaged over the main model set. Static Causal cases remain below Static Ontological cases, so the Ontological–Causal gap is not reducible to temporal demand alone.

Model	Setting	Task I		Task II		Task III		Task IV		Diag. Overall
Model	Setting	ACC	F1	Onto.	Caus.	Onto.	Caus.	Onto.	Caus.	Diag. Overall
GLM-4.6V-106B-FP8	Instruct	69.8	76.2	31.5	21.9	24.9	18.0	71.0	58.9	41.8
GLM-4.6V-106B-FP8	Instruct-Align	68.4	75.1	36.2	36.9	32.5	26.8	73.6	60.7	44.6
GLM-4.6V-106B-FP8	Thinking	68.5	75.7	45.7	59.8	78.9	72.6	78.3	62.0	54.2
Qwen3.5-122B-A10B-FP8	Instruct	88.0	87.5	45.0	41.0	85.9	79.2	73.1	61.4	57.6
Qwen3.5-122B-A10B-FP8	Instruct-Align	86.7	87.7	44.6	35.4	88.6	80.3	68.0	61.5	56.9

Table 3: Comparison of Instruct, Instruct-Align, and native Thinking decoding on Video-HOCA. Task I reports ACC/F1 (%); Task II and Task III report Ontological–Causal macro-F1 (%); Task IV reports rubric-based Ontological–Causal judge scores (0–100). Diag. Overall uses the same fixed main-table z-score normalization as Table 1 ; bold marks the best setting within each model block.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Video-HOCA annotation interface. The browser-based review tool used by annotators. It pairs a video player with editable fields for taxonomy type, category, sub-category, scene description, anomaly phenomenon, principle/expectation, and an optional temporal-localization range. The right panel tracks per-category dataset progress.

Figure 6: Taxonomy distribution. The sunburst chart visualizes the Ontological–Causal branches and their fine-grained L2 categories.

Figure 7: Video-source distribution. The generative sources used for Video-HOCA anomaly clips, supporting the source-diversity statistics reported in the main text.

Figure 8: Temporal annotation example. Annotated time intervals for the temporal-grounding probe. The probe localizes when anomalous events occur and is auxiliary to the main Ontological–Causal claims.

L2 group	L3 tag	Primary violation	Canonical example
Ontological anomalies (entity-centered violations)
Property	Shape	Geometrically impossible or contradictory shape.	Three-headed goat.
	Size	Object scale violates physical bounds.	Tiny elephant on a dinner plate.
	Visual Material	Surface appearance inconsistent with entity identity.	Turtle with a ruby-like shell; transparent wood.
State	Object Permanence	Spontaneous appearance or disappearance of the whole entity.	Chair appears out of thin air.
	Morphing	Shape or size changes without external cause.	Paper grows longer by itself.

Appendix

Table 4: Full Video-HOCA taxonomy. L1 is the Ontological–Causal split; L2 is the 9 anomaly groups; L3 lists the fine-grained tags actually used as labels. For the four L2 groups with no further subdivision (Bio-Behavior, Mass and Volume Consistency, Thermodynamics, Chemistry, Optics), the L3 tag is identical to the L2 name. Canonical examples are selected to have a stable primary violated unit.

Model	Temporal setting	Frames	Task I		Task II		Task III		Task IV
Model	Temporal setting	Frames	ACC	F1	Onto.	Caus.	Onto.	Caus.	Onto.	Caus.
Qwen3.5-122B-A10B-FP8	Original order	16f	88.0	87.5	45.0	41.0	85.9	79.2	73.1	61.4
Qwen3.5-122B-A10B-FP8	Middle frame	1f	80.6	82.0	39.5	40.2	77.9	70.7	58.0	59.0
Qwen3.5-122B-A10B-FP8	Shuffled order	16f	87.9	87.2	44.4	36.8	81.8	75.5	68.5	59.0

Appendix

Table 6: Temporal frame-order ablation on Video-HOCA. The experiment compares the original 16-frame input, a single middle-frame input, and a shuffled 16-frame input. Task I reports ACC/F1 (%); Task II and Task III report Ontological–Causal scores (%); Task IV reports rubric-based Ontological–Causal judge scores (0–100). Bold marks the best temporal setting in each column.

Model	Subset	Videos	Task I	Task II	Task III	Task IV
Labeled videos: Static=258, Dynamic=372
Open-weight dense models ( $<$ 7B)
InternVL-3.5-2B	Static	258	48.1	41.1	65.5	49.9
	Dynamic	372	27.2	39.8	57.0	41.4
Qwen3-VL-2B	Static	258	70.9	52.7	76.7	56.1
	Dynamic	372	49.7	37.4	65.1	47.0

Appendix

Table 7: Static vs. dynamic subset analysis on Video-HOCA. The subset labels come from static_dynamic_labels.jsonl . Only labeled anomaly videos are included, so Task I is reported as subset accuracy rather than F1. Task II/III report exact-match accuracy (%), and Task IV reports the rubric-based judge score (0–100).

Model	GPT-OSS z-score	DeepSeek z-score	GPT-OSS weighted	DeepSeek weighted
Qwen3.5-27B	61.2 (1)	61.3 (1) $\cdot$	74.4 (1) $\cdot$	73.4 (1) $\cdot$
Gemini-3-Flash	60.2 (2)	60.6 (2) $\cdot$	73.5 (2) $\cdot$	73.3 (2) $\cdot$
Qwen3.5-122B-A10B-FP8	57.6 (3)	57.5 (3) $\cdot$	68.9 (5) $\downarrow$ 2	67.6 (5) $\downarrow$ 2
Qwen3-VL-32B	57.4 (4)	57.4 (4) $\cdot$	70.7 (3) $\uparrow$ 1	69.4 (3) $\uparrow$ 1
Qwen3-VL-30B-A3B	55.6 (5)	55.2 (6) $\downarrow$ 1	68.7 (6) $\downarrow$ 1	66.5 (6) $\downarrow$ 1
Gemini-2.5-Flash	55.1 (6)	55.7 (5) $\uparrow$ 1	69.0 (4) $\uparrow$ 2	69.3 (4) $\uparrow$ 2

Appendix

Table 8: Ranking robustness under alternative Task IV judges and aggregation rules. Each cell reports score (rank); arrows show rank changes relative to the GPT-OSS z-score setting. The z-score variants standardize Task I F1, count-weighted Task II/III macro-F1, and the selected Task IV judge score over the same main-table models. The weighted variants use raw scores with weights 0.2/0.2/0.2/0.4 for Tasks I–IV.

Explore similar work

PhyCheck: Fine-Grained Evidence-Grounded Dataset for Physical Law Understanding in Video-LLMs
Aug 3, 2026Zhongjie Ba, Shengwang Xu, Peng Cheng +4Video-Language Model Evaluation Video QA
State Key Lab. of Blockchain and Data Security, Zhejiang University, Hangzhou, China · Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security Hangzhou, Zhejiang, China · Mohamed Bin Zayed University of Artificial Intelligence, Abu Dhabi, United Arab Emirates
From Priors to Perception: Grounding Video-LLMs in Physical Reality
May 6, 2026Zicheng Zhao, Chaofan Gan, Shijie Li +1VLM Adaptation VLM Reasoning
Shanghai Jiao Tong University
Video-Oasis: Rethinking Evaluation of Video Understanding
Mar 31, 2026Geuntaek Lim, Sungjune Park, Jaeyun Lee +5VLM Evaluation Temporal Video Understanding
Sejong University, South Korea · NAVER Cloud, South Korea

Aug 3, 2026J/K move · Enter open · S save

PhyCheck: Fine-Grained Evidence-Grounded Dataset for Physical Law Understanding in Video-LLMs

Zhongjie Ba, Shengwang Xu, Peng Cheng, Jinyang Zou+3

State Key Lab. of Blockchain and Data Security, Zhejiang University, Hangzhou, China · Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security Hangzhou, Zhejiang, China · Mohamed Bin Zayed University of Artificial Intelligence, Abu Dhabi, United Arab Emirates

Video-Language Model Evaluation Video QA Physical Reasoning in Video

Open paper AskPDF arXiv

Embodied intelligence and world models require video understanding systems to go beyond recognizing objects and actions and develop an understanding of physical regularities. However, despite their strong performance on general video understanding tasks, current video-language models still struggle to reliably determine whether an observed event conforms to specific physical laws. Existing benchmarks primarily assess the physical quality of generated videos, providing limited support for systematically evaluating and improving the physical-law understanding of Video Large Language Models (VideoLLMs). To address this gap, we introduce PhyCheck, a video question answering dataset organized at two complementary levels of granularity. The coarse-grained subset asks models to determine whether the phenomenon shown in a video conforms to or violates physical laws, while the fine-grained subset further examines whether models can capture physical details responsible for the violation or compliance. We use these subsets as structured supervision to improve physical understanding. In addition, the dataset contains a diagnostic subset with external causal context that reveal hidden factors affecting physical plausibility, assessing whether models can recalibrate their judgments accordingly. Experiments with Fine-tune Qwen2.5-VL show that training with the proposed data substantially improves the understanding of physical-consistency, while evaluations in the diagnostic subset reveal that current models still have difficulty incorporating additional causal conditions into their decisions. These findings highlight the gap between recognizing surface-level inconsistencies and understanding underlying physical mechanisms, and provide a foundation for evaluating and improving physical understanding in Video-LLMs.

Abstract

Explore similar work

PhyCheck: Fine-Grained Evidence-Grounded Dataset for Physical Law Understanding in Video-LLMs

From Priors to Perception: Grounding Video-LLMs in Physical Reality

Video-Oasis: Rethinking Evaluation of Video Understanding