Multi-turn Physics-informed Vision-language Model for Physics-grounded Anomaly Detection

Yao Gu; Xiaohao Xu; Yingna Wu

cs.CVMar 16, 2026

Multi-turn Physics-informed Vision-language Model for Physics-grounded Anomaly Detection

Authors: Yao Gu, Xiaohao Xu, Yingna Wu

Organizations: Shanghaitech University · University of Michigan, Ann Arbor

Abstract

Vision-Language Models (VLMs) demonstrate strong general-purpose reasoning but remain limited in physics-grounded anomaly detection, where causal understanding of dynamics is essential. Existing VLMs, trained predominantly on appearance-centric correlations, fail to capture kinematic constraints, leading to poor performance on anomalies such as irregular rotations or violated mechanical motions. We introduce a physics-informed instruction tuning framework that explicitly encodes object properties, motion paradigms, and dynamic constraints into structured prompts. By delivering these physical priors through multi-turn dialogues, our method decomposes causal reasoning into incremental steps, enabling robust internal representations of normal and abnormal dynamics. Evaluated on the Phys-AD benchmark, our approach achieves 96.7% AUROC in video-level detection--substantially outperforming prior SOTA (66.9%)--and yields superior causal explanations (0.777 LLM score). This work highlights how structured physics priors can transform VLMs into reliable detectors of dynamic anomalies.

Figures & tables

Figure 1: Comparison of (a) conventional instruction fine-tuning and (b) our multi-turn physics-informed approach. Our method uses multi-turn dialogue to incorporate structured physical information as prior knowledge, enhancing the LLM’s autoregressive generation with improved logic and structure. This universal physical prior reduces dataset construction burden.

Figure 2: The construction process of our structured physics information. This explicit knowledge base, $\mathcal{P}_{c}=(S_{com},S_{dyn},S_{mot})$ , guides the model through a logical reasoning chain for physics-grounded anomaly detection.

Figure 3: Our model architecture for physics-grounded anomaly detection and explanation. Based on Video-LLaVA, only the LLM components (QKV matrices and FFN layer) are fine-tuned, while the vision tower and multi-modal projector remain frozen. The multi-turn dialogue injects physical priors, enabling robust anomaly detection and causal explanation in two stages.

Category	LAVAD	ZS Clip	ZS ImageBind	Video-ChatGPT	Video-LLaMA	Video-LLaVA	Ours
Car	0.557	0.500	0.500	0.500	0.678	0.522	0.999
Fan	0.510	0.500	0.500	0.549	0.592	0.611	0.912
Rolling Bearing	0.532	0.500	0.500	0.300	0.500	0.500	1.000
Spherical Bearing	0.435	0.500	0.500	0.450	0.550	0.500	1.000
Servo	0.502	0.500	0.500	0.506	0.683	0.464	0.926
Clip	0.516	0.500	0.500	0.669	0.556	0.458	0.974

Table 1: Video-level AUROC (

\uparrow

) result on Phys-AD.

Category	Video-ChatGPT		Video-LLaMA		Video-LLaVA		Ours
Category	SBERT	LLM	SBERT	LLM	SBERT	LLM	SBERT	LLM
Car	0.703	0.074	0.577	0.108	0.580	0.052	0.758	0.875
Fan	0.775	0.234	0.715	0.557	0.619	0.135	0.844	0.875
Rolling Bearing	0.755	0.330	0.583	0.200	0.729	0.268	0.746	0.997
Spherical Bearing	0.836	0.050	0.634	0.158	0.826	0.213	0.883	1.000
Servo	0.738	0.255	0.642	0.304	0.706	0.277	0.743	0.491

Table 2: Video-level SBERT (

\uparrow

) result on Phys-AD dataset.

Figure 4: Qualitative comparison of anomaly explanation methods on a ‘leaking ball’ scene. Our multi-turn physics-informed VLM provides a more accurate and causally relevant explanation than prior SOTA methods.

	Car	Fan	Rolling Bearing	Spherical Bearing	Servo	Clip	USB	Hinge	Sticky Roller	Ball	Magnet
No-phys	0.994	0.912	1.000	1.000	0.877	1.000	0.958	0.557	1.000	0.935	1.000
Single-turn	1.000	0.868	1.000	1.000	0.898	1.000	0.969	0.500	0.500	0.919	0.500
Multi-turn	0.999	0.912	1.000	1.000	0.926	0.974	0.964	0.818	1.000	0.973	1.000

Table 3: Detection performance comparison between single-turn and multi-turn conversation. Evaluated with video-level AUROC (

\uparrow

) result of 21 categories on Phys-AD dataset. Failure cases (AUROC

\approx

0.5) are highlighted in red .

Explore similar work

Video-HOCA: A Diagnostic Benchmark for Physical Anomaly Reasoning in Video-LLMs
Feb 23, 2026Chang Liu, Yunfan Ye, Qingyang Zhou +5VLM Reasoning Physical Reasoning in Video
National University of Defense Technology, Changsha, China · Hunan University, Changsha, China
LATERN: Test-Time Context-Aware Explainable Video Anomaly Detection
May 14, 2026Mitchell Piehl, Muchao YeInterpretable Anomaly Detection VLM Reasoning
The University of Iowa, Iowa City IA 52245, USA
From Priors to Perception: Grounding Video-LLMs in Physical Reality
May 6, 2026Zicheng Zhao, Chaofan Gan, Shijie Li +1VLM Adaptation VLM Reasoning
Shanghai Jiao Tong University

Feb 23, 2026J/K move · Enter open · S save

Video-HOCA: A Diagnostic Benchmark for Physical Anomaly Reasoning in Video-LLMs

Chang Liu, Yunfan Ye, Qingyang Zhou, Xichen Tan+4

National University of Defense Technology, Changsha, China · Hunan University, Changsha, China

VLM Reasoning Physical Reasoning in Video Video-Language Model Evaluation Causal Reasoning in Videos

Open paper AskPDF arXiv

We introduce Video-HOCA, a diagnostic benchmark for physical anomaly reasoning in videos. Video-HOCA uses an Ontological-Causal taxonomy to distinguish violations of an entity's own properties or capabilities from violations of physical relations among entities and the environment. It contains more than 1,400 generated and real-world videos and 3,470 question-answer pairs, with human verification of labels and reference answers. The benchmark evaluates four levels of reasoning: plausibility checking, anomaly attribution, fine-grained recognition, and open-ended physical explanation. Across 20 Instruct-mode Video-LLMs, we find that recognition outpaces explanation: Task I scores cluster at 75-88, while Task II macro-F1 stays mostly below 50. We also find that the Ontological-Causal gap depends on the task and model configuration, and that Thinking-mode gains are not explained by sampling or output budget alone. Annotation agreement, Task-IV human-judge and judge-judge checks, alternative metrics, and temporal/decoding controls validate the evaluation pipeline and bound the claims supported by the benchmark.

Abstract

Explore similar work

Video-HOCA: A Diagnostic Benchmark for Physical Anomaly Reasoning in Video-LLMs

LATERN: Test-Time Context-Aware Explainable Video Anomaly Detection

From Priors to Perception: Grounding Video-LLMs in Physical Reality