INFACT: A Diagnostic Benchmark for Induced Faithfulness and Factuality Hallucinations in Video-LLMs

Junqi Yang; Yuecong Min; Jie Zhang; Shiguang Shan; Xilin Chen

cs.CVMar 12, 2026

INFACT: A Diagnostic Benchmark for Induced Faithfulness and Factuality Hallucinations in Video-LLMs

Authors: Junqi Yang, Yuecong Min, Jie Zhang, Shiguang Shan, Xilin Chen

Organizations: State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences · School of Advanced Interdisciplinary Sciences, UCAS

Abstract

Despite rapid progress, Video Large Language Models (Video-LLMs) remain unreliable due to hallucinations, which are outputs that contradict either video evidence (faithfulness) or verifiable world knowledge (factuality). Existing benchmarks provide limited coverage of factuality hallucinations and predominantly evaluate models only in clean settings. We introduce INFACT, a diagnostic benchmark comprising 9{,}800 QA instances with fine-grained taxonomies for faithfulness and factuality, spanning real and synthetic videos. INFACT evaluates models in four modes: Base (clean), Visual Degradation, Evidence Corruption, and Temporal Intervention for order-sensitive items. Reliability under induced modes is quantified using Resist Rate (RR) and Temporal Sensitivity Score (TSS). Experiments on 14 representative Video-LLMs reveal that higher Base-mode accuracy does not reliably translate to higher reliability in the induced modes, with evidence corruption reducing stability and temporal intervention yielding the largest degradation. Notably, many open-source baselines exhibit near-zero TSS on factuality, indicating pronounced temporal inertia on order-sensitive questions.

Figures & tables

Figure 1: Examples of Faithfulness and Factuality Hallucinations. Top: Faithfulness items are verified by video evidence, covering Static Entities & Attributes, Dynamic Actions & Motions, and Spatio-Temporal Relations. Bottom: Factuality items require consistency with world knowledge and cover Domain Knowledge (Know-WHAT), Procedural Knowledge (Know-HOW), and Physical Knowledge (Know-WHY).

Benchmark	# Ques. / # Videos	Faithfulness	Factuality	Source	Visual Degradation		Evidence Corruption		Temporal Intervention
Benchmark	# Ques. / # Videos	Faithfulness	Factuality	Source	Motion Blur	Gaussian Noise	Caption Injection	Adversarial Noise	Shuffle	Reverse
VIDHAL Choong et al. (2024)	– / 400	✓(5)	✗	Real	✗	✗	✗	✗	✗	✗
EventHallusion Zhang et al. (2024)	– / 400	✓(3)	✗	Real	✗	✗	✗	✗	✗	✗
VideoHallucer Wang et al. (2024b)	1,800 / 948	✓(3)	✓(3)	Real	✗	✗	✗	✗	✗	✗
VIDHALLUC Li et al. (2025a)	9,295 / 5,002	✓(3)	✗	Real	✗	✗	✗	✗	✗	✗
VideoHallu Li et al. (2025b)	3,233 / 3,233	✓(2)	✓(4)	Synthetic	✗	✗	✗	✗	✗	✗

Table 1: Comparison of INFACT with recent hallucination benchmarks for video understanding. INFACT covers video-grounded faithfulness and world-knowledge-grounded factuality with fine-grained taxonomies. It also supports controlled evaluation modes for visual degradation, evidence corruption, and temporal intervention within a unified protocol.

Figure 2: Overview of the INFACT construction process. Left: Candidate videos and QA pairs are collected from multiple sources, including video QA datasets, instructional datasets, and synthetic videos. Middle: Samples are organized into fine-grained faithfulness and factuality dimensions, and filtered to remove ambiguous or non-video-grounded items, followed by human-in-the-loop quality verification. Right: The resulting benchmark supports four evaluation modes: Base, Visual Degradation, Evidence Corruption, and Temporal Intervention.

Figure 3: Dataset composition of INFACT. Distribution over the fine-grained taxonomy for faithfulness (top) and factuality (bottom).

			Evidence Corruption				Visual Degradation				Temporal Intervention
Model	Text-only	Base	$RR_{\mathrm{adv}}$	$RR_{\mathrm{cap}}$	$RR_{\mathrm{sub}}$	$RR_{\mathrm{cor}}$	$RR_{\mathrm{cmp}}$	$RR_{\mathrm{gau}}$	$RR_{\mathrm{mb}}$	$RR_{\mathrm{deg}}$	$TSS_{\mathrm{shu}}$	$TSS_{\mathrm{rev}}$	$\bar{\mathrm{TSS}}$	Avg Score
PLLaVA-13B Xu et al. (2024)	0.285	0.458	0.782	0.678	0.922	0.794	0.777	0.813	0.699	0.763	0.010	0.014	0.012	0.523
VideoLLaMA2-7B Cheng et al. (2024)	0.261	0.434	0.751	0.564	0.910	0.742	0.818	0.725	0.749	0.764	0.113	0.091	0.102	0.536
ShareGPT4Video-8B Chen et al. (2024)	0.292	0.462	0.719	0.688	0.894	0.767	0.840	0.762	0.754	0.785	0.101	0.094	0.098	0.550
PLLaVA-34B Xu et al. (2024)	0.287	0.505	0.757	0.618	0.920	0.765	0.772	0.874	0.848	0.831	0.070	0.073	0.072	0.556
Tarsier-34B Wang et al. (2024a)	0.302	0.568	0.785	0.644	0.960	0.796	0.759	0.846	0.612	0.739	0.158	0.208	0.183	0.573

Table 2: Faithfulness results on INFACT. Text-only: question-only accuracy. Base: Mode I. RR: Modes II–III. TSS: Mode IV (order-sensitive subset). Models are grouped by availability and sorted by Avg Score.

			Evidence Corruption				Visual Degradation				Temporal Intervention
Model	Text-only	Base	$RR_{\mathrm{adv}}$	$RR_{\mathrm{cap}}$	$RR_{\mathrm{sub}}$	$RR_{\mathrm{cor}}$	$RR_{\mathrm{cmp}}$	$RR_{\mathrm{gau}}$	$RR_{\mathrm{mb}}$	$RR_{\mathrm{deg}}$	$TSS_{\mathrm{shu}}$	$TSS_{\mathrm{rev}}$	$\bar{\mathrm{TSS}}$	Avg Score
VideoLLaMA2-7B Cheng et al. (2024)	0.272	0.407	0.676	0.590	0.917	0.728	0.818	0.723	0.772	0.771	0.000	0.000	0.000	0.500
ShareGPT4Video-8B Chen et al. (2024)	0.294	0.489	0.715	0.519	0.883	0.706	0.840	0.816	0.798	0.818	0.000	0.000	0.000	0.508
PLLaVA-13B Xu et al. (2024)	0.279	0.410	0.788	0.585	0.919	0.764	0.777	0.947	0.705	0.810	0.000	0.000	0.000	0.525
NVILA-8B Liu et al. (2024c)	0.291	0.424	0.818	0.541	0.961	0.773	0.797	0.888	0.759	0.815	0.000	0.000	0.000	0.529
PLLaVA-34B Xu et al. (2024)	0.265	0.431	0.782	0.603	0.920	0.768	0.772	0.897	0.812	0.827	0.000	0.000	0.000	0.532

Table 3: Factuality results on INFACT. Same evaluation protocol and metrics as Table 2 .

Figure 4: Base accuracy vs. average reliability score under inductions. Base accuracy is measured in Mode I and averaged over faithfulness and factuality. The average reliability score under induction aggregates RR over Modes II–III and TSS over Mode IV.

Figure 5: Comparison of four representative models on fine-grained evaluation dimensions. The left radar plot shows performance on faithfulness dimensions, while the right radar plot shows performance on factuality dimensions. Each axis corresponds to a fine-grained category in the INFACT taxonomy, and each curve represents one representative model. Higher values indicate better performance on the corresponding dimension.

Figure 6: Effect of the number of sampled frames. Left: Base accuracy under

\{2,8,16,24,32\}

uniformly sampled frames. Right: induced metrics (

RR_{\mathrm{deg}}

RR_{\mathrm{cor}}

, TSS) under

\{8,16,24,32\}

frames.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Open-Source Video-LLMs	HF Checkpoint
VideoLLaMA2-7B	DAMO-NLP-SG/VideoLLaMA2-7B
Qwen2.5VL-7B	Qwen/Qwen2.5-VL-7B-Instruct
Qwen3VL-8B	Qwen/Qwen3-VL-8B-Instruct
ShareGPT4Video-8B	Lin-Chen/sharegpt4video-8b
NVILA-8B	Efficient-Large-Model/NVILA-8B
InternVL3-8B	OpenGVLab/InternVL3-8B

Appendix

Table A1: Simplified configurations for open-source multimodal foundation models used in the evaluation.

Model	TSS ( $\mathcal{S}_{\text{order}}$ )	TSS ( $\mathcal{S}_{\text{nonorder}}$ )
Qwen2.5-VL-7B	20.5%	4.5%
Qwen2.5-VL-32B	19.5%	6.0%
Qwen3-VL-8B	28.0%	9.0%
Qwen3-VL-32B	25.0%	8.0%
InternVL3-8B	14.0%	6.0%
InternVL3.5-8B	22.5%	5.5%

Appendix

Table A2: TSS on order-sensitive vs. non-order-sensitive items. Models with non-trivial temporal responsiveness show meaningful separation between subsets.

Operator	Answerability	Label-pres.
Compression (cmp)	99.5%	100.0%
Subtitle corruption (sub)	99.0%	99.5%
Gaussian noise (gau)	100.0%	100.0%
Motion blur (mb)	100.0%	99.5%
Adversarial noise (adv)	98.5%	99.5%

Appendix

Table A3: Human validity audit for invariant-label operators. Each operator is audited on 200 perturbed video–QA pairs with three blinded annotators and majority-vote aggregation. High answerability and label-preservation rates indicate strong agreement between the automatic invariant-label evaluation and human judgments.

Subtype	Answerability	Label-pres.
Irrelevant	100.0%	100.0%
Misleading	94.0%	94.5%

Appendix

Table A4: Caption injection validity by subtype. Irrelevant captions fully preserve answerability and labels. Misleading captions show slightly lower rates, consistent with their adversarial design.

Task & Definition	Example
1. Static Entities & Attributes
1.1 Entity Recognition Identify an entity label supported by video evidence.	Which human organ is visible in the video? (A) Stomach (B) Liver (C) Lungs (D) Kidneys
1.2 Unique Entity Counting Count the number of distinct entity instances.	How many babies does the lion mother have in the video? (A) 4 (B) 3 (C) 2 (D) 5
1.3 Temporal Attributes Recognition Attribute query with an explicit temporal/event anchor.	What is the color of the second object that enters the scene? (A) Brown (B) Cyan (C) Gray (D) Purple
1.4 Static Attributes Recognition Attribute query without explicit temporal/event anchor.	What color is the T-shirt worn by the boy? (A) Yellow (B) White (C) Black (D) Blue
1.5 Scene Text Recognition Detecting and reading text that appears in the video.	What is written on the first made keychain? (A) Google (B) YouTube (C) Facebook (D) Spotify

Appendix

Table A5: Faithfulness Hallucination Taxonomy.

Category & Definition	Example
1. Domain Knowledge (Know-WHAT)
1.1 Cultural Event Recognition Knowledge of traditions, festivals, artistic heritage, and customs.	When is the festival in the video usually celebrated? (A) 9th day of 9th lunar month (B) 8th day of 12th lunar month (C)5th day of 5th lunar month (D) 1st day of 1st lunar month
1.2 Historical Background Identification Knowledge of past events, chronologies, archaeology, and historical figures.	When was the Site shown in the video discovered? (A) 1928-1929 (B) 1925-1926 (C) 1920-1922 (D) 1934-1938
1.3 Geospatial Localization Knowledge of geospatial locations, landmarks, and regional characteristics.	Which city is this video most likely recorded in? (A) San Diego (B) Phoenix (C) Tucson (D) Las Vegas
1.4 Entertainment-Related Recognition Knowledge of movies, TV, music, celebrities, and sports.	What genre of movie is portrayed in the video? (A) Reality TV (B) Documentary (C) Thriller (D) Romance
2. Procedural Knowledge (Know-HOW)

Appendix

Table A6: Factual Hallucination Taxonomy. The framework evaluates world knowledge across three dimensions: Domain Knowledge (Know-WHAT), Procedural Knowledge (Know-HOW), and Physical Knowledge (Know-WHY).

Video-Language Model Evaluation

Video-language model evaluation measures whether multimodal systems understand, ground, and reliably use visual and audio evidence over time.

See all 99 papers

MoHallBench: A Benchmark for Motion Hallucination in Video Large Language Models
Jul 1, 2026Sihan Chen, Jiale Li, Jianghang Lin +1VLM Evaluation Hallucination Detection in VLMs
State Key Laboratory of General Artificial Intelligence, Peking University, Shenzhen Graduate School, Shenzhen, China · South China University of Technology, Guangzhou, China · Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, Xiamen, China
No Place to Hide: Benchmarking Video Hallucination with Background-Controlled Pairs
Jun 30, 2026Haojian Huang, Harold Haodong Chen, Meng Luo +6VLM Evaluation Multimodal Hallucination
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · Knowin AI, China · National University of Singapore, Singapore +4
Catching Hallucinated Citations in Video-LLM Question Answering: A Self-Verification Pipeline and Verifier Ablation Study
Aug 16, 2026Yogesh KumarHallucination Detection in VLMs Video QA

Jul 1, 2026J/K move · Enter open · S save

MoHallBench: A Benchmark for Motion Hallucination in Video Large Language Models

Sihan Chen, Jiale Li, Jianghang Lin, Mengyuan Liu

State Key Laboratory of General Artificial Intelligence, Peking University, Shenzhen Graduate School, Shenzhen, China · South China University of Technology, Guangzhou, China · Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, Xiamen, China

VLM Evaluation Hallucination Detection in VLMs Video-Language Models

Open paper AskPDF arXiv

Video Large Language Models (VideoLLMs) have shown strong progress in video understanding, yet they still suffer from hallucinations that are inconsistent with visual evidence. Existing benchmarks mainly focus on object hallucination or coarse action perception, leaving a key video-specific problem underexplored: motion hallucination, in which models infer human motions that are absent from the video. We present MoHallBench, a benchmark for diagnosing motion hallucination in VideoLLMs. MoHallBench systematically evaluates three major sources of hallucination: co-occurrence priors, sequential inference, and similarity confusion. It contains 11,306 video clips and 40,493 question-answer pairs, covering binary-choice, multiple-choice, and generative settings. We further introduce a bi-directional questioning protocol with bias-aware metrics to reduce affirmation bias in binary evaluation. Experiments on ten recent open-source VideoLLMs reveal a clear decoupling between action recognition and hallucination resistance, as models that perform well on positive action recognition often fail on adversarial negatives. Among all settings, sequential inference hallucination is the most severe, showing that current models tend to over-infer expected outcomes from partial motion cues. Our analyses further confirm that stronger priors and finer-grained similarity substantially amplify hallucination. We hope MoHallBench can facilitate future evaluation and mitigation of motion hallucination in VideoLLMs.

Abstract

Video-Language Model Evaluation

Explore similar work

MoHallBench: A Benchmark for Motion Hallucination in Video Large Language Models

No Place to Hide: Benchmarking Video Hallucination with Background-Controlled Pairs

Catching Hallucinated Citations in Video-LLM Question Answering: A Self-Verification Pipeline and Verifier Ablation Study