VIGIL: Tackling Hallucination Detection in Image Recontextualization

Joanna Wojciechowicz; Maria Łubniewska; Jakub Antczak; Justyna Baczyńska; Wojciech Gromski; Wojciech Kozłowski; Maciej Zieba

cs.CVFeb 16, 2026

VIGIL: Tackling Hallucination Detection in Image Recontextualization

Authors: Joanna Wojciechowicz, Maria Łubniewska, Jakub Antczak, Justyna Baczyńska, Wojciech Gromski, Wojciech Kozłowski, Maciej Zieba

Organizations: Wroclaw University of Science and Technology · Tooploox

Abstract

We introduce VIGIL (Visual Inconsistency & Generative In-context Lucidity), a benchmark dataset and framework that provides a fine-grained categorization of hallucinations in the multimodal image recontextualization task for large multimodal models. Most existing methods treat hallucinations as a single undifferentiated error. We instead decompose them into five categories, namely Object Visual Fidelity, Background Fidelity, Spatial and Instructional Fidelity, Physical and Integration Fidelity, and Object Omission. We propose a multi-stage detection pipeline that processes recontextualized images through specialized steps targeting all five categories with a coordinated set of open-source models. We evaluate the pipeline on the VIGIL dataset of 1,269 manually annotated samples across five product domains and report macro-F1 per category. The decomposed pipeline reaches the best macro-F1 among open-source detectors. It returns a textual explanation for each detected error, which prior methods for this task do not provide. We openly release VIGIL, along with the detection pipeline and benchmark code, through our GitHub repository: https://github.com/mlubneuskaya/vigil and Data repository: https://huggingface.co/datasets/joannaww/VIGIL.

Figures & tables

Panel of Figure 2 — Figure 2: Exemplary sample from the VIGIL Benchmark Dataset. Each sample includes reference images for object 1 ( 2(a) ) and object 2 ( 2(b) ), the target background image ( 2(c) ), the final generated image ( 2(d) ), and the corresponding textual recontextualization prompt used for generation.

Figure 3: The first chart shows the distribution of the categories, while the second shows the proportion of different hallucination types considered in our framework.

Method	Clothing	Furniture	Cosmetics	Electronics	Cars
Gemini 2.5 Flash	0.4261	0.3696	0.2302	0.2890	0.4283
Qwen3.5-9B	0.2927	0.4592	0.3455	0.3046	0.3747
Gemma 3 12B IT	0.2283	0.4712	0.2932	0.3115	0.3743
CIGEval	0.1697	0.1845	0.1303	0.1845	0.2455
VIEScore	0.0348	0.0139	0.0034	0.0036	0.0952
VIGIL (ours)	0.3891	0.5064	0.3442	0.3029	0.4154

Table 1: Macro-F1 for multi-label classification (higher is better), using the best parameter set on each test category. The remaining categories were used for calibration. Bold values indicate the best score among all non-Gemini methods. Underlined values indicate the second-best score among all non-Gemini methods.

Method	Clothing	Furniture	Cosmetics	Electronics	Cars
Gemini 2.5 Flash	0.3185	0.2141	0.0911	0.1559	0.2724
Qwen3.5-9B	0.1528	0.1528	0.0633	0.0714	0.1683
Gemma 3 12B IT	0.0527	0.1175	0.0604	0.0427	0.0603
CIGEval	0.0756	0.0929	0.0415	0.0484	0.0977
VIEScore	0.0407	0.0103	0.0035	0.0022	0.0866
VIGIL (ours)	0.2455	0.1609	0.0902	0.0861	0.1461

Table 2: Macro-F1 for the semantic LLM-as-a-Judge evaluation (higher is better), using the best parameter set selected for each test category from the multi-label classification calibration. Bold values indicate the best score among all non-Gemini methods. Underlined values indicate the second-best score among all non-Gemini methods. Because Gemini is also used as the judge, these results are reported only as a reference.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

				Binary classification F1 score - all classes except
Threshold	Margin	Boxes	Padding	Cars	Clothes	Cosmetics	Electronics	Furniture
0.1	0.0	False	0.5	0.4070	0.3985	0.4282	0.4338	0.3875
0.1	0.0	False	1.0	0.4078	0.3996	0.4279	0.4346	0.3882
0.1	0.0	True	0.5	0.4026	0.4004	0.4221	0.4287	0.3806
0.1	0.0	True	1.0	0.4015	0.3995	0.4219	0.4277	0.3795
0.1	0.1	False	0.5	0.4099	0.4010	0.4285	0.4359	0.3889

Appendix

Table S1 : Experiment Results: Binary Classification F1 Scores

Figure S1 : Architecture of the multi-stage detection pipeline. Entity extraction, segmentation and embedding produce a paired representation of the sample, which is then consumed by five category-specific verification modules.

Category	Objects	Background	Spatial&Instructional	Physical&Integration	Object Omission
Furniture	73.3	10.5	51.4	41.9	8.6
Clothing	45.7	12.5	29.8	3.0	10.5
Cosmetics	23.8	15.2	28.3	51.1	0.0
Electronics	46.6	1.9	16.3	42.0	1.1
Cars	52.2	8.3	52.2	25.6	2.8

Appendix

Table S2 : Percentage of samples in each domain that exhibit at least one error of each top-level category. Rows correspond to the five semantic domains of the dataset, and columns correspond to the five categories of the taxonomy.

Figure S2 : Representative domain-specific failure cases. In the top row, for the Cars category, the generated vehicle preserves the global appearance of the reference car, but the license plate text is distorted, which illustrates a localized object-fidelity error. In the bottom row, for the Electronics category, the generated laptop loses fine-grained identity cues such as screen content, logo or text, and keyboard details, because of the low effective resolution of the generated crop.

Category	Det.% (C)	Det.% (I)	[email protected] (C)	[email protected] (I)	[email protected] (C)	[email protected] (I)
Background	71%	31%	0.701	0.712	0.725	0.718
Object Omission	94%	49%	0.742	0.942	0.690	0.797
Objects	42%	31%	0.546	0.423	0.611	0.500
Physical&Integration	4%	4%	0.464	0.038	0.500	0.071
Spatial&Instructional	33%	24%	0.526	0.327	0.607	0.425

Appendix

Table S3 : SAM 3 detection performance per error category for the manually annotated subset. Det.% is the fraction of images in each group where the pipeline correctly detected at least one instance of the hallucination type. C is SAM-correct and I is SAM-incorrect.

Metric	Med. (C)	Med. (I)	$p_{\text{adj}}$	Sig.
Background TP	0.115	0.022	0.336	ns
Background FP	0.288	0.800	0.000	∗∗∗
Background FN	0.058	0.089	0.884	ns
Background F1	0.283	0.011	0.041	∗
Object Omission TP	0.000	0.022	0.511	ns
Object Omission FP	0.058	0.911	0.000	∗∗∗

Appendix

Table S4 : Mann–Whitney U test comparing per-image pipeline error metrics between the SAM-correct (

n_{C}

) and SAM-incorrect (

n_{I}

) groups. The

p

-values are Holm–Bonferroni adjusted. For F1, only images with at least one positive instance are included (

n

varies). Significance levels are

{}^{***}p<0.001

{}^{*}p<0.05

, and ns not significant.

Panel of Figure 11 — Figure S3 : Exemplary system prompt used for the Gemini 2.5 Flash baseline. The prompt assigns a specific role and breaks the task into five distinct categories with semantic definitions, ensuring consistent evaluation.

Panel of Figure 12 — Figure S4 : System prompt used for the LLM-as-a-Judge semantic evaluation. The prompt instructs the model to extract and count True Positives (TP), False Positives (FP), and False Negatives (FN) based on a direct comparison between the human ground-truth descriptions and the AI pipeline predictions.

Method	Objects	Background	Spatial&Instructional	Physical&Integration	Object Omission
Gemini 2.5 Flash	0.1422	0.0442	0.4866	0.1347	0.5495
Qwen3.5-9B	0.1781	0.0284	0.1979	0.0532	0.2158
Gemma 3 12B IT	0.0578	0.0110	0.1132	0.1535	0.0233
CIGEval	0.1369	0.0066	0.0972	0.0145	0.1022
VIEScore	0.0074	0.0000	0.0000	0.0250	0.1750
VIGIL (ours)	0.2627	0.1449	0.2379	0.0603	0.1028

Appendix

Table S5 : Macro-F1 for the semantic LLM-as-a-Judge evaluation (higher is better) per hallucination category. Bold values indicate the best score among all non-Gemini methods. Because Gemini is also used as the judge, these results are reported only as a reference.

Hallucination Detection in VLMs

Hallucination detection in vision-language models identifies claims that are unsupported by images, video, or other visual evidence.

See all 72 papers

HalluScope: Fine-grained Hallucination Diagnosis for Multimodal Large Language Models
Jul 23, 2026Weilin Jin, Mingyu Wang, Wenbo Li +5Hallucination Detection in VLMs Multimodal Hallucination
Peking University Beijing, China · Joy Future Academy Beijing, China
Fine-Grained Multi Image Object Hallucination Benchmark
Aug 31, 2026Joonki Min, Chaeyun Kim, Hyungwook Choi +4VLM Evaluation Object Hallucination in VLMs
Seoul National University · AIM Intelligence
Beyond Language Priors: Diagnosing and Fixing Visual-Origin Hallucinations in Multimodal LLM
Aug 31, 2026Peiyang Xu, Xiaopei Zhu, Jun Zhu +1Object Hallucination in VLMs Multimodal Contrastive Learning
Tsinghua University Beijing, China

Jul 23, 2026J/K move · Enter open · S save

HalluScope: Fine-grained Hallucination Diagnosis for Multimodal Large Language Models

Weilin Jin, Mingyu Wang, Wenbo Li, Haoyang Huang+4

Peking University Beijing, China · Joy Future Academy Beijing, China

Hallucination Detection in VLMs Multimodal Hallucination LLM Hallucination Detection Multimodal Model Interpretability VLM Hallucination Mitigation

Open paper AskPDF arXiv

Although Multimodal Large Language Models have achieved strong performance across a wide range of vision-language tasks, they still suffer from hallucinations, where model outputs become inconsistent with the visual content, textual context, or commonsense knowledge. Existing studies primarily address this problem through coarse-grained detection. However, these approaches often provide insufficient diagnostic information for understanding hallucination types and supporting downstream hallucination mitigation. To bridge this gap, we propose fine-grained hallucination diagnosis for MLLMs, a new unified task that jointly performs hallucination detection, classification, and interpretable explanation generation. We develop an automated data generation pipeline and construct HalluScope-30K, a large-scale diagnostic dataset covering eight sources and five task categories. Based on this dataset, we design a multi-granular joint reward function and train two diagnosis models, HalluScope-4B and HalluScope-8B, which achieve state-of-the-art performance on both the MHALO benchmark and our fine-grained hallucination classification benchmark. Notably, detection and classification are mutually beneficial under joint optimization. Furthermore, diagnosis-driven feedback experiments show that the fine-grained diagnostic explanations produced by our model effectively guide target models to correct their hallucinations, with full diagnosis substantially outperforming all baselines on both Qwen3-VL-8B-Instruct and LLaVA-1.5-7B. Our code, data, and models are available at https://github.com/wkinglin/HalluScope.

Abstract

Hallucination Detection in VLMs

Explore similar work

HalluScope: Fine-grained Hallucination Diagnosis for Multimodal Large Language Models

Fine-Grained Multi Image Object Hallucination Benchmark

Beyond Language Priors: Diagnosing and Fixing Visual-Origin Hallucinations in Multimodal LLM