DetailVerifyBench: A Benchmark for Dense Hallucination Localization in Long Image Captions

Xinran Wang; Yuxuan Zhang; Xiao Zhang; Haolong Yan; Muxi Diao; Songyu Xu; Zhonghao Yan; Hongbing Li

cs.CVApr 7, 2026

DetailVerifyBench: A Benchmark for Dense Hallucination Localization in Long Image Captions

Authors: Xinran Wang, Yuxuan Zhang, Xiao Zhang, Haolong Yan, Muxi Diao, Songyu Xu, Zhonghao Yan, Hongbing Li,

Organizations: Beijing University of Posts and Telecommunications, Beijing, China

Abstract

Accurately detecting and localizing hallucinations is a critical task for ensuring high reliability of image captions. In the era of Multimodal Large Language Models (MLLMs), captions have evolved from brief sentences into comprehensive narratives, often spanning hundreds of words. This shift exponentially increases the challenge: models must now pinpoint specific erroneous spans or words within extensive contexts, rather than merely flag response-level inconsistencies. However, existing benchmarks lack the fine granularity and domain diversity required to evaluate this capability. To bridge this gap, we introduce DetailVerifyBench, a rigorous benchmark comprising 1,000 high-quality images across five distinct domains. With an average caption length of over 200 words and dense, token-level annotations of multiple hallucination types, it stands as the most challenging benchmark for precise hallucination localization in the field of long image captioning to date. Our benchmark is available at https://zyx-hhnkh.github.io/DetailVerifyBench/.

Figures & tables

Figure 1. The difference between the task of hallucination detection and hallucination localization.

Benchmark	Gran.	#Samp.	Avg. Len.	#Type	#Dom.	Real.	Syn.
HaELM ( Wang et al., 2023 )	Resp.	5,000	$\sim$ 50 w	2	1		✓
ALOHa ( Petryk et al., 2024 )	Span.	490	$\sim$ 14 w	1	1	✓
M-HalDet ( Gunjal et al., 2024 )	Seg.	4,000	$\sim$ 80 w	3	1	✓
HalLoc ( Park et al., 2025 )	Token	155,953	$\sim$ 50 w	3	1		✓
TLDR ( Fu et al., 2025 )	Token	–	–	–	1		✓
HLVC ( Nakada et al., 2025 )	Span	1,167	$\sim$ 60 w	3	1	✓

Table 1. Comparison with existing hallucination localization benchmarks. “-” means undisclosed. Gran.: annotation granularity; Real.: contains real hallucinations; Syn.: contains synthesized hallucinations.

Domain	Source	#Img	Avg. Len	#Hallu locations	Hallu. Rate
GUI	Screenspot Pro ( Li et al., 2025 )	200	196	274	68%
Nature	DOCCI ( Onoe et al., 2024 )	200	148	69	26%
Chart	Echarts Examples 1 1 1 https://echarts.apache.org/examples/en/index.html	200	197	192	41%
Movie	CineTechBench ( Wang et al., 2025 )	200	214	613	88%
Movie	ShotBench ( Liu et al., 2025a )	200	214	613	88%
Poster	IMDB	200	257	576	90%

Table 2. Information of each domain in our benchmark. Hallu. Rate: The ratio of captions contain hallucination.

Figure 2. The pipeline for building the DetailVerifyBench.

Model	Real Overall			Real Dimension- $R_{tok}$								Synthetic Overall			Synthetic Dimension- $R_{tok}$
Model	$\mathrm{P_{tok}}$	$\mathrm{R_{tok}}$	$\mathrm{F1_{tok}}$	Num	Clr	Cat	Shp	Mat	Spat	OCR	Other	$\mathrm{P_{tok}}$	$\mathrm{R_{tok}}$	$\mathrm{F1_{tok}}$	Num	Clr	Cat	Shp	Mat	Spat	OCR	Scene	Cam	Other
Open-source MLLMs
GLM-4.6V-Flash	0.01	0.01	0.01	0.00	0.00	0.01	0.01	0.00	0.00	0.01	0.01	0.41	0.22	0.26	0.17	0.40	0.27	0.17	0.40	0.06	0.34	0.31	0.00	0.22
Step3-VL-10B	0.03	0.04	0.03	0.03	0.04	0.13	0.07	0.13	0.04	0.15	0.04	0.36	0.48	0.35	0.49	0.66	0.48	0.43	0.53	0.37	0.77	0.68	0.04	0.39
Qwen3-VL-8B-Thinking	0.04	0.03	0.03	0.02	0.04	0.03	0.15	0.02	0.01	0.10	0.03	0.45	0.47	0.39	0.45	0.68	0.52	0.41	0.63	0.25	0.76	0.64	0.05	0.49
Qwen3.5-9B	0.10	0.09	0.08	0.16	0.10	0.11	0.12	0.28	0.05	0.24	0.11	0.66	0.60	0.58	0.49	0.74	0.58	0.48	0.61	0.50	0.86	0.61	0.22	0.63

Table 3. Evaluation results of advanced open-source and closed-source MLLMs on real and synthetic hallucinations. Dimension abbreviations: Num (Number), Clr (Color), Cat (Category), Shp (Shape), Mat (Material), Spat (Spatial), OCR (Optical Character Recognition), Scene (Scene), Cam (Camera). The best results are in bold type.

Figure 3. Distribution of 10 hallucination dimensions across five domains for (a) real and (b) synthetic hallucinations. Each cell shows the count (top) and percentage within the domain (bottom). The rightmost column indicates the total count per dimension. Color intensity reflects the within-domain percentage.

Figure 4. (a) Comparison of injection methods; (b) Hallucination counts & $\mathrm{R_{tok}}$ of different adversarial iteration $K$ .

Figure 5. Visualization of a GUI image example with real hallucinations in our dataset. Tiny areas were enlarged and the judgments were annotated beside. The red bold text is the ground truth localization annotation. [ ] with specific color indicates the localization result of each MLLM. For more visualization examples, please visit our website.

Model	Metric	Iteration Round $K$
Model	Metric	0	1	2	3
GPT-5.2	$\mathrm{P}_{\mathrm{tok}}$	0.60	0.48 (-0.12)	0.46 (-0.14)	0.46 (-0.14)
	$\mathrm{R}_{\mathrm{tok}}$	0.75	0.64 (-0.11)	0.53 (-0.22)	0.52 (-0.23)
	$\mathrm{F}_{\mathrm{tok}}$	0.63	0.48 (-0.15)	0.43 (-0.20)	0.43 (-0.20)
Qwen3-VL-8B-Thinking	$\mathrm{P}_{\mathrm{tok}}$	0.53	0.45 (-0.08)	0.45 (-0.08)	0.44 (-0.09)
	$\mathrm{R}_{\mathrm{tok}}$	0.71	0.56 (-0.15)	0.47 (-0.24)	0.47 (-0.24)

Table 4. Adversarial injection quality across iteration rounds. We report the token-level precision (

\mathrm{P}_{\mathrm{tok}}

), recall (

\mathrm{R}_{\mathrm{tok}}

), and F1 score (

\mathrm{F}_{\mathrm{tok}}

) for different models (GPT-5.2 and Qwen3-VL-8B-Thinking), measured at each iteration round

K

. Lower values indicate harder hallucinations.

VLM Hallucination

Vision-language model hallucination is the production of claims unsupported by or in conflict with visual evidence.

See all 71 papers

HalDec-Bench: Benchmarking Hallucination Detector in Image Captioning
Mar 16, 2026Kuniaki Saito, Risa Shinoda, Shohei Tanaka +3Image Captioning Evaluation Hallucination Detection in VLMs
OMRON SINICX · The University of Tokyo · The University of Osaka
Where Do Long Captions Fail? Position-Aware Diagnosis and Reinforcement Learning for Detailed Image Captioning
Oct 22, 2026Yuanze Hu, Zhichao Yang, Junwei Jing +9Image Captioning Faithfulness in VLMs
Beihang University · Kuaishou Technology · Fudan University +1
CapProbe: Evaluating Detailed Image Captions via Full-Scene Dense Question Answering
Aug 11, 2026Mouxiao Huang, Qiangyu Yan, Borui Jiang +1VLM Evaluation Image Captioning
Huawei Technologies

Mar 16, 2026J/K move · Enter open · S save

HalDec-Bench: Benchmarking Hallucination Detector in Image Captioning

Kuniaki Saito, Risa Shinoda, Shohei Tanaka, Tosho Hirasawa+2

OMRON SINICX · The University of Tokyo · The University of Osaka

Image Captioning Evaluation Hallucination Detection in VLMs VLM Evaluation Image Captioning

Open paper AskPDF arXiv

Hallucination detection in captions (HalDec) assesses a vision-language model's ability to correctly align image content with text by identifying errors in captions that misrepresent the image. Beyond evaluation, effective hallucination detection is also essential for curating high-quality image-caption pairs used to train VLMs. However, the generalizability of VLMs as hallucination detectors across different captioning models and hallucination types remains unclear due to the lack of a comprehensive benchmark. In this work, we introduce HalDec-Bench, a benchmark designed to evaluate hallucination detectors in a principled and interpretable manner. HalDec-Bench contains captions generated by diverse VLMs together with human annotations indicating the presence of hallucinations, detailed hallucination-type categories, and segment-level labels. The benchmark provides tasks with a wide range of difficulty levels and reveals performance differences across models that are not visible in existing multimodal reasoning or alignment benchmarks. Our analysis further uncovers two key findings. First, detectors tend to recognize sentences appearing at the beginning of a response as correct, regardless of their actual correctness. Second, our experiments suggest that dataset noise can be substantially reduced by using strong VLMs as filters while employing recent VLMs as caption generators. Our project page is available at https://dahlian00.github.io/HalDec-Bench-Page/.

Abstract

VLM Hallucination

Explore similar work

HalDec-Bench: Benchmarking Hallucination Detector in Image Captioning

Where Do Long Captions Fail? Position-Aware Diagnosis and Reinforcement Learning for Detailed Image Captioning

CapProbe: Evaluating Detailed Image Captions via Full-Scene Dense Question Answering