CDH-Bench: A Commonsense-Driven Hallucination Benchmark for Evaluating Visual Fidelity in Vision-Language Models

Kesheng Chen; Yamin Hu; Qi Zhou; Zhenqian Zhu; Wenjian Luo

cs.CVMar 30, 2026

CDH-Bench: A Commonsense-Driven Hallucination Benchmark for Evaluating Visual Fidelity in Vision-Language Models

Authors: Kesheng Chen, Yamin Hu, Qi Zhou, Zhenqian Zhu, Wenjian Luo

Organizations: Guangdong Provincial Key Laboratory of Novel Security Intelligence Technologies, Institute of Cyberspace Security, School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen, China

Abstract

Vision-language models (VLMs) achieve strong performance on many benchmarks, yet a basic reliability question remains underexplored: when visual evidence conflicts with commonsense, do models follow what is shown or what commonsense suggests? A characteristic failure in this setting is that the model overrides visual evidence and outputs the commonsense alternative. We term this phenomenon commonsense-driven hallucination (CDH). To evaluate it, we introduce CDH-Bench, a benchmark designed to create explicit visual evidence--commonsense conflicts. CDH-Bench covers three dimensions: counting anomalies, relational anomalies, and attribute anomalies. We evaluate frontier VLMs under binary Question Answering (QA) and multiple-choice QA, and report metrics including Counterfactual Accuracy (CF-Acc), Commonsense Accuracy (CS-Acc), Counterfactual Accuracy Drop (CFAD), Commonsense Collapse Rate (CCR), and Relative Prior Dependency (RPD). Results show that even strong models remain vulnerable to prior-driven normalization under visual evidence--commonsense conflict. CDH-Bench provides a controlled diagnostic of visual fidelity under visual evidence--commonsense conflict.

Figures & tables

Figure 1: Taxonomy and distribution of CDH-Bench.

Panel of Figure 2 — Figure 2: CDH-Bench illustration: examples from diverse dimensions. Each subfigure displays a counterfactual (CF) image on the left and its commonsense (CS) counterpart on the right, together with the associated binary QA and multi-choice QA evaluation details.

Figure 3: Model performance comparison on counterfactual accuracy (CF-Acc) across different subcategories in the binary QA task.

Figure 4: Counterfactual Accuracy Drop (CFAD) across subcategories in the binary QA task.

Figure 5: Model performance comparison on counterfactual accuracy (CF-Acc) across different subcategories in the multiple-choice QA task.

Figure 6: Counterfactual Accuracy Drop (CFAD) across subcategories in the multiple-choice QA task.

Model	Cat.	Binary QA				Multiple-choice QA
Model	Cat.	CF $\uparrow$	CS $\uparrow$	CFAD $\downarrow$	RPD $\downarrow$	CF $\uparrow$	CS $\uparrow$	CFAD $\downarrow$	CCR $\downarrow$	RPD $\downarrow$
gemini-3.1-pro-preview	Overall	90.33%	87.67%	-2.67%	-3.04%	81.67%	89.00%	7.33%	58.18%	8.24%
	Attr.	85.00%	83.00%	-2.00%	-2.41%	84.00%	93.00%	9.00%	81.25%	9.68%
	Count.	96.00%	87.00%	-9.00%	-10.34%	79.00%	80.00%	1.00%	42.86%	1.25%
	Rel.	90.00%	93.00%	3.00%	3.23%	82.00%	94.00%	12.00%	55.56%	12.77%
gemini-3.1-flash-lite-preview	Overall	83.67%	91.33%	7.67%	8.39%	71.67%	94.33%	22.67%	71.76%	24.03%

Table 1: Main results on CDH-Bench. Higher CF-Acc and CS-Acc are better; lower CFAD , CCR , and RPD are better. To emphasize failure analysis rather than leaderboard ranking, selected notably weak results are marked in red. CCR is reported only for Multiple-choice QA in the main table. All values are percentages. Gray rows denote Overall results.

Model	Binary QA				Multiple-choice QA					Overall
Model	CF (%)	CS (%)	CFAD (%)	RPD (%)	CF (%)	CS (%)	CFAD (%)	CCR (%)	RPD (%)	CF (%)	CS (%)	CFAD (%)	CCR (%)	RPD (%)
Qwen3-VL-2B-Instruct	40.33	87.67	47.33	53.99	32.00	74.00	42.00	73.53	56.76	36.17	80.83	44.67	86.76	55.37
Qwen3-VL-4B-Instruct	52.00	91.67	39.67	43.27	34.33	89.33	55.00	84.77	61.57	43.17	90.50	47.33	92.39	52.42
Qwen3-VL-8B-Instruct	45.00	93.67	48.67	51.96	42.00	93.67	51.67	78.74	55.16	43.50	93.67	50.17	89.37	53.56
Qwen3-VL-32B-Instruct	55.67	93.33	37.67	40.36	46.00	92.67	46.67	84.57	50.36	50.83	93.00	42.17	92.28	45.36
Qwen3-VL-2B-Thinking	58.00	91.33	33.33	36.50	48.33	93.00	44.67	77.42	48.03	53.17	92.17	39.00	88.31	42.26

Table 2: Results of the Qwen3-VL series on CDH-Bench. All values are reported as percentages. Higher CF-Acc and CS-Acc are better; lower CFAD, CCR, and RPD are better. We do not highlight best values; instead, selected notably weak results are marked in red.

Figure 7: Model size vs. performance metrics for the MC task on CDH-Bench. The plot shows the scaling behavior of CF-Acc, CFAD, and CS-Acc for Qwen3-VL Instruct and Thinking models.

Figure 8: Model size vs. performance metrics for the QA task on CDH-Bench. The plot shows the scaling behavior of CF-Acc, CFAD, and CS-Acc for Qwen3-VL Instruct and Thinking models.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 9 — Figure 9: Images for Pair 5. The anomaly is an extra finger on the right hand (personś perspective).

Emotional Hallucination in VLMs

See all 8 papers

When Model Priors Conflict with Visual Evidence: Mitigating Commonsense-Driven Hallucinations by Selective Prior Calibration
Jul 31, 2026Kesheng Chen, Yamin Hu, Wenjian LuoVLM Hallucination Emotional Hallucination in VLMs
Guangdong Provincial Key Laboratory of Novel Security Intelligence Technologies Institute of Cyberspace Security School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen, China
DO-Bench: An Attributable Benchmark for Diagnosing Object Hallucination in Vision-Language Models
Apr 18, 2026JiYang Wang, Jiawei Chen, Mengqi Xiao +3VLM Evaluation Hallucination Detection in VLMs
Shanghai Key Laboratory of Multidimensional Information Processing, East China Normal University · Zhongguancun Academy
EDCT-Bench: Uncovering Faithfulness Gaps in VLMs via Explanation-Driven Counterfactual Testing
Sep 16, 2026Sihao Ding, Santosh Vasa, Aditi Ramadwar +1VLM Evaluation VLM Robustness
Mercedes-Benz Research & Development North America

Jul 31, 2026J/K move · Enter open · S save

When Model Priors Conflict with Visual Evidence: Mitigating Commonsense-Driven Hallucinations by Selective Prior Calibration

Kesheng Chen, Yamin Hu, Wenjian Luo

Guangdong Provincial Key Laboratory of Novel Security Intelligence Technologies Institute of Cyberspace Security School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen, China

VLM Hallucination Emotional Hallucination in VLMs VLM Calibration VLM Hallucination Mitigation

Open paper AskPDF arXiv

In vision--language models, commonsense-driven hallucination (CDH) occurs when a model's commonsense prior overrides clear visual evidence of an atypical state. For example, a model may report that a visibly six-fingered hand has five fingers. We show that these errors are systematically directed: when a model answers a question about a counterfactual (CF) image incorrectly, its answer often coincides with the candidate it prefers without access to the image. Suppressing this prior indiscriminately can repair CF errors, but may also disrupt correct answers on matched commonsense (CS) images, where the same prior is helpful. We therefore propose Selective Prior Calibration (SPC), which subtracts candidate-level prior-preference estimates from image-conditioned scores with an instance-dependent strength and revises the original prediction only when the resulting score pattern strongly supports an alternative. Extensive experiments demonstrate that SPC substantially improves accuracy on CF images while largely preserving accuracy on matched CS images. Furthermore, these gains generalize across CDH categories, candidate-answer permutations, and other conflict benchmarks, while SPC rarely alters predictions on benchmarks without such conflicts.

Abstract

Emotional Hallucination in VLMs

Explore similar work

When Model Priors Conflict with Visual Evidence: Mitigating Commonsense-Driven Hallucinations by Selective Prior Calibration

DO-Bench: An Attributable Benchmark for Diagnosing Object Hallucination in Vision-Language Models

EDCT-Bench: Uncovering Faithfulness Gaps in VLMs via Explanation-Driven Counterfactual Testing