Visually Irrelevant Hallucinations in Medical Vision-Language Models

Ke Zou; Nan Zhou; Tao Yang; Meng Liu; Linchao He; Zehui Liao; Ling Huang; Guangren Huang

cs.CVOct 28, 2026

Visually Irrelevant Hallucinations in Medical Vision-Language Models

Authors: Ke Zou, Nan Zhou, Tao Yang, Meng Liu, Linchao He, Zehui Liao, Ling Huang, Guangren Huang,

Organizations: K. Zou is with the Department of Ophthalmology, Yong Loo Lin School of Medicine, National University of Singapore, Singapore 119228. · N. Zhou, T. Yang, M. Liu, L. He, and H. Chen are with the College of Computer Science, Sichuan University, Chengdu 610065, China. · Z. Liao is with the Faculty of Information Technology, Monash University, Clayton, Victoria 3800, Australia. · L. Huang is with the Institute of Clinical Sciences, Imperial College London, London W12 0NN, UK. · G. Huang is with the Department of Radiology, Beihai People’s Hospital, Beihai, Guangxi 536000, China. · Y. Chen is with the Department of Radiology, The First Affiliated Hospital of Guangxi Medical University, Nanning, Guangxi 530021, China. · Y. Zhang is with the School of Cyber Science and Engineering and the Key Laboratory of Data Protection and Intelligent Management, Ministry of Education, Sichuan University, Chengdu 610065, China. · H. Fu is with the institute of Advanced Intelligence and Computing, Agency for Science, Technology and Research (A*STAR), Singapore 138632.

Abstract

Medical vision-language models (VLMs) show strong performance in radiology report generation (RRG) and medical visual question answering (Med-VQA), producing free-form clinical descriptions and answers from medical images. However, they may generate clinically meaningful statements that lack visual evidence or contradict image content. We refer to these errors as visually irrelevant hallucinations and investigate them at the phrase level. We propose CLEAR (Counterfactual Localized Evaluation of Alignment and Response), a model-agnostic post-hoc framework for phrase-level hallucination risk estimation. CLEAR decomposes generated responses into visually assessable clinical phrases. For each phrase, phrase-conditioned grounding identifies a candidate visual region, which is selectively perturbed by a localized counterfactual intervention. CLEAR characterizes the intervention using two complementary auditing cues: response cues, capturing semantic and clinically meaningful changes in the generated response, and alignment cues, measuring changes in image-phrase compatibility. These cues are normalized and jointly modeled to estimate phrase-level hallucination risk. Across four datasets spanning RRG and Med-VQA, CLEAR outperforms state-of-the-art hallucination detection baselines across multiple medical and general-purpose VLMs. On MIMIC-CXR, CLEAR improves AUPRC from 85.10% to 90.23%. Its effectiveness is further validated on a GT-derived controlled VIH benchmark. These results establish visually irrelevant hallucination as an important fine-grained reliability failure mode in medical VLMs and demonstrate the value of model-agnostic visual-support auditing for more trustworthy multimodal clinical generation.

Figures & tables

Fig. 1: Motivation and overview of phrase-level visually irrelevant hallucination (VIH) detection. (a) A fluent VLM response may contain both visually supported and irrelevant clinical phrases, including positive and negative statements; therefore, response-level assessment cannot localize unreliable content. (b) Given the image $I$ , query $Q$ , generated response $R$ , CLEAR performs post-hoc auditing and estimates a VIH risk for each atomic, visually assessable phrase $S_{i}$ . The same formulation applies to both RRG and Med-VQA.

Fig. 2: Overview of CLEAR. The VLM-generated response is decomposed into clinical phrases. For each phrase, a candidate visual region is localized and perturbed, after which the same VLM generates a counterfactual response.

Fig. 3: Architectural overview of Counterfactual Auditing Representation and Calibration in CLEAR.

Method	MIMIC-CXR			IU-Xray			VQA-RAD			MIMIC-Diff-VQA
Method	AUC $\uparrow$	PRC $\uparrow$	F1 $\uparrow$	AUC $\uparrow$	PRC $\uparrow$	F1 $\uparrow$	AUC $\uparrow$	PRC $\uparrow$	F1 $\uparrow$	AUC $\uparrow$	PRC $\uparrow$	F1 $\uparrow$
MedGemma-4B-it [ 7 ]
AvgProb	67.33	76.00	66.88	55.86	71.70	55.57	60.06	62.80	56.12	47.39	78.01	69.60
AvgEnt	68.06	78.84	68.90	60.84	74.23	59.50	65.69	78.12	64.97	54.94	80.95	75.71
MaxProb	65.28	75.36	66.97	58.74	73.78	57.98	61.58	60.12	54.92	49.60	79.20	71.52
MaxEnt	66.21	77.62	68.33	61.59	75.31	64.20	65.06	78.21	72.68	60.87	80.71	78.29

TABLE I: HALLUCINATION DETECTION PERFORMANCE OF CLEAR AND EIGHT COMPETING METHODS ON OPEN-ENDED VQA AND RRG BENCHMARKS. BEST RESULTS ARE SHOWN IN BOLD , AND THE SECOND-BESTRE SULTS ARE UNDERLINED .Statistical significance using pairedt-tests are reported,wherenumbers in blue and red indicate p-values

<

0.05 and

<

0.01, respectively.AUC and PRC denote Area Under the Receiver Operating Characteristic Curve (AUROC) and Area Under the Precision-Recall Curve (AUPRC), respectively.

Method	RRG			VQA
Method	AUROC	AUPRC	F1	AUROC	AUPRC	F1
RadFlag	76.69	76.51	79.83	79.03	85.82	85.88
Cross-Checking	74.62	79.66	80.62	73.67	82.18	79.49
CLEAR	80.10	89.08	85.28	82.11	87.43	91.55

TABLE II: Hallucination detection performance on the GT-derived controlled evaluation set. The best result in each column is shown in bold .

Fig. 4: Qualitative phrase-level hallucination detection examples of CLEAR. For each radiology image, multiple generated clinical phrases are grounded to their corresponding visual regions and independently assigned hallucination risk scores. Red boxes indicate phrases predicted as hallucinated, while green boxes indicate phrases predicted as supported.

CLEAR Components			RRG			VQA
Resp.	Align.	Norm.	AUROC	AUPRC	F1	AUROC	AUPRC	F1
✓	✗	✗	70.87	83.49	72.48	68.02	85.01	76.18
✗	✓	✗	65.46	78.71	66.41	65.18	80.79	67.80
✓	✓	✗	72.02	84.13	74.44	70.89	86.60	80.84
✓	✓	✓	75.06	87.83	78.84	72.56	89.95	83.43

TABLE III: Ablation study of CLEAR. Resp., Align., and Norm. denote the counterfactual response cues, counterfactual alignment cues, and percentile normalization, respectively. Results are macro-averaged across the evaluated VLM–dataset settings within the RRG and Med-VQA tasks. The best result in each column is shown in bold .

Fig. 5: Hard-case analysis of CLEAR. Bars report the prevalence of five non-exclusive phrase and evidence patterns among false-positive and false-negative cases. Negated or normal phrases are the most prevalent pattern, followed by limited counterfactual alignment variation.

Fig. 6: Counterfactual auditing-cue attribution of CLEAR. Absolute feature attributions are aggregated into response cues, including semantic persistence and clinically meaningful response changes, and alignment cues, capturing image–phrase compatibility and its intervention-induced variation. Bars show the mean normalized absolute attribution for false-positive and false-negative cases. Both error types exhibit similar attribution patterns, with response cues contributing approximately 65% and alignment cues approximately 35%.

Backbone	Perturbation	MIMIC-CXR			IU-Xray
Backbone	Perturbation	AUROC	AUPRC	F1	AUROC	AUPRC	F1
MedGemma	Blur	79.75	90.23	84.07	76.00	81.40	79.79
	Mask	79.60	90.09	84.06	76.52	81.72	80.01
	Mean	79.60	90.08	84.12	76.55	82.01	79.95
	Noise	79.62	90.11	84.11	76.59	81.83	79.89
LLaVA-Med	Blur	79.12	94.83	91.22	75.67	89.29	88.83

TABLE IV: Robustness analysis of CLEAR under different localized counterfactual perturbations across backbone models. Blur, Mask, Mean, and Noise denote alternative operators for constructing counterfactual images. Comparable performance across these perturbations indicates that CLEAR is not strongly dependent on a specific intervention strategy.

Backbone	Blur Radius	MIMIC-CXR			IU-Xray
Backbone	Blur Radius	AUROC	AUPRC	F1	AUROC	AUPRC	F1
MedGemma	5	79.68	90.12	84.08	76.16	81.80	79.81
	10	79.61	90.10	84.06	76.55	82.16	79.95
	20	79.75	90.23	84.07	76.00	81.40	79.79
LLaVA-Med	5	78.79	94.73	90.98	78.45	89.50	88.83
	10	79.01	94.72	91.00	78.56	89.46	88.90

TABLE V: Sensitivity analysis of CLEAR with respect to the Gaussian blur radius on the MIMIC-CXR and IU-Xray benchmarks. The results show that CLEAR maintains stable detection performance across different blur scales. The best result for each backbone and evaluation metric is highlighted in bold .

Ke Zou received the Ph.D. degree in Computer Science from Sichuan University, Chengdu, China, in 2024. From 2022 to 2023, he was a Visiting Ph.D. Student with the Agency for Science, Technology and Research (A*STAR), Singapore. He is currently a Research Fellow with the Yong Loo Lin School of Medicine, National University of Singapore, Singapore. His research interests include medical image analysis, vision-language models, foundation models, and agentic AI. His work has been published in leading journals and conferences in medical imaging and artificial intelligence, including The Lancet Digital Health , IEEE TPAMI , IEEE TCYB , MedIA , and MICCAI . He has served a Guest Editor for the IEEE JBHI and served as an Area Chair for MICCAI 2025 and 2026.

Table 12

Nan Zhou received the B.E. degree from Shanghai Ocean University, Shanghai, China, in 2022. She is currently pursuing the Ph.D. degree at Sichuan University, Chengdu, China. Her research interests include medical image analysis, deep learning, and uncertainty estimation.

Table 13

Tao Yang received the B.E. degree from Sichuan Agricultural University, Ya’an, China, in 2025. He is currently pursuing the Ph.D. degree in computer science and technology at Sichuan University, Chengdu, China. His research interests include medical image analysis, multimodal large language models, and multi-agent systems, with a particular focus on their applications in healthcare.

Table 14

Meng Liu is a Ph.D. candidate at Sichuan University. He received his Master degree from Sichuan University in 2023. He is now focusing on medical image analysis, multimodal learning and uncertainty estimation.

Table 15

Linchao He received the B.S. degree from Sichuan Agricultural University, Ya’an, China, in 2020. He is currently pursuing the Ph.D. degree at Sichuan University, Chengdu, China, where he is affiliated with the National Key Laboratory of Fundamental Science on Synthetic Vision. His research interests include medical image analysis, deep learning, and computational imaging inverse problems.

Table 16

Zehui Liao is a Research Fellow at the Faculty of Information Technology, Monash University, Australia, where she works with Prof. Jianfei Cai. Before joining Monash University, she received her PhD in Computer Science from Northwestern Polytechnical University under the supervision of Prof. Yong Xia. From 2024 to 2025, she was a visiting PhD student at the Institute of High Performance Computing, Agency for Science, Technology and Research, Singapore, under the supervision of Dr. Huazhu Fu and Dr. Liangli Zhen. Her research interests include medical image analysis, with a particular focus on multimodal large language models and learning under label uncertainty.

Table 17

Ling Huang received the Ph.D. degree in 2023 from Université de Technologie de Compiègne, Compiègne, France, where she conducted her doctoral research in a laboratory affiliated with the French National Centre for Scientific Research (CNRS). She is currently a Postdoctoral Researcher at Imperial College London, London, U.K., and a Visiting Research Fellow at the National University of Singapore, Singapore. Her research interests include the theory of belief functions and its applications to uncertainty quantification, causal inference, and decision explanation in medicine. She has authored or coauthored more than 40 papers, including over 20 papers published in leading journals and conferences, such as Information Fusion, IEEE Transactions on Fuzzy Systems, IEEE Transactions on Cybernetics, Medical Image Analysis, ICML, ECCV, ACL, ACM MM, and MICCAI. She serves as an Editorial Board Member for Information Fusion.

Table 18

Guangren Huang is a Deputy Chief Physician in the Department of Radiology at Beihai People’s Hospital, Guangxi, China. He is currently pursuing a Ph.D. degree at Guangxi Medical University. His research focuses on lung cancer, glioma, and medical imaging.

Table 19

Yidi Chen received his Ph.D. degree from The First Affiliated Hospital of Guangxi Medical University and is currently an Associate Professor at the West China Hospital of Sichuan University. He was previously a joint doctoral candidate at the University of Wisconsin School of Medicine and Public Health. His research focuses on hepatocellular carcinoma and magnetic resonance imaging.

Table 20

Yi Zhang received the Ph.D. degree in computer science and technology from Sichuan University, Chengdu, China, in 2012. He is currently a Full Professor with the School of Cyber Science and Engineering, Sichuan University. His research interests include medical imaging, compressive sensing, and deep learning. He serves as a Guest Editor for the International Journal of Biomedical Imaging and Sensing and Imaging, and as an Associate Editor for the IEEE Transactions on Medical Imaging and the IEEE Transactions on Radiation and Plasma Medical Sciences.

Table 21

Hu Chen received the Ph.D. degree from the University of Electronic Science and Technology of China, Chengdu, China, in 2011. From 2008 to 2009, he was a visiting doctoral student at the University of California, San Diego, CA, USA. He is currently an Associate Professor with the School of Computer Science, Sichuan University, Chengdu, China. His research interests include image and video processing, computer vision, and artificial intelligence. He has authored or coauthored more than 40 papers in these areas.

Table 22

Huazhu Fu (SM’18) is a Principal Scientist at the Institute of High Performance Computing (IHPC), ASTAR, Singapore. He earned his Ph.D. from Tianjin University in 2013. Prior to his current position, he was a Research Fellow at NTU, Singapore from 2013 to 2015, a Research Scientist at I2R, ASTAR, Singapore from 2015 to 2018, and a Senior Scientist at the Inception Institute of Artificial Intelligence, UAE from 2018 to 2021. His research focuses on medical image analysis, AI for healthcare, and trustworthy AI. He is an Associate Editor for several IEEE journals, including IEEE TMI, IEEE TNNLS, IEEE TAI, and IEEE JBHI.

Table 23

VLM Hallucination

Vision-language model hallucination is the production of claims unsupported by or in conflict with visual evidence.

See all 72 papers

Hallucination Detection and Correction in Medical VLMs via Counter-Evidence Verification
Jun 17, 2026Nan Zhou, Ke Zou, Meng Liu +5Hallucination Detection in VLMs Vision-Language Models
the College of Computer Science, Sichuan University. · Yong Loo Lin School of Medicine, National University of Singapore. · National Key Laboratory of Autonomous Intelligent Unmanned Systems, Beijing2026 Institute of Technology. +2
HalluCXR: Benchmarking and Mitigating Hallucinations in Medical Vision-Language Models for Chest Radiograph Interpretation
May 19, 2026Haoyu Wang, Zitong LiVision-Language Models Audio-Language Model Hallucination Detection
Department of Biostatistics & Health Informatics, Institute of Psychiatry, Psychology & Neuroscience, King’s College London, London, UK
VIHD: Visual Intervention-based Hallucination Detection for Medical Visual Question Answering
May 20, 2026Jiayi Chen, Benteng Ma, Zehui Liao +3Hallucination Detection in VLMs Healthcare
Department of Data Science & AI, Faculty of Information Technology, Monash University, Melbourne, VIC 3800, Australia · Hong Kong Polytechnic University, Hong Kong SAR, China · Alfred Health Radiology, Alfred Health, Melbourne, VIC 3004, Australia +1

Jun 17, 2026J/K move · Enter open · S save

Hallucination Detection and Correction in Medical VLMs via Counter-Evidence Verification

Nan Zhou, Ke Zou, Meng Liu, Linchao He+4

the College of Computer Science, Sichuan University. · Yong Loo Lin School of Medicine, National University of Singapore. · National Key Laboratory of Autonomous Intelligent Unmanned Systems, Beijing2026 Institute of Technology.+2

Hallucination Detection in VLMs Vision-Language Models Audio-Language Model Hallucination Detection Medical VLMs VLM Hallucination Mitigation

Open paper AskPDF arXiv

Vision-Language models (VLMs) reliability in medical diagnosis is challenged by trust-undermining hallucinations. Existing hallucination detection approaches mainly focus on identifying factual inconsistencies between generated text and reference data. While some studies analyze where models attend in images, they seldom verify whether such attention truly reflects the visual evidence supporting the generated text. To address this gap, we propose Co}unter-Evidence Verification (CoEV), a training-free plug-and-play framework that detects and corrects hallucinations through evidence-based factual consistency verification. CoEV performs bidirectional verification between textual assertions and visual evidence, testing whether each statement is supported by its corresponding evidence region, and assigns each statement into a four-quadrant diagnostic map capturing combinations of text factuality and visual grounding. CoEV detects hallucinated content and serves as a post hoc refinement tool, correcting hallucinations without retraining. Extensive experiments on four medical datasets show that CoEV combats hallucinations in VLMs.For hallucination detection, CoEV consistently outperforms existing methods, improving average PR-AUC and ROC-AUC by 3.0% and 3.9% absolute points respectively, with notable gains of up to 18.5% in specific VQA scenarios. For hallucination correction, it improves Micro-F1 by up to 12.5%, reduces hallucination rates by over 11.9% on medical report generation, and also boosts medical VQA accuracy. These results show that CoEV enables reliable detection and correction of hallucinations, providing clinicians with dependable, evidence-based cues for diagnosis. Code will be released upon acceptance.

Abstract

VLM Hallucination

Explore similar work

Hallucination Detection and Correction in Medical VLMs via Counter-Evidence Verification

HalluCXR: Benchmarking and Mitigating Hallucinations in Medical Vision-Language Models for Chest Radiograph Interpretation

VIHD: Visual Intervention-based Hallucination Detection for Medical Visual Question Answering