MedHal: a Synthetic Dataset for Medical Hallucination Detection

Fabrice Lamarche; Gaya Mehenni; Neshat Elhami Fard; Odette Rios-Ibacache; Li Ming Wang; John Kildea; Amal Zouaq

cs.CLApr 11, 2025

MedHal: a Synthetic Dataset for Medical Hallucination Detection

Authors: Fabrice Lamarche, Gaya Mehenni, Neshat Elhami Fard, Odette Rios-Ibacache, Li Ming Wang, John Kildea, Amal Zouaq

Organizations: LAMA-WeST Lab, Polytechnique Montréal, Mila – Quebec AI Institute · Medical Physics Unit – RI-MUHC, McGill University · Department of Medicine, Division of Experimental Medicine, McGill University

Abstract

Hallucination, the generation of non factual content by AI systems, poses serious risks in medical contexts, where errors can directly affect patient outcomes. We present MedHal, a large-scale dataset specifically designed to assess capabilities and train models on the task of hallucination detection in medical texts. Current hallucination detection methods face significant limitations when applied to specialized domains like medicine, where they can have disastrous consequences. MedHal addresses this issue by incorporating diverse medical text sources and tasks covering both intrinsic and extrinsic hallucinations, and by providing a substantial volume of data samples suitable for training medical hallucination detection models. We demonstrate MedHal's utility by training and evaluating a baseline medical hallucination detection model, showing improvements over general-purpose hallucination detection approaches. This resource enables more efficient evaluation and training of medical text generation systems while reducing reliance on costly expert review, potentially accelerating the development of medical AI research.

Figures & tables

Task	Sample	Generated Statement
Information Extraction	A 10-year-old girl first noted a swollen left knee and underwent repeated … $\rightarrow$ age: 10 years old	The patient is 10 years old.
Summarization	the large genotyping studies in the last decade have revolutionize genetic studies. our current ability to … $\rightarrow$ genetic admixture is a common caveat for …	genetic admixture is a common caveat for…
NLI	Labs were notable for Cr 1.7 (baseline 0.5 per old records) and lactate 2.4. $\rightarrow$ Patient has normal Cr	Patient has normal Cr
QA	Which of the following medications is most commonly used as treatment for type 2 diabetes? $\rightarrow$ Metformin	Metformin is most commonly used as treatment for type 2 diabetes.

Table 1: Example of samples used to generate statements for each task

Dataset	Task	S	Content Type	Source	# Samples	# Gen
MedMCQA	QA	✗	Medical Content	( Pal et al., 2022 )	183,000	278,021
MedNLI	NLI	✗	Clinical Notes	( Romanov and Shivade, 2018 )	11,232	7,488
ACM	IE	✓	Clinical Notes	( Bonnet and Boulenger, 2024 )	22,000	327,568
MedQA	QA	✗	Medical Content	( Jin et al., 2021 )	12,723	18,680
SumPubMed	Sum	✗	Clinical Trials	( Gupta et al., 2021 )	33,772	195,339

Table 2: Description of datasets, after filtering, used to create the MedHal benchmark (S is for Synthetic, # Gen is the number of generated MedHal-format samples).

Table 3

Task	A1	A2	Avg
Common Knowledge	78.0	90.0	84.0
Information Extraction	86.0	98.0	92.0
NLI	81.0	97.0	89.0
Summarization	100.0	100.0	100.0
Overall	86.3	96.3	91.3

Table 5: Expert validation results per task. Values are percentages. NLI = Natural Language Inference.

Type	Model	Hallucination Detection				Explanation
Type	Model	H-F1	NH-F1	Mac-F1	$\Delta$ F1	ECS	N
General	Qwen3-8B	0.75	0.76	0.76		0.85	16,013
	MedHal -Qwen3-8B	0.89	0.88	0.89	(+0.13)	0.94	20,719
	Llama-3-8B	0.66	0.74	0.70		0.83	18,724
	MedHal -Llama-3-8B	0.90	0.90	0.90	(+0.20)	0.94	21,055
	LLaMA3-70B	0.86	0.88	0.87		0.83	18,046

Table 6: Performance of models on MedHal’s test set. N refers to the number of samples where an explanation was extracted when prompting the model, H is for ‘Hallucinated’, NH for ‘Non-Hallucinated’ and ECS for ‘Embedding Cosine Similarity’,

\Delta

F1 is the improvement in Macro-F1 between the base and MedHal finetuned models.

Type	Model	ACI-NH	Medec-UW	Hall-Gen-DI
General LLM	Llama3-8B	0.54 $\pm$ 0.07	0.91 $\pm$ 0.03	0.50 $\pm$ 0.05
	MedHal -Llama3-8B	0.52 $\pm$ 0.07	0.98 $\pm$ 0.01	0.63 $\pm$ 0.06
	Llama3-70B	0.63 $\pm$ 0.07	0.98 $\pm$ 0.01	0.51 $\pm$ 0.05
	Qwen3-8B	0.62 $\pm$ 0.07	0.95 $\pm$ 0.02	0.65 $\pm$ 0.06
	MedHal -Qwen3-8B	0.63 $\pm$ 0.07	0.98 $\pm$ 0.01	0.63 $\pm$ 0.06
	GPT-OSS-20B	0.46 $\pm$ 0.08	0.64 $\pm$ 0.05	0.55 $\pm$ 0.06

Table 7: Downstream performance (Macro-F1) on the three human-annotated hallucination detection datasets, with

\pm

95% stratified bootstraped confidence interval over resampling of the datasets.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Keywords
question
answer
statement
however
above
shown

Appendix

Table 8: List of Keywords used for QA filtering

Sample Type	Question	Answer	Generated Statement
Factual	Which of the following medications is most commonly used as first-line treatment for newly diagnosed type 2 diabetes mellitus in patients without contraindications?	Metformin	Metformin is most commonly used as first-line treatment for newly diagnosed type 2 diabetes mellitus in patients without contraindications.
Non-Factual		Insulin	Insulin is most commonly used as first-line treatment for newly diagnosed type 2 diabetes mellitus in patients without contraindications.

Appendix

Table 9: Example of Question-Answering Dataset Transformation

Sample Type	Source Document	Extraction	Statement	Explanation
Factual	A 10-year-old girl first noted a swollen left knee and underwent repeated arthrocentesis…	age: 10 years old	The patient is 10 years old	-
Non-Factual	A 10-year-old girl first noted a swollen left knee and underwent repeated arthrocentesis…	age: 16 years old	The patient is 16 years old	The patient is 10 years old

Appendix

Table 10: Example of Information Extraction Dataset Transformation (the extraction from the non-factual statement is taken from another original sample)

Sample Type	Source Document	Summary	Statement	Explanation
Factual	a central feature in the maturation of hearing is a transition in the electrical signature of cochlear hair cells from spontaneous calcium…	cochlear hair cells are high-frequency sensory receptors…	cochlear hair cells are high-frequency sensory receptors…	-
Non-Factual			cochlear hair cells are low-frequency sensory receptors…	According to the source document, cochlear hair cells are high-frequency sensory receptors…

Appendix

Table 11: Example of Summarization Dataset Transformation

Dataset	Input	Output / task	Non-factuality / hallucination	Unified transformation	Final samples
ACI-NH	Clinician–patient transcript and LLM-generated summary statements	Factuality classification of each summary statement	Summary information that is not present in, or is misinterpreted from, the source transcript. Hallucination , Inference , and Misunderstanding are mapped to non-factual; No Factual Error is factual.	context : transcript; statement : each summary statement; label : YES for No Factual Error , NO otherwise; explanation : annotator notes when available.	170 (88 YES, 82 NO)
Medec-UW	Clinical note text	Error flag prediction, error sentence extraction, and correction generation	An inaccurate medical statement introduced into the note, e.g., an error in diagnosis, management, treatment, pharmacotherapy, or causal organism; in the Medec-UW subset, injected errors are designed to contradict at least two other parts of the note.	In our unified transformation, we remove one target sentence from the note; context : remaining note text; statement : removed sentence; we use the annotated error sentence when available, otherwise a randomly selected sentence; label : YES for Error Flag =0, NO for Error Flag =1; explanation : corrected sentence when available, otherwise “The statement is factual.”	328 (164 YES, 164 NO)
Hall-Gen DI	Original abstract and LLM-generated summary	Binary factual consistency classification	Summary content not contained in, or not logically derivable from, the abstract; a summary is consistent only if all its information is supported by the source.	context : original abstract; statement : generated summary; label : YES for score 3, NO otherwise; explanation : expert inconsistency rationale when available, otherwise “The statement is factual.”	248 (132 YES, 116 NO)

Appendix

Table 12: Downstream evaluation datasets and their transformation into the unified factuality format.

Statement
Serine, not threonine, is typically involved in the synthesis of ceramide; however, according to the given answer: Threonine is involved in the synthesis of ceramide.
Vitamin K should be administered to reverse the effects of the heparin, however protamine is typically used to reverse heparin, but since the answer is Vitamin K, the statement will be: Vitamin K should be administered to reverse the effects of the heparin.
The normal value of ankle brachial index is 0.8 or greater, however since 0.8 is given as the answer, the statement would be: The normal value of ankle brachial index is 0.8, but to be more precise: The normal value of ankle brachial index is at least 0.8, but since the exact wording of the answer is 0.8, the statement is: The normal value of ankle brachial index is 0.8.

Appendix

Table 13: Example of faulty Statements from the QA task.

Checkpoint	Link
Qwen3-8B ( Yang et al., 2025 )	https://huggingface.co/Qwen/Qwen3-8B
Llama-3-8B ( Grattafiori et al., 2024 )	https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct
LLaMA3-70B ( Grattafiori et al., 2024 )	https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct
GPT-OSS-20B ( Agarwal et al., 2025 )	https://huggingface.co/openai/gpt-oss-20b
OpenBioLLM-8B ( Ankit Pal, 2024 )	https://huggingface.co/aaditya/Llama3-OpenBioLLM-8B
OpenBioLLM-70B ( Ankit Pal, 2024 )	https://huggingface.co/aaditya/Llama3-OpenBioLLM-70B

Appendix

Table 14: List of model checkpoints and their download links

Model	Parameters
Llama-3-8B	Temperature = 0.6, Top P = 0.9
Llama-3-70B	Temperature = 0.6, Top P = 0.9
Qwen3-8B	Temperature = 0.7, Top P = 0.8, Top K = 20, Min P = 0
GPT-OSS-20B	Temperature = 1.0, Top P = 1.0
OpenBioLLM-8B	Temperature = 0.0, Top P = 0.9
OpenBioLLM-70B	Temperature = 0.0, Top P = 0.9

Appendix

Table 15: Models and Configurations

Source	Type	HallOumi	OpenBioLLM	Llama-3-8B	Qwen3-8B	Llama-3.3-70B
QA	MedHal	8.42	8.90	7.11	8.53	7.80
Summarization	MedHal	7.78	8.64	7.09	7.85	7.62
Information Extraction	MedHal	6.98	7.76	6.01	7.28	6.45
NLI	MedHal	6.23	7.06	5.42	6.26	5.85
MEDEC-UW	Control	8.74	9.74	8.16	9.56	8.40
ACI-NH	Control	7.80	8.66	6.72	8.23	6.99

Appendix

Table 16: Min-K% Prob. (

K=20

) computed on the first 128 tokens of the context.

Source	Type	HallOumi	OpenBioLLM	Llama-3-8B	Qwen3-8B	Llama-3.3-70B
QA	MedHal	6.8	8.7	6.2	6.8	5.9
Summarization	MedHal	14.5	18.6	12.0	13.8	12.0
Information Extraction	MedHal	10.1	12.5	8.1	10.5	8.2
NLI	MedHal	19.8	18.7	12.9	16.8	15.5
MEDEC-UW	Control	26.2	39.9	21.2	32.2	20.6
ACI-NH	Control	13.4	17.4	10.7	13.9	10.3

Appendix

Table 17: Median perplexity computed on the first 128 tokens of the context.

Source	Type	HallOumi	OpenBioLLM	Llama-3-8B	Qwen3-8B	Llama-3.3-70B
QA	MedHal	$-0.067$	$-0.001$	$+0.012^{*}$	$+0.032^{*}$	$+0.022^{*}$
Summarization	MedHal	$-0.039$	$-0.034$	$-0.084$	$-0.009$	$-0.046$
Information Extraction	MedHal	$-0.057$	$-0.022$	$-0.001$	$-0.004$	$-0.039$
NLI	MedHal	$-0.071$	$-0.007$	$-0.004$	$+0.005$	$-0.001$
MEDEC-UW	Control	$-0.009$	$+0.005$	$-0.003$	$+0.004$	$-0.048$
ACI-NH	Control	$-0.052$	$-0.011$	$+0.000$	$+0.005$	$-0.024$

Appendix

Table 18: Time Travel mean ROUGE-L delta (guided

-

general). Significant cells (bootstrap

p\leq 0.05

) are marked with *.

Task	Hallucination Detection				Explanation
Task	H-F1	NH-F1	Mac-F1	$\Delta$ F1	ECS	N
Summarization	0.757	0.829	0.794	+0.09	0.72	20,142
Question Answering	0.759	0.754	0.756	+0.06	0.71	19,974
Information Extraction	0.758	0.732	0.745	+0.05	0.68	18,013

Appendix

Table 19: Performance of Llama-3-8B on MedHal’s test set after fine-tuning using only samples associated to certain tasks. N refers to the number of samples where an explanation was extracted when prompting the model, H is for ‘Hallucinated’, NH for ‘Non-Hallucinated’ and ECS for ‘Embedding Cosine Similarity’,

\Delta

F1 is the improvement in Macro-F1 between the base and fine-tuned models.

Hallucination in RAG

See all 31 papers

Med-HEAL: Analyzing and Mitigating Hallucinations in Medical LLMs with Hallucination-Aware In-Context Learning
May 31, 2026Yiming Liao, Zeno Franco, Jose Eduardo Lizarraga Mazaba +1LLM Self-Correction LLM Hallucination Mitigation
University of Maryland, Baltimore County Baltimore, Maryland, USA · Concordia University Wisconsin · Medical College of Wisconsin Milwaukee, Wisconsin USA
MedFabric and EtHER: A Data-Centric Framework for Word-Level Fabrication Generation and Detection in Medical LLMs
May 5, 2026Tung Sum Thomas Kwok, Qian Qian, Xiaofeng Lin +8Healthcare Hallucination Detection
University of California, Los Angeles CA 90095, USA · Optum AI, USA
ClinHallu: A Benchmark for Diagnosing Stage-Wise Hallucinations in Medical MLLM Reasoning
Jun 12, 2026Sicheng Yang, Hangjie Yuan, Wenjun Zhang +5Multimodal Hallucination Clinical Reasoning
The Hong Kong University of Science and Technology (Guangzhou) · DAMO Academy, Alibaba Group · Hupan Lab +1

May 31, 2026J/K move · Enter open · S save

Med-HEAL: Analyzing and Mitigating Hallucinations in Medical LLMs with Hallucination-Aware In-Context Learning

Yiming Liao, Zeno Franco, Jose Eduardo Lizarraga Mazaba, Keke Chen

University of Maryland, Baltimore County Baltimore, Maryland, USA · Concordia University Wisconsin · Medical College of Wisconsin Milwaukee, Wisconsin USA

LLM Self-Correction LLM Hallucination Mitigation Clinical QA In-Context Learning LLM Hallucination Detection Retrieval-Augmented ICL+1

Open paper AskPDF arXiv

Hallucinations in medical large language models (LLMs) pose serious risks for clinical decision support, particularly when models must reason over complex electronic health records (EHRs). However, existing benchmarks often lack a realistic clinical context and provide limited insight into how hallucinations can be mitigated in practice. We introduce Med-HEAL, a framework for systematically identifying, analyzing, and mitigating hallucinations in medical LLMs using clinically grounded data. Building on the EHRNoteQA benchmark derived from MIMIC-IV discharge summaries, we construct a hallucination dataset by evaluating BioMistral-7B on open-ended clinical question answering tasks. Model outputs are labeled through a dual evaluation pipeline that combines LLM-as-a-Judge assessment (GPT-4o) with human auditing by medical student reviewers, producing correctness judgments and annotations of reasoning errors via a custom web-based evaluation system. We then leverage this dataset to investigate mitigation strategies: a self-critique pipeline, in which the test model reviews its own answers to detect potential errors and regenerates responses for flagged cases, and retrieval-augmented in-context learning (RA-ICL), which exposes the model to hallucinated and corrected examples. Experiments across five open-source LLMs-BioMistral, Llama-3.1, DeepSeek, Qwen2.5, and Qwen3, show that the self-critique strategy improves accuracy for three of five models (p < 0.05) without requiring parameter updates. Med-HEAL provides both a reusable hallucination dataset and a practical framework for studying and mitigating hallucinations in medical LLMs, supporting safer deployment of AI systems in clinical environments. Our code and data are publicly available at https://github.com/yimingliao-blad/med-heal.git.

Abstract

Hallucination in RAG

Explore similar work

Med-HEAL: Analyzing and Mitigating Hallucinations in Medical LLMs with Hallucination-Aware In-Context Learning

MedFabric and EtHER: A Data-Centric Framework for Word-Level Fabrication Generation and Detection in Medical LLMs

ClinHallu: A Benchmark for Diagnosing Stage-Wise Hallucinations in Medical MLLM Reasoning