Blending Human and LLM Expertise to Detect Hallucinations and Omissions in Mental Health Chatbot Responses

Khizar Hussain; Bradley A. Malin; Zhijun Yin; Susannah Leigh Rose; Murat Kantarcioglu

cs.CLMar 17, 2026

Blending Human and LLM Expertise to Detect Hallucinations and Omissions in Mental Health Chatbot Responses

Authors: Khizar Hussain, Bradley A. Malin, Zhijun Yin, Susannah Leigh Rose, Murat Kantarcioglu

Organizations: Virginia Tech, Blacksburg, VA, USA · Vanderbilt University Medical Center, Nashville, TN, USA

Abstract

As LLM-powered chatbots are increasingly deployed in mental health services, detecting hallucinations and omissions has become critical for user safety. However, state-of-the-art LLM-as-a-judge methods often fail in high-risk healthcare contexts, where subtle errors can have serious consequences. We show that leading LLM judges achieve only 52% accuracy on mental health counseling data, with some hallucination detection approaches exhibiting near-zero recall. We identify the root cause as LLMs' inability to capture nuanced linguistic and therapeutic patterns recognized by domain experts. To address this, we propose a framework that integrates human expertise with LLMs to extract interpretable, domain-informed features across five analytical dimensions: logical consistency, entity verification, factual accuracy, linguistic uncertainty, and professional appropriateness. Experiments on a public mental health dataset and a new human-annotated dataset show that traditional machine learning models trained on these features achieve 0.717 F1 on our custom dataset and 0.849 F1 on a public benchmark for hallucination detection, with 0.59-0.64 F1 for omission detection across both datasets. Our results demonstrate that combining domain expertise with automated methods yields more reliable and transparent evaluation than black-box LLM judging in high-stakes mental health applications.

Figures & tables

Figure 1: The healthcare AI evaluation problem. Current methods fail in mental health contexts (top), while our human-informed approach achieves reliable performance through systematic integration of domain expertise and machine learning models (bottom).

Dataset	Samples	Hal.	Omis.
Custom (hal)	4,418	87	0
Cutom (omis)	3,368	0	124
Kaggle	994	243	511

Table 1: Dataset Distribution

Figure 2: An overview of the ensemble hallucination and omission detection framework that processes original prompt-response pairs through two parallel pathways: LLM-as-a-judge (e.g., Gpt-4o) evaluation that generates hallucination and omission scores, and multi-dimensional feature extraction using a LLM. The ensemble module integrates outputs from supervised ML classifiers trained on the engineered features with the LLM-as-a-judge scores to produce final classification results.

Traditional Methods
Method	DS	Task	Acc	Prc	Rec	F1
SelfCheckGPT	C	Hal	0.523	0.576	0.172	0.265
SelfCheckGPT	C	Omi	0.496	0.484	0.129	0.203
SelfCheckGPT	K	Hal	0.524	0.573	0.193	0.289
SelfCheckGPT	K	Omi	0.501	0.503	0.159	0.242
SAC3 (L3.3-70B)	C	Hal	0.900	0.000	0.000	0.000

Table 2: Baseline Methods Comparison

LLM	Classifier	F1	ROC-AUC
GPT-4o	MLP	0.680 $\pm$ 0.101	0.737 $\pm$ 0.098
	Logistic Reg.	0.641 $\pm$ 0.063	0.737 $\pm$ 0.096
	SAINT	0.609 $\pm$ 0.027	0.713 $\pm$ 0.101
GPT-5	SVM	0.601 $\pm$ 0.095	0.594 $\pm$ 0.117
	Random Forest	0.587 $\pm$ 0.058	0.615 $\pm$ 0.064
	Logistic Reg.	0.505 $\pm$ 0.086	0.633 $\pm$ 0.145

Table 3: Hallucination Detection: Top 3 Classifiers per LLM (Custom Dataset)

LLM	Classifier	F1	ROC-AUC
GPT-4o	SAINT	0.637 $\pm$ 0.030	0.661 $\pm$ 0.025
	Logistic Reg.	0.621 $\pm$ 0.071	0.683 $\pm$ 0.054
	SVM	0.620 $\pm$ 0.091	0.673 $\pm$ 0.051
GPT-5	Logistic Reg.	0.614 $\pm$ 0.033	0.644 $\pm$ 0.063
	Random Forest	0.606 $\pm$ 0.077	0.653 $\pm$ 0.083
	SVM	0.598 $\pm$ 0.043	0.602 $\pm$ 0.066

Table 4: Omission Detection: Top 3 Classifiers per LLM (Custom Dataset)

Hallucination Detection
Model	F1-Score	ROC-AUC
LightGBM	84.6 $\pm$ 1.7%	91.3 $\pm$ 1.9%
XGBoost	84.9 $\pm$ 1.2%	91.3 $\pm$ 1.2%
MLP	81.3 $\pm$ 1.5%	91.0 $\pm$ 1.7%
CatBoost	83.9 $\pm$ 1.3%	91.0 $\pm$ 0.9%
Random Forest	82.7 $\pm$ 1.7%	90.8 $\pm$ 1.0%

Table 5: Cross-Dataset Validation: Kaggle Mental Health Dataset Results

Custom-Hal		Custom-Omis
#	Feature (AUC)	#	Feature (AUC)
1	omis (0.585)	1	omis (0.648)
2	hal (0.621)	2	hal (0.649)
3	emotion (0.632)	3	emotion (0.650)
4	professional (0.646)	4	vague (0.654)
5	relation_fab (0.647)	5	certainty (0.658)

Table 6: Top 5 Most Critical Features by Performance Drop When Removed (ROC-AUC)

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

LLM	Classifier	Acc	Prc	Rec	F1	PR-AUC	ROC-AUC
GPT-4o	MLP	0.683 $\pm$ 0.099	0.695 $\pm$ 0.116	0.677 $\pm$ 0.119	0.680 $\pm$ 0.101	0.753 $\pm$ 0.107	0.737 $\pm$ 0.098
	Logistic Reg.	0.661 $\pm$ 0.052	0.684 $\pm$ 0.068	0.611 $\pm$ 0.096	0.641 $\pm$ 0.063	0.765 $\pm$ 0.108	0.737 $\pm$ 0.096
	SAINT	0.638 $\pm$ 0.036	0.673 $\pm$ 0.086	0.564 $\pm$ 0.057	0.609 $\pm$ 0.027	0.731 $\pm$ 0.108	0.713 $\pm$ 0.101
GPT-5	SVM	0.537 $\pm$ 0.101	0.534 $\pm$ 0.090	0.711 $\pm$ 0.176	0.601 $\pm$ 0.095	—	0.594 $\pm$ 0.117
	Random Forest	0.572 $\pm$ 0.069	0.577 $\pm$ 0.092	0.611 $\pm$ 0.094	0.587 $\pm$ 0.058	0.698 $\pm$ 0.085	0.615 $\pm$ 0.064
	Logistic Reg.	0.535 $\pm$ 0.066	0.548 $\pm$ 0.091	0.489 $\pm$ 0.155	0.505 $\pm$ 0.086	0.673 $\pm$ 0.110	0.633 $\pm$ 0.145

Appendix

Table 7: Hallucination Detection - Complete Metrics

LLM	Classifier	Acc	Prc	Rec	F1	PR-AUC	ROC-AUC
GPT-4o	SAINT	0.645 $\pm$ 0.026	0.660 $\pm$ 0.064	0.629 $\pm$ 0.095	0.637 $\pm$ 0.030	0.674 $\pm$ 0.039	0.661 $\pm$ 0.025
	Logistic Reg.	0.637 $\pm$ 0.066	0.662 $\pm$ 0.104	0.604 $\pm$ 0.127	0.621 $\pm$ 0.071	0.713 $\pm$ 0.046	0.683 $\pm$ 0.054
	SVM	0.657 $\pm$ 0.037	0.694 $\pm$ 0.049	0.581 $\pm$ 0.155	0.620 $\pm$ 0.091	0.675 $\pm$ 0.026	0.673 $\pm$ 0.051
GPT-5	Logistic Reg.	0.635 $\pm$ 0.023	0.653 $\pm$ 0.027	0.584 $\pm$ 0.062	0.614 $\pm$ 0.033	0.696 $\pm$ 0.057	0.644 $\pm$ 0.063
	Random Forest	0.633 $\pm$ 0.067	0.656 $\pm$ 0.077	0.568 $\pm$ 0.090	0.606 $\pm$ 0.077	0.674 $\pm$ 0.072	0.653 $\pm$ 0.083
	SVM	0.565 $\pm$ 0.033	0.555 $\pm$ 0.025	0.651 $\pm$ 0.078	0.598 $\pm$ 0.043	—	0.602 $\pm$ 0.066

Appendix

Table 8: Omission Detection - Complete Metrics

Model	Acc	Prc	Rec	F1	PR-AUC	ROC-AUC
Hallucination Detection
LightGBM	85.4 $\pm$ 1.9%	89.2 $\pm$ 1.7%	80.7 $\pm$ 1.9%	84.6 $\pm$ 1.7%	93.3 $\pm$ 1.7%	91.3 $\pm$ 1.9%
XGBoost	85.4 $\pm$ 1.2%	87.7 $\pm$ 1.2%	82.3 $\pm$ 1.2%	84.9 $\pm$ 1.2%	93.6 $\pm$ 1.5%	91.3 $\pm$ 1.2%
MLP	81.9 $\pm$ 1.7%	84.3 $\pm$ 1.5%	78.6 $\pm$ 1.7%	81.3 $\pm$ 1.5%	93.1 $\pm$ 1.5%	91.0 $\pm$ 1.7%
CatBoost	84.6 $\pm$ 0.9%	87.3 $\pm$ 1.3%	81.1 $\pm$ 0.9%	83.9 $\pm$ 1.3%	93.1 $\pm$ 1.3%	91.0 $\pm$ 0.9%
Random Forest	83.8 $\pm$ 1.0%	88.5 $\pm$ 1.7%	77.8 $\pm$ 1.0%	82.7 $\pm$ 1.7%	92.7 $\pm$ 1.7%	90.8 $\pm$ 1.0%

Appendix

Table 9: Supplementary Table S3: Kaggle Dataset - Complete Metrics

Hallucination Detection
Feature	ROC-AUC	PR-AUC	Accuracy	F1
hal_score	0.644	0.647	0.633	0.484
relevance_score	0.616	0.605	0.536	0.453
omis_score	0.578	0.584	0.539	0.554
entity_fab	0.519	0.519	0.523	0.581
professional_score	0.565	0.627	0.551	0.474

Appendix

Table 10: Supplementary Table S4: Individual Feature Performance (Averaged Across All Classifiers)

Task	GPT-4o	GPT-5	O3	L3.3-70B	OSS-20B	OSS-120B	Mean (CV)
Hal.	0.680	0.601	0.678	0.695	0.656	0.707	0.670 (0.057)
Omis.	0.637	0.614	0.614	0.591	—	—	0.614 (0.031)

Appendix

Table 11: Cross-Architecture Feature Extraction Stability (F1 Scores)

Adversarial Advocate-Critic Pipeline
Approach	Variant	F1	ROC-AUC
Adversarial	Baseline	0.520	—
Adversarial	+Features	0.370	—
Hybrid Rule-Based Detection
OR Rules	CatBoost	0.717	—
AND Rules	Random Forest	0.689	—

Appendix

Table 12: Specialized Approaches (Hallucination)

Method	Model	k=0	k=2	k=4	k=6	k=8
Balanced Omission Dataset (n=248)
Judge	GPT-5	0.678	0.700	0.690	0.670	0.677
Features+ML	GPT-5+RF	0.663	0.609	0.627	0.600	0.647
KMHC Omission Dataset (n=966)
Judge	GPT-4o	0.606	0.647	0.649	0.664	0.652
Judge	GPT-5	0.609	0.581	0.593	0.585	0.593

Appendix

Table 13: Few-Shot Learning Comparison: LLM-as-a-judge vs Features+ML (Omission Detection)

Balanced Hallucination Dataset (n=174)
Method	Model	k=0	k=2	k=4	k=6	k=8
Judge	GPT-4-Turbo	0.204	0.204	0.198	0.179	0.194
Judge	GPT-4o	0.126	0.163	0.125	0.143	0.148
Judge	GPT-4o-mini	0.043	0.147	0.147	0.109	0.129
KMHC Hallucination Dataset (n=486)
Judge	GPT-4o	0.093	0.186	0.221	0.282	0.290

Appendix

Table 14: Few-Shot Learning: LLM-as-a-judge (Hallucination Detection)

Language Model Error Detection

Language model error detection identifies unreliable answers, reasoning steps, and actions so systems can flag, verify, or repair them.

See all 90 papers

Disentangling Prompt Element Level Risk Factors for Hallucinations and Omissions in Mental Health LLM Responses
Mar 10, 2026Congning Ni, Sarvech Qadir, Bryan Steitz +14Hallucination in Language Models LLM Safety Evaluation
Vanderbilt University Medical Center, Nashville, TN, USA · Vanderbilt University, Nashville, TN, USA · Lirio, Knoxville, TN, USA +1
HalluScan: A Systematic Benchmark for Detecting and Mitigating Hallucinations in Instruction-Following LLMs
May 4, 2026Ahmed CherifLLM Evaluation LLM Hallucination Mitigation
1Sofrecom Tunisia, Orange Innovation, Tunis, 1053, Tunisia.
Logical Consistency as a Bridge: Improving LLM Hallucination Detection via Label Constraint Modeling between Responses and Self-Judgments
May 5, 2026Hao Mi, Qiang Sheng, Shaofei Wang +7Hallucination in Language Models LLM Hallucination Detection
Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences

Mar 10, 2026J/K move · Enter open · S save

Disentangling Prompt Element Level Risk Factors for Hallucinations and Omissions in Mental Health LLM Responses

Congning Ni, Sarvech Qadir, Bryan Steitz, Mihir Sachin Vaidya+13

Vanderbilt University Medical Center, Nashville, TN, USA · Vanderbilt University, Nashville, TN, USA · Lirio, Knoxville, TN, USA+1

Hallucination in Language Models LLM Safety Evaluation Clinical Language Model Evaluation Question Answering

Open paper AskPDF arXiv

Mental health concerns are often expressed outside clinical settings, including in high-distress help seeking, where safety-critical guidance may be needed. Consumer health informatics systems increasingly incorporate large language models (LLMs) for mental health question answering, yet many evaluations underrepresent narrative, high-distress inquiries. We introduce UTCO (User, Topic, Context, Tone), a prompt construction framework that represents an inquiry as four controllable elements for systematic stress testing. Using 2,075 UTCO-generated prompts, we evaluated Llama 3.3 and annotated hallucinations (fabricated or incorrect clinical content) and omissions (missing clinically necessary or safety-critical guidance). Hallucinations occurred in 6.5% of responses and omissions in 13.2%, with omissions concentrated in crisis and suicidal ideation prompts. Across regression, element-specific matching, and similarity-matched comparisons, failures were most consistently associated with context and tone, while user-background indicators showed no systematic differences after balancing. These findings support evaluating omissions as a primary safety outcome and moving beyond static benchmark question sets.

Abstract

Language Model Error Detection

Explore similar work

Disentangling Prompt Element Level Risk Factors for Hallucinations and Omissions in Mental Health LLM Responses

HalluScan: A Systematic Benchmark for Detecting and Mitigating Hallucinations in Instruction-Following LLMs

Logical Consistency as a Bridge: Improving LLM Hallucination Detection via Label Constraint Modeling between Responses and Self-Judgments