Blending Human and LLM Expertise to Detect Hallucinations and Omissions in Mental Health Chatbot Responses

Khizar Hussain; Bradley A. Malin; Zhijun Yin; Susannah Leigh Rose; Murat Kantarcioglu

cs.CLMar 17, 2026

Blending Human and LLM Expertise to Detect Hallucinations and Omissions in Mental Health Chatbot Responses

Authors: Khizar Hussain, Bradley A. Malin, Zhijun Yin, Susannah Leigh Rose, Murat Kantarcioglu

Organizations: Virginia Tech, Blacksburg, VA, USA · Vanderbilt University Medical Center, Nashville, TN, USA

Abstract

As LLM-powered chatbots are increasingly deployed in mental health services, detecting hallucinations and omissions has become critical for user safety. However, state-of-the-art LLM-as-a-judge methods often fail in high-risk healthcare contexts, where subtle errors can have serious consequences. We show that leading LLM judges achieve only 52% accuracy on mental health counseling data, with some hallucination detection approaches exhibiting near-zero recall. We identify the root cause as LLMs' inability to capture nuanced linguistic and therapeutic patterns recognized by domain experts. To address this, we propose a framework that integrates human expertise with LLMs to extract interpretable, domain-informed features across five analytical dimensions: logical consistency, entity verification, factual accuracy, linguistic uncertainty, and professional appropriateness. Experiments on a public mental health dataset and a new human-annotated dataset show that traditional machine learning models trained on these features achieve 0.717 F1 on our custom dataset and 0.849 F1 on a public benchmark for hallucination detection, with 0.59-0.64 F1 for omission detection across both datasets. Our results demonstrate that combining domain expertise with automated methods yields more reliable and transparent evaluation than black-box LLM judging in high-stakes mental health applications.

Figures & tables

Figure 1: The healthcare AI evaluation problem. Current methods fail in mental health contexts (top), while our human-informed approach achieves reliable performance through systematic integration of domain expertise and machine learning models (bottom).

Dataset	Samples	Hal.	Omis.
Custom (hal)	4,418	87	0
Cutom (omis)	3,368	0	124
Kaggle	994	243	511

Table 1: Dataset Distribution

Figure 2: An overview of the ensemble hallucination and omission detection framework that processes original prompt-response pairs through two parallel pathways: LLM-as-a-judge (e.g., Gpt-4o) evaluation that generates hallucination and omission scores, and multi-dimensional feature extraction using a LLM. The ensemble module integrates outputs from supervised ML classifiers trained on the engineered features with the LLM-as-a-judge scores to produce final classification results.

Traditional Methods
Method	DS	Task	Acc	Prc	Rec	F1
SelfCheckGPT	C	Hal	0.523	0.576	0.172	0.265
SelfCheckGPT	C	Omi	0.496	0.484	0.129	0.203
SelfCheckGPT	K	Hal	0.524	0.573	0.193	0.289
SelfCheckGPT	K	Omi	0.501	0.503	0.159	0.242
SAC3 (L3.3-70B)	C	Hal	0.900	0.000	0.000	0.000

Table 2: Baseline Methods Comparison

LLM	Classifier	F1	ROC-AUC
GPT-4o	MLP	0.680 $\pm$ 0.101	0.737 $\pm$ 0.098
	Logistic Reg.	0.641 $\pm$ 0.063	0.737 $\pm$ 0.096
	SAINT	0.609 $\pm$ 0.027	0.713 $\pm$ 0.101
GPT-5	SVM	0.601 $\pm$ 0.095	0.594 $\pm$ 0.117
	Random Forest	0.587 $\pm$ 0.058	0.615 $\pm$ 0.064
	Logistic Reg.	0.505 $\pm$ 0.086	0.633 $\pm$ 0.145

Table 3: Hallucination Detection: Top 3 Classifiers per LLM (Custom Dataset)

LLM	Classifier	F1	ROC-AUC
GPT-4o	SAINT	0.637 $\pm$ 0.030	0.661 $\pm$ 0.025
	Logistic Reg.	0.621 $\pm$ 0.071	0.683 $\pm$ 0.054
	SVM	0.620 $\pm$ 0.091	0.673 $\pm$ 0.051
GPT-5	Logistic Reg.	0.614 $\pm$ 0.033	0.644 $\pm$ 0.063
	Random Forest	0.606 $\pm$ 0.077	0.653 $\pm$ 0.083
	SVM	0.598 $\pm$ 0.043	0.602 $\pm$ 0.066

Table 4: Omission Detection: Top 3 Classifiers per LLM (Custom Dataset)

Hallucination Detection
Model	F1-Score	ROC-AUC
LightGBM	84.6 $\pm$ 1.7%	91.3 $\pm$ 1.9%
XGBoost	84.9 $\pm$ 1.2%	91.3 $\pm$ 1.2%
MLP	81.3 $\pm$ 1.5%	91.0 $\pm$ 1.7%
CatBoost	83.9 $\pm$ 1.3%	91.0 $\pm$ 0.9%
Random Forest	82.7 $\pm$ 1.7%	90.8 $\pm$ 1.0%

Table 5: Cross-Dataset Validation: Kaggle Mental Health Dataset Results

Custom-Hal		Custom-Omis
#	Feature (AUC)	#	Feature (AUC)
1	omis (0.585)	1	omis (0.648)
2	hal (0.621)	2	hal (0.649)
3	emotion (0.632)	3	emotion (0.650)
4	professional (0.646)	4	vague (0.654)
5	relation_fab (0.647)	5	certainty (0.658)

Table 6: Top 5 Most Critical Features by Performance Drop When Removed (ROC-AUC)

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

LLM	Classifier	Acc	Prc	Rec	F1	PR-AUC	ROC-AUC
GPT-4o	MLP	0.683 $\pm$ 0.099	0.695 $\pm$ 0.116	0.677 $\pm$ 0.119	0.680 $\pm$ 0.101	0.753 $\pm$ 0.107	0.737 $\pm$ 0.098
	Logistic Reg.	0.661 $\pm$ 0.052	0.684 $\pm$ 0.068	0.611 $\pm$ 0.096	0.641 $\pm$ 0.063	0.765 $\pm$ 0.108	0.737 $\pm$ 0.096
	SAINT	0.638 $\pm$ 0.036	0.673 $\pm$ 0.086	0.564 $\pm$ 0.057	0.609 $\pm$ 0.027	0.731 $\pm$ 0.108	0.713 $\pm$ 0.101
GPT-5	SVM	0.537 $\pm$ 0.101	0.534 $\pm$ 0.090	0.711 $\pm$ 0.176	0.601 $\pm$ 0.095	—	0.594 $\pm$ 0.117
	Random Forest	0.572 $\pm$ 0.069	0.577 $\pm$ 0.092	0.611 $\pm$ 0.094	0.587 $\pm$ 0.058	0.698 $\pm$ 0.085	0.615 $\pm$ 0.064
	Logistic Reg.	0.535 $\pm$ 0.066	0.548 $\pm$ 0.091	0.489 $\pm$ 0.155	0.505 $\pm$ 0.086	0.673 $\pm$ 0.110	0.633 $\pm$ 0.145

Appendix

Table 7: Hallucination Detection - Complete Metrics

LLM	Classifier	Acc	Prc	Rec	F1	PR-AUC	ROC-AUC
GPT-4o	SAINT	0.645 $\pm$ 0.026	0.660 $\pm$ 0.064	0.629 $\pm$ 0.095	0.637 $\pm$ 0.030	0.674 $\pm$ 0.039	0.661 $\pm$ 0.025
	Logistic Reg.	0.637 $\pm$ 0.066	0.662 $\pm$ 0.104	0.604 $\pm$ 0.127	0.621 $\pm$ 0.071	0.713 $\pm$ 0.046	0.683 $\pm$ 0.054
	SVM	0.657 $\pm$ 0.037	0.694 $\pm$ 0.049	0.581 $\pm$ 0.155	0.620 $\pm$ 0.091	0.675 $\pm$ 0.026	0.673 $\pm$ 0.051
GPT-5	Logistic Reg.	0.635 $\pm$ 0.023	0.653 $\pm$ 0.027	0.584 $\pm$ 0.062	0.614 $\pm$ 0.033	0.696 $\pm$ 0.057	0.644 $\pm$ 0.063
	Random Forest	0.633 $\pm$ 0.067	0.656 $\pm$ 0.077	0.568 $\pm$ 0.090	0.606 $\pm$ 0.077	0.674 $\pm$ 0.072	0.653 $\pm$ 0.083
	SVM	0.565 $\pm$ 0.033	0.555 $\pm$ 0.025	0.651 $\pm$ 0.078	0.598 $\pm$ 0.043	—	0.602 $\pm$ 0.066

Appendix

Table 8: Omission Detection - Complete Metrics

Model	Acc	Prc	Rec	F1	PR-AUC	ROC-AUC
Hallucination Detection
LightGBM	85.4 $\pm$ 1.9%	89.2 $\pm$ 1.7%	80.7 $\pm$ 1.9%	84.6 $\pm$ 1.7%	93.3 $\pm$ 1.7%	91.3 $\pm$ 1.9%
XGBoost	85.4 $\pm$ 1.2%	87.7 $\pm$ 1.2%	82.3 $\pm$ 1.2%	84.9 $\pm$ 1.2%	93.6 $\pm$ 1.5%	91.3 $\pm$ 1.2%
MLP	81.9 $\pm$ 1.7%	84.3 $\pm$ 1.5%	78.6 $\pm$ 1.7%	81.3 $\pm$ 1.5%	93.1 $\pm$ 1.5%	91.0 $\pm$ 1.7%
CatBoost	84.6 $\pm$ 0.9%	87.3 $\pm$ 1.3%	81.1 $\pm$ 0.9%	83.9 $\pm$ 1.3%	93.1 $\pm$ 1.3%	91.0 $\pm$ 0.9%
Random Forest	83.8 $\pm$ 1.0%	88.5 $\pm$ 1.7%	77.8 $\pm$ 1.0%	82.7 $\pm$ 1.7%	92.7 $\pm$ 1.7%	90.8 $\pm$ 1.0%

Appendix

Table 9: Supplementary Table S3: Kaggle Dataset - Complete Metrics

Hallucination Detection
Feature	ROC-AUC	PR-AUC	Accuracy	F1
hal_score	0.644	0.647	0.633	0.484
relevance_score	0.616	0.605	0.536	0.453
omis_score	0.578	0.584	0.539	0.554
entity_fab	0.519	0.519	0.523	0.581
professional_score	0.565	0.627	0.551	0.474

Appendix

Table 10: Supplementary Table S4: Individual Feature Performance (Averaged Across All Classifiers)

Task	GPT-4o	GPT-5	O3	L3.3-70B	OSS-20B	OSS-120B	Mean (CV)
Hal.	0.680	0.601	0.678	0.695	0.656	0.707	0.670 (0.057)
Omis.	0.637	0.614	0.614	0.591	—	—	0.614 (0.031)

Appendix

Table 11: Cross-Architecture Feature Extraction Stability (F1 Scores)

Adversarial Advocate-Critic Pipeline
Approach	Variant	F1	ROC-AUC
Adversarial	Baseline	0.520	—
Adversarial	+Features	0.370	—
Hybrid Rule-Based Detection
OR Rules	CatBoost	0.717	—
AND Rules	Random Forest	0.689	—

Appendix

Table 12: Specialized Approaches (Hallucination)

Method	Model	k=0	k=2	k=4	k=6	k=8
Balanced Omission Dataset (n=248)
Judge	GPT-5	0.678	0.700	0.690	0.670	0.677
Features+ML	GPT-5+RF	0.663	0.609	0.627	0.600	0.647
KMHC Omission Dataset (n=966)
Judge	GPT-4o	0.606	0.647	0.649	0.664	0.652
Judge	GPT-5	0.609	0.581	0.593	0.585	0.593

Appendix

Table 13: Few-Shot Learning Comparison: LLM-as-a-judge vs Features+ML (Omission Detection)

Balanced Hallucination Dataset (n=174)
Method	Model	k=0	k=2	k=4	k=6	k=8
Judge	GPT-4-Turbo	0.204	0.204	0.198	0.179	0.194
Judge	GPT-4o	0.126	0.163	0.125	0.143	0.148
Judge	GPT-4o-mini	0.043	0.147	0.147	0.109	0.129
KMHC Hallucination Dataset (n=486)
Judge	GPT-4o	0.093	0.186	0.221	0.282	0.290

Appendix

Table 14: Few-Shot Learning: LLM-as-a-judge (Hallucination Detection)

Language Model Error Detection

Language model error detection identifies unreliable answers, reasoning steps, and actions so systems can flag, verify, or repair them.

See all 90 papers

Mar 10, 2026

Disentangling Prompt Element Level Risk Factors for Hallucinations and Omissions in Mental Health LLM Responses

Mental health concerns are often expressed outside clinical settings, including in high-distress help seeking, where safety-critical guidance may be needed. Consumer health informatics systems increasingly incorporate large language models (LLMs) for mental health question answering, yet many evaluations underrepresent narrative, high-distress inquiries. We introduce UTCO (User, Topic, Context, Tone), a prompt construction framework that represents an inquiry as four controllable elements for systematic stress testing. Using 2,075 UTCO-generated prompts, we evaluated Llama 3.3 and annotated hallucinations (fabricated or incorrect clinical content) and omissions (missing clinically necessary or safety-critical guidance). Hallucinations occurred in 6.5% of responses and omissions in 13.2%, with omissions concentrated in crisis and suicidal ideation prompts. Across regression, element-specific matching, and similarity-matched comparisons, failures were most consistently associated with context and tone, while user-background indicators showed no systematic differences after balancing. These findings support evaluating omissions as a primary safety outcome and moving beyond static benchmark question sets.

Congning Ni, Sarvech Qadir, Bryan Steitz +14

Vanderbilt University Medical Center, Nashville, TN, USA · Vanderbilt University, Nashville, TN, USA · Lirio, Knoxville, TN, USA +1

Hallucination in Language Models LLM Safety Evaluation Clinical Language Model Evaluation Question Answering

Ask PDF

May 4, 2026

HalluScan: A Systematic Benchmark for Detecting and Mitigating Hallucinations in Instruction-Following LLMs

Large Language Models (LLMs) have demonstrated remarkable capabilities across diverse natural language processing tasks, yet they remain susceptible to hallucinations -- generating content that is factually incorrect, unfaithful to provided context, or misaligned with user instructions. We present HalluScan, a comprehensive benchmark framework that systematically evaluates hallucination detection and mitigation across 72 configurations spanning 6 detection methods, 4 open-weight model families, and 3 diverse domains. We introduce three key contributions: (1) HalluScore, a novel composite metric that achieves a Pearson correlation of r = 0.41 with human expert judgments; (2) Adaptive Detection Routing (ADR), an intelligent routing algorithm achieving 2.0x cost reduction with only 0.1% AUROC degradation; and (3) systematic error cascade decomposition revealing substantial variation in hallucination error types across domains. Our experiments reveal that NLI Verification achieves the highest overall AUROC of 0.88, while RAV achieves the second-highest AUROC of 0.66.

Ahmed Cherif

1Sofrecom Tunisia, Orange Innovation, Tunis, 1053, Tunisia.

LLM Evaluation LLM Hallucination Mitigation Hallucination in Language Models LLM Reliability LLM Hallucination Detection

Ask PDF

May 5, 2026

Logical Consistency as a Bridge: Improving LLM Hallucination Detection via Label Constraint Modeling between Responses and Self-Judgments

Large Language Models (LLMs) are prone to factual hallucinations, risking their reliability in real-world applications. Existing hallucination detectors mainly extract micro-level intrinsic patterns for uncertainty quantification or elicit macro-level self-judgments through verbalized prompts. However, these methods address only a single facet of the hallucination, focusing either on implicit neural uncertainty or explicit symbolic reasoning, thereby treating these inherently coupled behaviors in isolation and failing to exploit their interdependence for a holistic view. In this paper, we propose LaaB (Logical Consistency-as-a-Bridge), a framework that bridges neural features and symbolic judgments for hallucination detection. LaaB introduces a "meta-judgment" process to map symbolic labels back into the feature space. By leveraging the inherent logical bridge where response and meta-judgment labels are either the same or opposite based on the self-judgment's semantics, LaaB aligns and integrates dual-view signals via mutual learning and enhances the hallucination detection. Extensive experiments on 4 public datasets, across 4 LLMs, against 8 baselines demonstrate the superiority of LaaB.

Hao Mi, Qiang Sheng, Shaofei Wang +7

Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences

Hallucination in Language Models LLM Hallucination Detection Generative AI Evaluation

Ask PDF

Abstract

Language Model Error Detection

Explore similar work

Disentangling Prompt Element Level Risk Factors for Hallucinations and Omissions in Mental Health LLM Responses

HalluScan: A Systematic Benchmark for Detecting and Mitigating Hallucinations in Instruction-Following LLMs

Logical Consistency as a Bridge: Improving LLM Hallucination Detection via Label Constraint Modeling between Responses and Self-Judgments