Disentangling Prompt Element Level Risk Factors for Hallucinations and Omissions in Mental Health LLM Responses

Congning Ni; Sarvech Qadir; Bryan Steitz; Mihir Sachin Vaidya; Qingyuan Song; Lantian Xia; Shelagh Mulvaney; Siru Liu

cs.CLMar 10, 2026

Disentangling Prompt Element Level Risk Factors for Hallucinations and Omissions in Mental Health LLM Responses

Authors: Congning Ni, Sarvech Qadir, Bryan Steitz, Mihir Sachin Vaidya, Qingyuan Song, Lantian Xia, Shelagh Mulvaney, Siru Liu,

Organizations: Vanderbilt University Medical Center, Nashville, TN, USA · Vanderbilt University, Nashville, TN, USA · Lirio, Knoxville, TN, USA · Virginia Tech, Blacksburg, VA, USA

Abstract

Mental health concerns are often expressed outside clinical settings, including in high-distress help seeking, where safety-critical guidance may be needed. Consumer health informatics systems increasingly incorporate large language models (LLMs) for mental health question answering, yet many evaluations underrepresent narrative, high-distress inquiries. We introduce UTCO (User, Topic, Context, Tone), a prompt construction framework that represents an inquiry as four controllable elements for systematic stress testing. Using 2,075 UTCO-generated prompts, we evaluated Llama 3.3 and annotated hallucinations (fabricated or incorrect clinical content) and omissions (missing clinically necessary or safety-critical guidance). Hallucinations occurred in 6.5% of responses and omissions in 13.2%, with omissions concentrated in crisis and suicidal ideation prompts. Across regression, element-specific matching, and similarity-matched comparisons, failures were most consistently associated with context and tone, while user-background indicators showed no systematic differences after balancing. These findings support evaluating omissions as a primary safety outcome and moving beyond static benchmark question sets.

Figures & tables

Clinical topic domain	Total $N$	Hallucination rate	Omission rate
Medication-Specific Issues	174	10.9% (19)	13.2% (23)
Professional Care Navigation	155	9.7% (15)	10.3% (16)
Anxiety Experience and Management	300	8.3% (25)	14.0% (42)
Clinical Depression Experience and Management	348	7.5% (26)	18.7% (65)
Social & Relationship Dynamics	179	7.3% (13)	7.3% (13)
Treatment Approaches	272	6.6% (18)	9.2% (25)

Table 1: Prevalence of hallucinations and omissions by clinical topic domain (

N=2{,}075

Figure 1: SHAP summary plots for the gradient-boosted tree models predicting hallucination (a) and omission (b). Features are ordered by overall contribution magnitude across inquiries.

Figure 2: Topic-stratified linguistic trigger profiles (descriptive). Heatmap of case-level mean LLM-judge severity scores (0 to 3) across topic domains. Topic-level summaries are descriptive and may be unstable in sparsely represented domains.

Linguistic Dimension (0–3)	Hallucination (n=61)	Omission (n=66)
Ambiguity / Underspecification	$2.77\pm 0.37$	$2.67\pm 0.45$
Missing Clinical Constraints	$2.65\pm 0.44$	$2.26\pm 0.58$
Emotional Load / Crisis Cues	$2.00\pm 0.76$	$2.46\pm 0.73$
Multi-intent / Scope Creep	$1.57\pm 0.75$	$1.46\pm 0.80$
Temporal Confusion	$0.76\pm 0.56$	$0.89\pm 0.58$
Logical Contradiction	$0.03\pm 0.13$	$0.15\pm 0.40$

Table 2: Linguistic taxonomy severity by failure mode (case-level). Scores (0 to 3) adjudicated by a blinded judge across up to

K=10

neighbors. Values are reported as mean

\pm

SD.

Hallucination in Language Models

Hallucination in language models studies how systems produce unsupported or false claims and how to detect, explain, and reduce them.

See all 254 papers

Blending Human and LLM Expertise to Detect Hallucinations and Omissions in Mental Health Chatbot Responses
Mar 17, 2026Khizar Hussain, Bradley A. Malin, Zhijun Yin +2Language Model Error Detection Hallucination Detection
Virginia Tech, Blacksburg, VA, USA · Vanderbilt University Medical Center, Nashville, TN, USA
From Clinical Judgment to Checkable Rules: Clinician-Led Guardrails for Mental-Health Chatbots
Oct 22, 2026Chenhan Lyu, Kit Wislocki, Sarah Elizabeth Stoeckl +2LLM Guardrails LLM Safety
University of California, Irvine, USA
Lost in Delusion: Examining LLM Safety Under User Delusions and Distress
May 31, 2026Andrew Aquilina, Chetna Nihalani, Vasudha Varadarajan +3Mental Health LLM Safety Alignment
University of Pittsburgh · Carnegie Mellon University · Fordham University

Mar 17, 2026J/K move · Enter open · S save

Blending Human and LLM Expertise to Detect Hallucinations and Omissions in Mental Health Chatbot Responses

Khizar Hussain, Bradley A. Malin, Zhijun Yin, Susannah Leigh Rose+1

Virginia Tech, Blacksburg, VA, USA · Vanderbilt University Medical Center, Nashville, TN, USA

Language Model Error Detection Hallucination Detection Clinical Language Model Evaluation

Open paper AskPDF arXiv

As LLM-powered chatbots are increasingly deployed in mental health services, detecting hallucinations and omissions has become critical for user safety. However, state-of-the-art LLM-as-a-judge methods often fail in high-risk healthcare contexts, where subtle errors can have serious consequences. We show that leading LLM judges achieve only 52% accuracy on mental health counseling data, with some hallucination detection approaches exhibiting near-zero recall. We identify the root cause as LLMs' inability to capture nuanced linguistic and therapeutic patterns recognized by domain experts. To address this, we propose a framework that integrates human expertise with LLMs to extract interpretable, domain-informed features across five analytical dimensions: logical consistency, entity verification, factual accuracy, linguistic uncertainty, and professional appropriateness. Experiments on a public mental health dataset and a new human-annotated dataset show that traditional machine learning models trained on these features achieve 0.717 F1 on our custom dataset and 0.849 F1 on a public benchmark for hallucination detection, with 0.59-0.64 F1 for omission detection across both datasets. Our results demonstrate that combining domain expertise with automated methods yields more reliable and transparent evaluation than black-box LLM judging in high-stakes mental health applications.

Abstract

Hallucination in Language Models

Explore similar work

Blending Human and LLM Expertise to Detect Hallucinations and Omissions in Mental Health Chatbot Responses

From Clinical Judgment to Checkable Rules: Clinician-Led Guardrails for Mental-Health Chatbots

Lost in Delusion: Examining LLM Safety Under User Delusions and Distress