PromptSense: Sentiment-Aware Mental Health Chatbot with Dynamic Prompting

Sugi Lee; Jianlong Zhou; Jing sUN

cs.AIOct 27, 2026

PromptSense: Sentiment-Aware Mental Health Chatbot with Dynamic Prompting

Authors: Sugi Lee, Jianlong Zhou, Jing sUN

Organizations: Transdisciplinary School, University of Technology Sydney, Sydney, Australia · Data Science Institute, University of Technology Sydney, Sydney, Australia

Abstract

Current Large Language Models (LLMs)-based mental health chatbots often lack sufficient empathy, contextual awareness, and safety mechanisms, potentially leading to ineffective or harmful interactions. This paper proposes a novel framework that integrates sentiment and depression analysis with a rule-based filtering mechanism prior to model inference. By dynamically constructing prompts based on emotional and risk-related insights, the framework aims to significantly improve the contextual relevance, empathy, and risk-aware of model-generated responses. We evaluated four model strategies and two prompt engineered variants across qualitative metrics such as empathy, helpfulness, clarity, relevance, safety, as well as toxicity assessments using the Perspective API. Experimental results indicate that the prompt engineering approaches outperforms DSFT and RLAIF by 1.8% and 1.1% respectively.

Figures & tables

Study	Objective	Dataset
[ 16 ]	Fine-tuning LLaMA2-7B with the sets of instructions	IMHI dataset
[ 17 ]	Fine-tuning LoRA-based approach on top of LLaMA2-7B	CounselChat dataset
[ 8 ]	Fine-tuning GPT-4 model with localised prompt engineering	CounselChat dataset, local Indonesian mental health discussions
[ 10 ]	Fine-tuning BERT and RoBERTa with Mental Health dataset	SWMH, Depression Reddit, and other data collected from Reddit

Table 1: Selected Related Work

Figure 1: Proposed Model Framework.

Figure 2: Mental Health LLM Fine Tuning Process.

Dataset	Application	Train	Val	Test
SWMH [ 11 ]	SAM’s Mental Head	34,823	8,706	10,883
GoEmotions [ 7 ]	SAM’s Emotion Head	43,410	5,426	5,427
NRC VAS Lexicon [ 14 ]	SAM’s VAD Head	-	-	-
Counsel Chat [ 1 ]	Fine Tuning LLM	2,612	-	-
Emphatic Dialogues [ 15 ]	Model Evaluation	-	-	418

Table 2: A Summary of Datasets

Metric	Description
Empathy	Does the response acknowledge and validate the user’s feelings?
Helpfulness	Does it offer useful strategies, advice, or insights?
Clarity	Is the response clearly written, easy to understand, and coherent?
Relevance	Does it directly address the user’s question or concern?
Safety	Does it avoid harmful, dismissive, or risky advice?
Toxicity	A rude, disrespectful, or unreasonable comment that is likely to make people leave a discussion

Table 3: Evaluation Metrics

Model	MH Accy	EH F1	V RMSE	A RMSE
Original (Frozen)	0.5086	0.0801	0.2762	0.1974
Base (Frozen)	0.5787	0.1088	0.2587	0.1932
Proposed Model (Frozen)	0.5997	0.3527	0.2420	0.1990
Proposed Model (Unfreeze)	0.6803	0.4840	0.2025	0.1835

Table 4: Evaluation of Sentiment Analysis Model

Model	MIXT [ 13 ]	MIST [ 12 ]	MELL [ 16 ]	AVG
Base	8.63	9.12	8.44	8.73
Base + PE-Q	8.70	9.26	8.54	8.83
Base + PE-P	8.78	9.25	8.52	8.85
DSFT	8.59	9.08	8.36	8.68
DSFT + PE-Q	8.70	9.23	8.46	8.80
DSFT + PE-P	8.66	9.20	8.46	8.77

Table 5: AI Judge Evaluation of Mental Health Model (Average Score)

Figure 3: Empathy vs. Other Metrics Trade-offs Across Models and Prompting Strategies.

Model	Toxicity	Identity Attack	Insult	Threat	Compassion	Respect	Nuance	Incoherent
Base	0.0166	0.0022	0.0094	0.0070	0.9250	0.8815	0.4387	0.1748
Base + PE-Q	0.0156	0.0021	0.0091	0.0069	0.9415	0.8838	0.4452	0.1506
Base + PE-P	0.0180	0.0022	0.0109	0.0070	0.9391	0.8814	0.4638	0.1564
DSFT	0.0174	0.0022	0.0096	0.0072	0.9246	0.8817	0.4092	0.1971
DSFT + PE-Q	0.0178	0.0023	0.0102	0.0069	0.9386	0.8841	0.4268	0.1690
DSFT + PE-P	0.0178	0.0023	0.0099	0.0071	0.9380	0.8813	0.4330	0.1747

Table 6: Perspective API Evaluation of Mental Health Model

Mental Health Conversational Agents

See all 25 papers

Lost in Delusion: Examining LLM Safety Under User Delusions and Distress
May 31, 2026Andrew Aquilina, Chetna Nihalani, Vasudha Varadarajan +3Mental Health LLM Safety Alignment
University of Pittsburgh · Carnegie Mellon University · Fordham University
Guiding Language Models to Be More Empathetic: Culturally Sensitive Mental Health Advice Generation Through Human-LLM Collaboration
Jul 26, 2026Fatema Tuj Johora Faria, Mukaffi Bin Moin, Md. Mahfuzur Rahman +3Mental Health Counseling LLM Prompting
Ahsanullah University of Science and Technology · Bangladesh University of Business and Technology · The University of New South Wales +2
Risk Governance for Generative AI Mental Health Support: A Multi-Turn Safety Architecture
Jul 17, 2026Anabela C. Areias, Catarina Botelho, António Farinhas +7Emotional Support Conversation Mental Health
Sword Health, New York, NY, United States · Yale Child Study Center, Yale School of Medicine, New Haven, CT, United States · Department of Psychology, University of Virginia, Charlottesville, VA, USA

May 31, 2026J/K move · Enter open · S save

Lost in Delusion: Examining LLM Safety Under User Delusions and Distress

Andrew Aquilina, Chetna Nihalani, Vasudha Varadarajan, Nathan S. Fishbein+2

University of Pittsburgh · Carnegie Mellon University · Fordham University

Mental Health LLM Safety Alignment LLM Safety LLM Safety Evaluation Mental Health Conversational Agents

Open paper AskPDF arXiv

LLM chatbots increasingly serve as a first source of support for people in psychological distress, including those whose distress is entangled with delusional beliefs. Prior work on LLM mental-health safety largely evaluates general therapeutic quality or single-turn crisis detection, leaving unclear how models behave when distress is intertwined with delusion over sustained conversations. We address this gap with matched multi-turn simulations, across clinically grounded personas and six LLMs, that pair each delusional conversation with a distress-only control to isolate the effect of delusional framing. This reveals a recognition-intervention gap: models detect distress at comparable rates regardless of framing, yet sharply fail to act on it once distress is embedded in delusion, with safety interventions suppressed by up to 4.5x. The failure tracks accumulated acceptance of the user's premises rather than emotional validation. Worse, the intuitive fix of prompting models to assess user distress backfires under delusional framing; only delusion-aware prompting with explicit response guidance closes the gap, and even this depends on a delusion classifier that is itself unreliable on the most vulnerable models. Safe deployment therefore requires treating delusional framing as a distinct risk signal that overrides conversational accommodation.

Abstract

Mental Health Conversational Agents

Explore similar work

Lost in Delusion: Examining LLM Safety Under User Delusions and Distress

Guiding Language Models to Be More Empathetic: Culturally Sensitive Mental Health Advice Generation Through Human-LLM Collaboration

Risk Governance for Generative AI Mental Health Support: A Multi-Turn Safety Architecture