GrandGuard: Taxonomy, Benchmark, and Safeguards for Elderly-Chatbot Interaction Safety

Changxuan Fan; Xi Yang; Yueyuan Zheng; Bin Zhou; Yuanping Wang; Wenbin Hu; Huihao Jing; Ki Sen Hung

cs.HCApr 7, 2026

GrandGuard: Taxonomy, Benchmark, and Safeguards for Elderly-Chatbot Interaction Safety

Authors: Changxuan Fan, Xi Yang, Yueyuan Zheng, Bin Zhou, Yuanping Wang, Wenbin Hu, Huihao Jing, Ki Sen Hung,

Organizations: The Hong Kong University of Science and Technology

Abstract

As older adults increasingly use LLM-based chatbots for companionship and assistance, a safety gap is emerging. Older adults may face vulnerabilities from social isolation, limited digital literacy, and cognitive decline, yet existing safety benchmarks largely target general harms and overlook elderly-specific risks. For example, a prompt such as "how to repair a ceiling light alone in the dark" may be benign for most users but poses a serious fall risk for older adults with mobility limitations. We introduce GrandGuard, the first comprehensive framework for assessing and mitigating elderly-specific contextual risks in LLM interactions. We develop a three-level taxonomy with 50 fine-grained risk types across mental well-being, financial, medical, toxicity, and privacy domains, grounded in real-world incidents, community discussions, and analysis of stakeholder studies. Using this taxonomy, we construct a benchmark of 10,404 labeled prompts and responses, showing that several leading LLMs mishandle elderly-specific contextual risks in over 50% of cases. We mitigate these failures with two safeguards: a fine-tuned Llama-Guard-3 and a policy-enhanced gpt-oss-safeguard-20b, achieving up to 96.2% and 90.9% unsafe-prompt detection accuracy, respectively. GrandGuard lays the groundwork for AI systems that move beyond general safety to support aging populations.

Figures & tables

Figure 1: GrandGuard safety evaluation criteria. Prompts are assessed for elderly-specific contextual risks. Responses are evaluated using dual criteria: Risk Indication (recognizing elderly-specific concerns) and Harm Avoidance (avoiding harmful enablement while suggesting safer alternatives).

Figure 2: Overview of the GrandGuard framework. GrandGuard combines an elderly-centric taxonomy, benchmark, and safeguards to improve LLM response safety.

Figure 3: Three-level taxonomy of elderly-specific risks in LLM interactions. It comprises 5 first-level risk types, 13 second-level risk types, and 50 third-level risk types derived from empirical analysis.

Category	Likely Safe for Younger Adults	Often Unsafe for Older Adults
Physical tasks	Moderate physical strain	High fall/injury risk
Medication/diet changes	Usually tolerable	Dangerous drug interactions
Financial/legal	Fast independent decisions	Scam susceptibility
Tech complexity	Manageable	Confusion $\rightarrow$ security risk
Emotional influence	Resisted	Heightened manipulation vulnerability
Urgency	Handled well	Poor coping under pressure

Table 1: Comparison of younger- and older-adult risks. Identical prompts may be safe for younger adults but unsafe for older adults across common interaction domains.

Figure 4: Severity-weighted risk distribution. Horizontal bars show counts of unsafe prompts and unsafe responses for each second-level risk type. Markers and whiskers show mean human severity on a 7-point Likert scale

\pm

1 s.d., mapped to the same axis.

Model	Safe	Unsafe (Partial + Complete)
claude-sonnet-4.5	89.8	10.2 (7.2 + 3.0)
claude-sonnet-3.7	87.4	13.6 (8.2 + 5.4)
gpt-5.1	56.2	43.8 (30.2 + 13.6)
gemini-2.5-flash	45.0	55.0 (39.6 + 15.4)
qwen3-max	43.8	56.2 (41.2 + 15.0)
deepseek-v3.2	39.6	60.4 (43.2 + 17.2)

Table 2: Response safety rates (%) on GrandGuard unsafe prompts. Safe : both Risk Indication and Harm Avoidance satisfied. Unsafe : failure of at least one criterion, reported as Total (Partial Unsafe + Complete Unsafe) .

Safeguard	F1	Acc	Prec	Rec
omni-moderation	0.108	0.528	0.989	0.057
llama-guard-2	0.415	0.596	0.749	0.287
llama-guard-3	0.492	0.633	0.797	0.356
llama-guard-4	0.489	0.630	0.792	0.353
gpt-oss-safeguard-20b	0.539	0.676	0.931	0.380

Table 3: Existing safeguard performance on GrandGuard unsafe prompts. Low recall indicates systematic failure to detect elderly-specific risks.

Model	Prompt Awareness	Response Safety	Gap $\Delta$	Response Critique
gemini-2.5-flash	95.2	45.0	+50.2	40.2
claude-sonnet-3.7	80.2	87.4	-7.2	3.4
claude-sonnet-4.5	79.6	89.8	-10.2	3.0
gpt-5.1	75.8	56.2	+19.6	22.0
llama-4-maverick	63.6	28.2	+35.4	21.6
qwen3-max	62.2	43.8	+18.4	30.6

Table 4: Self-diagnosis of elderly-specific safety. Prompt Awareness (PA, % of prompts flagged as unsafe) vs. Response Safety (RS, % of responses labeled as safe). Gap denotes

\Delta=PA-RS

, and Response Critique (% of models flag their own responses as unsafe).

Configuration	F1	Acc	Prec	Rec
Prompt Classification
llama-guard-3	0.492	0.633	0.797	0.356
+ w/ our taxonomy	0.631	0.697	0.808	0.518
+ fine-tuning	0.962	0.962	0.961	0.963
gpt-oss-safeguard-20b	0.539	0.676	0.931	0.380
+ our policy	0.904	0.909	0.952	0.860

Table 5: Safeguard performance on GrandGuard . Elderly-specific risk detection for prompt and response classification. Our taxonomy improves base models, and fine-tuning and policy-enhanced augmentation yield substantial gains across all metrics.

Model	Safe	+Agent	$\boldsymbol{\Delta}$
claude-sonnet-4.5	89.8	94.6	+4.8
claude-sonnet-3.7	87.4	93.8	+6.4
gpt-5.1	56.2	91.7	+35.5
gemini-2.5-flash	45.0	88.8	+43.8
qwen3-max	43.8	91.6	+47.8
deepseek-v3.2	39.6	91.8	+52.2

Table 6: Effect of the GrandGuard Agent on Response Safety. Response safety rates (%) on GrandGuard unsafe prompts before ( Safe ) and after ( +Agent ) applying agent-based safety reasoning.

\Delta

shows absolute improvement.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

LoRA setting	Value
Base model	meta-llama/Llama-Guard-3-8B
Adapter type	LoRA
Rank ( $r$ )	16
Alpha ( $\alpha$ )	16
Dropout	0
Bias	none

Appendix

Table 7: LoRA configuration. Adapters are attached to both attention projections and MLP projections to support robust classification-style instruction following.

Training setting	Value
Train/eval split	1:1 (balanced)
Per-device train batch size	4
Gradient accumulation steps	4
Max steps	500
Warmup steps	50
Learning rate	$2\times 10^{-5}$

Appendix

Table 8: Training schedule. Settings are summarized from exported trainer configuration and logs.

Step	Eval loss
100	$\approx$ 0.170
200	$\approx$ 0.104
300	$\approx$ 0.0866
400	$\approx$ 0.0743
500	$\approx$ 0.0753

Appendix

Table 9: Evaluation loss across training checkpoints. Values are reported at logged evaluation steps during LoRA fine-tuning.

Setting	F1	Acc	Prec	Rec
Prompt Classification
Ours	0.962	0.962	0.961	0.963
w/o general mix	0.957	0.958	0.963	0.951
w/o taxonomy map	0.937	0.940	0.956	0.920
w/o synthetic safe	0.678	0.552	0.523	0.965
Response Classification

Appendix

Table 10: Ablation on Elderly-Sensitive Llama-Guard-3 fine-tuning. Prompt- and response-level classification after removing the general-data mix, taxonomy mapping, or synthetic safe pairs.

Condition	Label	Routing / Decision Rule
Elderly context detected?	No	Apply standard safeguard only. Output unsafe if standard safeguard is unsafe; else safe .
Elderly context detected?	Yes	Apply standard safeguard and Elderly-Sensitive Policy. Output unsafe if either is unsafe; else safe .
Cognitive decline present?	Yes	Vulnerability override: any request involving making financial, medical, or legal decisions is unsafe unless it is informational without action.

Appendix

Table 11: Routing logic for the Elderly-Sensitive Safeguard Policy. The policy enforces high-sensitivity screening in elderly contexts and adds a vulnerability override for cognitive decline.

LLM Safety Evaluation

Large language model safety evaluation measures harmful compliance, robustness, and safe utility across prompts, tools, and real-world tasks.

See all 258 papers

CareGuardAI: Context-Aware Multi-Agent Guardrails for Clinical Safety & Hallucination Mitigation in Patient-Facing LLMs
Apr 7, 2026Elham Nasarian, Abhilash Neog, Kwok-Leung Tsui +1LLM Guardrails LLM Safety
Grado Department of Industrial & Systems Engineering, Virginia Tech, Blacksburg, VA 24061, USA · Department of Computer Science, Virginia Tech, Blacksburg, VA 24061, USA · Dept of Industrial, Manufacturing, and Systems Engineering at University of Texas at Arlington, Arlington, TX 76019, USA
RubRIX: Rubric-Driven Risk Mitigation in Caregiver-AI Interactions
Jan 19, 2026Drishti Goel, Jeongah Lee, Qiuyue Joy Zhong +5Language Model Safety Evaluation LLM Safety Evaluation
University of Illinois Urbana-Champaign · University of Massachusetts Amherst · OSF HealthCare +1
Inform, Coach, Relate, Listen: Auditing LLM Caregiving Support Roles
May 28, 2026Drishti Goel, Agam Goyal, Veda Duddu +8Healthcare LLM Auditing
University of Illinois Urbana-Champaign · University of Massachusetts Amherst · OSF HealthCare +1

Apr 7, 2026J/K move · Enter open · S save

CareGuardAI: Context-Aware Multi-Agent Guardrails for Clinical Safety & Hallucination Mitigation in Patient-Facing LLMs

Elham Nasarian, Abhilash Neog, Kwok-Leung Tsui, Niyousha HosseiniChimeh

Grado Department of Industrial & Systems Engineering, Virginia Tech, Blacksburg, VA 24061, USA · Department of Computer Science, Virginia Tech, Blacksburg, VA 24061, USA · Dept of Industrial, Manufacturing, and Systems Engineering at University of Texas at Arlington, Arlington, TX 76019, USA

LLM Guardrails LLM Safety LLM Agent Safety LLM Hallucination Mitigation Medical LLMs

Open paper AskPDF arXiv

Integrating large language models (LLMs) into patient-facing healthcare systems offers significant potential to improve access to medical information. However, ensuring clinical safety and factual reliability remains a critical challenge. In practice, AI-generated responses may be conditionally correct yet medically inappropriate, as models often fail to interpret patient context and tend to produce agreeable responses rather than challenge unsafe assumptions. Unlike clinicians, who infer risk from incomplete information, LLMs frequently lack contextual awareness. Moreover, real-world patient interactions are open-ended and underspecified, unlike structured benchmark settings. We present CareGuardAI, a risk-aware safety framework for patient-facing medical question answering that addresses two key failure modes: clinical safety risk and hallucination risk. The framework introduces Clinical Safety Risk Assessment (SRA), inspired by ISO 14971, and Hallucination Risk Assessment (HRA) to evaluate medical risk and factual reliability. At inference time, CareGuardAI employs a multi-stage pipeline consisting of a controller agent, safety-constrained generation, and dual risk evaluation, followed by iterative refinement when necessary. Responses are released only when both SRA and HRA are less than or equal to 2, ensuring clinically acceptable outputs with bounded latency. We evaluate CareGuardAI on PatientSafeBench, MedSafetyBench, and MedHallu, covering both safety and hallucination detection. Across these benchmarks, the framework consistently outperforms strong baseline models, including GPT-4o-mini, demonstrating the importance of context-aware, risk-based, inference-time safety mechanisms for reliable deployment in healthcare.

Abstract

LLM Safety Evaluation

Explore similar work

CareGuardAI: Context-Aware Multi-Agent Guardrails for Clinical Safety & Hallucination Mitigation in Patient-Facing LLMs

RubRIX: Rubric-Driven Risk Mitigation in Caregiver-AI Interactions

Inform, Coach, Relate, Listen: Auditing LLM Caregiving Support Roles