From Clinical Judgment to Checkable Rules: Clinician-Led Guardrails for Mental-Health Chatbots

Chenhan Lyu; Kit Wislocki; Sarah Elizabeth Stoeckl; Maitreyi Sinha; Amir Rahmani

cs.CLOct 22, 2026

From Clinical Judgment to Checkable Rules: Clinician-Led Guardrails for Mental-Health Chatbots

Authors: Chenhan Lyu, Kit Wislocki, Sarah Elizabeth Stoeckl, Maitreyi Sinha, Amir Rahmani

Organizations: University of California, Irvine, USA

Abstract

Large language models (LLMs) deployed in mental-health contexts can be evaluated for safety failures, but the resulting judgments are often difficult to inspect or explain. Common approaches to improving safety rely on prompt engineering, LLM-as-a-Judge, and Constitutional AI. In this work, we describe a clinician-led alternative in which two clinicians define, finalize, and adjudicate the four safety checks of a rule base, which is formally verified for internal consistency and applied as a guardrail over LLM outputs. Here and in the title, clinician-led refers to this rule base, not to the fact extractor or the uncertainty default, which are ours. Every verdict of the resulting verification layer is recomputable from a named rule and sixteen facts. Against an LLM judge matched in model, revision, quantization, rule text, and decoding but without the layer's computed 988/911 fact, the largest disagreement class, the satisfied-rule class, is structured: the Llama-3.3-70B judge, on its primary prompt, demands a crisis resource from 55 of the 65 replies (84.6%) that already contain one, while the same model, asked as a fact extractor, reports a recognized resource present on 54 of the 55. The verification layer never raises that demand, by construction. We report with equal prominence where this judge is stronger: fewer false crisis alarms on the one reference-scored check. We also report the measured cost (a specification revision raised block-class outcomes on held-out boundary prompts of an over-refusal benchmark from 6.0% to 16.0%, chiefly by changing fact extraction), the clinicians' decision to keep our uncertainty default, and the analysis of the result. The contribution is not a safer verdict. It is a guardrail whose disagreements with the judge localize to named rules and facts and whose measured costs trace to named facts and the uncertainty default.

Figures & tables

Panel of Figure 1 — Figure 1: Two Types of Clinician Engagement

	Pilot (19)		All (94)
Dimension	Raw	$\kappa$	Raw	$\kappa$
Category (7-class)	17/19	0.842	81/94	0.793
C1 crisis resources	15/19	0.513	80/94	0.589
C2 referral floor	17/19	0.689	76/94	0.501 †
C3 medication bound.	19/19	1.0 †	93/94	0.883 †
C4 non-reinforcement	19/19	1.0 †	85/94	0.723

Table 1: Two-rater agreement (Rater 1

\times

Rater 2) on the 19-item dual-rated pilot and pooled over all 94 dual-rated items. Within the pilot, severity disagreements never exceed one level. Pooled, 89 of 94 are within one level.

\dagger

= fragile: pilot C3 and C4 rest on 1 and 3 positive items, pooled C3 on 5 positive items, and pooled C2 on 14 negative items.

Figure 2: Pipeline of Predicate and Verdict

Endpoint	Layer	Judge
Fires C1 on a reply containing 988/911	0/294	55/294
Inconsistent with fact-extraction output (of those 55)	—	54/55
C1 false-fires †	39/81	23/81
C1 misses †	2/13	2/13
Block-class, direct-benign stratum	22/50	4/50
Outcome flips at $T{=}0.7$ , three seeds	37/294	30/294

Table 2: Head-to-head of the layer under its fail-closed default and the Llama-3.3-70B judge on its deployed prompt (identical model, revision, quantization, and greedy decoding, one shared rule text, and no computed resource fact for the judge). Final outcomes agree on 104 of 294 (35.4%), which is agreement, not correctness.

\dagger

= reference-scored on the deterministic crisis-resource endpoint, scored against the clinician message labels (81 no-violation and 13 violation items); the miss counts are equal at

n{=}13

. The layer’s zero in the first row is by construction: the recognized-resource fact is computed by exact text match.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

check	agr.	stratum	agr.
C1	48.6%	direct benign	46.0%
C2	42.2%	direct crisis	36.0%
C3	95.2%	indirect knowledge	30.0%
C4	63.9%	gold-anchored	37.2%
		rare-check (C3/C4)	26.0%

Appendix

Table 3: Agreement between the two methods, by check and by sampling stratum, with the Llama-3.3-70B judge on its deployed prompt.

fact (what it gates)	sens.	spec.
composite crisis signal (C1)	83.3% (15/18)	60.5% (46/76)
context (C4, NSSI trigger)	80.0% (20/25)	39.1% (27/69)
severity $\geq$ moderate (C2)	94.9% (75/79)	20.0% (3/15)

Appendix

Table 4: Extractor facts against the clinician labels, message side only. The composite crisis signal is any of the four crisis facts, with an undetermined fact counted as present. The labels mark relevance to judging replies, not whether a check should fire, so these are fire-versus-relevance figures rather than accuracy.

reply model	$n$	agree	layer block	judge block
grok-4-fast	66	17 (25.8%)	28 (42.4%)	35 (53.0%)
gpt-4o-mini	58	27 (46.6%)	49 (84.5%)	32 (55.2%)
Llama-4-Scout-17B	57	29 (50.9%)	43 (75.4%)	23 (40.4%)
deepseek-chat	57	10 (17.5%)	27 (47.4%)	25 (43.9%)
gpt-5-nano	56	21 (37.5%)	19 (33.9%)	25 (44.6%)

Appendix

Table 5: Descriptive breakdown of the 294 by the model that produced the reply. Agreement is on the final outcome. Model assignment was seeded uniform per conversation, so the stratum mix differs by model and rows are not comparable. Satisfied-rule rows are 24, 0, 0, 14, and 17 in row order. Two models never place a literal 988 or 911 in a reply.

arm	agreement
judge, end to end (primary)	104/294 = 35.4%
own per-check answers, our resolver	115/294 = 39.1%
+ no C1 fires on 988/911 replies	122/294 = 41.5%
rerun with rationale permitted	121/294 = 41.2%

Appendix

Table 6: Judge-side controls for the Llama-3.3-70B judge, on its deployed prompt except in the rationale rerun. The first three rows are computed from archived verdicts with no further inference, and the rationale row is a rerun. The suppression row is a post hoc correction rather than a judge behavior, since it uses the computed 988/911 fact that the evaluated judge arm does not receive.

fact	type	fail-closed
message side
crisis_indicated	bool	true
acute_distress	bool	true
lethal_means_mentioned	bool	true
medication_quantity_question	bool	true
distress_level	level	high

Appendix

Table 7: The sixteen typed facts and their fail-closed grounding values. Levels are none

<

mild

<

moderate

<

high. emotion_validated appears in no predicate. It is recorded as protective context for audit only.

LLM Guardrails

Large language model guardrails detect or constrain harmful outputs, prompt attacks, and unsafe agent actions before they cause harm.

See all 194 papers

Mar 20, 2026

Using LLM-as-a-Judge/Jury to Advance Scalable, Clinically-Validated Safety Evaluations of Model Responses to Users Demonstrating Psychosis

General-purpose Large Language Models (LLMs) are becoming widely adopted by people for mental health support. Yet emerging evidence suggests there are significant risks associated with high-frequency use, particularly for individuals suffering from psychosis, as LLMs may reinforce delusions and hallucinations. Existing evaluations of LLMs in mental health contexts are limited by a lack of clinical validation and scalability of assessment. To address these issues, this research focuses on psychosis as a critical condition for LLM safety evaluation by (1) developing and validating seven clinician-informed safety criteria, (2) constructing a human-consensus dataset, and (3) testing automated assessment using an LLM as an evaluator (LLM-as-a-Judge) or taking the majority vote of several LLM judges (LLM-as-a-Jury). Results indicate that LLM-as-a-Judge aligns closely with the human consensus (Cohen's

κ_{\text{human} \times \text{gemini}} = 0.75

κ_{\text{human} \times \text{qwen}} = 0.68

κ_{\text{human} \times \text{kimi}} = 0.56

) and that the best judge slightly outperforms LLM-as-a-Jury (Cohen's

κ_{\text{human} \times \text{jury}} = 0.74

). Overall, these findings have promising implications for clinically grounded, scalable methods in LLM safety evaluations for mental health contexts.

May Lynn Reese, Markela Zeneli, Mindy Ng +3

Apart Research · Odyssean Institute · London School of Economics and Political Science +1

LLM Safety LLM Safety Evaluation Clinical Language Model Evaluation LLM-as-a-Judge

Ask PDF

Sep 14, 2026

K-Bench: a clinically calibrated benchmark for evaluating large language models in high-risk mental health conversations

People increasingly use large language models (LLMs) for mental health support, yet their safety in evolving, high-risk conversations remains poorly characterised. We developed K-Bench, a clinician-calibrated, protected benchmark evaluating 125 model configurations representing 33 base models from 14 providers across a fixed cohort of 200 multi-turn vignettes involving suicide, self-harm, domestic violence, substance misuse, and no-risk presentations. Synthetic patient conversations showed substantial distributional overlap with real human-AI conversations. A frozen GPT-4o judge achieved 94.2% exact agreement with clinician consensus across 6,751 eligible item comparisons from 151 clinician-rated transcripts. Leading models combined strong supportive conversation with combined-risk scores above 95, whereas risk exploration exposed substantial variation among lower-performing configurations. Therapeutic prompting produced configuration-specific gains concentrated among weaker models, while elevated reasoning produced no average improvement. K-Bench combines broader clinical coverage and configuration-scale comparison with a continuously updated public leaderboard whose operational test materials are protected from direct optimisation. The leaderboard is available at www.k-bench.ai.

Laura M. Vowels, Matthew J. Vowels, Shivali Sharma +9

School of Psychology, University of Roehampton, London, United Kingdom · Kivira Health, United Kingdom · University of Hertfordshire, Hatfield, United Kingdom +4

LLM Safety Benchmarks Mental Health Counseling Language Model Safety Evaluation LLM Safety Clinical Language Model Evaluation

Ask PDF

Apr 20, 2026

MHSafeEval: Role-Aware Interaction-Level Evaluation of Mental Health Safety in Large Language Models

Large language models (LLMs) are increasingly explored as scalable tools for mental health counseling, yet evaluating their safety remains challenging due to the interactional and context-dependent nature of clinical harm. Existing evaluation frameworks predominantly assess isolated responses using coarse-grained taxonomies or static datasets, limiting their ability to diagnose how harms emerge and accumulate over multi-turn counseling interactions. In this work, we introduce R-MHSafe, a role-aware mental health safety taxonomy that characterizes clinically significant harm in terms of the interactional roles an AI counselor adopts, including perpetrator, instigator, facilitator, or enabler, combined with clinically grounded harm categories. Then, we propose MHSafeEval, a closed-loop, agent-based evaluation framework that formulates safety assessment as trajectory-level discovery of harm through adversarial multi-turn interactions, guided by role-aware modeling. Using R-MHSafe and MHSafeEval, we conduct a large-scale evaluation across state-of-the-art LLMs. Our results reveal substantial role-dependent and cumulative safety failures that are systematically missed by existing static benchmarks, and show that our framework significantly improves failure-mode coverage and diagnostic granularity.

Suhyun Lee, Palakorn Achananuparp, Neemesh Yadav +2

Department of Artificial Intelligence, Hanyang University, Seoul, Republic of Korea · School of Computing and Information Systems, Singapore Management University, Singapore

Mental Health Counseling LLM Evaluation Mental Health LLM Safety Evaluation Clinical Language Model Evaluation

Ask PDF

Abstract

LLM Guardrails

Explore similar work

Using LLM-as-a-Judge/Jury to Advance Scalable, Clinically-Validated Safety Evaluations of Model Responses to Users Demonstrating Psychosis

K-Bench: a clinically calibrated benchmark for evaluating large language models in high-risk mental health conversations

MHSafeEval: Role-Aware Interaction-Level Evaluation of Mental Health Safety in Large Language Models