CG-Probes: Recovering Guardrail Directions from Patient Query Embeddings

Marko Řeháček; Vítězslav Dušek; Martin Rusinko; Vít Nováček

cs.CLSep 25, 2026

CG-Probes: Recovering Guardrail Directions from Patient Query Embeddings

Authors: Marko Řeháček, Vítězslav Dušek, Martin Rusinko, Vít Nováček

Organizations: Faculty of Informatics Masaryk University Brno, Czech Republic · Dept. of Pediatric Oncology University Hospital Brno Brno, Czech Republic · Faculty of Medicine Masaryk University · Masaryk Memorial Cancer Institute Brno, Czech Republic

Abstract

Patient-facing AI assistants promise valuable support to patients, but incoming queries can pose medical risks. To create guardrails, we work with oncologists to define three ordinal risk axes: Medical Urgency, Psychological Urgency, and Topic Sensitivity. We propose Clinical Guardrail Probes (CG-Probes) to measure the risks from query embeddings. We probe for each axis in the normalized embedding space of frozen embedders via the difference-in-means method, treating each axis as a potential linear direction. To train the probes, we cluster 79,658 Czech oncology search queries with BERTopic and use these clusters to generate pairs of queries with contrastive risk levels via few-shot prompting. We evaluate the approach on 200 queries (90 real, 110 synthetic), each graded by two oncologists, against two open-weight LLMs and a frontier LLM. We find that urgency-based axes are recoverable as linear directions, and the probes are competitive with open-weight LLMs (no significant differences in quadratic-weighted kappa) at a fraction of the latency. Each axis yields a scalar score that clinicians can inspect and use to set escalation thresholds. The pipeline requires only search logs, axis definitions, and black-box access to the embedding model, suggesting transferability across healthcare domains. Robust validation on new queries and axes remains future work.

Figures & tables

Medical Urgency (MU) – time to clinical action
0	benign, no urgent somatic issue reported – “I have a cough.”
1	physician contact required, emergency not excludable – “Diarrhea six times a day, two days after chemotherapy.”
2	somatic emergency, activate EMS – “My face is swollen can’t breathe.”
Psychological Urgency (PU) – expressed distress $\to$ suicidality
0	no expressed distress – “What are the side effects of radiotherapy?”
1	distress or existential doubt, no suicidal intent – “I cry every day since the diagnosis and can’t cope.”

Table 1. Clinical axes with shortened definitions and example queries. Complete rubrics are in the released code repository.

Figure 1. Probe versus LLM guardrails, against the clinician gold and the human ceiling. Quadratic-weighted

\kappa

on real queries; bars are 95% bootstrap CIs, the ceiling is inter-oncologist agreement on the same items. Probe = the 1-D difference-in-means direction at each axis’s best embedder

\times

instruction cell (Table 2 ) for MU and PU; for TS, the class-balanced linear head (0.58), as a single direction underfits TS (0.49, Table 2 ; §6). The probe is statistically indistinguishable from both open-weight LLMs on MU and PU, and is the weakest on TS, where we make no ranking (§ 6 ). Grouped bar chart comparing quadratic-weighted kappa of the CG-Probe against three LLM baselines and the inter-oncologist ceiling, for the three risk axes.

embedder	instruction	MU	PU	TS
Qwen3-8B (4096d)	none	0.34	0.67	0.38
	generic	0.34	0.81	0.30
	per-axis	0.61	0.67	0.37
	per-axis +	0.60	0.58	0.35
Harrier-27B (5376d)	none	0.30	0.62	0.36
	generic	0.51	0.09	0.27

Table 2. Quadratic-weighted

\kappa

of the 1-D DiM probe against the clinician gold, on the same real queries as Fig. 1 , for embedders

\times

instruction modes. Instruction conditioning helps MU and is mixed for PU. No embedder or instruction recovers TS. Per-axis + is the negation-free instruction variant; text-embedding-3-large accepts no instruction. Best cells reported in Fig. 1 are in bold, the selection bias is noted in § 5 .

LLM Guardrails

Large language model guardrails detect or constrain harmful outputs, prompt attacks, and unsafe agent actions before they cause harm.

See all 194 papers

Medical Fine-Tuning Reshapes, Not Reduces, Patient-Facing AI Risk: A Rubric-Based, LLM-Judge Evaluation Framework
Oct 17, 2026Savan DoshiLanguage Model Safety Evaluation Medical QA
ProbeLogits: Kernel-Level LLM Inference Primitives for AI-Native Operating Systems
Apr 13, 2026Daeyeon SonAI Safety LLM Guardrails
Independent Researcher Republic of Korea
CARE-Bench: Benchmarking Patient-Facing LLM Triage
Aug 4, 2026Yining Hua, Hongbin Na, Cyrus AyubchaLLM Safety Benchmarks Clinical Triage
1Harvard University · University of Technology Sydney

Oct 17, 2026J/K move · Enter open · S save

Medical Fine-Tuning Reshapes, Not Reduces, Patient-Facing AI Risk: A Rubric-Based, LLM-Judge Evaluation Framework

Savan Doshi

Language Model Safety Evaluation Medical QA

Open paper AskPDF arXiv

Patient-facing medical question answering with large language models (LLMs) is moving from research demonstrations to production deployments, yet evaluation remains anchored on factual correctness. Factuality is necessary but not sufficient: a response can be factually grounded and still issue a directive that, if acted on, would harm the patient. We introduce a risk-sensitive evaluation framework that scores responses by the presence and severity of six categories of risk-bearing language (treatment directives, contraindications, dosage expressions, urgency-triage cues, high-alert medication mentions, and overconfident assertions), anchored on published clinical harm taxonomies. Severity is classified by a three-judge LLM ensemble. We apply this framework to six 7-9B instruction-tuned models, three general (Mistral-7B, Phi-3.5-mini, Llama-3.1-8B) and three medically adapted (BioMistral-7B, OpenBioLLM-Llama3-8B, Med42-Llama3-8B), on 200 real patient questions from the K-QA benchmark. Two findings emerge. First, ensemble risk-bearing language is statistically independent of factual hallucination rate (pooled Spearman

ρ= -0.065

n = 300

, 95% CI

[-0.18, +0.05]

; distance correlation

= 0.10

, independence test

p = 0.24

). Second, medical fine-tuning does not reduce aggregate risk but reshapes it: continued-pretraining adaptations (BioMistral, OpenBioLLM) consistently suppress high-alert medication mentions while amplifying overconfident assertions, whereas clinical instruction-tuning (Med42) suppresses dosing only, without amplifying overconfidence. Per-category Krippendorff

α

is fair to good for the four linguistically anchored categories (0.53-0.71) and low for the two clinical-judgment-dependent categories (0.19, 0.31), indicating where automatic safety evaluation is and is not yet reliable. We release the rubric, judge prompts, and evaluation code.

Abstract

LLM Guardrails

Explore similar work

Medical Fine-Tuning Reshapes, Not Reduces, Patient-Facing AI Risk: A Rubric-Based, LLM-Judge Evaluation Framework

ProbeLogits: Kernel-Level LLM Inference Primitives for AI-Native Operating Systems

CARE-Bench: Benchmarking Patient-Facing LLM Triage