Medical Fine-Tuning Reshapes, Not Reduces, Patient-Facing AI Risk: A Rubric-Based, LLM-Judge Evaluation Framework

Savan Doshi

cs.AIOct 17, 2026

Medical Fine-Tuning Reshapes, Not Reduces, Patient-Facing AI Risk: A Rubric-Based, LLM-Judge Evaluation Framework

Authors: Savan Doshi

Abstract

Patient-facing medical question answering with large language models (LLMs) is moving from research demonstrations to production deployments, yet evaluation remains anchored on factual correctness. Factuality is necessary but not sufficient: a response can be factually grounded and still issue a directive that, if acted on, would harm the patient. We introduce a risk-sensitive evaluation framework that scores responses by the presence and severity of six categories of risk-bearing language (treatment directives, contraindications, dosage expressions, urgency-triage cues, high-alert medication mentions, and overconfident assertions), anchored on published clinical harm taxonomies. Severity is classified by a three-judge LLM ensemble. We apply this framework to six 7-9B instruction-tuned models, three general (Mistral-7B, Phi-3.5-mini, Llama-3.1-8B) and three medically adapted (BioMistral-7B, OpenBioLLM-Llama3-8B, Med42-Llama3-8B), on 200 real patient questions from the K-QA benchmark. Two findings emerge. First, ensemble risk-bearing language is statistically independent of factual hallucination rate (pooled Spearman $ρ= -0.065$ , $n = 300$ , 95% CI $[-0.18, +0.05]$ ; distance correlation $= 0.10$ , independence test $p = 0.24$ ). Second, medical fine-tuning does not reduce aggregate risk but reshapes it: continued-pretraining adaptations (BioMistral, OpenBioLLM) consistently suppress high-alert medication mentions while amplifying overconfident assertions, whereas clinical instruction-tuning (Med42) suppresses dosing only, without amplifying overconfidence. Per-category Krippendorff $α$ is fair to good for the four linguistically anchored categories (0.53-0.71) and low for the two clinical-judgment-dependent categories (0.19, 0.31), indicating where automatic safety evaluation is and is not yet reliable. We release the rubric, judge prompts, and evaluation code.

Figures & tables

Category	Harm pathway	Key evidence
treatment_directive	Patient-initiated medication interruption	DOAC interruption: thromboembolic events rise from 1.0 to 21.6 per 100 patient-years ( Vene et al., 2016 ) ; statin discontinuation associated with near-doubling of all-cause mortality ( Peixoto et al., 2024 ) ; insulin self-administration errors cause severe hypoglycemia and DKA ( Grissinger, 2016 )
dosage_expression	Expression format drives error rate, not dose magnitude	84.4% caregiver dosing-error rate; teaspoon vs. mL units OR $=4.6$ ( Yin et al., 2016 ) ; metric-only instructions reduce error $>$ 50% ( van Ittersum & Wansink, 2016 ) ; AAP recommends metric-only patient-facing dosing ( American Academy of Pediatrics, 2015 )
urgency_triage	Over-reassurance delays time-critical care	Help-seeking delayed months–years for cancer after “all-clear” ( Renzi et al., 2015 ) ; routing acute MI through primary care nearly doubles time-to-hospital ( Rawles et al., 1993 ) ; symptom-checker apps miss $>$ 40% of emergencies ( Schmieding et al., 2022 )
high_alert_medication	Instruction structure drives harm, not drug misidentification	$\sim$ 100,000 insulin-related ED visits and 30,000 hospitalizations annually in the US ( Geller et al., 2014 ) ; capecitabine over-adherence without “stop-and-call” framing potentiates toxicity ( Bourmaud et al., 2015 ; Le Saux et al., 2018 )
overconfident_assertion	Confidence asserted independently of accuracy	Overconfidence is a leading contributor to diagnostic error ( Berner & Graber, 2008 ) ; LLMs assert wrong and right answers with nearly identical confidence ( Omar et al., 2025 ; Rosenbacke et al., 2025 )
contraindication	Closest evidence is clinician-side DDI alert fatigue	$>$ 90% DDI alert override rate with documented adverse-event consequences ( Phansalkar et al., 2013 ; Villa Zapata et al., 2022 ) . Patient-facing evidence is a research gap this framework helps surface.

Table 1: Each category corresponds to a class of patient-facing communication for which published patient-safety literature has documented harm consequences independent of factual accuracy. Five of six categories are supported by multi-decade clinical-safety literature; contraindication is retained as a research gap.

General model	Medical counterpart	Adaptation pipeline
Mistral-7B-Instruct-v0.3	BioMistral-7B ( Labrak et al., 2024 )	Continued pretraining (PMC, PubMed)
Phi-3.5-mini (3.8B)	OpenBioLLM-Llama3-8B ( Pal, 2024 )	Continued pretraining + medical SFT
Llama-3.1-8B-Instruct	Med42-Llama3-8B ( Christophe et al., 2024 )	Clinical instruction-tuning (full-param FT)

Table 2: Six 7–9B models as three medical/general pairs. Mistral/BioMistral and Llama-3.1/Med42 share base architecture; Phi-3.5/OpenBioLLM is a scale-matched comparison (3.8B vs. 8B) with different base architectures.

Figure 1: Ensemble RSHS vs. NLI hallucination rate across six models on 50 K-QA prompts ( $n=300$ ). Pooled Spearman $\rho=-0.065$ , 95% bootstrap CI $[-0.18,+0.05]$ : the two evaluation axes are statistically independent.

Model	$n$	Spearman $\rho$	95% bootstrap CI
Mistral-7B	50	$-0.135$	$[-0.40,+0.15]$
Phi-3.5-mini	50	$+0.274$	$[-0.04,+0.55]$
Llama-3.1-8B	50	$-0.184$	$[-0.44,+0.08]$
BioMistral-7B	50	$-0.013$	$[-0.27,+0.26]$
OpenBioLLM-Llama3-8B	50	$-0.139$	$[-0.41,+0.16]$
Med42-Llama3-8B	50	$-0.109$	$[-0.39,+0.20]$

Table 3: Spearman correlation between ensemble RSHS and NLI hallucination rate, per model and pooled. CIs from 5000-resample percentile bootstrap. No individual model shows a significant correlation; the pooled CI tightly bounds

|\rho|\leq 0.18

Figure 2: Per-category presence rates with 95% bootstrap CIs (5000 resamples) for the six models, grouped by family pair. BioMistral and OpenBioLLM show reduced presence of high-alert medication mentions relative to their general counterparts; severity-level amplification of overconfident assertion is reported separately in Table 5 .

Model (type)	$n$	Mean RSHS (95% CI)
Mistral-7B (gen)	200	1.43 [1.31, 1.56]
Phi-3.5-mini (gen)	200	1.24 [1.11, 1.38]
Llama-3.1-8B (gen)	200	1.28 [1.14, 1.43]
BioMistral-7B (med)	200	1.41 [1.26, 1.55]
OpenBioLLM-Llama3-8B (med)	200	1.33 [1.19, 1.47]
Med42-Llama3-8B (med)	200	1.25 [1.11, 1.39]

Table 4: Mean ensemble RSHS per model. CIs from 5000-resample percentile bootstrap. All six models fall within a 0.19 band; medical and general groups overlap.

	Mistral $\to$ BioMistral		Phi-3.5 $\to$ OpenBioLLM		Llama-3.1 $\to$ Med42
Category	$\Delta$	$p$	$\Delta$	$p$	$\Delta$	$p$
treatment_directive	$-0.034$	0.49	$-0.029$	0.51	$+0.032$	0.63
contraindication	$-0.009$	0.50	$+0.005$	1.00	$-0.002$	0.91
dosage_expression	$-0.083$	0.07	$\mathit{-0.169}$	0.0003	$\mathit{-0.125}$	0.013
urgency_triage	$\mathit{-0.155}$	0.005	$+0.028$	0.52	$-0.020$	0.80
high_alert_medication	$\mathit{-0.154}$	$\mathbf{<10^{-4}}$	$\mathit{-0.086}$	0.0004	$+0.020$	0.98

Table 5: Per-category paired Wilcoxon signed-rank tests (

\Delta

= medical minus general mean severity). Italic

\Delta

: significant suppression; bold

\Delta

: significant amplification.

n=200

per pair. Mistral and Phi pairs show the reshape pattern; Llama/Med42, trained via clinical instruction-tuning, suppresses dosing only. Bonferroni-Holm and Benjamini-Hochberg (FDR = 0.05) corrections applied post-hoc to the 18 paired tests. Under BH correction, 7 of 18 tests survive at FDR

\leq

0.05, including all headline reshape findings: overconfident_assertion amplification in Mistral/BioMistral (BH

p=0.024

) and Phi-3.5/OpenBioLLM (BH

p=0.0009

); dosage_expression suppression in Phi-3.5/OpenBioLLM (BH

p=0.002

) and Llama/Med42 (BH

p=0.034

); high_alert_medication suppression in Mistral/BioMistral (BH

p=0.0006

) and Phi-3.5/OpenBioLLM (BH

p=0.002

); urgency_triage suppression in Mistral/BioMistral (BH

p=0.019

Category	$\alpha$ (presence, nominal)	$\alpha$ (severity, ordinal)
dosage_expression	0.702	0.708
urgency_triage	0.644	0.659
contraindication	0.620	0.624
treatment_directive	0.526	0.529
overconfident_assertion	0.269	0.305
high_alert_medication	0.186	0.193

Table 6: Krippendorff

\alpha

for inter-judge agreement, computed separately for presence (nominal) and severity (ordinal) measurement levels. 95% CIs from 500-resample bootstrap over items;

n=1{,}200

(model, response) pairs; three LLM judges.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Pair	Gen HC-RSHS	Med HC-RSHS	$\Delta$	Wilcoxon $p$
Mistral / BioMistral	0.347	0.277	$-0.070$	0.0076 ∗∗
Phi-3.5 / OpenBioLLM	0.255	0.214	$-0.041$	0.084
Llama-3.1 / Med42	0.313	0.283	$-0.029$	0.985

Appendix

Table 7: HC-RSHS composite (mean severity across the four well-agreed categories). Medical

<

general direction is preserved in all three pairs; the suppression is significant for Mistral/BioMistral (

p=0.0076

) and directionally consistent but not significant for Phi/OpenBioLLM and Llama/Med42. Note that Phi/OpenBioLLM shows individually significant dosage_expression suppression (

p=0.0003

, Table 5 ) that is diluted in the composite mean.

Pair	Judge	Gen sev	Med sev	$\Delta$	$p$
Mistral / BioMistral	Gemini	0.270	0.640	$+0.370$	$<0.001^{***}$
	GPT-4o-mini	1.065	1.005	$-0.060$	0.576
	Claude Hk	0.750	1.025	$+0.275$	$0.009^{**}$
Phi-3.5 / OpenBioLLM	Gemini	0.275	0.575	$+0.300$	$<0.001^{***}$
	GPT-4o-mini	0.730	1.130	$+0.400$	$<0.001^{***}$
	Claude Hk	0.800	0.865	$+0.065$	0.416

Appendix

Table 8: Per-judge paired Wilcoxon tests for overconfident_assertion severity. For each continued-pretraining pair, at least 2 of 3 judges show significant amplification (positive

\Delta

). The dissenting judge differs across pairs (GPT-4o-mini for Mistral/BioMistral; Claude Haiku for Phi/OpenBioLLM), suggesting judge-idiosyncratic sensitivity rather than a systematic artifact. For Llama/Med42, Claude Haiku uniquely shows strong overconfidence suppression (

\Delta=-0.389

p<0.001

), while the other two judges show no effect. This may reflect a self-preference artifact: Med42’s instruction-tuning style closely resembles Claude Haiku’s hedged response pattern.

Language Model Safety Evaluation

Language model safety evaluation measures harmful behavior, refusal, robustness, and over-refusal across realistic prompts, users, and deployment settings.

See all 542 papers

When Medical Safety Alignment Fails: A Benchmark for Evaluating LLMs on High-Risk Medical Queries
May 26, 2026Yige Li, Jun Sun, Wei Zhao +5LLM Safety Benchmarks Healthcare
Singapore Management University · Deakin University · The University of Melbourne +2
MIRA: A Bilingual Benchmark for Medical Information Response Audit
May 27, 2026Mengyu Xu, Qiaoxin Yang, Qianqian Wang +3Multilingual Language Model Evaluation Healthcare
The University of Chicago · The Chinese University of Hong Kong, Shenzhen · Jinzhou Medical University +3
Measuring Epistemic Resilience of LLMs Under Misleading Medical Context
Jun 10, 2026Hongjian Zhou, Xinyu Zou, Jinge Wu +19LLM Reliability Medical QA
University of Oxford · University of Washington · University College London +1

May 26, 2026J/K move · Enter open · S save

When Medical Safety Alignment Fails: A Benchmark for Evaluating LLMs on High-Risk Medical Queries

Yige Li, Jun Sun, Wei Zhao, Zhe Li+4

Singapore Management University · Deakin University · The University of Melbourne+2

LLM Safety Benchmarks Healthcare LLM Guardrails LLM Safety Alignment LLM Safety Evaluation Medical LLMs

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly used for medical and health-related questions, yet their safety in high-risk medical scenarios remains poorly understood. We introduce MedHarm\footnote{Code and data will be released upon acceptance. Due to the sensitive nature of high-risk medical queries, data access will be available to qualified researchers upon request.}, a high-risk medical safety benchmark with 1,100 medically grounded queries across 10 safety-critical categories, including toxicology, pharmacology, covert poisoning, anesthesia, and fetal harm. Unlike broad medical QA benchmarks, MedHarm targets realistic clinical, educational, and technical prompts that require refusal, caution, or safe redirection rather than direct helpfulness. We evaluate 15 LLMs spanning general-purpose, medical-purpose, closed-source, and downstream SFT models, together with 4 representative guardrail models. Results reveal a substantial gap between apparent alignment and medical safety: aligned models can still produce unsafe or actionable responses, medical fine-tuning can amplify harmful specificity, and external guardrails reduce some failures while introducing brittle blocking and weak safe helpfulness. These findings show that medical safety cannot be inferred from general alignment or medical capability alone, highlighting the need for domain-specific stress testing before deploying LLMs in safety-critical medical applications.

Abstract

Language Model Safety Evaluation

Explore similar work

When Medical Safety Alignment Fails: A Benchmark for Evaluating LLMs on High-Risk Medical Queries

MIRA: A Bilingual Benchmark for Medical Information Response Audit

Measuring Epistemic Resilience of LLMs Under Misleading Medical Context