MedPriv-Bench: Benchmarking the Privacy-Utility Trade-off of Large Language Models in Medical Open-Ended Question Answering

Shaowei Guan; Yu Zhai; Hin Chi Kwok; Jiawei Du; Xinyu Feng; Jing Li; Harry Qin; Vivian Hui

cs.CLMar 15, 2026

MedPriv-Bench: Benchmarking the Privacy-Utility Trade-off of Large Language Models in Medical Open-Ended Question Answering

Authors: Shaowei Guan, Yu Zhai, Hin Chi Kwok, Jiawei Du, Xinyu Feng, Jing Li, Harry Qin, Vivian Hui

Organizations: Centre for Smart Health, School of Nursing, The Hong Kong Polytechnic University · Department of Language Science and Technology, The Hong Kong Polytechnic University · Centre for Frontier AI Research, Agency for Science, Technology and Research, Singapore · Department of Computing, The Hong Kong Polytechnic University ·

Abstract

Recent advances in Retrieval-Augmented Generation enable LLMs to ground outputs in clinical evidence, but connections to external databases create the risk of contextual leakage, where unique combinations of medical details enable patient re-identification without explicit identifiers. Existing healthcare benchmarks emphasize accuracy while overlooking this risk. To fill this gap, we present MedPriv-Bench, the first benchmark for jointly evaluating privacy preservation and clinical utility in medical open-ended question answering. Our framework utilizes a multi-agent, human-in-the-loop pipeline to synthesize sensitive medical contexts and clinically relevant queries that create realistic privacy pressure. We also establish an automated evaluation protocol using a fine-tuned RoBERTa-NLI model, which achieved an instance-level F1 score of 75.3%, sensitivity of 90.7%, and an average inference time of 0.056 s per sample against human annotations. Across nine LLMs and three privacy-preserving methods, we observed a pervasive privacy-utility trade-off. Relative to unprotected Med42-v2-8B (utility 3.87/5; leakage 72.8%), supervised fine-tuning improved utility to 4.25 and reduced leakage to 38.9%, whereas local differential privacy reduced leakage to 20.5% but lowered utility to 3.03. These results demonstrate the need for domain-specific benchmarks to validate medical AI systems in privacy-sensitive settings.

Figures & tables

Figure 1: Illustration of contextual leakage in healthcare retrieval-augmented generation (RAG) systems. The difference between a privacy-preserving response and a contextual leakage response is highlighted.

Figure 2: Overview of the four-phase benchmark construction pipeline: (1) Phase 1 uses an Injection Agent to embed contextually coherent PHIs into patient summaries based on a predefined taxonomy. (2) Phase 2 employs a White-box Attack strategy, where a Question Agent crafts queries relying on the injected PHIs. (3) Phase 3 feeds the sensitive notes and queries to an Answer Agent for response generation. (4) Phase 4 applies automated quality filtering and human validation to finalize the dataset.

Model	Response Utility	Overall Privacy		Fact-Level Leakage by PHI Type (%)
Model	Response Utility	LR inst (%)	LR fact (%)	SB	MH&CS	SDOH	QIV
Closed-Source Models
GPT-5-nano (w/o)	4.89/5	70.2	57.1	32.2	58.0	65.8	38.1
GPT-5-nano (w/)	4.88/5	54.6	42.7	16.9	43.5	52.7	19.6
Claude Haiku 4.5 (w/o)	4.71/5	89.0	80.0	75.4	88.3	80.5	36.1
Claude Haiku 4.5 (w/)	4.60/5	61.4	48.6	33.1	61.1	46.9	6.2

Table 1: Performance comparison of 9 LLMs with and without defense mechanisms on the Privacy-Utility Benchmark (Test set

n=738

). SB : Stigmatized Behaviors; MH&CS : Mental Health & Cognitive Status; SDOH : Socioeconomic Determinants of Health; QIV : Quasi-Identifiers / VIP Status.

Figure 3: Privacy-utility intervention effects. Changes within each model relative to its corresponding undefended configuration.

Question Category	$LR_{inst}$ (%)
SDOH & Psychosocial Support	55.3
Discharge Planning & Care Coordination	52.4
Medication Adherence & Management	49.4
Follow-up Care & Monitoring	47.8
Infection Control & Wound Care	46.7

Table 2: Privacy vulnerability across different clinical question types. SDOH: Socioeconomic Determinants of Health.

Figure 4: Impact of Context Volume and Word Constraints on Privacy vs. Utility.

Model	Fact-Level		Instance-Level
Model	IAA (%)	Cohen’s $\kappa$	IAA (%)	Cohen’s $\kappa$
Gemini	96.3	0.924	94.0	0.860
Med42	93.8	0.867	90.0	0.798
DeepSeek	82.7	0.639	80.0	0.602
Llama	91.4	0.772	88.0	0.752
GPT	81.5	0.545	76.0	0.520

Table 3: Inter-rater agreement between two independent human annotators. Average values are unweighted macro averages across models.

Model	Instance-Level LR (%)		Fact-Level LR (%)		Alignment (%)*			Length
Model	NLI	Human Avg.	NLI	Human Avg.	Accuracy	Sensitivity	$F_{1}$ -score	Length
Gemini 3 Flash	88.0	69.0	80.2	57.4	77.2 \| 81.0	100.0 \| 100.0	83.4 \| 87.9	276
Med42-v2-70B	70.0	55.0	64.2	36.4	67.3 \| 77.0	93.2 \| 92.7	67.5 \| 81.6	311
DeepSeek-V3.2	64.0	48.0	56.8	39.5	79.0 \| 80.0	95.4 \| 95.8	78.2 \| 82.0	332
Llama3-70B	62.0	38.0	54.3	24.1	68.5 \| 74.0	97.5 \| 97.5	59.8 \| 74.0	264
GPT-5-nano	28.0	41.0	19.8	29.6	77.8 \| 75.0	51.3 \| 57.8	56.0 \| 64.3	652

Table 4: Comparison of automatic (fine-tuned RoBERTa) and human leakage risk (LR) rates across models. Human LR values are arithmetic means across the two independent human raters. Alignment is computed separately against each rater and then averaged. *Alignment is reported as Fact-Level | Instance-Level. Length is the response word count.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

PHI Category	Description and Examples	Legal & Clinical Rationale (Justification)
Stigmatized Behaviors	Confidential behaviors affecting care (e.g., heavy alcohol use; non-prescribed opioid misuse; high-risk sexual behavior).	Legal Basis: 42 CFR Part 2. This federal law provides enhanced protection for substance use records beyond standard HIPAA, as disclosure carries severe risks of social discrimination or legal repercussions. In clinical notes, these unique narratives act as “semantic identifiers” that narrow the patient pool significantly.
Mental Health & Cognitive Status	Psychiatric conditions influencing adherence and safety (e.g., depression with suicidal ideation; bipolar disorder; PTSD; MCI).	Legal/Clinical Basis: HIPAA Psychotherapy Notes (45 CFR § 164.501) and ICD-10 F-codes (F01–F99). Mental health data is categorized as high-sensitivity under HIPAA. We use ICD-10 F-codes to standardize these risks, acknowledging that specific diagnostic patterns often serve as a “clinical fingerprint” for individuals in local healthcare settings.
Socioeconomic Determinants of Health (SDOH)	Social/economic factors affecting care (e.g., homelessness; inability to afford meds; recent incarceration; unsafe home).	Legal/Clinical Basis: HIPAA 45 CFR § 160.103 and ICD-10 Z-codes (Z55–Z65). HIPAA defines PHI to include any information regarding the “provision of healthcare.” Since SDOH (mapped via Z-codes) describes obstacles to care, it is legally PHI. Social vulnerabilities combined with medical conditions drastically reduce the k-anonymity of a record.
Quasi-Identifiers / VIP Status	Attributes that indirectly identify a patient (e.g., prominent occupation; rare residence; notable family ties; unique public incident).	Legal Basis: HIPAA 164.514(b) (Expert Determination). This category addresses the “Residual Risk” of re-identification. High-profile roles or rare residential details allow for “linkage attacks” where clinical data is cross-referenced with public data (e.g., news or LinkedIn), a key concern in privacy engineering.

Appendix

Table 5: Expanded PHI taxonomy

\mathcal{T}

. This table illustrates how each category is grounded in federal regulations (CFR) and clinical coding standards (ICD-10), justifying their inclusion as high-risk privacy vectors in medical LLM evaluation.

Model	Instance-Level LR (%)		Fact-Level LR (%)		Alignment (%)*
Model	NLI	Human Avg.	NLI	Human Avg.	Accuracy	Sensitivity	$F_{1}$ -score
GPT-5-nano	38.0	41.0	30.9	29.6	71.6 \| 71.0	63.4 \| 69.4	54.5 \| 65.2
Claude Haiku 4.5	14.0	39.0	12.3	29.0	77.2 \| 69.0	38.7 \| 35.3	48.6 \| 46.1
Gemini 3 Flash	42.0	69.0	32.1	57.4	74.7 \| 73.0	55.9 \| 60.9	71.7 \| 75.7
Llama3-70B	26.0	38.0	17.3	24.1	84.6 \| 80.0	53.9 \| 58.1	62.7 \| 68.8
Qwen 3.6	34.0	41.0	24.7	32.7	79.6 \| 77.0	56.7 \| 63.8	64.6 \| 69.4

Appendix

Table 6: Comparison of automatic (original RoBERTa) and human leakage risk (LR) rates across models. Human LR values are arithmetic means across the two independent human raters. Alignment is computed separately against each rater and then averaged. *Alignment is reported as Fact-Level | Instance-Level.

Setting	Value
Base model	Llama3-Med42-8B
Dataset Size	885
Max sequence length $L$	1024
Epochs $E$	2
Learning rate $\eta$	$2\times 10^{-4}$
Per-device batch size	2

Appendix

Table 7: Default SFT-LoRA configuration used in our experiments.

Panel A: Training-data-size ablation
Training condition	Mean utility (0–5), 95% CI	Good utility ( $\geq 4$ )	$\mathbf{LR_{inst}}$ (%)	$\mathbf{LR_{fact}}$ (%)
SFT, 25% of training subset	4.05 [3.99, 4.12]	74.0%	62.3	51.4
SFT, 50% of training subset	4.07 [4.01, 4.13]	74.7%	67.5	54.1
SFT, 75% of training subset	4.10 [4.04, 4.17]	77.2%	67.6	56.1
SFT, full 885-example subset	4.25 [4.19, 4.31]	82.0%	50.5	38.9
Panel B: Seed sensitivity using the full 885-example subset

Appendix

Table 8: Robustness analysis of SFT on Med42-v2-8B. Panel A evaluates different fractions of the 885-example SFT training subset. Panel B reports sensitivity to random seed using the full SFT subset. Lower

\mathrm{LR}_{inst}

and

\mathrm{LR}_{fact}

indicate better privacy preservation.

Attribute	Category	Count (%)
Gender	Male	1,550 (52.76%)
	Female	1,388 (47.24%)
Age Group	0–18 (Pediatric)	442 (15.04%)
	19–40 (Adult)	753 (25.63%)
	41–65 (Middle Age)	1,089 (37.07%)
	65+ (Senior)	654 (22.26%)

Appendix

Table 9: Demographic distribution of the patients involved in the benchmark.

Example 1

patient id: 0

gender: M

age: 60

phi categories: Mental Health and Cognitive Status; Socioeconomic Determinants of Health (SDOH).

(Phase 1: PHI Injection: The Agent reads the patient summary first, then determines which kinds of PHI are suitable here.)

phi contents: The patient has untreated major depression with episodic suicidal thoughts but refuses to discuss this with family.; The patient is currently living in a car and has no reliable access to refrigeration for biologic injections.

Appendix

Table 10: Benchmark Example (1) from the PMC-Patients Dataset. The highlighted sentences are the injected PHI statements in the original patient summary.

Example 2

patient id: 229

gender: M

age: 44

phi categories: Stigmatized Behaviors; Socioeconomic Determinants of Health (SDOH).

(Phase 1: PHI Injection: The Agent first identifies sensitive PHI categories that could plausibly appear in the patient’s clinical and social context.)

phi contents: The patient has a history of long-term opioid misuse, specifically oxycodone obtained without a prescription. The patient’s unstable employment status, as he works irregular hours as a freelance driver.

Appendix

Table 11: Benchmark Example (2) from the PMC-Patients Dataset. The highlighted sentences are the injected PHI statements in the original patient summary.

Privacy-Utility Tradeoff

Privacy-utility tradeoffs study how to limit disclosure of sensitive information while preserving useful data, models, and services.

See all 71 papers

Clinically Grounded Privacy Evaluation of Medical LMs
Jun 8, 2026Sasha Ronaghi, Sana Tonekaboni, Lena Stempfle +6Data Leakage Memorization in Language Models
1Stanford University · 2Massachusetts Institute of Technology · 3American Board of Family Medicine
Medmarks: A Comprehensive Open-Source LLM Benchmark Suite for Medical Tasks
May 2, 2026Benjamin Warner, Ratna Sagari Grandhi, Max Kieffer +32LLM Evaluation Healthcare
Sophont · MedARC · Prime Intellect
PrivMedChat: End-to-End Differentially Private RLHF for Medical Dialogue Systems
Mar 3, 2026Sudip BhujelPrivacy-Preserving Language Models RL from Human Feedback
Department of Computer Science, University of Kentucky, Lexington, KY, USA

Jun 8, 2026J/K move · Enter open · S save

Clinically Grounded Privacy Evaluation of Medical LMs

Sasha Ronaghi, Sana Tonekaboni, Lena Stempfle, Vivian Utti+5

1Stanford University · 2Massachusetts Institute of Technology · 3American Board of Family Medicine

Data Leakage Memorization in Language Models Privacy Leakage in Language Models Clinical Language Model Evaluation Medical LLMs

Open paper AskPDF arXiv

Medical language models (LMs) can memorize and reproduce protected health information, but privacy evaluations often focus on recovery of training text rather than disclosure under realistic threat models. We introduce a clinically grounded framework that evaluates leakage along a graded axis of adversarial access, ranging from publicly inferable demographics to leaked note fragments. At each tier, we measure verbatim memorization of patient-specific text and semantic leakage of sensitive diagnoses. Applying the framework to an LM pretrained on 378k clinical notes, we find that routine encounter metadata (i.e. name, date of birth, provider, practice, visit date) elicits high rates of verbatim memorization across a patient's timeline and sensitive-diagnosis recovery (AUROC 0.91 for abortion, 0.81 for HIV). At the same time, exact-match memorization can overstate disclosure: 36% of memorized tokens reflect templated documentation. Our work highlights the risks of training on longitudinal clinical data, providing a practical framework for contextual privacy evaluation of medical LMs.

Abstract

Privacy-Utility Tradeoff

Explore similar work

Clinically Grounded Privacy Evaluation of Medical LMs

Medmarks: A Comprehensive Open-Source LLM Benchmark Suite for Medical Tasks

PrivMedChat: End-to-End Differentially Private RLHF for Medical Dialogue Systems