MedPriv-Bench: Benchmarking the Privacy-Utility Trade-off of Large Language Models in Medical Open-Ended Question Answering
Authors: Shaowei Guan, Yu Zhai, Hin Chi Kwok, Jiawei Du, Xinyu Feng, Jing Li, Harry Qin, Vivian Hui
Organizations: Centre for Smart Health, School of Nursing, The Hong Kong Polytechnic University · Department of Language Science and Technology, The Hong Kong Polytechnic University · Centre for Frontier AI Research, Agency for Science, Technology and Research, Singapore · Department of Computing, The Hong Kong Polytechnic University · Research Institute for Federated Learning, PAAI, The Hong Kong Polytechnic University
Recent advances in Retrieval-Augmented Generation enable LLMs to ground outputs in clinical evidence, but connections to external databases create the risk of contextual leakage, where unique combinations of medical details enable patient re-identification without explicit identifiers. Existing healthcare benchmarks emphasize accuracy while overlooking this risk. To fill this gap, we present MedPriv-Bench, the first benchmark for jointly evaluating privacy preservation and clinical utility in medical open-ended question answering. Our framework utilizes a multi-agent, human-in-the-loop pipeline to synthesize sensitive medical contexts and clinically relevant queries that create realistic privacy pressure. We also establish an automated evaluation protocol using a fine-tuned RoBERTa-NLI model, which achieved an instance-level F1 score of 75.3%, sensitivity of 90.7%, and an average inference time of 0.056 s per sample against human annotations. Across nine LLMs and three privacy-preserving methods, we observed a pervasive privacy-utility trade-off. Relative to unprotected Med42-v2-8B (utility 3.87/5; leakage 72.8%), supervised fine-tuning improved utility to 4.25 and reduced leakage to 38.9%, whereas local differential privacy reduced leakage to 20.5% but lowered utility to 3.03. These results demonstrate the need for domain-specific benchmarks to validate medical AI systems in privacy-sensitive settings.
Figures & tables
Figure 1: Illustration of contextual leakage in healthcare retrieval-augmented generation (RAG) systems. The difference between a privacy-preserving response and a contextual leakage response is highlighted.
Figure 2: Overview of the four-phase benchmark construction pipeline: (1) Phase 1 uses an Injection Agent to embed contextually coherent PHIs into patient summaries based on a predefined taxonomy. (2) Phase 2 employs a White-box Attack strategy, where a Question Agent crafts queries relying on the injected PHIs. (3) Phase 3 feeds the sensitive notes and queries to an Answer Agent for response generation. (4) Phase 4 applies automated quality filtering and human validation to finalize the dataset.
Model
Response Utility
Overall Privacy
Fact-Level Leakage by PHI Type (%)
LR inst (%)
LR fact (%)
SB
MH&CS
SDOH
QIV
Closed-Source Models
GPT-5-nano (w/o)
4.89/5
70.2
57.1
32.2
58.0
65.8
38.1
GPT-5-nano (w/)
4.88/5
54.6
42.7
16.9
43.5
52.7
19.6
Claude Haiku 4.5 (w/o)
4.71/5
89.0
80.0
75.4
88.3
80.5
36.1
Claude Haiku 4.5 (w/)
4.60/5
61.4
48.6
33.1
61.1
46.9
6.2
Table 1: Performance comparison of 9 LLMs with and without defense mechanisms on the Privacy-Utility Benchmark (Test set n=738 ). SB : Stigmatized Behaviors; MH&CS : Mental Health & Cognitive Status; SDOH : Socioeconomic Determinants of Health; QIV : Quasi-Identifiers / VIP Status.
Figure 3: Privacy-utility intervention effects. Changes within each model relative to its corresponding undefended configuration.
Question Category
LRinst (%)
SDOH & Psychosocial Support
55.3
Discharge Planning & Care Coordination
52.4
Medication Adherence & Management
49.4
Follow-up Care & Monitoring
47.8
Infection Control & Wound Care
46.7
Table 2: Privacy vulnerability across different clinical question types. SDOH: Socioeconomic Determinants of Health.
Figure 4: Impact of Context Volume and Word Constraints on Privacy vs. Utility.
Model
Fact-Level
Instance-Level
IAA (%)
Cohen’s κ
IAA (%)
Cohen’s κ
Gemini
96.3
0.924
94.0
0.860
Med42
93.8
0.867
90.0
0.798
DeepSeek
82.7
0.639
80.0
0.602
Llama
91.4
0.772
88.0
0.752
GPT
81.5
0.545
76.0
0.520
Table 3: Inter-rater agreement between two independent human annotators. Average values are unweighted macro averages across models.
Model
Instance-Level LR (%)
Fact-Level LR (%)
Alignment (%)*
Length
NLI
Human Avg.
NLI
Human Avg.
Accuracy
Sensitivity
F1 -score
Gemini 3 Flash
88.0
69.0
80.2
57.4
77.2 | 81.0
100.0 | 100.0
83.4 | 87.9
276
Med42-v2-70B
70.0
55.0
64.2
36.4
67.3 | 77.0
93.2 | 92.7
67.5 | 81.6
311
DeepSeek-V3.2
64.0
48.0
56.8
39.5
79.0 | 80.0
95.4 | 95.8
78.2 | 82.0
332
Llama3-70B
62.0
38.0
54.3
24.1
68.5 | 74.0
97.5 | 97.5
59.8 | 74.0
264
GPT-5-nano
28.0
41.0
19.8
29.6
77.8 | 75.0
51.3 | 57.8
56.0 | 64.3
652
Table 4: Comparison of automatic (fine-tuned RoBERTa) and human leakage risk (LR) rates across models. Human LR values are arithmetic means across the two independent human raters. Alignment is computed separately against each rater and then averaged. *Alignment is reported as Fact-Level | Instance-Level. Length is the response word count.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
PHI Category
Description and Examples
Legal & Clinical Rationale (Justification)
Stigmatized Behaviors
Confidential behaviors affecting care (e.g., heavy alcohol use; non-prescribed opioid misuse; high-risk sexual behavior).
Legal Basis: 42 CFR Part 2. This federal law provides enhanced protection for substance use records beyond standard HIPAA, as disclosure carries severe risks of social discrimination or legal repercussions. In clinical notes, these unique narratives act as “semantic identifiers” that narrow the patient pool significantly.
Mental Health & Cognitive Status
Psychiatric conditions influencing adherence and safety (e.g., depression with suicidal ideation; bipolar disorder; PTSD; MCI).
Legal/Clinical Basis: HIPAA Psychotherapy Notes (45 CFR § 164.501) and ICD-10 F-codes (F01–F99). Mental health data is categorized as high-sensitivity under HIPAA. We use ICD-10 F-codes to standardize these risks, acknowledging that specific diagnostic patterns often serve as a “clinical fingerprint” for individuals in local healthcare settings.
Socioeconomic Determinants of Health (SDOH)
Social/economic factors affecting care (e.g., homelessness; inability to afford meds; recent incarceration; unsafe home).
Legal/Clinical Basis: HIPAA 45 CFR § 160.103 and ICD-10 Z-codes (Z55–Z65). HIPAA defines PHI to include any information regarding the “provision of healthcare.” Since SDOH (mapped via Z-codes) describes obstacles to care, it is legally PHI. Social vulnerabilities combined with medical conditions drastically reduce the k-anonymity of a record.
Quasi-Identifiers / VIP Status
Attributes that indirectly identify a patient (e.g., prominent occupation; rare residence; notable family ties; unique public incident).
Legal Basis: HIPAA 164.514(b) (Expert Determination). This category addresses the “Residual Risk” of re-identification. High-profile roles or rare residential details allow for “linkage attacks” where clinical data is cross-referenced with public data (e.g., news or LinkedIn), a key concern in privacy engineering.
Appendix
Table 5: Expanded PHI taxonomy T . This table illustrates how each category is grounded in federal regulations (CFR) and clinical coding standards (ICD-10), justifying their inclusion as high-risk privacy vectors in medical LLM evaluation.
Model
Instance-Level LR (%)
Fact-Level LR (%)
Alignment (%)*
NLI
Human Avg.
NLI
Human Avg.
Accuracy
Sensitivity
F1 -score
GPT-5-nano
38.0
41.0
30.9
29.6
71.6 | 71.0
63.4 | 69.4
54.5 | 65.2
Claude Haiku 4.5
14.0
39.0
12.3
29.0
77.2 | 69.0
38.7 | 35.3
48.6 | 46.1
Gemini 3 Flash
42.0
69.0
32.1
57.4
74.7 | 73.0
55.9 | 60.9
71.7 | 75.7
Llama3-70B
26.0
38.0
17.3
24.1
84.6 | 80.0
53.9 | 58.1
62.7 | 68.8
Qwen 3.6
34.0
41.0
24.7
32.7
79.6 | 77.0
56.7 | 63.8
64.6 | 69.4
Appendix
Table 6: Comparison of automatic (original RoBERTa) and human leakage risk (LR) rates across models. Human LR values are arithmetic means across the two independent human raters. Alignment is computed separately against each rater and then averaged. *Alignment is reported as Fact-Level | Instance-Level.
Setting
Value
Base model
Llama3-Med42-8B
Dataset Size
885
Max sequence length L
1024
Epochs E
2
Learning rate η
2×10−4
Per-device batch size
2
Appendix
Table 7: Default SFT-LoRA configuration used in our experiments.
Panel A: Training-data-size ablation
Training condition
Mean utility (0–5), 95% CI
Good utility ( ≥4 )
LRinst (%)
LRfact (%)
SFT, 25% of training subset
4.05 [3.99, 4.12]
74.0%
62.3
51.4
SFT, 50% of training subset
4.07 [4.01, 4.13]
74.7%
67.5
54.1
SFT, 75% of training subset
4.10 [4.04, 4.17]
77.2%
67.6
56.1
SFT, full 885-example subset
4.25 [4.19, 4.31]
82.0%
50.5
38.9
Panel B: Seed sensitivity using the full 885-example subset
Appendix
Table 8: Robustness analysis of SFT on Med42-v2-8B. Panel A evaluates different fractions of the 885-example SFT training subset. Panel B reports sensitivity to random seed using the full SFT subset. Lower LRinst and LRfact indicate better privacy preservation.
Attribute
Category
Count (%)
Gender
Male
1,550 (52.76%)
Female
1,388 (47.24%)
Age Group
0–18 (Pediatric)
442 (15.04%)
19–40 (Adult)
753 (25.63%)
41–65 (Middle Age)
1,089 (37.07%)
65+ (Senior)
654 (22.26%)
Appendix
Table 9: Demographic distribution of the patients involved in the benchmark.
Example 1
patient id: 0
gender: M
age: 60
phi categories: Mental Health and Cognitive Status; Socioeconomic Determinants of Health (SDOH).
(Phase 1: PHI Injection: The Agent reads the patient summary first, then determines which kinds of PHI are suitable here.)
phi contents: The patient has untreated major depression with episodic suicidal thoughts but refuses to discuss this with family.; The patient is currently living in a car and has no reliable access to refrigeration for biologic injections.
Appendix
Table 10: Benchmark Example (1) from the PMC-Patients Dataset. The highlighted sentences are the injected PHI statements in the original patient summary.
Example 2
patient id: 229
gender: M
age: 44
phi categories: Stigmatized Behaviors; Socioeconomic Determinants of Health (SDOH).
(Phase 1: PHI Injection: The Agent first identifies sensitive PHI categories that could plausibly appear in the patient’s clinical and social context.)
phi contents: The patient has a history of long-term opioid misuse, specifically oxycodone obtained without a prescription. The patient’s unstable employment status, as he works irregular hours as a freelance driver.
Appendix
Table 11: Benchmark Example (2) from the PMC-Patients Dataset. The highlighted sentences are the injected PHI statements in the original patient summary.
Medical language models (LMs) can memorize and reproduce protected health information, but privacy evaluations often focus on recovery of training text rather than disclosure under realistic threat models. We introduce a clinically grounded framework that evaluates leakage along a graded axis of adversarial access, ranging from publicly inferable demographics to leaked note fragments. At each tier, we measure verbatim memorization of patient-specific text and semantic leakage of sensitive diagnoses. Applying the framework to an LM pretrained on 378k clinical notes, we find that routine encounter metadata (i.e. name, date of birth, provider, practice, visit date) elicits high rates of verbatim memorization across a patient's timeline and sensitive-diagnosis recovery (AUROC 0.91 for abortion, 0.81 for HIV). At the same time, exact-match memorization can overstate disclosure: 36% of memorized tokens reflect templated documentation. Our work highlights the risks of training on longitudinal clinical data, providing a practical framework for contextual privacy evaluation of medical LMs.
Sasha Ronaghi, Sana Tonekaboni, Lena Stempfle +6
1Stanford University · 2Massachusetts Institute of Technology · 3American Board of Family Medicine
Evaluating large language models (LLMs) for medical applications remains challenging due to benchmark saturation, limited data accessibility, and insufficient coverage of relevant tasks. Existing suites have either saturated, heavily depend on restricted datasets, or lack comprehensive model coverage. We introduce Medmarks, a fully open-source evaluation suite with 30 benchmarks spanning question answering, information extraction, medical calculations, and open-ended clinical reasoning. We perform a systematic evaluation of 61 models across 71 configurations using verifiable metrics and LLM-as-a-Judge. Our results show that frontier reasoning models (Gemini 3 Pro Preview, GPT-5.1, & GPT-5.2) achieve the highest performance across both benchmarks, most frontier proprietary models are significantly more token efficient than open-weight alternatives, medically fine-tuned models outperform their generalist counterparts, and that models are susceptible to answer-order bias (particularly smaller models and Grok 4). A subset of our evals (Medmarks-T) can be directly used as reinforcement learning environments to post-train LLMs for medical reasoning. Code is available at https://github.com/MedARC-AI/Medmarks
Benjamin Warner, Ratna Sagari Grandhi, Max Kieffer +32
Large language models are increasingly used for patient-facing medical assistance and clinical decision support, but adapting them to clinical dialogue often requires supervision derived from doctor-patient conversations that may contain sensitive information. Conventional supervised fine-tuning and reinforcement learning from human feedback (RLHF) can amplify memorization, enabling membership inference and disclosure of rare training-set details. We present PrivMedChat (Private Medical Chat), an end-to-end framework for differentially private RLHF (DP-RLHF) for medical dialogue systems. Our approach enforces differential privacy at each training stage that accesses dialogue-derived supervision, combining DP-SGD for supervised fine-tuning and reward model learning from preference pairs, and DP-aware policy optimization for alignment. To avoid costly clinician labeling, we introduce an annotation-free preference construction strategy that pairs physician responses with filtered non-expert generations. We evaluate PrivMedChat across medical dialogue tasks and assess utility, safety, and privacy under consistent privacy accounting, thereby providing a practical pathway to align medical chatbots while offering formal privacy guarantees. We open-source our code at https://github.com/sudip-bhujel/privmedchat.
Sudip Bhujel
Department of Computer Science, University of Kentucky, Lexington, KY, USA