PrivMedChat: End-to-End Differentially Private RLHF for Medical Dialogue Systems

Sudip Bhujel

cs.CLMar 3, 2026

PrivMedChat: End-to-End Differentially Private RLHF for Medical Dialogue Systems

Authors: Sudip Bhujel

Organizations: Department of Computer Science, University of Kentucky, Lexington, KY, USA

Abstract

Large language models are increasingly used for patient-facing medical assistance and clinical decision support, but adapting them to clinical dialogue often requires supervision derived from doctor-patient conversations that may contain sensitive information. Conventional supervised fine-tuning and reinforcement learning from human feedback (RLHF) can amplify memorization, enabling membership inference and disclosure of rare training-set details. We present PrivMedChat (Private Medical Chat), an end-to-end framework for differentially private RLHF (DP-RLHF) for medical dialogue systems. Our approach enforces differential privacy at each training stage that accesses dialogue-derived supervision, combining DP-SGD for supervised fine-tuning and reward model learning from preference pairs, and DP-aware policy optimization for alignment. To avoid costly clinician labeling, we introduce an annotation-free preference construction strategy that pairs physician responses with filtered non-expert generations. We evaluate PrivMedChat across medical dialogue tasks and assess utility, safety, and privacy under consistent privacy accounting, thereby providing a practical pathway to align medical chatbots while offering formal privacy guarantees. We open-source our code at https://github.com/sudip-bhujel/privmedchat.

Figures & tables

Fig. 1 : A standard medical LLM that is fine-tuned without privacy safeguards may disclose the membership for patients with rare symptoms in its training set. In contrast, PrivMedChat’s application of differential privacy prevents high-confidence inferences.

Symbol	Meaning
$D_{\text{priv}}^{\text{SFT}}$	Private SFT corpus
$D_{\text{priv}}^{\text{pref}}$	Private preference pairs
$\mathcal{X}_{\text{pub}}$	Public/synthetic prompts (PPO)
$\varepsilon_{\text{SFT}}$	Privacy cost of DP-SFT
$\varepsilon_{\text{RM}}$	Privacy cost of reward model
$\varepsilon_{\text{total}}$	Total privacy cost (composed)

TABLE I : List of Notations

Panel of Figure 3 — Fig. 2 : Data pre-processing and system overview of PrivMedChat. (a) methodology for annotation-free medical preference pair construction (b) illustrates the three-zone design separating DP-protected training (Zone 1) from evaluation and deployment (Zones 2–3).

Parameter	SFT	Reward Model	PPO [ 5 ] (Actor/Critic)
Effective Batch Size	16 – 64	16 – 64	16 – 64
Learning Rate	$2\times 10^{-5}$	$5\times 10^{-5}$	$5\times 10^{-6}$
Training Duration	3 Epochs	3 Epochs	10 Iterations
Epochs per Iteration	–	–	3
Optimization Steps	–	–	$\approx 1{,}000$

TABLE II : Hyperparameter configurations for the PrivMedChat training stages. Parameters were selected to balance convergence stability with differential privacy noise requirements.

		Utility					Privacy (MIA AUC $\downarrow$ )
Model	$\boldsymbol{\varepsilon}$	ROUGE-L	BERTScore	Entity F1	PPL $\downarrow$	PubMedQA [ 20 ]	Loss [ 21 ]	Ref	Min-K% [ 22 ]	Min-K%++ [ 23 ]	Zlib [ 7 ]	Lowercase [ 7 ]
Base	–	0.133	0.830	0.112	37.34	0.554	0.518	0.500	0.513	0.505	0.513	0.502
SFT	–	0.163	0.829	0.096	19.95	0.642	0.532	0.542	0.535	0.548	0.532	0.515
Overfit SFT	–	0.162	0.823	0.093	198.61	0.536	0.527	0.518	0.528	0.531	0.523	0.505
PPO	–	0.148	0.825	0.102	20.13	0.636	0.532	0.543	0.534	0.547	0.533	0.516
DP-SFT	1	0.149	0.831	0.093	25.27	0.554	0.522	0.519	0.521	0.519	0.517	0.555

TABLE III : Utility and privacy results on the held-out test set. Bold denotes the best result per metric among DP models.

\varepsilon

denotes the cumulative privacy budget (

\delta{=}10^{-5}

). Utility metrics (ROUGE-L [ 18 ] , BERTScore [ 19 ] , Entity F1, PPL) are computed on 5,000 prompts. PubMedQA [ 20 ] accuracy is evaluated on 500 biomedical questions. Privacy columns report AUC-ROC of six membership inference attacks; values near

0.50

indicate random-guessing (ideal).

Model	$\boldsymbol{\varepsilon}$	Accuracy	Yes	No	Maybe	Unans.
Base	–	0.554	1.000	0.006	0.000	0
SFT	–	0.642	0.833	0.526	0.000	61
Overfit SFT	–	0.536	0.533	0.699	0.000	8
PPO	–	0.636	0.826	0.520	0.000	60
DP-SFT	1	0.554	0.975	0.046	0.000	3
DP-SFT	3	0.534	0.884	0.121	0.039	45

TABLE IV : PubMedQA [ 20 ] correctness evaluation (

n{=}500

). Per-class accuracy shows the model’s ability to distinguish “yes,” “no,” and “maybe” answers. Non-DP models achieve balanced predictions while DP models exhibit majority-class bias.

Model	$\varepsilon$	Fact.	Safety	Helpful.	Concise.	Empathy	Overall
SFT	–	2.76	2.53	2.32	2.36	2.04	2.4
DP-SFT	7	2.93	2.57	2.36	2.38	2.14	2.48
PrivMedChat	7	3.18	3.1	2.62	2.99	2.41	2.86

TABLE V : LLM-Jury Evaluation Results (3-model G-EVAL [ 24 ] jury on 400 held-out test prompts). Higher scores are better.

Model	$\boldsymbol{\varepsilon}$	Halluc.	Harmful	Med. Valid.	Emerg. Esc.
Base	–	1.0	0.2	0.2	5.0
SFT	–	2.2	0.2	0.4	4.6
Overfit SFT	–	1.4	0.0	0.6	3.8
PPO	–	0.8	1.2	1.4	4.8
DP-SFT	1	1.2	0.2	1.4	3.8
DP-SFT	3	2.2	0.2	0.4	5.0

TABLE VI : Safety evaluation results (

n{=}500

). All rates are reported as percentages (%).

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Budget	Stage	$n$	$q$	$T$	$C$	$\sigma$	$\varepsilon$	$\delta$
$\varepsilon_{\text{total}}{=}1$	SFT	174,957	$2.2863\times 10^{-5}$	131,217	1.0	0.5640	1.00	$10^{-5}$
	RM	27,410	$7.2966\times 10^{-5}$	41,115	1.0	0.7642	1.00	$10^{-5}$
	PPO	25,968	$3.9063\times 10^{-3}$	7,680	1.0	3.7891	1.00	$10^{-5}$
$\varepsilon_{\text{total}}{=}3$	SFT	174,957	$2.2863\times 10^{-5}$	131,217	1.0	0.4660	3.00	$10^{-5}$
	RM	27,410	$7.2966\times 10^{-5}$	41,115	1.0	0.5847	3.00	$10^{-5}$
	PPO	25,968	$3.9063\times 10^{-3}$	7,680	1.0	1.5625	3.00	$10^{-5}$

Appendix

TABLE VII : Differential Privacy Accounting Parameters per Stage. Total budget

\varepsilon_{\text{total}}

is obtained via composition over the three stages at fixed

\delta=10^{-5}

Privacy-Preserving Language Models

Privacy-preserving language models protect sensitive data during training, inference, personalization, and information sharing.

See all 178 papers

Privacy-Preserving Reinforcement Learning from Human Feedback via Decoupled Reward Modeling
Mar 23, 2026Young Hyun Cho, Will Wei SunPrivacy-Preserving ML Privacy-Preserving Language Models
Purdue University
MedPriv-Bench: Benchmarking the Privacy-Utility Trade-off of Large Language Models in Medical Open-Ended Question Answering
Mar 15, 2026Shaowei Guan, Yu Zhai, Hin Chi Kwok +5Privacy-Utility Tradeoff Privacy Leakage in Language Models
Centre for Smart Health, School of Nursing, The Hong Kong Polytechnic University · Department of Language Science and Technology, The Hong Kong Polytechnic University · Centre for Frontier AI Research, Agency for Science, Technology and Research, Singapore +2
PCQC: Privileged Counterfactual Question Credit for Multi-Turn Medical Dialogue
Sep 23, 2026Chenxuan Li, Jiayi Wan, Xinrong Chen +3Clinical Reasoning Medical LLMs
Peking University, Beijing, China · Department of Medical Bioinformatics, School of Basic Medical Sciences, Peking University, Beijing, China

Mar 23, 2026J/K move · Enter open · S save

Privacy-Preserving Reinforcement Learning from Human Feedback via Decoupled Reward Modeling

Young Hyun Cho, Will Wei Sun

Purdue University

Privacy-Preserving ML Privacy-Preserving Language Models RL from Human Feedback Preference-Based RL Reward Modeling

Open paper AskPDF arXiv

Preference-based fine-tuning has become an important component in training large language models, and the data used at this stage may contain sensitive user information. A central question is how to design a differentially private pipeline that is well suited to the distinct structure of reinforcement learning from human feedback. We propose a privacy-preserving framework that imposes differential privacy only on reward learning and derives the final policy from the resulting private reward model. Theoretically, we study the suboptimality gap and show that privacy contributes an additional additive term beyond the usual non-private statistical error. We also establish a minimax lower bound and show that the dominant term changes with sample size and privacy level, which in turn characterizes regimes in which the upper bound is rate-optimal up to logarithmic factors. Empirically, synthetic experiments confirm the scaling predicted by the theory, and experiments on the Anthropic HH-RLHF dataset using the Gemma-2B-IT model show stronger private alignment performance than existing differentially private baseline methods across privacy budgets.

Abstract

Privacy-Preserving Language Models

Explore similar work

Privacy-Preserving Reinforcement Learning from Human Feedback via Decoupled Reward Modeling

MedPriv-Bench: Benchmarking the Privacy-Utility Trade-off of Large Language Models in Medical Open-Ended Question Answering

PCQC: Privileged Counterfactual Question Credit for Multi-Turn Medical Dialogue