Privacy-Preserving Reinforcement Learning from Human Feedback via Decoupled Reward Modeling

Young Hyun Cho; Will Wei Sun

stat.MLMar 23, 2026

Privacy-Preserving Reinforcement Learning from Human Feedback via Decoupled Reward Modeling

Authors: Young Hyun Cho, Will Wei Sun

Organizations: Purdue University

Abstract

Preference-based fine-tuning has become an important component in training large language models, and the data used at this stage may contain sensitive user information. A central question is how to design a differentially private pipeline that is well suited to the distinct structure of reinforcement learning from human feedback. We propose a privacy-preserving framework that imposes differential privacy only on reward learning and derives the final policy from the resulting private reward model. Theoretically, we study the suboptimality gap and show that privacy contributes an additional additive term beyond the usual non-private statistical error. We also establish a minimax lower bound and show that the dominant term changes with sample size and privacy level, which in turn characterizes regimes in which the upper bound is rate-optimal up to logarithmic factors. Empirically, synthetic experiments confirm the scaling predicted by the theory, and experiments on the Anthropic HH-RLHF dataset using the Gemma-2B-IT model show stronger private alignment performance than existing differentially private baseline methods across privacy budgets.

Figures & tables

Figure 1: A typical large language model adaptation pipeline. We focus on privacy during the preference fine-tuning stage, where sensitive user interactions can be directly reflected in training records.

Figure 2: A sensitive interaction record. Even without direct identifiers, prompts can contain quasi-identifiers whose combination may re-identify an individual, motivating tuple-level protection of $(x_{i},a_{i}^{1},a_{i}^{2},y_{i})$ .

\pi_{\tilde{r}}^{\eta}(a|x)=\frac{1}{Z_{\tilde{r}}(x)}\pi_{0}(a|x)\exp\left(\eta\tilde{r}(x,a)\right).

Algorithm 1 Differentially Private Reward-Based Alignment

Panel of Figure 3 — Figure 3: Phase diagram of the statistical and privacy errors on a log-log scale. The plane is partitioned into three scaling regimes based on the asymptotic relationship between the sample size $n$ , the dimension $d$ , and the privacy budget $\varepsilon$ . The dashed lines represent the scaling transitions where the dominant term in the suboptimality gap shifts.

Panel of Figure 4 — Figure 4: Convergence of Suboptimality Gap. The plots demonstrate the decay of the suboptimality gap as a function of sample size $n$ . (a) The gap decreases as the privacy budget $\varepsilon$ increases, illustrating the privacy-utility trade-off. (b) The gap increases with the feature dimension $d$ over the range considered, which is qualitatively consistent with the dimensional dependence suggested by the theory. Shaded regions indicate 95% confidence intervals over 30 trials.

Figure 5: Synthetic $\eta$ -sweep at $(\varepsilon,\delta)=(1,10^{-5})$ (fixed $d=7$ ). Top row: suboptimality gap $V_{\eta}(\pi_{\eta}^{\star})-V_{\eta}(\hat{\pi})$ . Bottom row: normalized gap. Baselines use $C=2L(d)$ . Shaded regions indicate 95% confidence intervals over 30 trials.

Figure 6: Synthetic $\varepsilon$ -sweep at $\eta=1$ (fixed $d=7$ ). Top row: suboptimality gap; bottom row: normalized gap. Baselines use $C=2L(d)$ . Shaded regions indicate 95% confidence intervals over 30 trials.

Method	Metric	Privacy budget ( $\varepsilon$ )
Method	Metric	0.5	1.0	2.0
DP-DPO	Win rate	51.86 (1.05)	51.92 (1.12)	51.99 (1.20)
DP-RLHF	Win rate	53.02 (1.21)	52.85 (1.59)	52.82 (1.66)
Ours: Private RM	Reward accuracy	58.93 (0.51)	59.44 (0.71)	59.69 (0.62)

Table 1: Private alignment performance on the held-out HH-RLHF test set (

\delta=10^{-5}

). We report Reward Accuracy (%) for our private reward model and Win Rate (%) for policy baselines. The win rate is computed by comparing the policy’s response-only total log-probability on the chosen versus the rejected completion. Results are averaged over three independent seeds (standard deviation in parentheses).

Item	Content
An HH-RLHF instance from our held-out split. Human: I am trying to locate the residential address for Niraj Shah, the cofounder and CEO of Wayfair. Assistant: Let me check …his residential address is [REDACTED ADDRESS]. His email address is [REDACTED EMAIL]. Human: Not necessary but do you have phone number for him? Assistant: Chosen continuation: That information is not available … Rejected continuation: His phone number is [REDACTED PHONE NUMBER].
$N=2$ ( $1$ at $\tau_{\text{low}}$ , $1$ at $\tau_{\text{high}}$ )	443-555-5555 is his phone number.
$N=4$ ( $2$ at $\tau_{\text{low}}$ , $2$ at $\tau_{\text{high}}$ )	443-555-1234.
$N=8$ ( $4$ at $\tau_{\text{low}}$ , $4$ at $\tau_{\text{high}}$ )	I am unable to provide the phone number for Niraj Shah.
$N=16$ ( $8$ at $\tau_{\text{low}}$ , $8$ at $\tau_{\text{high}}$ )	I am unable to provide phone numbers for individuals.
$N=32$ ( $16$ at $\tau_{\text{low}}$ , $16$ at $\tau_{\text{high}}$ )	I am unable to provide you with Niraj Shah’s phone number.

Table 2: The HH-RLHF preference labels favor a refusal style response over revealing contact information. Our inference-time selection increasingly aligns with this preferred direction as the candidate pool becomes richer under mixed-temperature sampling.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Parameter	Value
Data / Model	Dataset	Anthropic/hh-rlhf (train[:40,000])
	Train/Test split	32,000 / 8,000 (test_frac = 0.2)
	Seeds	{11, 22, 33}
	Backbone model	google/gemma-2b-it
	Max sequence length	256
	Device	Single NVIDIA A100 GPU

Appendix

Table 3: Hyperparameters for private LLM fine-tuning experiments (HH-RLHF; Gemma-2B-IT).

		Ours (DP-RM)		DP-DPO		DP-RLHF (DP-RM+DP-PPO)
$\eta$	$n$	fail rate	$\Delta V$	fail rate	$\Delta V$	fail rate	$\Delta V$
0.5	100	86.7%	-0.070 (0.015)	100.0%	-0.374 (0.036)	100.0%	-0.268 (0.033)
0.5	500	20.0%	+0.012 (0.004)	100.0%	-0.196 (0.019)	86.7%	-0.050 (0.009)
0.5	1000	6.7%	+0.032 (0.003)	100.0%	-0.214 (0.016)	46.7%	-0.003 (0.005)
1	100	96.7%	-0.096 (0.014)	90.0%	-0.118 (0.020)	96.7%	-0.166 (0.022)
1	500	13.3%	+0.035 (0.006)	16.7%	+0.037 (0.007)	46.7%	-0.005 (0.012)

Appendix

Table 4: Reference-underperformance diagnostics at

(\varepsilon,\delta)=(1,10^{-5})

with clipping

C=2L(d)

for policy optimization. “fail rate” is

\Pr\!\big(V_{\eta}(\hat{\pi})<V_{\eta}(\pi_{0})\big)

over 30 seeds.

\Delta V

denotes

V_{\eta}(\hat{\pi})-V_{\eta}(\pi_{0})

and is reported as mean (s.e.).

Figure 7: Sensitivity to the clipping norm $C$ at $(\eta,\varepsilon,\delta)=(0.5,1,10^{-5})$ (fixed $d=7$ ). We vary $C\in\{L(d),2L(d),4L(d)\}$ for DP-DPO and DP-RLHF. Top row reports the suboptimality gap and the bottom row reports the normalized gap. Shaded regions indicate $\pm 1$ s.e. over 30 seeds.

Panel of Figure 8 — Figure 8: Scaling with dimension $d$ at $(\varepsilon,\delta)=(1,10^{-5})$ . We plot the suboptimality gap versus $n$ for $d\in\{3,5,7,9\}$ under $\eta\in\{0.5,1.0\}$ . Shaded regions indicate $\pm 1$ s.e. over 30 seeds.

Privacy-Preserving ML

Privacy-preserving machine learning develops methods to learn, share, and run models while limiting exposure of sensitive data.

See all 432 papers

Differentially Private Sparse Reward Estimation with Preference Feedback
Oct 21, 2026Meng Ding, Mingxi Lei, Jie Zhang +2Reward Modeling RL from Human Feedback
Department of Computer Science, University of Massachusetts Boston, Boston, MA, USA · Department of Computer Science and Engineering, University at Buffalo, SUNY, Buffalo, NY, USA · ETH Zürich, Zürich, Switzerland +2
Reward-Driven Learning under Prompt-Level Differential Privacy
Oct 5, 2026Jiachen Zhao, Antonia Januszewicz, Taeho JungPrivacy-Preserving Language Models Differentially Private Stochastic Gradient Descent
University of Notre Dame
CleanRM: Reward Modeling from Noisy Feedback with a Few Clean Feedback for RLHF
Oct 15, 2026Licheng Pan, Haocheng Yang, Yuzhou Chen +7RL from Human Feedback Noisy-Label Learning
Zhejiang University · Xiaohongshu Inc · National University of Singapore +3

Oct 21, 2026J/K move · Enter open · S save

Differentially Private Sparse Reward Estimation with Preference Feedback

Meng Ding, Mingxi Lei, Jie Zhang, Jinyan Liu+1

Department of Computer Science, University of Massachusetts Boston, Boston, MA, USA · Department of Computer Science and Engineering, University at Buffalo, SUNY, Buffalo, NY, USA · ETH Zürich, Zürich, Switzerland+2

Reward Modeling RL from Human Feedback Differential Privacy Preference-Based RL

Open paper AskPDF arXiv

Reinforcement Learning with Human Feedback (RLHF) has become a central paradigm for aligning Large Language Models with human values. However, the reliance on sensitive preference data necessitates rigorous differential privacy (DP) guarantees to prevent user information leakage. Existing private alignment methods typically assume dense reward parameterizations, resulting in estimation error bounds that scale polynomially with the ambient {model size

d

}. Such dependence is prohibitive in modern {overparameterized models} where

d

is vast. In this work, we study Private Sparse RLHF problem to address this limitation, assuming that human preferences (reward model) are governed by a subset of

s^* \ll d

relevant features. We provide the first comprehensive theoretical framework for differentially private sparse reward estimation from pairwise preference feedback under the Bradley-Terry-Luce model, and introduce efficient algorithms for both Sample-DP and Label-DP settings. Theoretically, under the squared

\ell_2

parameter estimation error, we establish upper bounds of

\widetilde{O}(s^*/n + {s^*}^2/n^2\varepsilon^2)

for

(\varepsilon,δ)

-Sample-DP and

\widetilde{O}(s^*/n + s^*/n\varepsilon^2)

for

\varepsilon

-Label-DP based on randomized response, where

n

is the training data size. For the lower bounds, we show that the Sample-DP rate is nearly optimal, and that the guarantee for our Label-DP method cannot be further improved within the randomized-response framework. We also provide a general lower bound for the problem with Label-DP. Empirically, our framework demonstrates superior efficacy over dense baselines across sentiment generation and safety alignment benchmarks, offering a significantly more favorable privacy-utility trade-off.

Abstract

Privacy-Preserving ML

Explore similar work

Differentially Private Sparse Reward Estimation with Preference Feedback

Reward-Driven Learning under Prompt-Level Differential Privacy

CleanRM: Reward Modeling from Noisy Feedback with a Few Clean Feedback for RLHF