Differentially Private Sparse Reward Estimation with Preference Feedback

Meng Ding; Mingxi Lei; Jie Zhang; Jinyan Liu; Di Wang

cs.LGOct 21, 2026

Differentially Private Sparse Reward Estimation with Preference Feedback

Authors: Meng Ding, Mingxi Lei, Jie Zhang, Jinyan Liu, Di Wang

Organizations: Department of Computer Science, University of Massachusetts Boston, Boston, MA, USA · Department of Computer Science and Engineering, University at Buffalo, SUNY, Buffalo, NY, USA · ETH Zürich, Zürich, Switzerland · Beijing Institute of Technology, Beijing, China ·

Abstract

Reinforcement Learning with Human Feedback (RLHF) has become a central paradigm for aligning Large Language Models with human values. However, the reliance on sensitive preference data necessitates rigorous differential privacy (DP) guarantees to prevent user information leakage. Existing private alignment methods typically assume dense reward parameterizations, resulting in estimation error bounds that scale polynomially with the ambient {model size $d$ }. Such dependence is prohibitive in modern {overparameterized models} where $d$ is vast. In this work, we study Private Sparse RLHF problem to address this limitation, assuming that human preferences (reward model) are governed by a subset of $s^* \ll d$ relevant features. We provide the first comprehensive theoretical framework for differentially private sparse reward estimation from pairwise preference feedback under the Bradley-Terry-Luce model, and introduce efficient algorithms for both Sample-DP and Label-DP settings. Theoretically, under the squared $\ell_2$ parameter estimation error, we establish upper bounds of $\widetilde{O}(s^*/n + {s^*}^2/n^2\varepsilon^2)$ for $(\varepsilon,δ)$ -Sample-DP and $\widetilde{O}(s^*/n + s^*/n\varepsilon^2)$ for $\varepsilon$ -Label-DP based on randomized response, where $n$ is the training data size. For the lower bounds, we show that the Sample-DP rate is nearly optimal, and that the guarantee for our Label-DP method cannot be further improved within the randomized-response framework. We also provide a general lower bound for the problem with Label-DP. Empirically, our framework demonstrates superior efficacy over dense baselines across sentiment generation and safety alignment benchmarks, offering a significantly more favorable privacy-utility trade-off.

Figures & tables

	Sample-DP	Label-DP	Upper Bound	Lower Bound
Sparse ( Ours )	✓		$\widetilde{O}\!\left(\frac{s^{}}{n}+\frac{(s^{})^{2}}{n^{2}\varepsilon^{2}}\right)$	$\widetilde{\Omega}\!\left(\frac{s^{}}{n}+\frac{(s^{})^{2}}{n^{2}\varepsilon^{2}}\right)$
Sparse ( Ours )		✓	$\widetilde{O}\!\left(\frac{s^{}}{n}+\frac{s^{}}{n\varepsilon^{2}}\right)$	$\widetilde{\Omega}\!\left(\frac{s^{}}{n}+\frac{s^{}}{n\varepsilon^{2}}\right)\dagger$	$\widetilde{\Omega}\!\left(\frac{s^{}}{n}+\frac{s^{}}{n^{2}\varepsilon^{2}}\right)\ddagger$
Non-Sparse [ 15 ]	✓	✓	$\widetilde{O}\!\left(\frac{d^{2}}{n}+\frac{d^{3}}{n^{2}\varepsilon^{2}}\right)\lx@sectionsign$	$\widetilde{\Omega}\!\left(\frac{d}{n}+\frac{d}{n^{2}\varepsilon^{2}}\right)\lx@paragraphsign$

Table 1: Estimation error rates for private RLHF in non-sparse and sparse settings with error metric

\|\cdot\|_{2}^{2}

. Here

d

is the ambient dimension,

s^{*}

is the sparsity level,

n

is the sample size, and

(\varepsilon,\delta)

are the privacy parameters.

\dagger

indicates that the lower bound is for the randomized-response mechanism, and

\ddagger

denotes a general central Label-DP lower bound for arbitrary Label-DP mechanisms.

\lx@sectionsign

denotes a best-case upper bound and

\lx@paragraphsign

indicates lower bound only for Label-DP.

\quad\theta^{t+1}\leftarrow\Pi_{B}\left(\mathrm{Peeling}\left(\theta^{t+\frac{1}{2}},s,\varepsilon,\delta,\frac{\eta}{m}\right)\right),

Algorithm 1 Sample-DP-IHT

Panel of Figure 1 — Figure 1: IMDb Sentiment Generation with Llama-7B under different privacy budgets ( $\varepsilon$ ).

	DP-IHT	RR-IHT	DP-SGD	RR
$\varepsilon$ = 2	2.37	2.44	1.98	2.12
$\varepsilon$ = 8	2.87	2.88	2.26	2.20

Table 2: Performance comparison under different privacy budgets. Higher is better.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Gemma-3-1B	Llama-2-7B
Batch Size	64	64
Mini-batch Size	64	64
Learning Rate	1e-5	1e-5
Optimizer	RMSProp	RMSProp
KL Penalty Coefficient	0.15	0.15
LoRA Rank	16	16

Appendix

Table 3: PPO parameter settings for Gemma-3-1B and Llama-2-7B.

Privacy budget	Non-private	DP-IHT	RR-IHT	DP-SGD	RR
$\varepsilon=2$	3.07	2.37	2.44	1.98	2.12
$\varepsilon=8$	3.07	2.87	2.88	2.26	2.20

Appendix

Table 4: Sentiment-generation policy scores with a non-private reference. Higher is better. The same non-private reference is shown in both rows.

Method	Mean loss/NLL $\downarrow$	Loss std.	Accuracy $\uparrow$
DP-SGD	0.5154	0.0090	0.8302
RR	0.4944	0.0012	0.8568
DP-IHT	0.3316	0.0008	0.8583
RR-IHT	0.3209	0.0006	0.8646
Non-private	0.3181	0.0008	0.8650

Appendix

Table 5: Direct reward-model evaluation reported in the rebuttal. Lower mean loss/NLL and higher held-out preference accuracy are better.

	$10^{-2}$		$10^{-3}$		$10^{-4}$
Method	Loss	Acc.	Loss	Acc.	Loss	Acc.
DP-SGD	0.5378	0.8398	0.5328	0.8336	0.5154	0.8302
RR	0.5188	0.8476	0.5247	0.8437	0.4944	0.8568
DP-IHT	0.3654	0.8425	0.3316	0.8583	0.3587	0.8554
RR-IHT	0.3209	0.8646	0.3384	0.8596	0.3328	0.8621

Appendix

Table 6: Reward-model learning-rate sweep. Loss is lower-is-better and preference accuracy (Acc.) is higher-is-better.

Learning rate	$1\times 10^{-5}$	$2\times 10^{-5}$	$4\times 10^{-5}$
Label-DP RR-IHT	$57\%$	$57\%$	$59\%$
Sample-DP IHT	$55\%$	$53\%$	$57\%$
RR	$51\%$	$55\%$	$53\%$
DP-SGD	$47\%$	$45\%$	$41\%$

Appendix

Table 7: PPO learning-rate sweep for safety alignment with Gemma-3-1B (

\varepsilon=8

), as reported in the rebuttal. Higher win rate is better. The comparison policy and evaluation sample size remain to be specified.

Reward Modeling

Reward modeling learns scores or rankings that capture human goals and task success, guiding AI selection, reinforcement learning, and alignment.

See all 325 papers

Mar 23, 2026

Privacy-Preserving Reinforcement Learning from Human Feedback via Decoupled Reward Modeling

Preference-based fine-tuning has become an important component in training large language models, and the data used at this stage may contain sensitive user information. A central question is how to design a differentially private pipeline that is well suited to the distinct structure of reinforcement learning from human feedback. We propose a privacy-preserving framework that imposes differential privacy only on reward learning and derives the final policy from the resulting private reward model. Theoretically, we study the suboptimality gap and show that privacy contributes an additional additive term beyond the usual non-private statistical error. We also establish a minimax lower bound and show that the dominant term changes with sample size and privacy level, which in turn characterizes regimes in which the upper bound is rate-optimal up to logarithmic factors. Empirically, synthetic experiments confirm the scaling predicted by the theory, and experiments on the Anthropic HH-RLHF dataset using the Gemma-2B-IT model show stronger private alignment performance than existing differentially private baseline methods across privacy budgets.

Young Hyun Cho, Will Wei Sun

Purdue University

Privacy-Preserving ML Privacy-Preserving Language Models RL from Human Feedback Preference-Based RL Reward Modeling

Ask PDF

Oct 5, 2026

Reward-Driven Learning under Prompt-Level Differential Privacy

Reinforcement learning with verifiable rewards (RLVR) trains a language model on problems that may themselves be confidential, and the trained model can reveal which problems it saw. We study RLVR under prompt-level differential privacy: the released weights must be (ε,δ)-differentially private with respect to the presence of any one training problem. Taking the group of responses to one prompt as the privacy record, our method aggregates their gradients, clips the prompt's contribution once, adds Gaussian noise, and composes the privacy loss across updates, so the budget depends on neither the number of responses per prompt nor the clipping norm; to our knowledge this is the first differential privacy guarantee for RLVR training. We train Qwen2.5-1.5B-Instruct with LoRA at a per-run budget of ε=8 and compare, on the same prompts and at the same budget, a control that removes only the reward signal and two private supervised fine-tuning recipes. The reward signal improves accuracy over the control by 2.65 points on MATH and 3.24 on GSM8K, in every seed; the improvement survives a format-robust scorer, at 1.3 points on MATH, and is not explained by response length. At the same budget the private model outperforms both supervised recipes on MATH and GSM8K by 2.3 to 3.8 points, retains 85--90% of the gain of non-private GRPO on these tasks, and on MATH the noise of an eightfold tighter budget costs at most 1.2 points. The reward effect also carries to CommonsenseQA, an exploratory non-mathematical task. Verifier feedback thus remains a usable learning signal under prompt-level privacy.

Jiachen Zhao, Antonia Januszewicz, Taeho Jung

University of Notre Dame

Privacy-Preserving Language Models Differentially Private Stochastic Gradient Descent Reinforcement Learning with Verifiable Rewards Differential Privacy

Ask PDF

Aug 26, 2026

Privacy Without Regret: Differentially Private Inference-Time Alignment

Best-of-N (BoN) sampling is the simplest and most widely deployed inference-time alignment strategy, but it suffers from two distinct problems: reward hacking, in which the selected response exploits errors in the proxy reward model, and the absence of any privacy protection for the sensitive human preference data used to train that reward model. We show that a single intervention-adding calibrated noise to reward scores before selection-resolves both. Our first result, Private Best-of-N (PrivBoN), establishes that Gumbel noise at an appropriate scale simultaneously provides

ε

-differential privacy and implements KL-regularized alignment. Whenever the privacy budget exceeds a critical threshold

ε^*

, the privacy-mandated noise is the regret-optimal regularization, and privacy imposes zero additional alignment cost-matching the information-theoretic skyline of Huang et al. (2025). Because

ε^*

depends on an unknown coverage coefficient, we introduce Private Inference-Time Pessimism (PrivITP), which combines

χ^2

-regularized rejection sampling with a two-phase Gaussian mechanism. PrivITP achieves ex-post

(ε,δ)

-DP with a privacy cost independent of the number of responses

n

, cleanly decouples the regularization parameter from the privacy parameter, and attains the skyline up to a noise-inflation term. Experiments across several language models, datasets, and reward models confirm our results: PrivBoN and PrivITP are scaling-monotonic (unlike BoN, which degrades past a critical

n

), and PrivITP matches or outperforms PrivBoN at equivalent privacy levels, with the largest gains in the strong-privacy regime.

Ishi Jain, Nandini Bhattad, Sayak Ray Chowdhury

Indian Institute of Technology Kanpur

Inference-Time Language Model Alignment Best-of-N Sampling Privacy-Preserving ML Differential Privacy Reward Hacking

Ask PDF

Abstract

Reward Modeling

Explore similar work

Privacy-Preserving Reinforcement Learning from Human Feedback via Decoupled Reward Modeling

Reward-Driven Learning under Prompt-Level Differential Privacy

Privacy Without Regret: Differentially Private Inference-Time Alignment