Robust Direct Preference Optimization under Preference Label-Flip Attacks

Chenye Yang; Weiyu Xu; Lifeng Lai

cs.LGOct 18, 2026

Robust Direct Preference Optimization under Preference Label-Flip Attacks

Authors: Chenye Yang, Weiyu Xu, Lifeng Lai

Organizations: Department of Electrical and Computer Engineering, University of California, Davis, Davis, CA 95616, USA · Department of Electrical and Computer Engineering, University of Iowa, Iowa City, IA 52242, USA

Abstract

Direct Preference Optimization (DPO) is widely used for offline Reinforcement Learning from Human Feedback. However, recent works have shown that DPO can be easily manipulated by adversarial preference label flip attacks, which are specifically designed to mislead the learner toward an attacker-chosen target policy. To address this issue, we design robust DPO methods that can defend against such attacks. Building on the property that lable flips induce parameter-independent gradient shifts in log-linear DPO, we propose Trim- $K$ Robust-DPO, which removes the $K$ highest-impact pairs, and Soft-Trim- $K$ Robust-DPO, which smoothly downweights them. We establish deterministic gradient perturbation bounds uniformly over all attack sets containing at most $K$ flips, explicitly separating defense-induced bias from surviving attack effects. Under bounded-region assumptions, these bounds yield time-uniform parameter and policy deviation guarantees. We further quantify how quadratic regularization improves curvature while introducing additional bias. Experiments on Stanford Human Preferences, HH-RLHF, and UltraFeedback-Binarized validate our analysis.

Figures & tables

Panel of Figure 1 — Figure 1: Candidate-set $\mathrm{KL}(\pi^{\star}\|\cdot)$ under two fixed $30\%$ label-flip attacks. Solid curves: our proposed methods. Dashed curves: DPO (left) and matched DPO_REG (right). $\lambda=10^{-3}$ . Lower is better.

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

\tilde{F}_{G,A}(\theta_{t})=\frac{1}{N-K}\sum_{i\in G}\ell_{\mathrm{DPO}}(\theta_{t};x^{i},\tilde{y}_{w}^{i},\tilde{y}_{l}^{i})

Appendix

Algorithm 1 Trim-

K

Robust-DPO

\tilde{F}_{\tau,A}(\theta_{t})=\frac{1}{W_{\tau}}\sum_{i=1}^{N}w_{i}^{(\tau)}\ell_{\mathrm{DPO}}(\theta_{t};x^{i},\tilde{y}_{w}^{i},\tilde{y}_{l}^{i})

Appendix

Algorithm 2 Soft-Trim-

K

Robust-DPO

Figure 2: Left vertical axis: Soft-Trim- $K$ weights $w_{i}^{(\tau)}$ as a function of the rank $r(i)$ , for different temperatures $\tau$ . Right vertical axis: the impact scores $s_{i}=\beta\|v_{i}\|_{2}$ as a function of rank, with $\beta=1$ . The trimming cutoff at $K=48{,}000$ is indicated by the vertical dashed line.

Panel of Figure 3 — Figure 3: No-attack scenario. $\mathrm{candKL}$ , plotted as $\mathrm{KL}(\pi^{\star}\|\cdot)$ (left), and $\|\theta-\theta^{\star}\|_{2}$ (right). The unregularized proposed defenses use $K=0$ , while the REG variants keep $\lambda=10^{-3}$ .

Panel of Figure 4 — Figure 4: No-attack scenario with conservative wrong $K$ . $\mathrm{candKL}$ , plotted as $\mathrm{KL}(\pi^{\star}\|\cdot)$ (left), and $\|\theta-\theta^{\star}\|_{2}$ (right). The true attack budget is zero, but the defender uses $K=48{,}000$ , so hard trimming discards clean high-impact pairs and soft trimming keeps a smaller clean-data bias.

Panel of Figure 5 — Figure 5: Random-corruption scenario ( $30\%$ of pairs flipped uniformly at random), averaged over ten seeds. Left: candKL from clean DPO. Right: parameter distance from $\theta^{\star}$ .

Panel of Figure 6 — Figure 6: Top-impact attack. $\mathrm{candKL}$ , plotted as $\mathrm{KL}(\pi^{\star}\|\cdot)$ (left), and $\|\theta-\theta^{\star}\|_{2}$ (right). TRIM matches its clean-data counterpart with the same $K$ .

Panel of Figure 7 — Figure 7: Targeted BMP-A attack. $\mathrm{candKL}$ , plotted as $\mathrm{KL}(\pi^{\star}\|\cdot)$ (left), and $\|\theta-\theta^{\star}\|_{2}$ (right). Both REG variants reduce deviation from clean DPO.

Panel of Figure 8 — Figure 8: No-attack scenario with all baselines. Left: candKL from clean DPO. Right: parameter distance from $\theta^{\star}$ . The proposed defenses use $K=0$ , and the REG variants keep $\lambda=10^{-3}$ . NR_DPO (with $\epsilon=0$ ) and DR_DPO also converge to near-zero $\mathrm{candKL}$ . WDPO and KLDPO converge to a moderately different solution ( $\mathrm{candKL}\approx 0.09$ – $0.10$ ).

Panel of Figure 9 — Figure 9: Random-corruption scenario with all baselines, averaged over ten seeds. Left: candKL from clean DPO. Right: parameter distance from $\theta^{\star}$ .

Panel of Figure 10 — Figure 10: Top-impact attack with all baselines. Left: candKL from clean DPO. Right: parameter distance from $\theta^{\star}$ .

Panel of Figure 11 — Figure 11: Targeted BMP-A attack with all baselines. Left: candKL from clean DPO. Right: parameter distance from $\theta^{\star}$ .

Method	No-attack	Random ( $30\%$ )	Top-impact	Targeted BMP-A
DPO (no defense)	$0$ (ref.)	$0.041$	$0.155$	$0.289$
DPO_REG	$0.069$	$0.087$	$0.109$	$0.130$
TRIM	$0$	$0.039$	$0.018$	$0.335$
TRIM_REG	$0.069$	$0.092$	$0.080$	$0.118$
SOFT_TRIM	$0$	$0.037$	$0.064$	$0.320$
SOFT_TRIM_REG	$0.069$	$0.089$	$0.093$	$0.123$

Appendix

Table 1: Final-step

\mathrm{candKL}

across all four baseline-comparison scenarios. The random-corruption column reports ten-seed means for all methods. Appendix I reports the corresponding mean and standard deviation for the proposed methods. In the no-attack control, the unregularized proposed defenses use

K=0

and the regularized ones keep

\lambda=10^{-3}

. NR_DPO uses

\epsilon=0

without attacks and

\epsilon=0.3

in attacked scenarios.

Attack	DPO_REG	TRIM_REG	Reduction	SOFT_TRIM_REG	Reduction
Top-impact	0.1089	0.0803	26.3%	0.0934	14.2%
BMP-A	0.1296	0.1176	9.3%	0.1232	4.9%

Appendix

Table 2: Final SGD candidate-set KL from full-data clean DPO in the

160{,}000

-pair SHP setting. Percentages are computed from unrounded values relative to matched full-data DPO_REG .

Scenario	TRIM_REG vs. Random TRIM_REG	SOFT_TRIM_REG vs. Permuted SOFT_TRIM_REG
Clean, wrong $K$	$-0.0086\pm 0.0002$	$-0.0042\pm 0.0001$
Random flips	$-0.0038\pm 0.0005$	$-0.0019\pm 0.0004$
Top-impact	$+0.0278\pm 0.0003$	$+0.0150\pm 0.0001$
BMP-A	$+0.0090\pm 0.0004$	$+0.0048\pm 0.0002$

Appendix

Table 3: Nonzero-conditioned control candKL minus impact-score candKL after

100{,}000

SGD updates. Positive values favor the impact score. Mean

\pm

sample standard deviation is over ten paired trials.

Panel of Figure 12 — Figure 12: Score-assignment ablation: candKL versus SGD steps, with the same quadratic regularizer for every method. Random TRIM_REG and Permuted SOFT_TRIM_REG are the nonzero-conditioned controls. Lines show means and shaded bands show $\pm$ one sample standard deviation over ten random assignments with fixed minibatch seeds. The impact-based methods and DPO_REG use one fixed SGD run for each attack.

Dataset	Budget	DPO	DPO_REG	TRIM	TRIM_REG	SOFT_TRIM	SOFT_TRIM_REG	BMP-A Residual
SHP	5%	.015	.030	.018	.030	.017	.030	.674
	10%	.035	.034	.042	.034	.039	.034	.493
	20%	.076	.040	.087	.038	.081	.039	.244
	30%	.110	.043	.126	.039	.117	.041	.102
HH-RLHF	5%	.015	.039	.015	.039	.014	.039	.698
	10%	.034	.048	.038	.048	.035	.048	.520

Appendix

Table 4: Three-seed mean two-candidate policy KL under nested BMP-A budgets. Residual is BMP-A’s relative target-gradient residual.

Panel of Figure 13 — Figure 13: Three-dataset log-linear BMP-A budget sweep. Lines show means and shaded bands show $\pm$ one sample standard deviation over three seeds.

Dataset	Assumed $K/N$	TRIM / TRIM_REG	SOFT_TRIM / SOFT_TRIM_REG
SHP	0%	.0743 / .0390	.0743 / .0390
	1%	.0767 / .0390	.0758 / .0390
	2%	.0770 / .0388	.0760 / .0389
	4%	.0788 / .0386	.0765 / .0388
	8%	.0806 / .0381	.0780 / .0385
	16%	.0875 / .0375	.0801 / .0381

Appendix

Table 5: Mean test two-candidate KL for a fixed true

20\%

attack while assumed

K

changes. Each entry is unregularized / regularized methods.

Dataset	TRIM	TRIM_REG	SOFT_TRIM	SOFT_TRIM_REG
SHP	0/0/0	32/32/32	0/0/0	32/32/32
HH-RLHF	0/0/0	32/32/4	0/0/0	32/16/4
UltraFeedback-Binarized	2/0/0	32/32/32	8/0/0	32/32/32

Appendix

Table 6: Validation-selected assumed

K/N

for seeds

0/1/2

. The true attack fraction is

20\%

in every run.

Panel of Figure 14 — Figure 14: Held-out test KL for a fixed- $20\%$ BMP-A support with a varied defender budget. Lines show means and shaded bands show $\pm$ one sample standard deviation over three seeds. The vertical dashed line marks the true attack fraction.

DPO	DPO_REG	TRIM	TRIM_REG	SOFT_TRIM	SOFT_TRIM_REG
$.213\pm.025$	$.141\pm.019$	$.297\pm.029$	$.115\pm.013$	$.241\pm.009$	$.128\pm.015$

Appendix

Table 7: Mean

\pm

sample standard deviation candidate-set

D_{\mathrm{KL}}(\pi_{\mathrm{clean}}\|\pi_{\mathrm{method}})

over three nonlinear-head seeds under the fixed

30\%

neural BMP-A setting.

Panel of Figure 15 — Figure 15: Small nonlinear-head sweep over frozen SHP representations. Left: nested $5\%,10\%,20\%,30\%$ neural BMP-A attacks. Right: clean data with the same assumed budgets. Lines show means and shaded bands show $\pm$ one sample standard deviation over three seeds.

Method	Static	Refreshed
TRIM	.2974	.2941
TRIM_REG	.1149	.1056
SOFT_TRIM	.2410	.2266
SOFT_TRIM_REG	.1277	.1216

Appendix

Table 8: Static versus periodically refreshed candidate-set KL, averaged over three nonlinear-head seeds.

Direct Preference Optimization

Direct Preference Optimization trains generative models from preferred and rejected outputs without fitting an explicit reward model.

See all 165 papers

Efficient Preference Poisoning Attack on Offline RLHF
May 4, 2026Chenye Yang, Weiyu Xu, Lifeng LaiDirect Preference Optimization Data Poisoning Attacks
Department of Electrical and Computer Engineering, University of California, Davis, Davis, CA, USA · Department of Electrical and Computer Engineering, University of Iowa, Iowa City, IA, USA.
Conditional Equivalence of DPO and RLHF: Implicit Assumption, Failure Modes, and Provable Alignment
May 20, 2026Zhiqin Yang, Yonggang Zhang, Wei Xue +3LLM Alignment Direct Preference Optimization
1The Hong Kong University of Science and Technology · 2LIGHTSPEED · 3Hong Kong Baptist University.
CompassDPO: Dynamics-Controlled Direct Preference Optimization for Robust Safety Alignment
Mar 7, 2026Jilong Liu, Yonghui Yang, Pengyang Shao +5Direct Preference Optimization LLM Safety Alignment
Hefei University of Technology · National University of Singapore

May 4, 2026J/K move · Enter open · S save

Efficient Preference Poisoning Attack on Offline RLHF

Chenye Yang, Weiyu Xu, Lifeng Lai

Department of Electrical and Computer Engineering, University of California, Davis, Davis, CA, USA · Department of Electrical and Computer Engineering, University of Iowa, Iowa City, IA, USA.

Direct Preference Optimization Data Poisoning Attacks RL from Human Feedback

Open paper AskPDF arXiv

Offline Reinforcement Learning from Human Feedback (RLHF) pipelines such as Direct Preference Optimization (DPO) train on a pre-collected preference dataset, which makes them vulnerable to preference poisoning attack. We study label flip attacks against log-linear DPO. We first illustrate that flipping one preference label induces a parameter-independent shift in the DPO gradient. Using this key property, we can then convert the targeted poisoning problem into a structured binary sparse approximation problem. To solve this problem, we develop two attack methods: Binary-Aware Lattice Attack (BAL-A) and Binary Matching Pursuit Attack (BMP-A). BAL-A embeds the binary flip selection problem into a binary-aware lattice and applies Lenstra-Lenstra-Lovász reduction and Babai's nearest plane algorithm; we provide sufficient conditions that enforce binary coefficients and recover the minimum-flip objective. BMP-A adapts binary matching pursuit to our non-normalized gradient dictionary and yields coherence-based recovery guarantees and robustness (impossibility) certificates for

K

-flip budgets. Experiments on synthetic dictionaries and the Stanford Human Preferences dataset validate the theory and highlight how dictionary geometry governs attack success.

Abstract

Direct Preference Optimization

Explore similar work

Efficient Preference Poisoning Attack on Offline RLHF

Conditional Equivalence of DPO and RLHF: Implicit Assumption, Failure Modes, and Provable Alignment

CompassDPO: Dynamics-Controlled Direct Preference Optimization for Robust Safety Alignment