Robust Direct Preference Optimization under Preference Label-Flip Attacks
Authors: Chenye Yang, Weiyu Xu, Lifeng Lai
Organizations: Department of Electrical and Computer Engineering, University of California, Davis, Davis, CA 95616, USA · Department of Electrical and Computer Engineering, University of Iowa, Iowa City, IA 52242, USA
Direct Preference Optimization (DPO) is widely used for offline Reinforcement Learning from Human Feedback. However, recent works have shown that DPO can be easily manipulated by adversarial preference label flip attacks, which are specifically designed to mislead the learner toward an attacker-chosen target policy. To address this issue, we design robust DPO methods that can defend against such attacks. Building on the property that lable flips induce parameter-independent gradient shifts in log-linear DPO, we propose Trim-K Robust-DPO, which removes the K highest-impact pairs, and Soft-Trim-K Robust-DPO, which smoothly downweights them. We establish deterministic gradient perturbation bounds uniformly over all attack sets containing at most K flips, explicitly separating defense-induced bias from surviving attack effects. Under bounded-region assumptions, these bounds yield time-uniform parameter and policy deviation guarantees. We further quantify how quadratic regularization improves curvature while introducing additional bias. Experiments on Stanford Human Preferences, HH-RLHF, and UltraFeedback-Binarized validate our analysis.
Figures & tables
Figure 1: Candidate-set KL(π⋆∥⋅) under two fixed 30% label-flip attacks. Solid curves: our proposed methods. Dashed curves: DPO (left) and matched DPO_REG (right). λ=10−3 . Lower is better.
Figure 2: Left vertical axis: Soft-Trim- K weights wi(τ) as a function of the rank r(i) , for different temperatures τ . Right vertical axis: the impact scores si=β∥vi∥2 as a function of rank, with β=1 . The trimming cutoff at K=48,000 is indicated by the vertical dashed line.
Figure 3: No-attack scenario. candKL , plotted as KL(π⋆∥⋅) (left), and ∥θ−θ⋆∥2 (right). The unregularized proposed defenses use K=0 , while the REG variants keep λ=10−3 .
Figure 4: No-attack scenario with conservative wrong K . candKL , plotted as KL(π⋆∥⋅) (left), and ∥θ−θ⋆∥2 (right). The true attack budget is zero, but the defender uses K=48,000 , so hard trimming discards clean high-impact pairs and soft trimming keeps a smaller clean-data bias.
Figure 5: Random-corruption scenario ( 30% of pairs flipped uniformly at random), averaged over ten seeds. Left: candKL from clean DPO. Right: parameter distance from θ⋆ .
Figure 6: Top-impact attack. candKL , plotted as KL(π⋆∥⋅) (left), and ∥θ−θ⋆∥2 (right). TRIM matches its clean-data counterpart with the same K .
Figure 7: Targeted BMP-A attack. candKL , plotted as KL(π⋆∥⋅) (left), and ∥θ−θ⋆∥2 (right). Both REG variants reduce deviation from clean DPO.
Figure 8: No-attack scenario with all baselines. Left: candKL from clean DPO. Right: parameter distance from θ⋆ . The proposed defenses use K=0 , and the REG variants keep λ=10−3 . NR_DPO (with ϵ=0 ) and DR_DPO also converge to near-zero candKL . WDPO and KLDPO converge to a moderately different solution ( candKL≈0.09 – 0.10 ).
Figure 9: Random-corruption scenario with all baselines, averaged over ten seeds. Left: candKL from clean DPO. Right: parameter distance from θ⋆ .
Figure 10: Top-impact attack with all baselines. Left: candKL from clean DPO. Right: parameter distance from θ⋆ .
Figure 11: Targeted BMP-A attack with all baselines. Left: candKL from clean DPO. Right: parameter distance from θ⋆ .
Method
No-attack
Random ( 30% )
Top-impact
Targeted BMP-A
DPO (no defense)
0 (ref.)
0.041
0.155
0.289
DPO_REG
0.069
0.087
0.109
0.130
TRIM
0
0.039
0.018
0.335
TRIM_REG
0.069
0.092
0.080
0.118
SOFT_TRIM
0
0.037
0.064
0.320
SOFT_TRIM_REG
0.069
0.089
0.093
0.123
Appendix
Table 1: Final-step candKL across all four baseline-comparison scenarios. The random-corruption column reports ten-seed means for all methods. Appendix I reports the corresponding mean and standard deviation for the proposed methods. In the no-attack control, the unregularized proposed defenses use K=0 and the regularized ones keep λ=10−3 . NR_DPO uses ϵ=0 without attacks and ϵ=0.3 in attacked scenarios.
Attack
DPO_REG
TRIM_REG
Reduction
SOFT_TRIM_REG
Reduction
Top-impact
0.1089
0.0803
26.3%
0.0934
14.2%
BMP-A
0.1296
0.1176
9.3%
0.1232
4.9%
Appendix
Table 2: Final SGD candidate-set KL from full-data clean DPO in the 160,000 -pair SHP setting. Percentages are computed from unrounded values relative to matched full-data DPO_REG .
Scenario
TRIM_REG vs. Random TRIM_REG
SOFT_TRIM_REG vs. Permuted SOFT_TRIM_REG
Clean, wrong K
−0.0086±0.0002
−0.0042±0.0001
Random flips
−0.0038±0.0005
−0.0019±0.0004
Top-impact
+0.0278±0.0003
+0.0150±0.0001
BMP-A
+0.0090±0.0004
+0.0048±0.0002
Appendix
Table 3: Nonzero-conditioned control candKL minus impact-score candKL after 100,000 SGD updates. Positive values favor the impact score. Mean ± sample standard deviation is over ten paired trials.
Figure 12: Score-assignment ablation: candKL versus SGD steps, with the same quadratic regularizer for every method. Random TRIM_REG and Permuted SOFT_TRIM_REG are the nonzero-conditioned controls. Lines show means and shaded bands show ± one sample standard deviation over ten random assignments with fixed minibatch seeds. The impact-based methods and DPO_REG use one fixed SGD run for each attack.
Dataset
Budget
DPO
DPO_REG
TRIM
TRIM_REG
SOFT_TRIM
SOFT_TRIM_REG
BMP-A Residual
SHP
5%
.015
.030
.018
.030
.017
.030
.674
10%
.035
.034
.042
.034
.039
.034
.493
20%
.076
.040
.087
.038
.081
.039
.244
30%
.110
.043
.126
.039
.117
.041
.102
HH-RLHF
5%
.015
.039
.015
.039
.014
.039
.698
10%
.034
.048
.038
.048
.035
.048
.520
Appendix
Table 4: Three-seed mean two-candidate policy KL under nested BMP-A budgets. Residual is BMP-A’s relative target-gradient residual.
Figure 13: Three-dataset log-linear BMP-A budget sweep. Lines show means and shaded bands show ± one sample standard deviation over three seeds.
Dataset
Assumed K/N
TRIM / TRIM_REG
SOFT_TRIM / SOFT_TRIM_REG
SHP
0%
.0743 / .0390
.0743 / .0390
1%
.0767 / .0390
.0758 / .0390
2%
.0770 / .0388
.0760 / .0389
4%
.0788 / .0386
.0765 / .0388
8%
.0806 / .0381
.0780 / .0385
16%
.0875 / .0375
.0801 / .0381
Appendix
Table 5: Mean test two-candidate KL for a fixed true 20% attack while assumed K changes. Each entry is unregularized / regularized methods.
Dataset
TRIM
TRIM_REG
SOFT_TRIM
SOFT_TRIM_REG
SHP
0/0/0
32/32/32
0/0/0
32/32/32
HH-RLHF
0/0/0
32/32/4
0/0/0
32/16/4
UltraFeedback-Binarized
2/0/0
32/32/32
8/0/0
32/32/32
Appendix
Table 6: Validation-selected assumed K/N for seeds 0/1/2 . The true attack fraction is 20% in every run.
Figure 14: Held-out test KL for a fixed- 20% BMP-A support with a varied defender budget. Lines show means and shaded bands show ± one sample standard deviation over three seeds. The vertical dashed line marks the true attack fraction.
DPO
DPO_REG
TRIM
TRIM_REG
SOFT_TRIM
SOFT_TRIM_REG
.213±.025
.141±.019
.297±.029
.115±.013
.241±.009
.128±.015
Appendix
Table 7: Mean ± sample standard deviation candidate-set DKL(πclean∥πmethod) over three nonlinear-head seeds under the fixed 30% neural BMP-A setting.
Figure 15: Small nonlinear-head sweep over frozen SHP representations. Left: nested 5%,10%,20%,30% neural BMP-A attacks. Right: clean data with the same assumed budgets. Lines show means and shaded bands show ± one sample standard deviation over three seeds.
Method
Static
Refreshed
TRIM
.2974
.2941
TRIM_REG
.1149
.1056
SOFT_TRIM
.2410
.2266
SOFT_TRIM_REG
.1277
.1216
Appendix
Table 8: Static versus periodically refreshed candidate-set KL, averaged over three nonlinear-head seeds.
Offline Reinforcement Learning from Human Feedback (RLHF) pipelines such as Direct Preference Optimization (DPO) train on a pre-collected preference dataset, which makes them vulnerable to preference poisoning attack. We study label flip attacks against log-linear DPO. We first illustrate that flipping one preference label induces a parameter-independent shift in the DPO gradient. Using this key property, we can then convert the targeted poisoning problem into a structured binary sparse approximation problem. To solve this problem, we develop two attack methods: Binary-Aware Lattice Attack (BAL-A) and Binary Matching Pursuit Attack (BMP-A). BAL-A embeds the binary flip selection problem into a binary-aware lattice and applies Lenstra-Lenstra-Lovász reduction and Babai's nearest plane algorithm; we provide sufficient conditions that enforce binary coefficients and recover the minimum-flip objective. BMP-A adapts binary matching pursuit to our non-normalized gradient dictionary and yields coherence-based recovery guarantees and robustness (impossibility) certificates for K-flip budgets. Experiments on synthetic dictionaries and the Stanford Human Preferences dataset validate the theory and highlight how dictionary geometry governs attack success.
Chenye Yang, Weiyu Xu, Lifeng Lai
Department of Electrical and Computer Engineering, University of California, Davis, Davis, CA, USA · Department of Electrical and Computer Engineering, University of Iowa, Iowa City, IA, USA.
Direct Preference Optimization (DPO) has emerged as a popular alternative to Reinforcement Learning from Human Feedback (RLHF), offering theoretical equivalence with simpler implementation. We prove this equivalence is conditional rather than universal, depending on an implicit assumption frequently violated in practice: the RLHF-optimal policy must prefer human-preferred responses. When this assumption fails, DPO optimizes relative advantage over the reference policy rather than absolute alignment with human preferences, leading to pathological convergence where policies decrease DPO loss while preferring dispreferred responses. We characterize when this assumption is violated, show the existence of an undesirable solution space, and prove that DPO and RLHF optimize fundamentally different objectives in such cases. To address this, we introduce Constrained Preference Optimization (CPO), augmenting RLHF with constraints for provable alignment. We further provide a geometric interpretation through soft margin ranking, revealing that DPO implements margin ranking with potentially negative targets. Our theoretical analysis establishes when DPOs' guarantees hold and provides solutions preserving simplicity with provable alignment. Comprehensive experiments on standard benchmarks demonstrate that CPO achieves state-of-the-art performance. Code is available at: https://github.com/visitworld123/CPO.
Zhiqin Yang, Yonggang Zhang, Wei Xue +3
1The Hong Kong University of Science and Technology · 2LIGHTSPEED · 3Hong Kong Baptist University.
Direct Preference Optimization (DPO) has become a standard framework for safety alignment, but its reliance on pairwise preference updates makes training sensitive to imperfect supervision. Existing robust DPO methods often address this sensitivity through global loss corrections or external data-level interventions, while largely overlooking how unreliable comparisons distort batch-level optimization dynamics. We propose CompassDPO, a reward-free DPO framework that stabilizes preference optimization through dynamics control. Using the implicit DPO reward margin as a training-time compass, CompassDPO regulates sample influence along two complementary axes: update direction and update magnitude. For directional control, it applies sparse, budgeted, and warm-up delayed loss mixing to attenuate update components that conflict with the emerging preference direction. For magnitude control, it adaptively soft-winsorizes high-loss tail contributions, reducing tail dominance while preserving useful gradients from hard examples. Both mechanisms use only signals available during standard DPO training and require no external reward model or additional supervision. Experiments on PKU-SafeRLHF across four backbones and multiple out-of-distribution safety benchmarks show that CompassDPO consistently improves robustness over vanilla DPO and strong DPO-family baselines, especially under controlled label-flip noise. Code is available at https://anonymous.4open.science/r/CompassDPO-4D00
Jilong Liu, Yonghui Yang, Pengyang Shao +5
Hefei University of Technology · National University of Singapore