Reliability-Weighted Direct Preference Optimization: Penalized Weights for Robust Preference Learning

Bin Luo; Param Venkat Vivek Kesireddy; Anil Vallepu

stat.MLOct 30, 2026

Reliability-Weighted Direct Preference Optimization: Penalized Weights for Robust Preference Learning

Authors: Bin Luo, Param Venkat Vivek Kesireddy, Anil Vallepu

Organizations: School of Data Science and Analytics, Kennesaw State University

Abstract

Direct Preference Optimization (DPO) treats every reported comparison as correct, yet preference data contain reversed, tied or otherwise noisy comparisons. RW-DPO gives each comparison an unknown reliability weight $w_i\in[0,1]$ that enters the likelihood linearly -- DPO is $w\equiv1$ , a reversed comparison $w_i=0$ -- and estimates the weights jointly with the policy under a minimax-concave penalty on $-\log w_i$ . The penalty keeps every comparison the model agrees with at exactly $w_i=1$ and acts only on those it contradicts, through closed-form weight updates at negligible cost over DPO. We prove that recovering all flipped comparisons from a single annotation per comparison requires margins of order $\log n$ . Furthermore, in a linear-index model, starting from an initial fit pointing in the correct direction, our estimator jointly consistently recovers the preference direction and weights, as well as identifies the flipped comparisons above this margin scale, even when a fixed fraction of comparisons is flipped. In simulations RW-DPO has the lowest average estimation error among DPO, noise-rate corrections and recent robust methods, with far more precise flags than recent label-correction methods. In DPO fine-tuning of a language model with 10-30% flipped labels, its win rate does not fall as the flip rate grows and exceeds that of DPO trained on clean labels, matching the strongest recent method with corrections concentrated on the flipped labels.

Figures & tables

Figure 1: Error relative to DPO (1 = no gain; bars

\pm 2

s.e.) against the flip rate, every method tuned on clean validation data. Left: relative error under random flips (log scale). Middle and right: angle and relative error under feature-dependent flips with skewed features. Settings in Appendix F .

	random flips		feature-dependent flips		clear-cut flips
Method	rel. err.	angle	rel. err.	angle	detection	swamping	precision	angle
Oracle labels	0.162	0.150	0.167	0.153	–	–	–	0.078
DPO (logistic)	0.632	0.383	0.666	0.591	–	–	–	0.150
rDPO	0.406	0.383	0.556	0.592	–	–	–	0.147
Copas (noise-aware)	0.385	0.374	0.530	0.571	–	–	–	0.093
Trimmed logistic	0.410	0.383	0.560	0.576	0.94	0.004	0.95	0.095

Table 2: RW-DPO has the lowest relative error under both flip types and, on clear-cut flips, far more precise flags than PLC-DPO and mean-shift. Linear-index model, clean-validation tuning. Random flips and feature-dependent flips with skewed features: relative and angle error, means over

\rho\in\{0.1,\dots,0.4\}

, 12 settings, 100 replicates (s.e.

\leq 0.005

). Clear-cut flips (flips only where

|d_{i}^{*}|\geq 3.5

\rho\in\{0.05,0.1\}

): detection, swamping and precision of the flags, and angle error. Designs, methods and metrics: Appendix F . Results by flip rate: Table 5 . Bold: best non-oracle.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

		$w_{\mathrm{sat}}$
$c$	reversal needs $w_{\mathrm{sat}}>$	$0.005$	$0.1$	$0.2$	$0.3$	$0.5$
0.5	0.075	f 0.34	$1.75$	$0.79$	$0.60$	$0.50$
0.85	0.173	f 0.54	f 0.48	$2.48$	$1.32$	$0.91$
1	0.224	f 0.61	f 0.56	f 0.49	$1.87$	$1.12$
1.5	0.408	f 0.77	f 0.75	f 0.74	f 0.72	$2.29$

Appendix

Table 4: Regimes of the

W_{\infty}

weight step at a fixed cut. A number is the reversal threshold

c^{\infty}

(a comparison with

z_{i}<-c^{\infty}

is reversed, one with

-c^{\infty}\leq z_{i}<-c

attenuated); an entry f

x

means reversal is impossible (condition ( C∞ ) fails) and the most contradicted comparisons stop at weight

x

. Computed by src/regimes.py .

Method	random flips	feature-dependent flips
Copas (noise-aware)	$-0.0039$ (0.0008)	$-0.0044$ (0.0007)
Mean-shift $\ell_{1}$	$-0.0120$ (0.0009)	$-0.0188$ (0.0009)
PLC-DPO	$-0.0135$ (0.0008)	$-0.0103$ (0.0007)
rDPO	$-0.0256$ (0.0011)	$-0.0299$ (0.0011)
Trimmed logistic	$-0.0293$ (0.0010)	$-0.0341$ (0.0011)
DPO (logistic)	$-0.2511$ (0.0026)	$-0.1405$ (0.0019)

Appendix

Table 6: RW-DPO’s advantage in relative error is systematic: mean paired difference (RW-DPO minus method; negative favours RW-DPO) over the same replicates, with its standard error, every method tuned on clean validation data (Table 2 ).

Epochs	Method	$\rho=0.0$	$\rho=0.1$	$\rho=0.2$	$\rho=0.3$	mean over $\rho>0$
1	DPO	37.8 $\pm$ 0.2	33.6 $\pm$ 0.5	30.1 $\pm$ 0.7	26.4 $\pm$ 0.2	30.0
	rDPO ( $\varepsilon=\rho$ )	–	34.1 $\pm$ 0.5	30.1 $\pm$ 0.1	26.8 $\pm$ 0.3	30.4
	cDPO ( $\varepsilon=\rho$ )	–	32.7 $\pm$ 0.7	29.5 $\pm$ 0.7	25.7 $\pm$ 0.2	29.3
	PLC-DPO	48.5 $\pm$ 0.7	44.8 $\pm$ 0.9	40.2 $\pm$ 1.5	32.6 $\pm$ 0.1	39.2
	RW-DPO	48.8 $\pm$ 0.3	46.2 $\pm$ 0.4	40.1 $\pm$ 0.5	31.6 $\pm$ 2.2	39.3
2	DPO	50.0 $\pm$ 1.1	45.9 $\pm$ 1.0	41.7 $\pm$ 1.8	34.5 $\pm$ 0.2	40.7

Appendix

Table 8: Win rate (%) against DPO trained on clean labels for two epochs, averaged over its three seeds (the reference of Table 3 );

50

= as good as clean-label DPO. UltraFeedback with a share

\rho

of flipped labels, Qwen2.5-1.5B (LoRA), mean

\pm

s.d. over three seeds (fewer where marked).

Epochs	Method	$\rho=0.0$	$\rho=0.1$	$\rho=0.2$	$\rho=0.3$	mean over $\rho>0$
1	DPO	64.7 $\pm$ 0.2	65.5 $\pm$ 0.1	65.7 $\pm$ 0.4	65.9 $\pm$ 0.2	65.7
	rDPO ( $\varepsilon=\rho$ )	–	65.3 $\pm$ 0.2	65.7 $\pm$ 0.0	66.0 $\pm$ 0.9	65.7
	cDPO ( $\varepsilon=\rho$ )	–	65.5 $\pm$ 0.1	65.7 $\pm$ 0.3	65.7 $\pm$ 0.9	65.6
	PLC-DPO	61.6 $\pm$ 0.1	62.0 $\pm$ 0.3	63.1 $\pm$ 0.1	64.1 $\pm$ 0.3	63.1
	RW-DPO	62.4 $\pm$ 0.1	61.9 $\pm$ 0.3	62.3 $\pm$ 1.1	63.0 $\pm$ 1.5	62.4
2	DPO	66.7 $\pm$ 0.2	65.5 $\pm$ 0.3	64.8 $\pm$ 0.4	65.3 $\pm$ 0.4	65.2

Appendix

Table 9: Preference accuracy (%) on the clean held-out test set. UltraFeedback with a share

\rho

of flipped labels, Qwen2.5-1.5B (LoRA), mean

\pm

s.d. over three seeds (fewer where marked).

$\rho$	flagged	detection	swamping	precision
0.1	0.128	0.308	0.108	0.24
0.2	0.152	0.307	0.114	0.40
0.3	0.179	0.296	0.129	0.50

Appendix

Table 10: RW-DPO’s flags on the language model (two epochs,

c=0.85

w_{\mathrm{sat}}=0.005

), by flip rate: share of comparisons with

\hat{w}_{i}<1

in the weights used in the second epoch, detection (share of flips flagged), swamping (share of clean comparisons flagged) and precision (share of flags that are flips); means over seeds. Flip masks are used only for this evaluation.

Direct Preference Optimization

Direct Preference Optimization trains generative models from preferred and rejected outputs without fitting an explicit reward model.

See all 173 papers

Aug 31, 2026

PLC-DPO: Posterior Label Correction in Noisy and Ambiguous Preference Optimization

Direct Preference Optimization (DPO) simplifies alignment through pairwise comparisons but assumes all observed preferences are reliable. Real data often violates this assumption, leading to reversed, weak, or ambiguous labels that cause harmful policy updates. To address this, we propose Posterior Label Correction DPO (PLC-DPO) to robustly optimize preferences by routing each pair's training signal as a clean, flip, or tie case. The key idea is to use the calibrated policy-reference margin as online evidence to take appropriate correction actions. This reframes noisy preference learning as actively correcting supervision direction and strength rather than merely filtering suspicious examples. Across 57 dataset-model-benchmark cells, PLC-DPO obtains the best mean win rate against DPO (60.5 vs. 55.5 for the next-best method). Injected-noise and tie stress tests, human disagreement analysis, and self-confirmation diagnostics further show that the routing remains stable and distinguishes flipped from weakly directional pairs.

Boryeong Cho, Sumyeong Ahn, Se-Young Yun

KAIST AI · KENTECH

Pairwise Preference Learning Direct Preference Optimization Preference Alignment Preference Learning

Ask PDF

Oct 18, 2026

Robust Direct Preference Optimization under Preference Label-Flip Attacks

Direct Preference Optimization (DPO) is widely used for offline Reinforcement Learning from Human Feedback. However, recent works have shown that DPO can be easily manipulated by adversarial preference label flip attacks, which are specifically designed to mislead the learner toward an attacker-chosen target policy. To address this issue, we design robust DPO methods that can defend against such attacks. Building on the property that lable flips induce parameter-independent gradient shifts in log-linear DPO, we propose Trim-

K

Robust-DPO, which removes the

K

highest-impact pairs, and Soft-Trim-

K

Robust-DPO, which smoothly downweights them. We establish deterministic gradient perturbation bounds uniformly over all attack sets containing at most

K

flips, explicitly separating defense-induced bias from surviving attack effects. Under bounded-region assumptions, these bounds yield time-uniform parameter and policy deviation guarantees. We further quantify how quadratic regularization improves curvature while introducing additional bias. Experiments on Stanford Human Preferences, HH-RLHF, and UltraFeedback-Binarized validate our analysis.

Chenye Yang, Weiyu Xu, Lifeng Lai

Department of Electrical and Computer Engineering, University of California, Davis, Davis, CA 95616, USA · Department of Electrical and Computer Engineering, University of Iowa, Iowa City, IA 52242, USA

Direct Preference Optimization Adversarial Robustness Preference-Based RL Adversarial Attacks

Ask PDF

Jun 10, 2026

Boosting Direct Preference Optimization with Penalization

Offline preference optimization has become a practical substitute for reinforcement learning from human feedback, but pairwise objectives such as Direct Preference Optimization (DPO) and its variants use only the chosen and rejected responses stored in a static dataset. This leaves a useful signal unused: the response that the reference model itself would generate for the same prompt. We propose Direct Preference Optimization with Penalization (DPOP), a simple extension of DPO that augments the base preference loss with a gated penalty on reference-greedy responses. DPOP activates this penalty only when the current policy still assigns a lower likelihood to the preferred response than to the rejected response. On AlpacaEval 2.0, DPOP improves length-controlled win rate over DPO, SimPO, and AlphaDPO on both Llama-3-8b-it and Gemma-2-9b-it, achieving relative gains of 5.3% and 4.4% over baselines on the two models, respectively. Ablations further show that a SimNPO-style length-normalized penalty is stronger than NPO and token-level unlikelihood in this setting.

Pengwei Sun

Stanford University.

Pairwise Preference Learning LLM Alignment Direct Preference Optimization Offline Preference Optimization

Ask PDF

Abstract

Direct Preference Optimization

Explore similar work

PLC-DPO: Posterior Label Correction in Noisy and Ambiguous Preference Optimization

Robust Direct Preference Optimization under Preference Label-Flip Attacks

Boosting Direct Preference Optimization with Penalization