Reliability-Weighted Direct Preference Optimization: Penalized Weights for Robust Preference Learning

Bin Luo; Param Venkat Vivek Kesireddy; Anil Vallepu

stat.MLOct 30, 2026

Reliability-Weighted Direct Preference Optimization: Penalized Weights for Robust Preference Learning

Authors: Bin Luo, Param Venkat Vivek Kesireddy, Anil Vallepu

Organizations: School of Data Science and Analytics, Kennesaw State University

Abstract

Direct Preference Optimization (DPO) treats every reported comparison as correct, yet preference data contain reversed, tied or otherwise noisy comparisons. RW-DPO gives each comparison an unknown reliability weight $w_i\in[0,1]$ that enters the likelihood linearly -- DPO is $w\equiv1$ , a reversed comparison $w_i=0$ -- and estimates the weights jointly with the policy under a minimax-concave penalty on $-\log w_i$ . The penalty keeps every comparison the model agrees with at exactly $w_i=1$ and acts only on those it contradicts, through closed-form weight updates at negligible cost over DPO. We prove that recovering all flipped comparisons from a single annotation per comparison requires margins of order $\log n$ . Furthermore, in a linear-index model, starting from an initial fit pointing in the correct direction, our estimator jointly consistently recovers the preference direction and weights, as well as identifies the flipped comparisons above this margin scale, even when a fixed fraction of comparisons is flipped. In simulations RW-DPO has the lowest average estimation error among DPO, noise-rate corrections and recent robust methods, with far more precise flags than recent label-correction methods. In DPO fine-tuning of a language model with 10-30% flipped labels, its win rate does not fall as the flip rate grows and exceeds that of DPO trained on clean labels, matching the strongest recent method with corrections concentrated on the flipped labels.

Figures & tables

Figure 1: Error relative to DPO (1 = no gain; bars

\pm 2

s.e.) against the flip rate, every method tuned on clean validation data. Left: relative error under random flips (log scale). Middle and right: angle and relative error under feature-dependent flips with skewed features. Settings in Appendix F .

	random flips		feature-dependent flips		clear-cut flips
Method	rel. err.	angle	rel. err.	angle	detection	swamping	precision	angle
Oracle labels	0.162	0.150	0.167	0.153	–	–	–	0.078
DPO (logistic)	0.632	0.383	0.666	0.591	–	–	–	0.150
rDPO	0.406	0.383	0.556	0.592	–	–	–	0.147
Copas (noise-aware)	0.385	0.374	0.530	0.571	–	–	–	0.093
Trimmed logistic	0.410	0.383	0.560	0.576	0.94	0.004	0.95	0.095

Table 2: RW-DPO has the lowest relative error under both flip types and, on clear-cut flips, far more precise flags than PLC-DPO and mean-shift. Linear-index model, clean-validation tuning. Random flips and feature-dependent flips with skewed features: relative and angle error, means over

\rho\in\{0.1,\dots,0.4\}

, 12 settings, 100 replicates (s.e.

\leq 0.005

). Clear-cut flips (flips only where

|d_{i}^{*}|\geq 3.5

\rho\in\{0.05,0.1\}

): detection, swamping and precision of the flags, and angle error. Designs, methods and metrics: Appendix F . Results by flip rate: Table 5 . Bold: best non-oracle.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

		$w_{\mathrm{sat}}$
$c$	reversal needs $w_{\mathrm{sat}}>$	$0.005$	$0.1$	$0.2$	$0.3$	$0.5$
0.5	0.075	f 0.34	$1.75$	$0.79$	$0.60$	$0.50$
0.85	0.173	f 0.54	f 0.48	$2.48$	$1.32$	$0.91$
1	0.224	f 0.61	f 0.56	f 0.49	$1.87$	$1.12$
1.5	0.408	f 0.77	f 0.75	f 0.74	f 0.72	$2.29$

Appendix

Table 4: Regimes of the

W_{\infty}

weight step at a fixed cut. A number is the reversal threshold

c^{\infty}

(a comparison with

z_{i}<-c^{\infty}

is reversed, one with

-c^{\infty}\leq z_{i}<-c

attenuated); an entry f

x

means reversal is impossible (condition ( C∞ ) fails) and the most contradicted comparisons stop at weight

x

. Computed by src/regimes.py .

Method	random flips	feature-dependent flips
Copas (noise-aware)	$-0.0039$ (0.0008)	$-0.0044$ (0.0007)
Mean-shift $\ell_{1}$	$-0.0120$ (0.0009)	$-0.0188$ (0.0009)
PLC-DPO	$-0.0135$ (0.0008)	$-0.0103$ (0.0007)
rDPO	$-0.0256$ (0.0011)	$-0.0299$ (0.0011)
Trimmed logistic	$-0.0293$ (0.0010)	$-0.0341$ (0.0011)
DPO (logistic)	$-0.2511$ (0.0026)	$-0.1405$ (0.0019)

Appendix

Table 6: RW-DPO’s advantage in relative error is systematic: mean paired difference (RW-DPO minus method; negative favours RW-DPO) over the same replicates, with its standard error, every method tuned on clean validation data (Table 2 ).

Epochs	Method	$\rho=0.0$	$\rho=0.1$	$\rho=0.2$	$\rho=0.3$	mean over $\rho>0$
1	DPO	37.8 $\pm$ 0.2	33.6 $\pm$ 0.5	30.1 $\pm$ 0.7	26.4 $\pm$ 0.2	30.0
	rDPO ( $\varepsilon=\rho$ )	–	34.1 $\pm$ 0.5	30.1 $\pm$ 0.1	26.8 $\pm$ 0.3	30.4
	cDPO ( $\varepsilon=\rho$ )	–	32.7 $\pm$ 0.7	29.5 $\pm$ 0.7	25.7 $\pm$ 0.2	29.3
	PLC-DPO	48.5 $\pm$ 0.7	44.8 $\pm$ 0.9	40.2 $\pm$ 1.5	32.6 $\pm$ 0.1	39.2
	RW-DPO	48.8 $\pm$ 0.3	46.2 $\pm$ 0.4	40.1 $\pm$ 0.5	31.6 $\pm$ 2.2	39.3
2	DPO	50.0 $\pm$ 1.1	45.9 $\pm$ 1.0	41.7 $\pm$ 1.8	34.5 $\pm$ 0.2	40.7

Appendix

Table 8: Win rate (%) against DPO trained on clean labels for two epochs, averaged over its three seeds (the reference of Table 3 );

50

= as good as clean-label DPO. UltraFeedback with a share

\rho

of flipped labels, Qwen2.5-1.5B (LoRA), mean

\pm

s.d. over three seeds (fewer where marked).

Epochs	Method	$\rho=0.0$	$\rho=0.1$	$\rho=0.2$	$\rho=0.3$	mean over $\rho>0$
1	DPO	64.7 $\pm$ 0.2	65.5 $\pm$ 0.1	65.7 $\pm$ 0.4	65.9 $\pm$ 0.2	65.7
	rDPO ( $\varepsilon=\rho$ )	–	65.3 $\pm$ 0.2	65.7 $\pm$ 0.0	66.0 $\pm$ 0.9	65.7
	cDPO ( $\varepsilon=\rho$ )	–	65.5 $\pm$ 0.1	65.7 $\pm$ 0.3	65.7 $\pm$ 0.9	65.6
	PLC-DPO	61.6 $\pm$ 0.1	62.0 $\pm$ 0.3	63.1 $\pm$ 0.1	64.1 $\pm$ 0.3	63.1
	RW-DPO	62.4 $\pm$ 0.1	61.9 $\pm$ 0.3	62.3 $\pm$ 1.1	63.0 $\pm$ 1.5	62.4
2	DPO	66.7 $\pm$ 0.2	65.5 $\pm$ 0.3	64.8 $\pm$ 0.4	65.3 $\pm$ 0.4	65.2

Appendix

Table 9: Preference accuracy (%) on the clean held-out test set. UltraFeedback with a share

\rho

of flipped labels, Qwen2.5-1.5B (LoRA), mean

\pm

s.d. over three seeds (fewer where marked).

$\rho$	flagged	detection	swamping	precision
0.1	0.128	0.308	0.108	0.24
0.2	0.152	0.307	0.114	0.40
0.3	0.179	0.296	0.129	0.50

Appendix

Table 10: RW-DPO’s flags on the language model (two epochs,

c=0.85

w_{\mathrm{sat}}=0.005

), by flip rate: share of comparisons with

\hat{w}_{i}<1

in the weights used in the second epoch, detection (share of flips flagged), swamping (share of clean comparisons flagged) and precision (share of flags that are flips); means over seeds. Flip masks are used only for this evaluation.

Direct Preference Optimization

Direct Preference Optimization trains generative models from preferred and rejected outputs without fitting an explicit reward model.

See all 173 papers

PLC-DPO: Posterior Label Correction in Noisy and Ambiguous Preference Optimization
Aug 31, 2026Boryeong Cho, Sumyeong Ahn, Se-Young YunPairwise Preference Learning Direct Preference Optimization
KAIST AI · KENTECH
Robust Direct Preference Optimization under Preference Label-Flip Attacks
Oct 18, 2026Chenye Yang, Weiyu Xu, Lifeng LaiDirect Preference Optimization Adversarial Robustness
Department of Electrical and Computer Engineering, University of California, Davis, Davis, CA 95616, USA · Department of Electrical and Computer Engineering, University of Iowa, Iowa City, IA 52242, USA
Boosting Direct Preference Optimization with Penalization
Jun 10, 2026Pengwei SunPairwise Preference Learning LLM Alignment
Stanford University.

Aug 31, 2026J/K move · Enter open · S save

PLC-DPO: Posterior Label Correction in Noisy and Ambiguous Preference Optimization

Boryeong Cho, Sumyeong Ahn, Se-Young Yun

KAIST AI · KENTECH

Pairwise Preference Learning Direct Preference Optimization Preference Alignment Preference Learning

Open paper AskPDF arXiv

Direct Preference Optimization (DPO) simplifies alignment through pairwise comparisons but assumes all observed preferences are reliable. Real data often violates this assumption, leading to reversed, weak, or ambiguous labels that cause harmful policy updates. To address this, we propose Posterior Label Correction DPO (PLC-DPO) to robustly optimize preferences by routing each pair's training signal as a clean, flip, or tie case. The key idea is to use the calibrated policy-reference margin as online evidence to take appropriate correction actions. This reframes noisy preference learning as actively correcting supervision direction and strength rather than merely filtering suspicious examples. Across 57 dataset-model-benchmark cells, PLC-DPO obtains the best mean win rate against DPO (60.5 vs. 55.5 for the next-best method). Injected-noise and tie stress tests, human disagreement analysis, and self-confirmation diagnostics further show that the routing remains stable and distinguishes flipped from weakly directional pairs.

Abstract

Direct Preference Optimization

Explore similar work

PLC-DPO: Posterior Label Correction in Noisy and Ambiguous Preference Optimization

Robust Direct Preference Optimization under Preference Label-Flip Attacks

Boosting Direct Preference Optimization with Penalization