Uncertainty-Normalized Margins for Direct Preference Optimization

Sadegh Khorasani; Petrus Mikkola; Matthias Grossglauser

cs.LGSep 29, 2026

Uncertainty-Normalized Margins for Direct Preference Optimization

Authors: Sadegh Khorasani, Petrus Mikkola, Matthias Grossglauser

Organizations: School of Computer and Communication Sciences, EPFL, Lausanne, Switzerland · University of Helsinki, Helsinki, Finland

Abstract

Direct preference optimization (DPO) models binary preferences through a Bradley-Terry model with a common noise scale, without explicitly accounting for preference strength or prompt-dependent uncertainty from human feedback. We introduce uncertainty-normalized margin DPO (UNM-DPO), which combines strength-dependent margins with a learned prompt scale. Motivated by a heteroskedastic Bradley-Terry model, we develop two training objectives. Both compare the implicit rewards of preferred and rejected responses, derived from response log-probability ratios to a reference policy. Advantage-only (AO) divides this reward difference by the prompt scale before subtracting the margin; whole-residual (WR) subtracts the margin before dividing by the scale. For the WR comparison model, we establish a necessary and sufficient condition under which known margins make the prompt scale identifiable. We introduce a practical procedure for learning the scale. Building on WR, we introduce ULNM-DPO-WR, which normalizes each response's implicit reward by its length. We evaluate our methods against DPO and related baselines on HelpSteer2 and HelpSteer3, using the Skywork reward model as a judge. With Llama-3.1-8B-Instruct, ULNM-DPO-WR achieves tie-adjusted win rates against matched DPO of 68.00% and 65.31% on evaluation panels, with higher mean rewards and shorter responses on average. On AlpacaEval with a GPT-4.1 judge and GPT-4-Turbo reference answers, the same 8B policy achieves a length-controlled win rate of 21.62%, compared with 16.39% for DPO and 15.30% for SimPO. These results demonstrate the potential of combining preference-strength margins, learned prompt scales, and length normalization for policy optimization.

Figures & tables

	400 prompts			800 prompts
Method	Win (%) $\uparrow$	$\Delta R\;\uparrow$	$L/L_{D}$	Win (%) $\uparrow$	$\Delta R\;\uparrow$	$L/L_{D}$
DPO (ref.)	50.00	0.000	1.000	50.00	0.000	1.000
Fixed-margin DPO	50.50 [45.63, 55.38]	+0.341 [-0.284, +0.988]	1.074	50.62 [47.19, 53.94]	+0.216 [-0.281, +0.716]	1.094
ODPO	53.88 [49.00, 58.63]	+0.529 [-0.092, +1.140]	1.056	51.38 [47.94, 54.81]	+0.272 [-0.219, +0.750]	1.069
MMPO	44.50 [39.63, 49.25]	-0.271 [-0.819, +0.297]	0.985	49.13 [45.75, 52.44]	-0.289 [-0.703, +0.119]	0.981
$\beta$ -DPO	25.13 [21.00, 29.50]	-9.018 [-10.216, -7.825]	1.056	24.50 [21.56, 27.56]	-8.606 [-9.410, -7.777]	1.050

Table 1: HelpSteer3 with Llama-3.2-1B-Instruct on the 400- and 800-prompt evaluation panels.

	400 prompts			800 prompts
Method	Win (%) $\uparrow$	$\Delta R\;\uparrow$	$L/L_{D}$	Win (%) $\uparrow$	$\Delta R\;\uparrow$	$L/L_{D}$
DPO (ref.)	50.00	0.000	1.000	50.00	0.000	1.000
Fixed-margin DPO	58.13 [53.25, 62.88]	+1.120 [+0.453, +1.799]	1.024	54.75 [51.38, 58.19]	+0.322 [-0.174, +0.816]	1.004
ODPO	55.50 [50.63, 60.25]	+1.042 [+0.412, +1.692]	1.012	52.81 [49.38, 56.19]	+0.073 [-0.407, +0.546]	1.016
MMPO	48.88 [44.13, 53.63]	-0.287 [-0.876, +0.306]	1.036	45.75 [42.44, 49.19]	-0.691 [-1.111, -0.266]	1.023
$\beta$ -DPO	8.63 [6.00, 11.38]	-25.450 [-26.980, -23.917]	2.266	8.69 [6.81, 10.69]	-25.533 [-26.604, -24.434]	2.246

Table 2: HelpSteer3 with Llama-3.1-8B-Instruct on the 400- and 800-prompt evaluation panels.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Evidence bin	Mean evidence	Held-out votes	Prompts	Agreement (95% CI)
Low $[1,1.5]$	$1.229$	$3{,}236$	$1{,}390$	$96.35\%\;[95.65,96.95]$
Middle $(1.5,2.5)$	$2.000$	$1{,}864$	$1{,}070$	$100.00\%\;[99.79,100.00]$
High $[2.5,3]$	$2.804$	$2{,}065$	$894$	$100.00\%\;[99.81,100.00]$

Appendix

Table 3: Held-out-annotator directional agreement on the HelpSteer3 development partition. The last column reports descriptive Wilson intervals; the high-minus-low contrast in the text uses prompt-clustered inference.

Model	Frozen revision
Llama-3.2-1B-Instruct	9213176726f574b556790deb65791e0c5aa438b6
Llama-3.1-8B-Instruct	0e9e39f249a16976918f6564b8830bc894c89659

Appendix

Table 4: Frozen initial-model revisions.

Method	Implemented per-pair loss
DPO	$\zeta(-\beta_{0}A_{i})$ , with $\beta_{0}=0.05$ .
ODPO	$\zeta(0.75k_{i}-\beta_{0}A_{i})$ .
MMPO	$\operatorname{CE}(\sigma(2.2k_{i}),\beta_{0}A_{i})$ .
SimPO	$\zeta(1.375-2.5\bar{P}_{i})$ .
SPO-basic	$\zeta(-0.01P_{i})/0.01$ .
$\beta$ -DPO	$\zeta(-\beta_{t}A_{i})$ on the selected batch subset.

Appendix

Table 5: Implemented baseline losses for one preference pair.

Method and parameter	Candidate values	Selected / fixed
Earlier AO: margin $\tau$	$\{1/3,\,0.5,\,0.75,\,1,\,1.5\}$	$1$
ODPO: strength multiplier	$\{1/3,\,0.5,\,0.75,\,1\}$	$0.75$
MMPO: target slope $\gamma$	$\{0.3,\,0.7,\,1.1,\,2.2\}$	$2.2$
$\beta$ -DPO: adaptation coefficient $\alpha$	$\{0.2,\,0.6,\,1\}$	$0.2$
$\gamma$ -PO (DPO): base margin $\gamma_{0}$	$\{0.02,\,0.05,\,0.20,\,0.40\}$	$0.02$
SimPO	$\beta\in\{2.5,10\}$ , $\gamma/\beta\in\{0.3,0.55\}$ , LR $\in\{1,2\}\times 10^{-6}$	$2.5$ , $0.55$ , $10^{-6}$

Appendix

Table 6: Hyperparameter search grids and selected values from the earlier 500-prompt tuning stage.

Setting	Median tokens	$\beta_{\mathrm{LN}}$
HS3 1B	364.5	18.225
HS2 1B	272	13.6
HS3 8B	364.5	18.225

Appendix

Table 7: Training-length calibration of the normalized score coefficient.

Scale estimator	Supervision and fitting schedule
Pooled OOF, WR-fit	One prompt head fitted to all pooled OOF values $\widetilde{B}_{i}$ using the WR loss, then frozen; this is the main formulation.
Pooled OOF, AO-fit	One prompt head fitted to all pooled OOF values $\widetilde{B}_{i}$ using the AO loss, then frozen.
Cross-fitted, AO-fit	Five prompt or pair heads fitted on four folds of OOF values $\widetilde{B}_{i}$ each; held-out predictions are pooled and frozen.
Joint, AO-fit	A prompt or pair head fitted alongside the final policy using detached current-policy values $B_{i}$ .
Entropy-CF	Five prompt or pair heads regress annotation dispersion on four folds; pooled held-out predictions are transformed and frozen.
EMA-joint	A prompt or pair head uses current-policy values $B_{i}$ and moving-average centering during policy training.

Appendix

Table 8: Scale estimators and their supervision and fitting schedules.

	400 prompts			800 prompts
Method	Win (%)	$\Delta R$	$L/L_{D}$	Win (%)	$\Delta R$	$L/L_{D}$
Fixed-margin DPO	50.50 [45.63, 55.38]	+0.341 [-0.284, +0.988]	1.074	50.62 [47.19, 53.94]	+0.216 [-0.281, +0.716]	1.094
Pooled prompt scale: AO-fitted $q$
Pooled prompt AO (AO-fit)	57.00 [52.13, 61.75]	+1.173 [+0.536, +1.825]	1.072	55.06 [51.56, 58.44]	+0.666 [+0.159, +1.170]	1.082
Pooled prompt WR (AO-fit)	54.50 [49.63, 59.38]	+0.818 [+0.155, +1.462]	1.085	55.38 [51.88, 58.81]	+0.633 [+0.115, +1.150]	1.089
Pooled prompt scale: WR-fitted $q$

Appendix

Table 9: Surrogate scale-learning formulations on the HS3 1B 400- and 800-prompt evaluation panels. All policies use seed 42 and 150 updates. Each panel uses outputs from the same DPO reference checkpoint.

Method	Win (%)	$\Delta R$	$L/L_{D}$
400 prompts, seed 42
EMA-joint pair AO	52.75 [47.88, 57.63]	$+0.534$	—
Entropy-CF pair AO	51.88 [47.13, 56.75]	$+0.518$	—
Entropy-CF prompt AO	52.00 [47.12, 56.88]	$+0.402$	—
Entropy-CF prompt WR ∗	56.00 [51.13, 60.88]	+0.996 [+0.309, +1.683]	1.070
Entropy-CF pair WR	48.25 [43.38, 53.00]	$+0.186$	—

Appendix

Table 10: Earlier entropy-supervised and EMA-centered HS3 1B estimators, grouped by evaluation panel and policy seed. Every row uses same-seed DPO; the seed-17 reference is independently trained.

Llama-3.2-1B-Instruct
	400 prompts		800 prompts
Scale fitting	Win (%)	$L/L_{D}$	Win (%)	$L/L_{D}$
Centered, regularized	62.63 [57.88, 67.25]	1.007	61.50 [58.13, 64.81]	0.998
Uncentered, regularized	64.13 [59.50, 68.88]	1.024	62.81 [59.50, 66.13]	1.007
Uncentered, unregularized	68.75 [64.25, 73.25]	1.039	61.00 [57.63, 64.38]	1.024
Llama-3.1-8B-Instruct

Appendix

Table 11: Centering and regularization in ULNM-DPO-WR , matched to the corresponding main-text configurations. Win rates compare each policy against the same-size DPO reference.

	400 prompts			800 prompts
Prompt scale	Win (%)	$\Delta R$	$L/L_{D}$	Win (%)	$\Delta R$	$L/L_{D}$
Constant $q=1$	62.00 [57.25, 66.63]	+1.641 [+0.890, +2.386]	0.991	56.56 [53.13, 60.00]	+0.527 [-0.071, +1.139]	0.992
Learned $q$	68.00 [63.38, 72.38]	+2.372 [+1.661, +3.080]	0.985	65.31 [62.00, 68.56]	+1.796 [+1.241, +2.356]	0.964

Appendix

Table 12: Constant-scale control for ULNM-DPO-WR on HelpSteer3 with Llama-3.1-8B-Instruct on the 400- and 800-prompt panels.

Panel	Scale fitting	Win (%)	$\Delta R$	$L/L_{D}$
400	Centered, regularized	58.25 [53.50, 63.00]	+1.040	1.064
400	Uncentered, regularized	57.13 [52.37, 62.00]	+1.233	1.103
400	Uncentered, unregularized	54.75 [49.88, 59.63]	+0.787	1.107
800	Centered, regularized	56.94 [53.56, 60.31]	+0.929	1.085
800	Uncentered, regularized	56.56 [53.19, 59.94]	+0.598	1.117
800	Uncentered, unregularized	55.88 [52.50, 59.25]	+0.725	1.112

Appendix

Table 13: Centering and regularization in UNM-DPO -AO on HelpSteer3 with Llama-3.2-1B-Instruct, matched to the main-text configuration. All policies are evaluated against the same DPO reference.

Method	Win (%) $\uparrow$	$\Delta R$ $\uparrow$	Length ratio
ULNM-DPO-WR	70.13	+2.949	1.034
	[66.94, 73.31]	[2.388, 3.511]
DPO	59.44	+1.152	1.072
	[56.19, 62.88]	[0.630, 1.679]
SimPO	62.94	+2.377	1.042
	[59.62, 66.19]	[1.707, 3.034]

Appendix

Table 14: Direct comparisons against the untouched Llama-3.1-8B-Instruct initialization on 800 HelpSteer3 prompts. Brackets indicate conditional 95% confidence intervals. Length ratios are relative to the initial model.

Method	Win (%) $\uparrow$	$\Delta R$ (point) $\uparrow$	$L/L_{D}$
DPO (reference)	50.00	0.000	1.000
Fixed-margin DPO	52.23 [45.76, 58.71]	+0.825	1.023
ODPO	49.33 [42.63, 55.80]	+0.759	1.018
MMPO	51.56 [44.87, 58.04]	$-0.066$	0.965
$\beta$ -DPO	57.59 [51.34, 63.84]	+1.173	1.016
$\gamma$ -PO (DPO)	55.80 [49.33, 62.28]	+0.786	1.020

Appendix

Table 15: HelpSteer2 with Llama-3.2-1B-Instruct on the 224-prompt panel.

Policy	Learning rate	Updates	Win (%)	$\Delta R$	$L/L_{D}$
AO	$10^{-6}$	52	54.69 [48.21, 61.16]	+0.241	0.965
AO	$2\times 10^{-6}$	52	56.92 [50.45, 63.17]	+1.041	0.967
AO	$5\times 10^{-6}$	52	55.36 [48.66, 61.61]	+0.203	0.958
AO	$10^{-6}$	104	56.03 [49.55, 62.50]	+0.784	0.984
AO	$2\times 10^{-6}$	104	55.80 [49.55, 62.05]	+0.883	0.974
AO	$5\times 10^{-6}$	104	54.69 [48.21, 61.17]	+0.873	0.996

Appendix

Table 16: Sensitivity to learning rate and training duration for HelpSteer2 with Llama-3.2-1B-Instruct on the 224-prompt panel. All other settings are held fixed.

$\tau$	Win (%)	$\Delta R$	$L/L_{D}$
0.5	54.46 [48.21, 60.71]	+0.639	0.979
1.0	60.27 [54.01, 66.52]	+0.824	0.975
1.5	54.24 [47.54, 60.49]	+0.717	0.989

Appendix

Table 17: HelpSteer2 WR margin sensitivity on the same 224-prompt panel. Learning rate, update budget and fitted scale are fixed.

Method	LC win (%) $\uparrow$	Mean characters
DPO	16.39	2148
Fixed-margin DPO	15.72	2348
ODPO	17.29	2284
MMPO	13.71	2492
$\beta$ -DPO	0.49	8445
$\gamma$ -PO (DPO)	14.80	2198

Appendix

Table 18: AlpacaEval results for HelpSteer3-trained Llama-3.1-8B-Instruct policies, using GPT-4-Turbo reference answers and a GPT-4.1 judge. All methods use the same 805 instructions.

Explore similar work

Boosting Direct Preference Optimization with Penalization
Jun 10, 2026Pengwei SunBacktranslation Augmented Direct Preference Optimization Penalties
Stanford University.
$ξ$ -DPO: Direct Preference Optimization via Ratio Reward Margin
May 9, 2026Zhengyuan Fan, Zhonghua Wu, Yuxuan Du +1Backtranslation Augmented Direct Preference Optimization
School of Computer Science, Northwestern Polytechnical University
TUR-DPO: Topology- and Uncertainty-Aware Direct Preference Optimization
Apr 30, 2026Abdulhady Abas Abdullah, Fatemeh Daneshfar, Seyedali Mirjalili +1Reasoning Skills Topology
Artificial Intelligence and Innovation Centre, University of Kurdistan, Erbil, Iraq · Department of Computer Engineering, University of Kurdistan, Iran · Centre for Artificial Intelligence Research and Optimisation, Torrens University Australia, Brisbane, Australia +2

Jun 10, 2026, cs.LGJ/K move · Enter open · S save

Boosting Direct Preference Optimization with Penalization

Pengwei Sun

Stanford University.

Backtranslation Augmented Direct Preference Optimization Penalties

Open paper AskPDF arXiv

Offline preference optimization has become a practical substitute for reinforcement learning from human feedback, but pairwise objectives such as Direct Preference Optimization (DPO) and its variants use only the chosen and rejected responses stored in a static dataset. This leaves a useful signal unused: the response that the reference model itself would generate for the same prompt. We propose Direct Preference Optimization with Penalization (DPOP), a simple extension of DPO that augments the base preference loss with a gated penalty on reference-greedy responses. DPOP activates this penalty only when the current policy still assigns a lower likelihood to the preferred response than to the rejected response. On AlpacaEval 2.0, DPOP improves length-controlled win rate over DPO, SimPO, and AlphaDPO on both Llama-3-8b-it and Gemma-2-9b-it, achieving relative gains of 5.3% and 4.4% over baselines on the two models, respectively. Ablations further show that a SimNPO-style length-normalized penalty is stronger than NPO and token-level unlikelihood in this setting.

Abstract

Explore similar work

Boosting Direct Preference Optimization with Penalization

ξξξ-DPO: Direct Preference Optimization via Ratio Reward Margin

TUR-DPO: Topology- and Uncertainty-Aware Direct Preference Optimization

$ξ$ -DPO: Direct Preference Optimization via Ratio Reward Margin