Uncertainty-Normalized Margins for Direct Preference Optimization

Sadegh Khorasani; Petrus Mikkola; Matthias Grossglauser

cs.LGSep 29, 2026

Uncertainty-Normalized Margins for Direct Preference Optimization

Authors: Sadegh Khorasani, Petrus Mikkola, Matthias Grossglauser

Organizations: School of Computer and Communication Sciences, EPFL, Lausanne, Switzerland · University of Helsinki, Helsinki, Finland

Abstract

Direct preference optimization (DPO) models binary preferences through a Bradley-Terry model with a common noise scale, without explicitly accounting for preference strength or prompt-dependent uncertainty from human feedback. We introduce uncertainty-normalized margin DPO (UNM-DPO), which combines strength-dependent margins with a learned prompt scale. Motivated by a heteroskedastic Bradley-Terry model, we develop two training objectives. Both compare the implicit rewards of preferred and rejected responses, derived from response log-probability ratios to a reference policy. Advantage-only (AO) divides this reward difference by the prompt scale before subtracting the margin; whole-residual (WR) subtracts the margin before dividing by the scale. For the WR comparison model, we establish a necessary and sufficient condition under which known margins make the prompt scale identifiable. We introduce a practical procedure for learning the scale. Building on WR, we introduce ULNM-DPO-WR, which normalizes each response's implicit reward by its length. We evaluate our methods against DPO and related baselines on HelpSteer2 and HelpSteer3, using the Skywork reward model as a judge. With Llama-3.1-8B-Instruct, ULNM-DPO-WR achieves tie-adjusted win rates against matched DPO of 68.00% and 65.31% on evaluation panels, with higher mean rewards and shorter responses on average. On AlpacaEval with a GPT-4.1 judge and GPT-4-Turbo reference answers, the same 8B policy achieves a length-controlled win rate of 21.62%, compared with 16.39% for DPO and 15.30% for SimPO. These results demonstrate the potential of combining preference-strength margins, learned prompt scales, and length normalization for policy optimization.

Figures & tables

	400 prompts			800 prompts
Method	Win (%) $\uparrow$	$\Delta R\;\uparrow$	$L/L_{D}$	Win (%) $\uparrow$	$\Delta R\;\uparrow$	$L/L_{D}$
DPO (ref.)	50.00	0.000	1.000	50.00	0.000	1.000
Fixed-margin DPO	50.50 [45.63, 55.38]	+0.341 [-0.284, +0.988]	1.074	50.62 [47.19, 53.94]	+0.216 [-0.281, +0.716]	1.094
ODPO	53.88 [49.00, 58.63]	+0.529 [-0.092, +1.140]	1.056	51.38 [47.94, 54.81]	+0.272 [-0.219, +0.750]	1.069
MMPO	44.50 [39.63, 49.25]	-0.271 [-0.819, +0.297]	0.985	49.13 [45.75, 52.44]	-0.289 [-0.703, +0.119]	0.981
$\beta$ -DPO	25.13 [21.00, 29.50]	-9.018 [-10.216, -7.825]	1.056	24.50 [21.56, 27.56]	-8.606 [-9.410, -7.777]	1.050

Table 1: HelpSteer3 with Llama-3.2-1B-Instruct on the 400- and 800-prompt evaluation panels.

	400 prompts			800 prompts
Method	Win (%) $\uparrow$	$\Delta R\;\uparrow$	$L/L_{D}$	Win (%) $\uparrow$	$\Delta R\;\uparrow$	$L/L_{D}$
DPO (ref.)	50.00	0.000	1.000	50.00	0.000	1.000
Fixed-margin DPO	58.13 [53.25, 62.88]	+1.120 [+0.453, +1.799]	1.024	54.75 [51.38, 58.19]	+0.322 [-0.174, +0.816]	1.004
ODPO	55.50 [50.63, 60.25]	+1.042 [+0.412, +1.692]	1.012	52.81 [49.38, 56.19]	+0.073 [-0.407, +0.546]	1.016
MMPO	48.88 [44.13, 53.63]	-0.287 [-0.876, +0.306]	1.036	45.75 [42.44, 49.19]	-0.691 [-1.111, -0.266]	1.023
$\beta$ -DPO	8.63 [6.00, 11.38]	-25.450 [-26.980, -23.917]	2.266	8.69 [6.81, 10.69]	-25.533 [-26.604, -24.434]	2.246

Table 2: HelpSteer3 with Llama-3.1-8B-Instruct on the 400- and 800-prompt evaluation panels.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Evidence bin	Mean evidence	Held-out votes	Prompts	Agreement (95% CI)
Low $[1,1.5]$	$1.229$	$3{,}236$	$1{,}390$	$96.35\%\;[95.65,96.95]$
Middle $(1.5,2.5)$	$2.000$	$1{,}864$	$1{,}070$	$100.00\%\;[99.79,100.00]$
High $[2.5,3]$	$2.804$	$2{,}065$	$894$	$100.00\%\;[99.81,100.00]$

Appendix

Table 3: Held-out-annotator directional agreement on the HelpSteer3 development partition. The last column reports descriptive Wilson intervals; the high-minus-low contrast in the text uses prompt-clustered inference.

Model	Frozen revision
Llama-3.2-1B-Instruct	9213176726f574b556790deb65791e0c5aa438b6
Llama-3.1-8B-Instruct	0e9e39f249a16976918f6564b8830bc894c89659

Appendix

Table 4: Frozen initial-model revisions.

Method	Implemented per-pair loss
DPO	$\zeta(-\beta_{0}A_{i})$ , with $\beta_{0}=0.05$ .
ODPO	$\zeta(0.75k_{i}-\beta_{0}A_{i})$ .
MMPO	$\operatorname{CE}(\sigma(2.2k_{i}),\beta_{0}A_{i})$ .
SimPO	$\zeta(1.375-2.5\bar{P}_{i})$ .
SPO-basic	$\zeta(-0.01P_{i})/0.01$ .
$\beta$ -DPO	$\zeta(-\beta_{t}A_{i})$ on the selected batch subset.

Appendix

Table 5: Implemented baseline losses for one preference pair.

Method and parameter	Candidate values	Selected / fixed
Earlier AO: margin $\tau$	$\{1/3,\,0.5,\,0.75,\,1,\,1.5\}$	$1$
ODPO: strength multiplier	$\{1/3,\,0.5,\,0.75,\,1\}$	$0.75$
MMPO: target slope $\gamma$	$\{0.3,\,0.7,\,1.1,\,2.2\}$	$2.2$
$\beta$ -DPO: adaptation coefficient $\alpha$	$\{0.2,\,0.6,\,1\}$	$0.2$
$\gamma$ -PO (DPO): base margin $\gamma_{0}$	$\{0.02,\,0.05,\,0.20,\,0.40\}$	$0.02$
SimPO	$\beta\in\{2.5,10\}$ , $\gamma/\beta\in\{0.3,0.55\}$ , LR $\in\{1,2\}\times 10^{-6}$	$2.5$ , $0.55$ , $10^{-6}$

Appendix

Table 6: Hyperparameter search grids and selected values from the earlier 500-prompt tuning stage.

Setting	Median tokens	$\beta_{\mathrm{LN}}$
HS3 1B	364.5	18.225
HS2 1B	272	13.6
HS3 8B	364.5	18.225

Appendix

Table 7: Training-length calibration of the normalized score coefficient.

Scale estimator	Supervision and fitting schedule
Pooled OOF, WR-fit	One prompt head fitted to all pooled OOF values $\widetilde{B}_{i}$ using the WR loss, then frozen; this is the main formulation.
Pooled OOF, AO-fit	One prompt head fitted to all pooled OOF values $\widetilde{B}_{i}$ using the AO loss, then frozen.
Cross-fitted, AO-fit	Five prompt or pair heads fitted on four folds of OOF values $\widetilde{B}_{i}$ each; held-out predictions are pooled and frozen.
Joint, AO-fit	A prompt or pair head fitted alongside the final policy using detached current-policy values $B_{i}$ .
Entropy-CF	Five prompt or pair heads regress annotation dispersion on four folds; pooled held-out predictions are transformed and frozen.
EMA-joint	A prompt or pair head uses current-policy values $B_{i}$ and moving-average centering during policy training.

Appendix

Table 8: Scale estimators and their supervision and fitting schedules.

	400 prompts			800 prompts
Method	Win (%)	$\Delta R$	$L/L_{D}$	Win (%)	$\Delta R$	$L/L_{D}$
Fixed-margin DPO	50.50 [45.63, 55.38]	+0.341 [-0.284, +0.988]	1.074	50.62 [47.19, 53.94]	+0.216 [-0.281, +0.716]	1.094
Pooled prompt scale: AO-fitted $q$
Pooled prompt AO (AO-fit)	57.00 [52.13, 61.75]	+1.173 [+0.536, +1.825]	1.072	55.06 [51.56, 58.44]	+0.666 [+0.159, +1.170]	1.082
Pooled prompt WR (AO-fit)	54.50 [49.63, 59.38]	+0.818 [+0.155, +1.462]	1.085	55.38 [51.88, 58.81]	+0.633 [+0.115, +1.150]	1.089
Pooled prompt scale: WR-fitted $q$

Appendix

Table 9: Surrogate scale-learning formulations on the HS3 1B 400- and 800-prompt evaluation panels. All policies use seed 42 and 150 updates. Each panel uses outputs from the same DPO reference checkpoint.

Method	Win (%)	$\Delta R$	$L/L_{D}$
400 prompts, seed 42
EMA-joint pair AO	52.75 [47.88, 57.63]	$+0.534$	—
Entropy-CF pair AO	51.88 [47.13, 56.75]	$+0.518$	—
Entropy-CF prompt AO	52.00 [47.12, 56.88]	$+0.402$	—
Entropy-CF prompt WR ∗	56.00 [51.13, 60.88]	+0.996 [+0.309, +1.683]	1.070
Entropy-CF pair WR	48.25 [43.38, 53.00]	$+0.186$	—

Appendix

Table 10: Earlier entropy-supervised and EMA-centered HS3 1B estimators, grouped by evaluation panel and policy seed. Every row uses same-seed DPO; the seed-17 reference is independently trained.

Llama-3.2-1B-Instruct
	400 prompts		800 prompts
Scale fitting	Win (%)	$L/L_{D}$	Win (%)	$L/L_{D}$
Centered, regularized	62.63 [57.88, 67.25]	1.007	61.50 [58.13, 64.81]	0.998
Uncentered, regularized	64.13 [59.50, 68.88]	1.024	62.81 [59.50, 66.13]	1.007
Uncentered, unregularized	68.75 [64.25, 73.25]	1.039	61.00 [57.63, 64.38]	1.024
Llama-3.1-8B-Instruct

Appendix

Table 11: Centering and regularization in ULNM-DPO-WR , matched to the corresponding main-text configurations. Win rates compare each policy against the same-size DPO reference.

	400 prompts			800 prompts
Prompt scale	Win (%)	$\Delta R$	$L/L_{D}$	Win (%)	$\Delta R$	$L/L_{D}$
Constant $q=1$	62.00 [57.25, 66.63]	+1.641 [+0.890, +2.386]	0.991	56.56 [53.13, 60.00]	+0.527 [-0.071, +1.139]	0.992
Learned $q$	68.00 [63.38, 72.38]	+2.372 [+1.661, +3.080]	0.985	65.31 [62.00, 68.56]	+1.796 [+1.241, +2.356]	0.964

Appendix

Table 12: Constant-scale control for ULNM-DPO-WR on HelpSteer3 with Llama-3.1-8B-Instruct on the 400- and 800-prompt panels.

Panel	Scale fitting	Win (%)	$\Delta R$	$L/L_{D}$
400	Centered, regularized	58.25 [53.50, 63.00]	+1.040	1.064
400	Uncentered, regularized	57.13 [52.37, 62.00]	+1.233	1.103
400	Uncentered, unregularized	54.75 [49.88, 59.63]	+0.787	1.107
800	Centered, regularized	56.94 [53.56, 60.31]	+0.929	1.085
800	Uncentered, regularized	56.56 [53.19, 59.94]	+0.598	1.117
800	Uncentered, unregularized	55.88 [52.50, 59.25]	+0.725	1.112

Appendix

Table 13: Centering and regularization in UNM-DPO -AO on HelpSteer3 with Llama-3.2-1B-Instruct, matched to the main-text configuration. All policies are evaluated against the same DPO reference.

Method	Win (%) $\uparrow$	$\Delta R$ $\uparrow$	Length ratio
ULNM-DPO-WR	70.13	+2.949	1.034
	[66.94, 73.31]	[2.388, 3.511]
DPO	59.44	+1.152	1.072
	[56.19, 62.88]	[0.630, 1.679]
SimPO	62.94	+2.377	1.042
	[59.62, 66.19]	[1.707, 3.034]

Appendix

Table 14: Direct comparisons against the untouched Llama-3.1-8B-Instruct initialization on 800 HelpSteer3 prompts. Brackets indicate conditional 95% confidence intervals. Length ratios are relative to the initial model.

Method	Win (%) $\uparrow$	$\Delta R$ (point) $\uparrow$	$L/L_{D}$
DPO (reference)	50.00	0.000	1.000
Fixed-margin DPO	52.23 [45.76, 58.71]	+0.825	1.023
ODPO	49.33 [42.63, 55.80]	+0.759	1.018
MMPO	51.56 [44.87, 58.04]	$-0.066$	0.965
$\beta$ -DPO	57.59 [51.34, 63.84]	+1.173	1.016
$\gamma$ -PO (DPO)	55.80 [49.33, 62.28]	+0.786	1.020

Appendix

Table 15: HelpSteer2 with Llama-3.2-1B-Instruct on the 224-prompt panel.

Policy	Learning rate	Updates	Win (%)	$\Delta R$	$L/L_{D}$
AO	$10^{-6}$	52	54.69 [48.21, 61.16]	+0.241	0.965
AO	$2\times 10^{-6}$	52	56.92 [50.45, 63.17]	+1.041	0.967
AO	$5\times 10^{-6}$	52	55.36 [48.66, 61.61]	+0.203	0.958
AO	$10^{-6}$	104	56.03 [49.55, 62.50]	+0.784	0.984
AO	$2\times 10^{-6}$	104	55.80 [49.55, 62.05]	+0.883	0.974
AO	$5\times 10^{-6}$	104	54.69 [48.21, 61.17]	+0.873	0.996

Appendix

Table 16: Sensitivity to learning rate and training duration for HelpSteer2 with Llama-3.2-1B-Instruct on the 224-prompt panel. All other settings are held fixed.

$\tau$	Win (%)	$\Delta R$	$L/L_{D}$
0.5	54.46 [48.21, 60.71]	+0.639	0.979
1.0	60.27 [54.01, 66.52]	+0.824	0.975
1.5	54.24 [47.54, 60.49]	+0.717	0.989

Appendix

Table 17: HelpSteer2 WR margin sensitivity on the same 224-prompt panel. Learning rate, update budget and fitted scale are fixed.

Method	LC win (%) $\uparrow$	Mean characters
DPO	16.39	2148
Fixed-margin DPO	15.72	2348
ODPO	17.29	2284
MMPO	13.71	2492
$\beta$ -DPO	0.49	8445
$\gamma$ -PO (DPO)	14.80	2198

Appendix

Table 18: AlpacaEval results for HelpSteer3-trained Llama-3.1-8B-Instruct policies, using GPT-4-Turbo reference answers and a GPT-4.1 judge. All methods use the same 805 instructions.

Explore similar work

Jun 10, 2026cs.LG

Boosting Direct Preference Optimization with Penalization

Offline preference optimization has become a practical substitute for reinforcement learning from human feedback, but pairwise objectives such as Direct Preference Optimization (DPO) and its variants use only the chosen and rejected responses stored in a static dataset. This leaves a useful signal unused: the response that the reference model itself would generate for the same prompt. We propose Direct Preference Optimization with Penalization (DPOP), a simple extension of DPO that augments the base preference loss with a gated penalty on reference-greedy responses. DPOP activates this penalty only when the current policy still assigns a lower likelihood to the preferred response than to the rejected response. On AlpacaEval 2.0, DPOP improves length-controlled win rate over DPO, SimPO, and AlphaDPO on both Llama-3-8b-it and Gemma-2-9b-it, achieving relative gains of 5.3% and 4.4% over baselines on the two models, respectively. Ablations further show that a SimNPO-style length-normalized penalty is stronger than NPO and token-level unlikelihood in this setting.

Pengwei Sun

Stanford University.

Backtranslation Augmented Direct Preference Optimization Penalties

Ask PDF

May 9, 2026cs.LG

$ξ$ -DPO: Direct Preference Optimization via Ratio Reward Margin

Reference-free preference optimization has emerged as an efficient alternative to reinforcement learning from human feedback, with Simple Preference Optimization(SimPO) demonstrating strong performance by eliminating the explicit reference model through a simple objective. However, the joint tuning of the hyperparameters

β

and

γ

in SimPO remains a central challenge. We argue that this difficulty arises because the margin formulation in SimPO is not easily interpretable across datasets with different reward gap structures. To better understand this issue, we conduct a comprehensive analysis of SimPO and find that

β

implicitly controls sample filtering, while the effect of

γ

depends on the reward gap structure of the dataset. Motivated by these observations, we propose

ξ

-DPO: Direct preference optimization via ratio reward margin. We first reformulate the preference objective through an equivalent transformation, changing the optimization target from maximizing the likelihood of reward gaps to minimizing the distance between reward gaps and optimal margins. Then, we redefine the reward in a ratio form between the chosen and rejected, which effectively cancels the effect of

β

and yields a bounded and interpretable margin. This margin is called the ratio reward margin and is denoted by

ξ

. Unlike the margin

γ

in SimPO,

ξ

explicitly represents the desired relative separation between chosen and rejected responses and can be determined from the initial reward gap distribution, avoiding repeated trial-and-error tuning. ....

Zhengyuan Fan, Zhonghua Wu, Yuxuan Du +1

School of Computer Science, Northwestern Polytechnical University

Backtranslation Augmented Direct Preference Optimization

Ask PDF

Apr 30, 2026cs.AI

TUR-DPO: Topology- and Uncertainty-Aware Direct Preference Optimization

Aligning large language models (LLMs) with human preferences is commonly done via reinforcement learning from human feedback (RLHF) with Proximal Policy Optimization (PPO) or, more simply, via Direct Preference Optimization (DPO). While DPO is stable and RL-free, it treats preferences as flat winner vs. loser signals and is sensitive to noisy or brittle preferences arising from fragile chains of thought. We propose TUR-DPO, a topology- and uncertainty-aware variant of DPO that rewards how answers are derived, not only what they say, by eliciting lightweight reasoning topologies and combining semantic faithfulness, utility, and topology quality into a calibrated uncertainty signal. A small learnable reward is factorized over these signals and incorporated into an uncertainty-weighted DPO objective that remains RL-free and relies only on a fixed or moving reference policy. Empirically, across open 7-8B models and benchmarks spanning mathematical reasoning, factual question answering, summarization, and helpful/harmless dialogue, TUR-DPO improves judge win-rates, faithfulness, and calibration relative to DPO while preserving training simplicity and avoiding online rollouts. We further observe consistent gains in multimodal and long-context settings, and show that TUR-DPO matches or exceeds PPO on reasoning-centric tasks while maintaining operational simplicity.

Abdulhady Abas Abdullah, Fatemeh Daneshfar, Seyedali Mirjalili +1

Artificial Intelligence and Innovation Centre, University of Kurdistan, Erbil, Iraq · Department of Computer Engineering, University of Kurdistan, Iran · Centre for Artificial Intelligence Research and Optimisation, Torrens University Australia, Brisbane, Australia +2

Reasoning Skills Topology

Ask PDF

Abstract

Explore similar work

Boosting Direct Preference Optimization with Penalization

ξξξ-DPO: Direct Preference Optimization via Ratio Reward Margin

TUR-DPO: Topology- and Uncertainty-Aware Direct Preference Optimization

$ξ$ -DPO: Direct Preference Optimization via Ratio Reward Margin