When Scores Conflict with Preferences: Pre-Alignment Diagnosis and Adaptive Correction of Constraint Drift in DPO
Authors: Ping Liu, Yan Yan
Organizations: Department of Computer Science and Engineering, University of Nevada, Reno · School of Electrical Engineering and Computer Science, Washington State University
Direct Preference Optimization (DPO) is increasingly used in settings where pairwise preferences coexist with pointwise criteria such as response length, answer correctness, or safety scores. When the two disagree, standard DPO treats pairs whose preferred output scores worse on the pointwise criterion (conflict pairs) in the same way as the remaining (matching) pairs. We characterize a systematic failure mode, constraint drift: for sufficiently small updates, conflict pairs tend to increase constraint cost while matching pairs tend to repair it, and the net change depends on their balance. This balance yields a closed-form pre-alignment diagnostic, ρeq∗, computable from dataset statistics, that indicates whether a dataset is self-correcting or requires intervention. Building on this diagnostic, we propose Conflict-aware DPO (CoDPO), which down-weights conflict pairs with a strength set from ρeq∗ rather than tuned, and adds an adaptive margin shift that addresses residual drift during training. Empirically, drift grows monotonically with conflict exposure on the conflict-dominated domains, and the intervention decision given by ρeq∗ matches the observed drift on every domain, including one that it identifies as self-correcting. On all four domains of the main comparison, CoDPO reduces drift while largely preserving the preference learning of vanilla DPO. Compared with hard filtering and recent constrained and multi-objective baselines, CoDPO also achieves a more favorable drift-preference trade-off.
Figures & tables
Figure 1: Overview of CoDPO. (a) Under DPO, conflict pairs push the constraint cost up and matching pairs pull it down; ρeq∗ , computed before training, compares the two. If the push dominates ( ρeq∗<1 ), CoDPO intervenes; otherwise the data are self-correcting and plain DPO is used. (b) Conflict pairs are down-weighted by severity, with τ∗ set from ρeq∗ . (c) During training, the DPO margin is shifted, more strongly when the constraint-risk estimate rises. Curves in (b) and (c) are schematic.
Domain
Pairwise preference
Pointwise constraint
Pairs
pc
ρeq∗
Zero-cross.
HelpSteer2
Helpfulness rating
Response length
7,220
57%
0.71
∼ 0.76
GSM8K
Explanation detail
Answer correctness
14,284
71%
0.40
∼ 0.27
ARC
Reward-model score
Answer correctness
12,580
58%
0.72
∼ 0.61
Math-Calc
Calculation density
Answer correctness
10,387
56%
0.80
∼ 0.50
TL;DR
Human preference
Summary length
92,534
58%
0.59
∼ 0.43
PKU-SafeRLHF
Helpfulness
Safety score
73,907
2.5%
1.0
—
Table 1: Experimental domains. pc : conflict rate. ρeq∗ : pre-alignment diagnostic (clipped to [0,1] ). Zero-cross.: estimated value of ρ at which drift changes sign in Table 2 .
HelpSteer2
GSM8K
ARC
Math-Calc
TL;DR
PKU-SafeRLHF
ρ
E[c]
Drift
P(ch)
E[c]
Drift
P(ch)
E[c]
Drift
P(ch)
E[c]
Drift
P(ch)
E[c]
Drift
P(ch)
E[c]
Drift
P(ch)
Base
1227
—
.454
.363
—
.351
.525
—
.439
.363
—
.420
126.5
—
.511
.4956
—
.396
0.0
1182
− 46
.489
.225
− .138
.358
.409
− .116
.473
.227
− .136
.516
121.4
− 5.1
.479
.4915
− .004
.418
0.3
1197
− 30
.532
.379
+.016
.423
.506
− .019
.514
.337
− .026
.597
125.9
− 0.6
.556
.4922
− .003
.418
0.5
1212
− 15
.550
.392
+.029
.434
.519
− .006
.523
.363
.000
.623
126.8
+0.3
.566
.4923
− .003
.418
0.7
1223
− 5
.560
.404
+.041
.443
.530
+.005
.528
.375
+.012
.631
127.2
+0.7
.568
.4926
− .003
.417
Table 2: Drift curves across six domains. Each row corresponds to a different average conflict weight ρ ; Base denotes the reference model before alignment. The constraint cost is response length (characters) on HelpSteer2 and TL;DR, binary incorrectness probability on GSM8K, ARC, and Math-Calc, and based on the safety score on PKU-SafeRLHF (Appendix A).
HelpSteer2
GSM8K
ARC
Math-Calc
Method
Drift
P(ch)
Drift
P(ch)
Drift
P(ch)
Drift
P(ch)
Base
—
.454
—
.351
—
.439
—
.420
Vanilla
+ 59
.566
+ .129
.453
+ .055
.532
+ .071
.652
SafeDPO
− 132
.437
− .353
.284
− .493
.412
− .352
.442
PD-DPO
− 128
.452
− .360
.289
− .509
.427
− .357
.449
C2-DPO
≈ 0
.457
+ .004
.353
+ .003
.442
≈ 0
.423
Table 3: Comparison with prior work across four domains. Drift =E[c(θ)]−E[c(θref)] . P(ch) =Pθ(yw≻yl∣x) . MODPO ( w=1.0 ) and Soups ( α=0.75 ) use the setting with the smallest ∣ drift ∣ . Full hyperparameter sweeps are in Appendix D. Implementation details are given in Appendix A.
Domains (Qwen2.5-7B)
Base models (HelpSteer2)
Losses (HelpSteer2)
HelpSteer2
GSM8K
ARC
Math-Calc
DeepSeek-8B
Mistral-7B
Qwen3-8B
IPO
SimPO
KTO
Vanilla
+ 59
+ .129
+ .055
+ .071
+ 18
+ 92
+ 34
+ 15
+ 74
+ 72
Coarse
+ 20
+ .105
+ .049
+ .068
− 7
+ 45
+ 0.4
− 36
− 1
+ 16
Combined
+ 18.7
+ .082
+ .046
+ .066
− 7
+ 41
+ 0.4
—
—
—
Table 4: Drift under vanilla training, coarse control only, and full CoDPO, across domains (Qwen2.5-7B), base models (HelpSteer2), and preference losses (HelpSteer2; IPO with β=0.5 , SimPO with β=2 , γ=0.5 ; coarse control at ρeq∗ ). Coarse control reduces drift in every setting.
Vanilla DPO
SafeDPO
PD-DPO
C2-DPO
MODPO
Soups
CoDPO wins
14.1%
86.3%
73.9%
12.7%
52.2%
13.5%
Opp. wins
10.5%
0.0%
3.0%
10.7%
7.7%
10.0%
Tie
75.5%
13.6%
23.2%
76.6%
40.1%
76.5%
Table 5: LLM-as-judge evaluation (JudgeLM-13B) on HelpSteer2: CoDPO compared pairwise against each method.