We introduce a proximal preference optimization framework that unifies the implicit reward formulation of Direct Preference Optimization (DPO) with the trust-region stability of Proximal Policy Optimization (PPO). Our framework yields two complementary algorithms. P2O (Proximal Preference Optimization) introduces a clipped importance ratio directly into the DPO preference margin, producing a reward-model-free algorithm that supports safe dataset reuse, bounded policy divergence, and monotonic preference improvement guarantees. PKTO (Proximal Kahneman-Tversky Optimization) extends P2O by replacing the DPO margin with KTO's prospect-theoretic utility margins, adding a self-regulating adaptive baseline that tightens the effective trust region as the policy drifts, and an asymmetric weighting that handles imbalanced or asymmetric preference datasets. We provide a unified theoretical analysis, including Kullback Leibler (KL) trust-region bounds, a policy-improvement lower bound, and convergence guarantees under multi-epoch proximal updates. Empirically, P2O and PKTO match or exceed DPO and KTO in alignment quality while maintaining better stability under repeated dataset reuse across multiple LLMs on three preference benchmarks. The code base is available here.
Figures & tables
Property
PPO
DPO
IPO
KTO
P 2 O
PKTO
Reward
Explicit
Implicit
Implicit
Utility
Implicit
Utility+Implicit
Trust region
✓
✗
✗
✗
PPO Clip+KL
PPO Clip+KL
Adaptive baseline
✗
✗
Partial
✓
✗
✓
Asymmetric weights
✗
✗
✗
✓
✗
✓
Multi-epoch reuse
✗
Unstable
Limited
Limited
✓
✓
On-policy sampling
✓
✗
✗
✗
✗
✗
TABLE I : Comparison of preference-optimization methods. ε>0 : clip radius, ϵ>0 : accuracy
Fig. 1 : Left (DPO): uses implicit reward log-ratios; no trust region leads to KL drift under multi-epoch training. Middle (PPO-RLHF): stable via KL constraints but requires a reward model and on-policy sampling. Right (P 2 O / PKTO): combines DPO/KTO implicit rewards with PPO-style clipping; stable, reward-free, multi-epoch safe.
Model
Method
Preference Accuracy
HH-RLHF
UltraFeedback
Orca
GPT-2 (125M)
DPO
0.58
0.50
0.80
IPO
0.48
0.45
0.59
KTO
0.59
0.48
0.69
P 2 O (ours)
0.62
0.52
0.85
PKTO (ours)
0.63
0.51
0.82
TABLE II : Preference accuracy across models and datasets. Bold indicates the best result, underline indicates second‑best.
Fig. 2 : Ablation studies for P 2 O with components and ε .
Method
Accuracy ↑
Reward Margin ↑
Token-mean KL ↓
Multi-epoch safe
DPO
0.560
0.329
1.125
✗
IPO
0.487
-0.071
0.865
✗
KTO
0.527
0.396
2.764
✗
P 2 O (ours)
0.547
0.459
0.871
✓
PKTO (ours)
0.593
0.343
0.409
✓
TABLE III : Stability metrics on HH-RLHF with GPT-2 (125M) at end of training. Token-mean KL measures policy drift from πref ; lower is more stable. Reward margin measures confidence of preference separation.
Fig. 3 : Ablation studies for P 2 O with K and β .
Direct preference optimization DPO is a promising offline approach for aligning large language models (LLMs) due to its simplicity, computational efficiency, and implicit modeling of human preferences. Interestingly, iterative extensions of DPO have achieved stronger performance on academic benchmarks, raising two key questions: (i) Why do iterative methods generally outperform offline ones? (ii) Can their advantages be incorporated into offline alignment? To answer the first question, our controlled experiments reveal that the explicit preference model, additionally introduced in the iterative procedure, is a key factor behind its superiority over offline methods. This insight leads us to answer the second question affirmatively and propose Distilled Preference Probability Policy Optimization (DP3O), an effective and efficient offline alignment algorithm. DP3O first learns an explicit preference model using a helper class of LLMs and then distills its knowledge into policy optimization. Theoretically, we show that explicit preference modeling admits better estimation error control than implicit formulations, and that DP3O achieves a tighter generalization bound than hard-label DPO through variance reduction. Empirically, we evaluate DP3O on a wide range of chat-based and downstream tasks and show that it outperforms state-of-the-art offline methods, achieves performance comparable to iterative DPO, and reduces training time by about 42%, demonstrating both its effectiveness and efficiency.
Wenbo Zhang, Wenzhuo Zhou, Hengrui Cai +1
Department of Statistics University of California, Irvine · School of Business George Washington University
Direct Preference Optimization (DPO) has emerged as a popular alternative to Reinforcement Learning from Human Feedback (RLHF), offering theoretical equivalence with simpler implementation. We prove this equivalence is conditional rather than universal, depending on an implicit assumption frequently violated in practice: the RLHF-optimal policy must prefer human-preferred responses. When this assumption fails, DPO optimizes relative advantage over the reference policy rather than absolute alignment with human preferences, leading to pathological convergence where policies decrease DPO loss while preferring dispreferred responses. We characterize when this assumption is violated, show the existence of an undesirable solution space, and prove that DPO and RLHF optimize fundamentally different objectives in such cases. To address this, we introduce Constrained Preference Optimization (CPO), augmenting RLHF with constraints for provable alignment. We further provide a geometric interpretation through soft margin ranking, revealing that DPO implements margin ranking with potentially negative targets. Our theoretical analysis establishes when DPOs' guarantees hold and provides solutions preserving simplicity with provable alignment. Comprehensive experiments on standard benchmarks demonstrate that CPO achieves state-of-the-art performance. Code is available at: https://github.com/visitworld123/CPO.
Zhiqin Yang, Yonggang Zhang, Wei Xue +3
1The Hong Kong University of Science and Technology · 2LIGHTSPEED · 3Hong Kong Baptist University.
Direct Preference Optimization (DPO) has become a standard framework for safety alignment, but its reliance on pairwise preference updates makes training sensitive to imperfect supervision. Existing robust DPO methods often address this sensitivity through global loss corrections or external data-level interventions, while largely overlooking how unreliable comparisons distort batch-level optimization dynamics. We propose CompassDPO, a reward-free DPO framework that stabilizes preference optimization through dynamics control. Using the implicit DPO reward margin as a training-time compass, CompassDPO regulates sample influence along two complementary axes: update direction and update magnitude. For directional control, it applies sparse, budgeted, and warm-up delayed loss mixing to attenuate update components that conflict with the emerging preference direction. For magnitude control, it adaptively soft-winsorizes high-loss tail contributions, reducing tail dominance while preserving useful gradients from hard examples. Both mechanisms use only signals available during standard DPO training and require no external reward model or additional supervision. Experiments on PKU-SafeRLHF across four backbones and multiple out-of-distribution safety benchmarks show that CompassDPO consistently improves robustness over vanilla DPO and strong DPO-family baselines, especially under controlled label-flip noise. Code is available at https://anonymous.4open.science/r/CompassDPO-4D00
Jilong Liu, Yonghui Yang, Pengyang Shao +5
Hefei University of Technology · National University of Singapore