Proximal Preference Optimization: A Unified Framework for Stable and Reward-Free Alignment

Qisai Liu; Zhanhong Jiang; Zahid Hasan; Soumik Sarkar

cs.LGNov 7, 2026

Proximal Preference Optimization: A Unified Framework for Stable and Reward-Free Alignment

Authors: Qisai Liu, Zhanhong Jiang, Zahid Hasan, Soumik Sarkar

Organizations: Iowa State University, Ames, USA

Abstract

We introduce a proximal preference optimization framework that unifies the implicit reward formulation of Direct Preference Optimization (DPO) with the trust-region stability of Proximal Policy Optimization (PPO). Our framework yields two complementary algorithms. P $^2$ O (Proximal Preference Optimization) introduces a clipped importance ratio directly into the DPO preference margin, producing a reward-model-free algorithm that supports safe dataset reuse, bounded policy divergence, and monotonic preference improvement guarantees. PKTO (Proximal Kahneman-Tversky Optimization) extends P $^2$ O by replacing the DPO margin with KTO's prospect-theoretic utility margins, adding a self-regulating adaptive baseline that tightens the effective trust region as the policy drifts, and an asymmetric weighting that handles imbalanced or asymmetric preference datasets. We provide a unified theoretical analysis, including Kullback Leibler (KL) trust-region bounds, a policy-improvement lower bound, and convergence guarantees under multi-epoch proximal updates. Empirically, P $^2$ O and PKTO match or exceed DPO and KTO in alignment quality while maintaining better stability under repeated dataset reuse across multiple LLMs on three preference benchmarks. The code base is available here.

Figures & tables

Property	PPO	DPO	IPO	KTO	P 2 O	PKTO
Reward	Explicit	Implicit	Implicit	Utility	Implicit	Utility+Implicit
Trust region	✓	✗	✗	✗	PPO Clip+KL	PPO Clip+KL
Adaptive baseline	✗	✗	Partial	✓	✗	✓
Asymmetric weights	✗	✗	✗	✓	✗	✓
Multi-epoch reuse	✗	Unstable	Limited	Limited	✓	✓
On-policy sampling	✓	✗	✗	✗	✗	✗

TABLE I : Comparison of preference-optimization methods.

\varepsilon>0

: clip radius,

\epsilon>0

: accuracy

Fig. 1 : Left (DPO): uses implicit reward log-ratios; no trust region leads to KL drift under multi-epoch training. Middle (PPO-RLHF): stable via KL constraints but requires a reward model and on-policy sampling. Right (P 2 O / PKTO): combines DPO/KTO implicit rewards with PPO-style clipping; stable, reward-free, multi-epoch safe.

Model	Method	Preference Accuracy
Model	Method	HH-RLHF	UltraFeedback	Orca
GPT-2 (125M)	DPO	0.58	0.50	0.80
	IPO	0.48	0.45	0.59
	KTO	0.59	0.48	0.69
	P 2 O (ours)	0.62	0.52	0.85
	PKTO (ours)	0.63	0.51	0.82

TABLE II : Preference accuracy across models and datasets. Bold indicates the best result, underline indicates second‑best.

Panel of Figure 4 — Fig. 2 : Ablation studies for P 2 O with components and $\varepsilon$ .

Method	Accuracy $\uparrow$	Reward Margin $\uparrow$	Token-mean KL $\downarrow$	Multi-epoch safe
DPO	0.560	0.329	1.125	✗
IPO	0.487	-0.071	0.865	✗
KTO	0.527	0.396	2.764	✗
P 2 O (ours)	0.547	0.459	0.871	✓
PKTO (ours)	0.593	0.343	0.409	✓

TABLE III : Stability metrics on HH-RLHF with GPT-2 (125M) at end of training. Token-mean KL measures policy drift from

\pi_{\mathrm{ref}}

; lower is more stable. Reward margin measures confidence of preference separation.

Panel of Figure 6 — Fig. 3 : Ablation studies for P 2 O with $K$ and $\beta$ .

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
Max length	128
Max prompt length	64
# of Training	400
# of Evaluation	100
Learning rate	2 $e^{-5}$
Weight decay	0.01

Appendix

TABLE IV : Hyperparameter Setting.

Fig. 4 : Ablation on

\lambda

for P 2 O.

LLM Alignment

Large language model alignment studies methods for making models follow human intentions, values, and safety constraints across contexts.

See all 628 papers

Towards Bridging the Gap Between Offline and Iterative Alignment via Preference Distillation
Sep 7, 2026Wenbo Zhang, Wenzhuo Zhou, Hengrui Cai +1LLM Alignment Preference Alignment
Department of Statistics University of California, Irvine · School of Business George Washington University
Conditional Equivalence of DPO and RLHF: Implicit Assumption, Failure Modes, and Provable Alignment
May 20, 2026Zhiqin Yang, Yonggang Zhang, Wei Xue +3LLM Alignment Direct Preference Optimization
1The Hong Kong University of Science and Technology · 2LIGHTSPEED · 3Hong Kong Baptist University.
CompassDPO: Dynamics-Controlled Direct Preference Optimization for Robust Safety Alignment
Mar 7, 2026Jilong Liu, Yonghui Yang, Pengyang Shao +5Direct Preference Optimization LLM Safety Alignment
Hefei University of Technology · National University of Singapore

Sep 7, 2026J/K move · Enter open · S save

Towards Bridging the Gap Between Offline and Iterative Alignment via Preference Distillation

Wenbo Zhang, Wenzhuo Zhou, Hengrui Cai, Zhengling Qi

Department of Statistics University of California, Irvine · School of Business George Washington University

LLM Alignment Preference Alignment

Open paper AskPDF arXiv

Direct preference optimization DPO is a promising offline approach for aligning large language models (LLMs) due to its simplicity, computational efficiency, and implicit modeling of human preferences. Interestingly, iterative extensions of DPO have achieved stronger performance on academic benchmarks, raising two key questions: (i) Why do iterative methods generally outperform offline ones? (ii) Can their advantages be incorporated into offline alignment? To answer the first question, our controlled experiments reveal that the explicit preference model, additionally introduced in the iterative procedure, is a key factor behind its superiority over offline methods. This insight leads us to answer the second question affirmatively and propose Distilled Preference Probability Policy Optimization (DP3O), an effective and efficient offline alignment algorithm. DP3O first learns an explicit preference model using a helper class of LLMs and then distills its knowledge into policy optimization. Theoretically, we show that explicit preference modeling admits better estimation error control than implicit formulations, and that DP3O achieves a tighter generalization bound than hard-label DPO through variance reduction. Empirically, we evaluate DP3O on a wide range of chat-based and downstream tasks and show that it outperforms state-of-the-art offline methods, achieves performance comparable to iterative DPO, and reduces training time by about

42\%

, demonstrating both its effectiveness and efficiency.

Abstract

LLM Alignment

Explore similar work

Towards Bridging the Gap Between Offline and Iterative Alignment via Preference Distillation

Conditional Equivalence of DPO and RLHF: Implicit Assumption, Failure Modes, and Provable Alignment

CompassDPO: Dynamics-Controlled Direct Preference Optimization for Robust Safety Alignment