As the era of large language models (LLMs) unfolds, Preference Optimization (PO) methods have become a central approach to aligning LLMs with human preferences and improving performance. We propose Maximum a Posteriori Preference Optimization (MaPPO), a methodology for learning from preferences that explicitly incorporates prior reward knowledge into the optimization objective. Building on the paradigm employed by Direct Preference Optimization (DPO) and its variants of treating preference learning as a Maximum Likelihood Estimation (MLE) problem, MaPPO integrates prior reward estimates into a principled Maximum a Posteriori (MaP) objective. This not only generalizes DPO and its variants, but also enhances alignment by mitigating the oversimplified binary classification of responses. Additionally, MaPPO introduces no additional hyperparameters, and supports preference optimization in both offline and online settings. In addition, MaPPO can be used as a plugin for DPO variants, including widely used SimPO, IPO, and CPO, and produce consistent improvements. Extensive empirical evaluations of different model sizes and model series on three standard benchmarks (MT-Bench, AlpacaEval 2.0, and Arena-Hard) demonstrate consistent improvements in alignment performance without sacrificing computational efficiency.
Figures & tables
Figure 1: An example of (x,yw,yl) pair. Both responses yw and yl have good quality as they achieve high rewards, where r(x,yw)=0.95 , r(x,yl)=0.91 , and r∈[0,1] .
Model
logp(yw)
logp(yl)
logp(yw) - logp(yl)
IT
−14.3
−43.4
+29.1
DPO
−121.5
−443.2
+321.7
MaPPO
−10.4
−321.1
+310.7
Table 1: Comparison of log-probabilities for responses yw and yl in PO methods.
Figure 2: Under the standard MLE-based DPO (left), empirical studies [ 43 , 46 , 59 , 49 ] demonstrated that training tends to simultaneously downscale (with different magnitudes) both the chosen and rejected responses to increase their gap. Our MaP-based method (right) mitigates this harmful tendency by re-weighting the rejected response based on prior knowledge. Here, the x-axis denotes the initial model θ0 and a potentially harmful model θk that may arise during training, while the y-axis shows the log-likelihood of a fixed preference pair under different policies.
Method
AlpacaEval 2.0 ↑
Arena-Hard ↑
MT-Bench ↑
DPO
32.01
45.5
8.56
+MaPPO
38.24 (+6.23)
59.2 (+13.7)
8.79 (+0.23)
I-DPO
33.80
46.9
8.55
+MaPPO
39.10 (+5.30)
61.6 (+14.7)
8.54 (-0.01)
SimPO
25.15
64.2
9.02
+MaPPO
32.75 (+7.60)
69.5 (+5.3)
8.94 (-0.08)
Table 2: Evaluation results of DPO variants with the MaPPO plugin. ↑ indicates higher is better.
Model
Method
AlpacaEval 2.0 ↑
Arena-Hard ↑
MT-Bench ↑
Mistral-7B-Instruct
IT
15.35
13.1
5.40
DPO
18.24
14.2
6.86
+ MaPPO
30.56{\color[rgb]{0,0.88,0}+12.32}
18.4{\color[rgb]{0,0.88,0}+4.2}
7.51{\color[rgb]{0,0.88,0}+0.65}
I-DPO
17.11
14.3
6.92
+ MaPPO
33.28{\color[rgb]{0,0.88,0}+16.17}
19.6{\color[rgb]{0,0.88,0}+5.3}
7.59{\color[rgb]{0,0.88,0}+0.67}
Llama-3-8B-Instruct
IT
10.85
10.2
7.52
Table 3: Main evaluation results on three standard benchmarks. ↑ indicates the higher the better.
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Illustration of the iterative MaPPO pipeline in each iteration k .
Hyperparameter
Value
global batch size
128
learning rate η
5×10−7
warmup steps
100
weight decay
0.01
optimizer
AdamW
KL weight β
0.1
Appendix
Table 4: Hyperparameter settings in MaPPO.
Hyperparameter
Value
SimPO: γ
1
IPO: β
0.1
CPO: λ
0.2
Appendix
Table 5: Hyperparameter settings in DPO Variants.
Method
AlpacaEval 2.0 ↑
Arena-Hard ↑
MT-Bench ↑
DPO
32.01
45.5
8.56
+MaPPO
38.24{\color[rgb]{0,0.88,0}+6.23}
59.2{\color[rgb]{0,0.88,0}+13.7}
8.79{\color[rgb]{0,0.88,0}+0.23}
I-DPO
33.80
46.9
8.55
+MaPPO
39.10{\color[rgb]{0,0.88,0}+5.30}
61.6{\color[rgb]{0,0.88,0}+14.7}
8.54{\color[rgb]{1,0.68,0.48}-0.01}
SimPO
25.15
64.2
9.02
+MaPPO
32.75{\color[rgb]{0,0.88,0}+7.60}
69.5{\color[rgb]{0,0.88,0}+5.3}
8.94{\color[rgb]{1,0.68,0.48}-0.08}
Appendix
Table 6: Evaluation results of DPO variants with a MaPPO plugin. ↑ indicates the higher the better. The results confirm the broad applicability of MaPPO across existing methods.
Method
IFEval ↑
GPQA ↑
MMLU ↑
HellaSwag ↑
TruthfulQA ↑
GSM8K ↑
IT
70.4
30.2
62.4
78.6
53.7
73.4
DPO
77.0
27.5
62.7
79.5
51.5
75.5
+ MaPPO
82.0
29.5
63.2
80.1
58.2
79.5
I-DPO
74.6
29.8
63.1
80.5
60.7
81.3
+ MaPPO
76.4
28.8
63.5
80.7
63.7
82.4
Appendix
Table 7: Evaluation results on six academic benchmarks with Llama-3-8B-Instruct model.
Method
IFEval ↑
GPQA ↑
MMLU ↑
HellaSwag ↑
TruthfulQA ↑
GSM8K ↑
IT
73.5
31.5
71.8
62.1
56.4
81.7
DPO
73.2
32.0
71.9
62.0
57.1
71.3
+ MaPPO
73.8
33.1
72.0
62.1
59.2
80.1
I-DPO
72.9
33.0
71.9
62.2
55.9
73.2
+ MaPPO
72.6
33.3
72.9
62.2
56.2
82.0
Appendix
Table 8: Evaluation results on six academic benchmarks with Qwen2.5-7B-Instruct model.
Model
DPO Win Rate
DPO+MaPPO Win Rate
Standard Error
Qwen2.5-14B-Instruct
46.31
53.69
1.72
Qwen2.5-7B-Instruct
47.14
52.86
1.74
Qwen2.5-3B-Instruct
46.53
53.47
1.52
Qwen2.5-1.5B-Instruct
47.26
52.74
1.31
Appendix
Table 9: Head-to-head comparison of DPO and DPO+MaPPO win rates ( % ).
Model
Reward Model
Method
AlpacaEval 2.0 ↑
Arena-Hard ↑
MT-Bench ↑
Qwen2.5-7B-IT
BT
IT
27.03
42.9
8.61
I-DPO
33.80
46.9
8.55
+ MaPPO
39.10{\color[rgb]{0,0.88,0}+5.30}
61.6{\color[rgb]{0,0.88,0}+14.7}
8.54{\color[rgb]{1,0.68,0.48}-0.01}
MoE
IT
27.03
42.9
8.61
I-DPO
28.79
48.3
8.58
+ MaPPO
31.56{\color[rgb]{0,0.88,0}+2.77}
52.0{\color[rgb]{0,0.88,0}+3.7}
8.60{\color[rgb]{0,0.88,0}+0.02}
Appendix
Table 10: Ablation on reward models. ↑ indicates the higher the better.
Offline preference optimization methods, such as Direct Preference Optimization (DPO), offer significant advantages in aligning Large Language Models (LLMs) with human values. However, achieving optimal performance with these methods typically involves additional hyperparameter tuning, resulting in substantial time overhead. Although prior work has proposed a range of improvements, these methods remain limited in effectiveness and have not fully eliminated reliance on hyperparameter tuning. In this work, we propose RMiPO, a lightweight and efficient framework for offline preference optimization. RMiPO leverages intrinsic Response-level Mutual information for Preference Optimization with hyperparameter modulation, dynamically decoupling preference contributions at negligible additional computational cost. Extensive experimental results demonstrate that RMiPO achieves consistently superior performance over existing methods while reducing training overhead by more than 15%. Our code is available at https://github.com/liavonpenn/rmipo.
Peng Liao, Peijia Zheng, Lingbo Li +2
1Sun Yat-sen University · University of Warwick · 3Macao Polytechnic University
We introduce a proximal preference optimization framework that unifies the implicit reward formulation of Direct Preference Optimization (DPO) with the trust-region stability of Proximal Policy Optimization (PPO). Our framework yields two complementary algorithms. P2O (Proximal Preference Optimization) introduces a clipped importance ratio directly into the DPO preference margin, producing a reward-model-free algorithm that supports safe dataset reuse, bounded policy divergence, and monotonic preference improvement guarantees. PKTO (Proximal Kahneman-Tversky Optimization) extends P2O by replacing the DPO margin with KTO's prospect-theoretic utility margins, adding a self-regulating adaptive baseline that tightens the effective trust region as the policy drifts, and an asymmetric weighting that handles imbalanced or asymmetric preference datasets. We provide a unified theoretical analysis, including Kullback Leibler (KL) trust-region bounds, a policy-improvement lower bound, and convergence guarantees under multi-epoch proximal updates. Empirically, P2O and PKTO match or exceed DPO and KTO in alignment quality while maintaining better stability under repeated dataset reuse across multiple LLMs on three preference benchmarks. The code base is available here.
Direct preference optimization DPO is a promising offline approach for aligning large language models (LLMs) due to its simplicity, computational efficiency, and implicit modeling of human preferences. Interestingly, iterative extensions of DPO have achieved stronger performance on academic benchmarks, raising two key questions: (i) Why do iterative methods generally outperform offline ones? (ii) Can their advantages be incorporated into offline alignment? To answer the first question, our controlled experiments reveal that the explicit preference model, additionally introduced in the iterative procedure, is a key factor behind its superiority over offline methods. This insight leads us to answer the second question affirmatively and propose Distilled Preference Probability Policy Optimization (DP3O), an effective and efficient offline alignment algorithm. DP3O first learns an explicit preference model using a helper class of LLMs and then distills its knowledge into policy optimization. Theoretically, we show that explicit preference modeling admits better estimation error control than implicit formulations, and that DP3O achieves a tighter generalization bound than hard-label DPO through variance reduction. Empirically, we evaluate DP3O on a wide range of chat-based and downstream tasks and show that it outperforms state-of-the-art offline methods, achieves performance comparable to iterative DPO, and reduces training time by about 42%, demonstrating both its effectiveness and efficiency.
Wenbo Zhang, Wenzhuo Zhou, Hengrui Cai +1
Department of Statistics University of California, Irvine · School of Business George Washington University