As the era of large language models (LLMs) unfolds, Preference Optimization (PO) methods have become a central approach to aligning LLMs with human preferences and improving performance. We propose Maximum a Posteriori Preference Optimization (MaPPO), a methodology for learning from preferences that explicitly incorporates prior reward knowledge into the optimization objective. Building on the paradigm employed by Direct Preference Optimization (DPO) and its variants of treating preference learning as a Maximum Likelihood Estimation (MLE) problem, MaPPO integrates prior reward estimates into a principled Maximum a Posteriori (MaP) objective. This not only generalizes DPO and its variants, but also enhances alignment by mitigating the oversimplified binary classification of responses. Additionally, MaPPO introduces no additional hyperparameters, and supports preference optimization in both offline and online settings. In addition, MaPPO can be used as a plugin for DPO variants, including widely used SimPO, IPO, and CPO, and produce consistent improvements. Extensive empirical evaluations of different model sizes and model series on three standard benchmarks (MT-Bench, AlpacaEval 2.0, and Arena-Hard) demonstrate consistent improvements in alignment performance without sacrificing computational efficiency.
Figures & tables
Figure 1: An example of (x,yw,yl) pair. Both responses yw and yl have good quality as they achieve high rewards, where r(x,yw)=0.95 , r(x,yl)=0.91 , and r∈[0,1] .
Model
logp(yw)
logp(yl)
logp(yw) - logp(yl)
IT
−14.3
−43.4
+29.1
DPO
−121.5
−443.2
+321.7
MaPPO
−10.4
−321.1
+310.7
Table 1: Comparison of log-probabilities for responses yw and yl in PO methods.
Figure 2: Under the standard MLE-based DPO (left), empirical studies [ 43 , 46 , 59 , 49 ] demonstrated that training tends to simultaneously downscale (with different magnitudes) both the chosen and rejected responses to increase their gap. Our MaP-based method (right) mitigates this harmful tendency by re-weighting the rejected response based on prior knowledge. Here, the x-axis denotes the initial model θ0 and a potentially harmful model θk that may arise during training, while the y-axis shows the log-likelihood of a fixed preference pair under different policies.
Method
AlpacaEval 2.0 ↑
Arena-Hard ↑
MT-Bench ↑
DPO
32.01
45.5
8.56
+MaPPO
38.24 (+6.23)
59.2 (+13.7)
8.79 (+0.23)
I-DPO
33.80
46.9
8.55
+MaPPO
39.10 (+5.30)
61.6 (+14.7)
8.54 (-0.01)
SimPO
25.15
64.2
9.02
+MaPPO
32.75 (+7.60)
69.5 (+5.3)
8.94 (-0.08)
Table 2: Evaluation results of DPO variants with the MaPPO plugin. ↑ indicates higher is better.
Model
Method
AlpacaEval 2.0 ↑
Arena-Hard ↑
MT-Bench ↑
Mistral-7B-Instruct
IT
15.35
13.1
5.40
DPO
18.24
14.2
6.86
+ MaPPO
30.56{\color[rgb]{0,0.88,0}+12.32}
18.4{\color[rgb]{0,0.88,0}+4.2}
7.51{\color[rgb]{0,0.88,0}+0.65}
I-DPO
17.11
14.3
6.92
+ MaPPO
33.28{\color[rgb]{0,0.88,0}+16.17}
19.6{\color[rgb]{0,0.88,0}+5.3}
7.59{\color[rgb]{0,0.88,0}+0.67}
Llama-3-8B-Instruct
IT
10.85
10.2
7.52
Table 3: Main evaluation results on three standard benchmarks. ↑ indicates the higher the better.
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Illustration of the iterative MaPPO pipeline in each iteration k .
Hyperparameter
Value
global batch size
128
learning rate η
5×10−7
warmup steps
100
weight decay
0.01
optimizer
AdamW
KL weight β
0.1
Appendix
Table 4: Hyperparameter settings in MaPPO.
Hyperparameter
Value
SimPO: γ
1
IPO: β
0.1
CPO: λ
0.2
Appendix
Table 5: Hyperparameter settings in DPO Variants.
Method
AlpacaEval 2.0 ↑
Arena-Hard ↑
MT-Bench ↑
DPO
32.01
45.5
8.56
+MaPPO
38.24{\color[rgb]{0,0.88,0}+6.23}
59.2{\color[rgb]{0,0.88,0}+13.7}
8.79{\color[rgb]{0,0.88,0}+0.23}
I-DPO
33.80
46.9
8.55
+MaPPO
39.10{\color[rgb]{0,0.88,0}+5.30}
61.6{\color[rgb]{0,0.88,0}+14.7}
8.54{\color[rgb]{1,0.68,0.48}-0.01}
SimPO
25.15
64.2
9.02
+MaPPO
32.75{\color[rgb]{0,0.88,0}+7.60}
69.5{\color[rgb]{0,0.88,0}+5.3}
8.94{\color[rgb]{1,0.68,0.48}-0.08}
Appendix
Table 6: Evaluation results of DPO variants with a MaPPO plugin. ↑ indicates the higher the better. The results confirm the broad applicability of MaPPO across existing methods.
Method
IFEval ↑
GPQA ↑
MMLU ↑
HellaSwag ↑
TruthfulQA ↑
GSM8K ↑
IT
70.4
30.2
62.4
78.6
53.7
73.4
DPO
77.0
27.5
62.7
79.5
51.5
75.5
+ MaPPO
82.0
29.5
63.2
80.1
58.2
79.5
I-DPO
74.6
29.8
63.1
80.5
60.7
81.3
+ MaPPO
76.4
28.8
63.5
80.7
63.7
82.4
Appendix
Table 7: Evaluation results on six academic benchmarks with Llama-3-8B-Instruct model.
Method
IFEval ↑
GPQA ↑
MMLU ↑
HellaSwag ↑
TruthfulQA ↑
GSM8K ↑
IT
73.5
31.5
71.8
62.1
56.4
81.7
DPO
73.2
32.0
71.9
62.0
57.1
71.3
+ MaPPO
73.8
33.1
72.0
62.1
59.2
80.1
I-DPO
72.9
33.0
71.9
62.2
55.9
73.2
+ MaPPO
72.6
33.3
72.9
62.2
56.2
82.0
Appendix
Table 8: Evaluation results on six academic benchmarks with Qwen2.5-7B-Instruct model.
Model
DPO Win Rate
DPO+MaPPO Win Rate
Standard Error
Qwen2.5-14B-Instruct
46.31
53.69
1.72
Qwen2.5-7B-Instruct
47.14
52.86
1.74
Qwen2.5-3B-Instruct
46.53
53.47
1.52
Qwen2.5-1.5B-Instruct
47.26
52.74
1.31
Appendix
Table 9: Head-to-head comparison of DPO and DPO+MaPPO win rates ( % ).
Model
Reward Model
Method
AlpacaEval 2.0 ↑
Arena-Hard ↑
MT-Bench ↑
Qwen2.5-7B-IT
BT
IT
27.03
42.9
8.61
I-DPO
33.80
46.9
8.55
+ MaPPO
39.10{\color[rgb]{0,0.88,0}+5.30}
61.6{\color[rgb]{0,0.88,0}+14.7}
8.54{\color[rgb]{1,0.68,0.48}-0.01}
MoE
IT
27.03
42.9
8.61
I-DPO
28.79
48.3
8.58
+ MaPPO
31.56{\color[rgb]{0,0.88,0}+2.77}
52.0{\color[rgb]{0,0.88,0}+3.7}
8.60{\color[rgb]{0,0.88,0}+0.02}
Appendix
Table 10: Ablation on reward models. ↑ indicates the higher the better.