cs.LGJul 27, 2025

MaPPO: Maximum a Posteriori Preference Optimization with Prior Knowledge

Authors: Guangchen Lan, Sipeng Zhang, Tianle Wang, Yuwei Zhang, Daoan Zhang, Xinpeng Wei, Xiaoman Pan, Hongming Zhang, 

Organizations: Purdue University · University of California, San Diego · Amazon · Meta, FAIR · 

Abstract

As the era of large language models (LLMs) unfolds, Preference Optimization (PO) methods have become a central approach to aligning LLMs with human preferences and improving performance. We propose Maximum a Posteriori Preference Optimization (MaPPO), a methodology for learning from preferences that explicitly incorporates prior reward knowledge into the optimization objective. Building on the paradigm employed by Direct Preference Optimization (DPO) and its variants of treating preference learning as a Maximum Likelihood Estimation (MLE) problem, MaPPO integrates prior reward estimates into a principled Maximum a Posteriori (MaP) objective. This not only generalizes DPO and its variants, but also enhances alignment by mitigating the oversimplified binary classification of responses. Additionally, MaPPO introduces no additional hyperparameters, and supports preference optimization in both offline and online settings. In addition, MaPPO can be used as a plugin for DPO variants, including widely used SimPO, IPO, and CPO, and produce consistent improvements. Extensive empirical evaluations of different model sizes and model series on three standard benchmarks (MT-Bench, AlpacaEval 2.0, and Arena-Hard) demonstrate consistent improvements in alignment performance without sacrificing computational efficiency.

Figures & tables

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Intrinsic Mutual Information as a Modulator for Preference Optimization

    Apr 27, 2026Peng Liao, Peijia Zheng, Lingbo Li +2LLM AlignmentOffline Preference Optimization

  2. Proximal Preference Optimization: A Unified Framework for Stable and Reward-Free Alignment

    Nov 7, 2026Qisai Liu, Zhanhong Jiang, Zahid Hasan +1LLM AlignmentPreference Optimization

  3. Towards Bridging the Gap Between Offline and Iterative Alignment via Preference Distillation

    Sep 7, 2026Wenbo Zhang, Wenzhuo Zhou, Hengrui Cai +1LLM AlignmentPreference Alignment