cs.LGMay 26, 2025

Understanding the Performance Gap in Preference Learning: A Dichotomy of RLHF and DPO

Authors: Ruizhe Shi, Minhak Song, Runlong Zhou, Zihan Zhang, Maryam Fazel, Simon S. Du

Organizations: University of Washington · KAIST · KRAFTON · The Hong Kong University of Science and Technology · 

Abstract

We present a fine-grained theoretical analysis of the performance gap between two-stage reinforcement learning from human feedback (RLHF) and direct preference optimization (DPO). Our study decomposes this gap into two sources: the explicit representation gap under exact optimization and the implicit representation gap under finite samples. In the exact optimization setting, we characterize how the relative capacities of the reward and policy model classes influence the final policy qualities. We show that RLHF, DPO, or online DPO can outperform one another depending on type of model mis-specifications. Notably, online DPO can outperform both RLHF and standard DPO when the reward and policy model classes are isomorphic and both mis-specified. In the approximate optimization setting, we provide a concrete construction where the ground-truth reward is sparse and show that RLHF requires significantly fewer samples than DPO to recover an effective reward model, highlighting a statistical advantage of two-stage learning. Together, these results provide a comprehensive understanding of the performance gap between RLHF and DPO under various settings, and offer practical insights into when each method is preferred.

Figures & tables

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Conditional Equivalence of DPO and RLHF: Implicit Assumption, Failure Modes, and Provable Alignment

    May 20, 2026Zhiqin Yang, Yonggang Zhang, Wei Xue +3LLM AlignmentDirect Preference Optimization

  2. Adaptive Margin RLHF via Preference over Preferences

    Sep 26, 2025Yaswanth Chittepu, Prasann Singhal, Greg Durrett +1Pairwise Preference LearningReward Modeling