We present a fine-grained theoretical analysis of the performance gap between two-stage reinforcement learning from human feedback (RLHF) and direct preference optimization (DPO). Our study decomposes this gap into two sources: the explicit representation gap under exact optimization and the implicit representation gap under finite samples. In the exact optimization setting, we characterize how the relative capacities of the reward and policy model classes influence the final policy qualities. We show that RLHF, DPO, or online DPO can outperform one another depending on type of model mis-specifications. Notably, online DPO can outperform both RLHF and standard DPO when the reward and policy model classes are isomorphic and both mis-specified. In the approximate optimization setting, we provide a concrete construction where the ground-truth reward is sparse and show that RLHF requires significantly fewer samples than DPO to recover an effective reward model, highlighting a statistical advantage of two-stage learning. Together, these results provide a comprehensive understanding of the performance gap between RLHF and DPO under various settings, and offer practical insights into when each method is preferred.
It depends on qualities of rRLHF and r^DPO . isomorphic case: Vr⋆πRLHF=Vr⋆πDPO , and Vr⋆πRLHF≤Vr⋆πDPOonline.
Table 1: Main results on performance gap induced by model mis-specification scenarios.
Figure 1: Experimental Results for Condition 1 . Experiments with different reward scales {0.4,1,4} align with Theorem 2 : as the reward scale increases, the second-order deviation in the online DPO objective grows, giving RLHF a clear advantage.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Numerically Computed Curves of Gradient Functions and Value Functions. The gradient values are rescaled for clarity of presentation.
Preference-based reinforcement learning learns policies from comparisons or rankings of behaviors instead of requiring hand-designed numerical rewards.