cs.LGOct 19, 2026

Statistical Learning Theory for Direct Preference Optimization

Authors: Joel Wills, Yuanfan Li, Yiming Ying, Ding-Xuan Zhou

Organizations: School of Mathematics and Statistics, The University of Sydney

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Direct Preference Optimization (DPO) learns from pairwise preferences through a logistic objective whose score-difference parameterization follows the Bradley--Terry model. We study how failure of this modeling assumption limits DPO, distinguishing structural misspecification from limited model capacity and finite-sample estimation. Our main contribution is a characterization of structural approximation error through the triangle-curl of the preference log-odds, which vanishes exactly when preferences admit a Bradley--Terry representation. We upper bound the achievable excess logistic risk by a score-class approximation term and a triangle-curl term, and establish a lower bound proportional to triangle-curl under additional boundedness assumptions. For ReLU score models, including clipped models, we derive explicit approximation rates showing that increasing capacity can eliminate the score-approximation term while leaving the structural obstruction unchanged. We further establish finite-sample bounds for clipped empirical risk minimizers that separate network approximation, structural misspecification, and estimation error. We place these results within a surrogate-risk framework by identifying contextual preference accuracy (CPA) as a natural measure of agreement between preference judgments and changes in pairwise probability ratios relative to the reference policy. Standard logistic consistency and calibration results connect DPO's objective to this policy property and transfer our excess-risk upper bounds to CPA, without treating CPA as a complete measure of generated-response quality.

More in this topic

Direct Preference Optimization

Direct Preference Optimization trains generative models from preferred and rejected outputs without fitting an explicit reward model.

See all 162 papers

Explore similar work

CardsList
Aug 27, 2026

Disentangling Optimization Scale from Preference Scale in DPO

Direct Preference Optimization (DPO) is a widely used objective for aligning language models from preference data, with the coefficient ββ commonly interpreted as controlling the KL constraint to a reference policy. We show that ββ entangles two distinct roles: it governs the effective inverse preference-noise scale and simultaneously rescales the optimization dynamics, coupling this scale with the effective step size. As a consequence, at a fixed learning rate the achieved policy deviation is non-monotone in ββ: it vanishes in a dead zone at small ββ, reaches a peak at an intermediate value, and decreases again for larger ββ. Moreover, standard DPO loss values are not comparable across ββ: runs with nearly identical loss curves can differ several-fold in KL divergence from the reference model. This entanglement obscures the role of ββ, increases sensitivity to hyperparameter choices, and complicates learning-rate scheduling. We propose a centered-softplus reformulation that is argmin-equivalent to DPO for β>0β>0, while making the inverse preference-noise-scale and learning-rate effects explicit and independently tunable. The normalized centered-softplus objective also admits a continuous β→0β\to0 endpoint that reduces to a linear preference-margin objective.

Ivan Kruzhilov

Direct Preference OptimizationPreference LearningDeep Learning OptimizationRL for Language Models
AskPDF
Jun 10, 2026

Boosting Direct Preference Optimization with Penalization

Offline preference optimization has become a practical substitute for reinforcement learning from human feedback, but pairwise objectives such as Direct Preference Optimization (DPO) and its variants use only the chosen and rejected responses stored in a static dataset. This leaves a useful signal unused: the response that the reference model itself would generate for the same prompt. We propose Direct Preference Optimization with Penalization (DPOP), a simple extension of DPO that augments the base preference loss with a gated penalty on reference-greedy responses. DPOP activates this penalty only when the current policy still assigns a lower likelihood to the preferred response than to the rejected response. On AlpacaEval 2.0, DPOP improves length-controlled win rate over DPO, SimPO, and AlphaDPO on both Llama-3-8b-it and Gemma-2-9b-it, achieving relative gains of 5.3% and 4.4% over baselines on the two models, respectively. Ablations further show that a SimNPO-style length-normalized penalty is stronger than NPO and token-level unlikelihood in this setting.

Pengwei Sun

Stanford University.

Pairwise Preference LearningLLM AlignmentDirect Preference OptimizationOffline Preference Optimization
AskPDF
May 20, 2026

Conditional Equivalence of DPO and RLHF: Implicit Assumption, Failure Modes, and Provable Alignment

Direct Preference Optimization (DPO) has emerged as a popular alternative to Reinforcement Learning from Human Feedback (RLHF), offering theoretical equivalence with simpler implementation. We prove this equivalence is conditional rather than universal, depending on an implicit assumption frequently violated in practice: the RLHF-optimal policy must prefer human-preferred responses. When this assumption fails, DPO optimizes relative advantage over the reference policy rather than absolute alignment with human preferences, leading to pathological convergence where policies decrease DPO loss while preferring dispreferred responses. We characterize when this assumption is violated, show the existence of an undesirable solution space, and prove that DPO and RLHF optimize fundamentally different objectives in such cases. To address this, we introduce Constrained Preference Optimization (CPO), augmenting RLHF with constraints for provable alignment. We further provide a geometric interpretation through soft margin ranking, revealing that DPO implements margin ranking with potentially negative targets. Our theoretical analysis establishes when DPOs' guarantees hold and provides solutions preserving simplicity with provable alignment. Comprehensive experiments on standard benchmarks demonstrate that CPO achieves state-of-the-art performance. Code is available at: https://github.com/visitworld123/CPO.

Zhiqin Yang, Yonggang Zhang, Wei Xue +3

1The Hong Kong University of Science and Technology · 2LIGHTSPEED · 3Hong Kong Baptist University.

LLM AlignmentDirect Preference OptimizationPreference OptimizationPreference AlignmentRL from Human Feedback
AskPDF