cs.AIOct 6, 2026

LSC-DPO: Learning-Signal-Controlled Direct Preference Optimization

Authors: Yang Qu, Yusheng Han, Chengjia Feng, Handan Liu

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Direct Preference Optimization (DPO) has become a standard reward-model-free approach for aligning language models with preference data. However, as the scaled preference margin grows during training, the logistic DPO loss becomes progressively less sensitive to further changes. We study DPO from a loss-level geometric perspective and identify the sigmoid factor as a learning signal that characterizes the local sensitivity of the objective. Based on this view, we propose Learning-Signal-Controlled Direct Preference Optimization (LSC-DPO), which dynamically regulates the learning signal near a target regime. A log-space analysis establishes conditions for stable tracking of the target learning-signal regime. Experiments on AlpacaEval 2, MT-Bench, and Anthropic-HH show that LSC-DPO consistently improves over DPO and strong preference-optimization baselines. We further find that different coefficient initializations induce distinct transient learning-signal trajectories even when their later signal levels become similar. Based on this observation, we derive a signal-budget compensation rule that adjusts the target learning signal to compensate for these transient differences. The resulting compensation substantially reduces performance variation across coefficient initializations.

Explore similar work

CardsList
Aug 27, 2026cs.LG

Disentangling Optimization Scale from Preference Scale in DPO

Direct Preference Optimization (DPO) is a widely used objective for aligning language models from preference data, with the coefficient ββ commonly interpreted as controlling the KL constraint to a reference policy. We show that ββ entangles two distinct roles: it governs the effective inverse preference-noise scale and simultaneously rescales the optimization dynamics, coupling this scale with the effective step size. As a consequence, at a fixed learning rate the achieved policy deviation is non-monotone in ββ: it vanishes in a dead zone at small ββ, reaches a peak at an intermediate value, and decreases again for larger ββ. Moreover, standard DPO loss values are not comparable across ββ: runs with nearly identical loss curves can differ several-fold in KL divergence from the reference model. This entanglement obscures the role of ββ, increases sensitivity to hyperparameter choices, and complicates learning-rate scheduling. We propose a centered-softplus reformulation that is argmin-equivalent to DPO for β>0β>0, while making the inverse preference-noise-scale and learning-rate effects explicit and independently tunable. The normalized centered-softplus objective also admits a continuous β→0β\to0 endpoint that reduces to a linear preference-margin objective.

Ivan Kruzhilov

Topic pending
AskPDF
Jun 10, 2026cs.LG

Boosting Direct Preference Optimization with Penalization

Offline preference optimization has become a practical substitute for reinforcement learning from human feedback, but pairwise objectives such as Direct Preference Optimization (DPO) and its variants use only the chosen and rejected responses stored in a static dataset. This leaves a useful signal unused: the response that the reference model itself would generate for the same prompt. We propose Direct Preference Optimization with Penalization (DPOP), a simple extension of DPO that augments the base preference loss with a gated penalty on reference-greedy responses. DPOP activates this penalty only when the current policy still assigns a lower likelihood to the preferred response than to the rejected response. On AlpacaEval 2.0, DPOP improves length-controlled win rate over DPO, SimPO, and AlphaDPO on both Llama-3-8b-it and Gemma-2-9b-it, achieving relative gains of 5.3% and 4.4% over baselines on the two models, respectively. Ablations further show that a SimNPO-style length-normalized penalty is stronger than NPO and token-level unlikelihood in this setting.

Pengwei Sun

Stanford University.

Pairwise Preference LearningLLM AlignmentDirect Preference OptimizationOffline Preference Optimization
AskPDF
Jun 11, 2026cs.CL

Direct Preference Optimization for Chatbot Fine-Tuning: An Empirical Study

We present an approach to fine-tuning large language models using Direct Preference Optimization (DPO), a reinforcement learning technique. Our experimental results demonstrate that DPO simplifies the training pipeline, improves computational efficiency, and achieves competitive performance. The evaluation using BLEU, ROUGE, and cosine similarity metrics indicates effective learning and convergence, though further investigation is needed to address observed training instability.

Dezhi Yu, Yvonne Qiu, ShuoJia Fu

LLM AlignmentDirect Preference OptimizationLLM Fine-TuningLLM Post-Training
AskPDF