Preference optimization can learn spurious correlations, and augmenting the preference data with ties (equally preferred responses) can reduce spurious reliance. However, ties differ in their corrective effect, making their selection for augmentation important. For log-linear policies, we derive each tie's influence on spurious reliance and show that taking the largest predicted reductions is first-order optimal among subsets of equal size. Computing these influences requires knowledge of the spurious features and the curvature of the training objective, which is often unavailable or expensive to estimate. To this end, we explore using the magnitude of a tie's implicit reward margin and characterize the conditions under which it is a useful proxy for predicted influence. To test the theory and its qualitative transfer to neural models, we estimate influence with a counterfactual proxy for spurious reliance and use a generalized Gauss--Newton approximation to curvature for language models. Across log-linear policies, neural models, and large language models, our selection rules consistently achieve comparable reductions in spurious reliance with fewer ties than uniform selection.
Figures & tables
Figure 2: Selection beyond the log-linear setting. Qadv accuracy for a nonlinear reward model (a) and LoRA-DPO on hotel (b) and emoji (c) tasks. Practical influence uses a sampled GGN and solves the damped system in its row space for all three neural regimes. Curves are means across seeds; shading shows 95% Student- t confidence intervals.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Environment
a
t
Interpretation
P
+1
+1
Training environment; both shortcut channels align.
Qclean
0
0
Both shortcut channels are removed.
Qadv
−1
−1
Both shortcut channels reverse.
Appendix
Table 1: Environments used to train and evaluate the synthetic models. The parameters a and t are defined in ( 14 ).
Setting
Log-linear
Neural network
Feature dimensions (dc,ds)
(10,10)
(10,10)
Condition numbers κ(Ωc),κ(Ωs)
(20,20)
(20,20)
Singular values σi(R)
[0.20,0.65]
[0.20,0.65]
Signal strengths (γc,γs)
(2,2)
(2,2)
Teacher coefficient βteacher
4
4
DPO coefficient β
1
1
Appendix
Table 2: Settings for the synthetic experiments.
Figure 3: Log-linear results under fresh training (left) and continued training (right). Rows share axes; colours, markers, and line styles match across columns. Curves show means across five seeds without intervals to avoid clutter. Dotted grey lines mark strict baselines. Fresh and continued models agree to numerical precision.
Figure 4: Neural-network results under fresh training (left) and continued training (right). Rows share axes; colours, markers, and line styles match across columns. Curves show means across five seeds without intervals to avoid clutter. Dotted grey lines mark strict baselines.
Setting
Hotels
Emojis
Base policy πref
Llama-3.2-1B-Instruct
Arithmetic and attention
unquantized bfloat16; FlashAttention 2
LoRA rank; scale; dropout
16;32;0.05
LoRA targets
all attention and feed-forward projections
DPO coefficient β
0.1
Optimizer; step size η
AdamW; 10−5
Appendix
Table 3: Settings for the LLM experiments. Shared entries apply to both tasks unless separate values are shown.
Figure 5: Hotel results under fresh training (left) and continued training (right). Rows share axes; colours, markers, and line styles match across columns. Curves show means across three seeds without intervals to avoid clutter. Dotted grey lines mark strict baselines. Energy uses a logarithmic axis.
Figure 6: Emoji results under fresh training (left) and continued training (right). Rows share axes; colours, markers, and line styles match across columns. Curves show means across three seeds without intervals to avoid clutter. Dotted grey lines mark strict baselines. Energy uses a logarithmic axis.
Figure 7: Results after fresh training for Qwen3-4B-Instruct-2507 on hotels (left) and emojis (right). Y-axis ranges differ between columns to show differences within each task. Curves show seed 0 with one UTL draw; uncertainty intervals are not estimated. Dotted grey lines mark strict baselines. Energy uses a logarithmic axis.
Preference learning methods like Direct Preference Optimization (DPO) are known to induce reliance on spurious correlations, leading to sycophancy and length bias in today's language models and potentially severe goal misgeneralization in future systems. In this work, we provide a unified theoretical analysis of this phenomenon, characterizing the mechanisms of spurious learning, its consequences on deployment, and a provable mitigation strategy. Focusing on log-linear policies, we show that standard preference-learning objectives induce reliance on spurious features at the population level through two channels: mean spurious bias and causal-spurious correlation leakage. We then show that this reliance creates an irreducible vulnerability to distribution shift: more data from the same training distribution fails to reduce the model's dependence on spurious features. To address this, we propose tie training, a data augmentation strategy using ties (equal-utility preference pairs) to introduce data-driven regularization. We demonstrate that this approach selectively reduces spurious learning without degrading causal learning. Finally, we validate our theory on log-linear models and provide empirical evidence that both the spurious learning mechanisms and the benefits of tie training persist for neural networks and large language models.
Christian Moya, Alex Semendinger, Guang Lin +1
Department of Mathematics, Purdue University, West Lafayette IN, USA
Preference optimization is widely used to align large language models (LLMs) with human preferences. However, many margin-based methods also suppress the chosen response when they try to suppress the rejected one, and there is no general way to prevent this across different objectives. We address this issue with a unified incentive-score decomposition of preference optimization, revealing that different objectives share the same local update directions and differ only in their scalar weights. This decomposition provides a common framework for analyzing objectives that were previously studied in separate settings. Building on this decomposition, by analyzing the dynamics of the chosen/rejected likelihoods, we identify the disentanglement band (DB), a simple, testable condition that tells us when training can follow the desired path: suppress the loser while preserving the winner, possibly after an early stage. Using the DB, we propose reward calibration (RC), a plug-and-play method that adaptively rebalances the updates for chosen and rejected responses to satisfy the DB, without redesigning the base objective. Empirical results show that RC leads to more disentangled dynamics, with better downstream performance observed across several settings. Our code is available at https://github.com/IceyWuu/DisentangledPreferenceOptimization.
Wei Chen, Yubing Wu, Junmei Yang +5
1South China University of Technology · 2AIP, RIKEN · 3Columbia University +1
Preference optimization methods such as DPO and KTO are widely used for aligning language models, yet little is understood about what properties of preference data drive downstream reasoning gains. We ask: what aspects of a preference pair improve a reasoning model's performance on general reasoning tasks? We investigate two distinct notions of quality delta in preference data: generator-level delta, arising from the differences in capability between models that generate chosen and rejected reasoning traces, and sample-level delta, arising from differences in judged quality differences within an individual preference pair. To study generator-level delta, we vary the generator's scale and model family, and to study sample-level delta, we employ an LLM-as-a-judge to rate the quality of generated traces along multiple reasoning-quality dimensions. We find that increasing generator-level delta steadily improves performance on out-of-domain reasoning tasks and filtering data by sample-level delta can enable more data-efficient training. Our results suggest a twofold recipe for improving reasoning performance through preference optimization: maximize generator-level delta when constructing preference pairs and exploit sample-level delta to select the most informative training examples.