Selective Tie Learning: Mitigating Spurious Correlations in Preference Optimization

Christian Moya; Purav Matlia; Elliott Thornley; Guang Lin

cs.LGOct 26, 2026

Selective Tie Learning: Mitigating Spurious Correlations in Preference Optimization

Authors: Christian Moya, Purav Matlia, Elliott Thornley, Guang Lin

Organizations: Purdue University · National University of Singapore

Abstract

Preference optimization can learn spurious correlations, and augmenting the preference data with ties (equally preferred responses) can reduce spurious reliance. However, ties differ in their corrective effect, making their selection for augmentation important. For log-linear policies, we derive each tie's influence on spurious reliance and show that taking the largest predicted reductions is first-order optimal among subsets of equal size. Computing these influences requires knowledge of the spurious features and the curvature of the training objective, which is often unavailable or expensive to estimate. To this end, we explore using the magnitude of a tie's implicit reward margin and characterize the conditions under which it is a useful proxy for predicted influence. To test the theory and its qualitative transfer to neural models, we estimate influence with a counterfactual proxy for spurious reliance and use a generalized Gauss--Newton approximation to curvature for language models. Across log-linear policies, neural models, and large language models, our selection rules consistently achieve comparable reductions in spurious reliance with fewer ties than uniform selection.

Figures & tables

Panel of Figure 2 — Figure 2: Selection beyond the log-linear setting. $Q_{\rm adv}$ accuracy for a nonlinear reward model (a) and LoRA-DPO on hotel (b) and emoji (c) tasks. Practical influence uses a sampled GGN and solves the damped system in its row space for all three neural regimes. Curves are means across seeds; shading shows 95% Student- $t$ confidence intervals.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Environment	$a$	$t$	Interpretation
$P$	$+1$	$+1$	Training environment; both shortcut channels align.
$Q_{\rm clean}$	$0$	$0$	Both shortcut channels are removed.
$Q_{\rm adv}$	$-1$	$-1$	Both shortcut channels reverse.

Appendix

Table 1: Environments used to train and evaluate the synthetic models. The parameters

a

and

t

are defined in ( 14 ).

Setting	Log-linear	Neural network
Feature dimensions $(d_{c},d_{s})$	$(10,10)$	$(10,10)$
Condition numbers $\kappa(\Omega_{c}),\kappa(\Omega_{s})$	$(20,20)$	$(20,20)$
Singular values $\sigma_{i}(R)$	$[0.20,0.65]$	$[0.20,0.65]$
Signal strengths $(\gamma_{c},\gamma_{s})$	$(2,2)$	$(2,2)$
Teacher coefficient $\beta_{\rm teacher}$	$4$	$4$
DPO coefficient $\beta$	$1$	$1$

Appendix

Table 2: Settings for the synthetic experiments.

Panel of Figure 3 — Figure 3: Log-linear results under fresh training (left) and continued training (right). Rows share axes; colours, markers, and line styles match across columns. Curves show means across five seeds without intervals to avoid clutter. Dotted grey lines mark strict baselines. Fresh and continued models agree to numerical precision.

Panel of Figure 4 — Figure 4: Neural-network results under fresh training (left) and continued training (right). Rows share axes; colours, markers, and line styles match across columns. Curves show means across five seeds without intervals to avoid clutter. Dotted grey lines mark strict baselines.

Setting	Hotels	Emojis
Base policy $\pi_{\rm ref}$	Llama-3.2-1B-Instruct
Arithmetic and attention	unquantized bfloat16; FlashAttention 2
LoRA rank; scale; dropout	$16;32;0.05$
LoRA targets	all attention and feed-forward projections
DPO coefficient $\beta$	$0.1$
Optimizer; step size $\eta$	AdamW; $10^{-5}$

Appendix

Table 3: Settings for the LLM experiments. Shared entries apply to both tasks unless separate values are shown.

Panel of Figure 5 — Figure 5: Hotel results under fresh training (left) and continued training (right). Rows share axes; colours, markers, and line styles match across columns. Curves show means across three seeds without intervals to avoid clutter. Dotted grey lines mark strict baselines. Energy uses a logarithmic axis.

Panel of Figure 6 — Figure 6: Emoji results under fresh training (left) and continued training (right). Rows share axes; colours, markers, and line styles match across columns. Curves show means across three seeds without intervals to avoid clutter. Dotted grey lines mark strict baselines. Energy uses a logarithmic axis.

Panel of Figure 7 — Figure 7: Results after fresh training for Qwen3-4B-Instruct-2507 on hotels (left) and emojis (right). Y-axis ranges differ between columns to show differences within each task. Curves show seed 0 with one UTL draw; uncertainty intervals are not estimated. Dotted grey lines mark strict baselines. Energy uses a logarithmic axis.

Preference Optimization

Preference optimization aligns AI systems with human or task-specific judgments by training them to favor better-ranked outputs.

See all 155 papers

Spurious Correlation Learning in Preference Optimization: Mechanisms, Consequences, and Mitigation via Tie Training
May 11, 2026Christian Moya, Alex Semendinger, Guang Lin +1Pairwise Preference Learning LLM Alignment
Department of Mathematics, Purdue University, West Lafayette IN, USA
Towards Disentangled Preference Optimization Dynamics: Suppress the Loser, Preserve the Winner
Apr 20, 2026Wei Chen, Yubing Wu, Junmei Yang +5Pairwise Preference Learning LLM Alignment
1South China University of Technology · 2AIP, RIKEN · 3Columbia University +1
Decomposing the Delta: What Do Models Actually Learn from Preference Pairs?
Apr 9, 2026Chia-Hsuan Lee, Mingyang Zhou, Renkun Ni +6Preference Optimization Pairwise Preference Learning
Capital One

May 11, 2026J/K move · Enter open · S save

Spurious Correlation Learning in Preference Optimization: Mechanisms, Consequences, and Mitigation via Tie Training

Christian Moya, Alex Semendinger, Guang Lin, Elliott Thornley

Department of Mathematics, Purdue University, West Lafayette IN, USA

Pairwise Preference Learning LLM Alignment Preference Optimization Spurious Correlation Robustness Preference Learning Spurious Correlation

Open paper AskPDF arXiv

Preference learning methods like Direct Preference Optimization (DPO) are known to induce reliance on spurious correlations, leading to sycophancy and length bias in today's language models and potentially severe goal misgeneralization in future systems. In this work, we provide a unified theoretical analysis of this phenomenon, characterizing the mechanisms of spurious learning, its consequences on deployment, and a provable mitigation strategy. Focusing on log-linear policies, we show that standard preference-learning objectives induce reliance on spurious features at the population level through two channels: mean spurious bias and causal-spurious correlation leakage. We then show that this reliance creates an irreducible vulnerability to distribution shift: more data from the same training distribution fails to reduce the model's dependence on spurious features. To address this, we propose tie training, a data augmentation strategy using ties (equal-utility preference pairs) to introduce data-driven regularization. We demonstrate that this approach selectively reduces spurious learning without degrading causal learning. Finally, we validate our theory on log-linear models and provide empirical evidence that both the spurious learning mechanisms and the benefits of tie training persist for neural networks and large language models.

Abstract

Preference Optimization

Explore similar work

Spurious Correlation Learning in Preference Optimization: Mechanisms, Consequences, and Mitigation via Tie Training

Towards Disentangled Preference Optimization Dynamics: Suppress the Loser, Preserve the Winner

Decomposing the Delta: What Do Models Actually Learn from Preference Pairs?