Dynamics of Learning under User Choice: Overspecialization and Peer-Model Probing

Adhyyan Narang; Sarah Dean; Lillian J Ratliff; Maryam Fazel

cs.LGFeb 27, 2026

Dynamics of Learning under User Choice: Overspecialization and Peer-Model Probing

Authors: Adhyyan Narang, Sarah Dean, Lillian J Ratliff, Maryam Fazel

Organizations: Electrical and Computer Engineering, University of Washington · Computer Science, Cornell University

Abstract

In many economically relevant contexts where machine learning is deployed, multiple platforms obtain data from the same pool of users, each of whom selects the platform that best serves them. Prior work in this setting focuses exclusively on the "local" losses of learners on the distribution of data that they observe. We find that there exist instances where learners who use existing algorithms almost surely converge to models with arbitrarily poor global performance, even when models with low full-population loss exist. This happens through a feedback-induced mechanism, which we call the overspecialization trap: as learners optimize for users who already prefer them, they become less attractive to users outside this base, which further restricts the data they observe. Inspired by the recent use of knowledge distillation in modern ML, we propose an algorithm that allows learners to "probe" the predictions of peer models, enabling them to learn about users who do not select them. Our analysis characterizes when probing succeeds: this procedure converges almost surely to a stationary point with bounded full-population risk when probing sources are sufficiently informative, e.g., a known market leader or a majority of peers with good global performance. We verify our findings with semi-synthetic experiments on the MovieLens, Census, and Amazon Sentiment datasets.

Figures & tables

Figure 1: Illustration of our online multi-learner problem setting. The borders of users represent their highest ranked learner

\pi(z)

. For further details, see Section 3 .

Scenario	What $i$ knows	Peer requirement	$T_{i}(x)$	$B$
Majority-good	Nothing	$>50\%$ in $B_{r}(\theta^{\ast})$	$[m]$	$R^{2}r^{2}+2\epsilon$
Market-leader	Identity of $j^{\ast}$	$\mathcal{R}(\theta^{0}_{j^{\ast}})\leq\xi$	$\{j^{\ast}\}$	$\xi$
Partial knowledge	Subset $G$	$>50\%$ of $G$ in $B_{r}(\theta^{\ast})$	$G$	$R^{2}r^{2}+2\epsilon$
Preference-aware	$\pi(x)$	Nothing	$\{\pi(x)\}$	$\epsilon$

Table 1: Probing scenarios with corresponding rules and accuracy bounds. The preference-aware scenario is notable: it requires no assumption on peer quality, only knowledge of user preferences.

Figure 2: MSGD full-population performance with random initialization (Preference-aware scenario). Left: Census test accuracy. Mid: Amazon sentiment test accuracy Right: MovieLens test loss. The dashed black line represents the performance of a baseline $\theta^{\ast}$ trained on the full dataset. In all cases, $\tau=0.3$ ; other dataset-specific hyperparameters are given in Table 3 .

Figure 3: Effect of probing on full-population performance (Preference-aware scenario). Left: Census final accuracy vs probing weight $p$ . Mid: Amazon sentiment final accuracy vs probing weight $p$ Right: MovieLens final loss vs $p$ . In all cases, triangle markers indicate the probing learner. Here $\tau=0.7$ ; other dataset-specific hyperparameters are given in Table 3 .

Figure 4: Effect of noisy probing-source selection. Preference-aware probing routes each probe query to the preferred peer $\pi(x)$ with probability $1-\kappa$ and to a random peer with probability $\kappa$ . Left: Census final test accuracy vs. probing weight $p$ . Middle: Amazon sentiment final test accuracy vs. $p$ . Right: MovieLens final test MSE vs. $p$ . The green learner is the probing learner; its multiple curves show different $\kappa$ values.

Figure 5: Performance of probing learner on Census as a function of

n

. Error bars show one standard deviation over 10 random seeds.

Figure 6: Census experiments with two-layer neural learners. We replace each linear logistic learner with a two-layer ReLU MLP (

d\!\to\!64\!\to\!1

) trained by SGD with binary cross-entropy, while keeping the Census train/test split, K-means preference clustering, induced learner rankings, and offline pseudo-label construction fixed. Left: random initialization with standard MSGD and no probing (

\tau=0.3,p=0

). Right: partition-pretrained initialization with preference-aware probing by Learner 2 (

\tau=0.7,\kappa=0

). Shaded bands show standard error over three seeds.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Scenario	$T_{i}(x)$	$B$
Majority-good	$[m]$	$\epsilon+2Rr$
Market-leader	$\{j^{\ast}\}$	$\xi$
Partial knowledge	$G$	$\epsilon+2Rr$
Preference-aware	$\{\pi(x)\}$	$\epsilon$

Appendix

Table 2: Probing accuracy parameters for cross-entropy loss.

Figure 7: Performance of probing learner on Census as a function of

n

. Error bars show one standard deviation over 10 random seeds.

Parameter	Description	Census	MovieLens	Amazon
$m$	Number of learners	5	5	9
$T$	Total rounds	4000	4000	20000
$\lambda$	L2 regularization	$10^{-9}$	$10^{-3}$	$10^{-9}$
$n$	Offline probe dataset size	100	1000	500

Appendix

Table 3: Hyperparameters by dataset. Shared values are merged across columns.

Figure 8: MSGD full-population performance with random initialization (Market-leader scenario). Left: Census test accuracy. Right: MovieLens test loss. Here $\tau=0.3$ .

Figure 9: Effect of probing on full-population performance (Market-leader scenario) Left: Final accuracy vs probing weight $p$ on Census. Right: MovieLens final loss vs $p$ . The triangle markers indicate Learner $4$ probes the market leader, learner $0$ . Here $\tau=0.7$ .

Figure 10: MSGD full-population performance with random initialization (Majority good scenario). Left: Census test accuracy. Right: MovieLens test loss. Here $\tau=0.3$ .

Figure 11: Effect of probing on full-population performance (Majority Good Scenario) Left: Final accuracy vs probing weight $p$ on Census. Right: MovieLens final loss vs $p$ . Triangle markers indicate Learner $4$ is probing via median aggregation over all peers. Here $\tau=0.7$ .

Figure 12: Effect of probing on full-population performance when multiple learners probe (Preference-aware scenario). Census final accuracy vs probing weight $p$ . Triangle markers indicate the probing learners (Learners $2$ and $3$ ). The dashed black line denotes the full-data baseline.

Spurious Correlation Learning in Preference Optimization: Mechanisms, Consequences, and Mitigation via Tie Training
May 11, 2026Christian Moya, Alex Semendinger, Guang Lin +1Pairwise Preference Learning LLM Alignment
Department of Mathematics, Purdue University, West Lafayette IN, USA
Bandits with Probing: Optimal Coverage and the Role of Feedback
Sep 14, 2026Yongjie GuanMulti-Armed Bandits Minimax Regret
Zhejiang University of Technology
Global Federated Learning Strategies for Building Efficient Personalized Models
Aug 15, 2026Seongyoon KimFederated Learning Heterogeneous Federated Learning
한국과학기술원 Korea Advanced Institute of Science and Technology 산업및시스템공학과

May 11, 2026J/K move · Enter open · S save

Spurious Correlation Learning in Preference Optimization: Mechanisms, Consequences, and Mitigation via Tie Training

Christian Moya, Alex Semendinger, Guang Lin, Elliott Thornley

Department of Mathematics, Purdue University, West Lafayette IN, USA

Pairwise Preference Learning LLM Alignment Preference Optimization Spurious Correlation Robustness Preference Learning Spurious Correlation

Open paper AskPDF arXiv

Preference learning methods like Direct Preference Optimization (DPO) are known to induce reliance on spurious correlations, leading to sycophancy and length bias in today's language models and potentially severe goal misgeneralization in future systems. In this work, we provide a unified theoretical analysis of this phenomenon, characterizing the mechanisms of spurious learning, its consequences on deployment, and a provable mitigation strategy. Focusing on log-linear policies, we show that standard preference-learning objectives induce reliance on spurious features at the population level through two channels: mean spurious bias and causal-spurious correlation leakage. We then show that this reliance creates an irreducible vulnerability to distribution shift: more data from the same training distribution fails to reduce the model's dependence on spurious features. To address this, we propose tie training, a data augmentation strategy using ties (equal-utility preference pairs) to introduce data-driven regularization. We demonstrate that this approach selectively reduces spurious learning without degrading causal learning. Finally, we validate our theory on log-linear models and provide empirical evidence that both the spurious learning mechanisms and the benefits of tie training persist for neural networks and large language models.

Abstract

Explore similar work

Spurious Correlation Learning in Preference Optimization: Mechanisms, Consequences, and Mitigation via Tie Training

Bandits with Probing: Optimal Coverage and the Role of Feedback

Global Federated Learning Strategies for Building Efficient Personalized Models