Learning Stable Predictors from Weak Supervision under Distribution Shift

Mehrdad Shoeibi; Elias Hossain; Ivan Garibay; Niloofar Yousefi

cs.LGApr 5, 2026

Learning Stable Predictors from Weak Supervision under Distribution Shift

Authors: Mehrdad Shoeibi, Elias Hossain, Ivan Garibay, Niloofar Yousefi

Organizations: University of Central Florida

Abstract

Learning from weak, proxy, or relative supervision is common when ground-truth labels are unavailable, but robustness under distribution shift remains poorly understood because the supervision mechanism itself may change across environments. We formalize this phenomenon as supervision drift, defined as changes in $P(y \mid x, c)$ across contexts, and study it in CRISPR-Cas13d transcriptomic perturbation experiments where guide efficacy is inferred indirectly from RNA-seq responses. Using publicly available data spanning two human cell lines and multiple post-induction timepoints, we construct a controlled non-IID benchmark with explicit domain (cell line) and temporal shifts, while reusing a fixed weak-label construction across all contexts to avoid changing targets. Across linear and tree-based models, weak supervision supports meaningful learning in-domain (ridge $R^2 = 0.356$ , Spearman $ρ= 0.442$ ) and partial cross-cell-line transfer ( $ρ\approx 0.40$ ). In contrast, temporal transfer collapses across all model classes considered, yielding negative $R^2$ and weak or near-zero $ρ$ (ridge $R^2 = -0.145$ , $ρ= 0.008$ ; XGBoost $R^2 = -0.155$ , $ρ= 0.056$ ; random forest $R^2 = -0.322$ , $ρ= 0.139$ ). Additional robustness analyses using externally recomputed weak labels, shift-score quantification, and simple mitigation baselines preserve the same qualitative pattern. Feature-label association and feature-importance analyses remain relatively stable across cell lines but change sharply over time, indicating that failures arise from supervision drift rather than model capacity or simple covariate shift. These results show that strong in-domain performance under weak supervision can be misleading and motivate feature stability as a lightweight diagnostic for non-transferability before deployment.

Figures & tables

Figure 1: Overview of the experimental framework for studying generalization under weak supervision. Transcript-level features are extracted from multiple biological contexts (K562 Day 2, HEK293FT Day 2, and HEK293FT Day 7). A single fixed weak supervision signal $y$ is constructed once from the HEK293FT Day 7 minus Day 2 transcriptomic response and reused across all experiments, ensuring that the target remains unchanged across contexts. Models are trained within a single source context (e.g., HEK293FT Day 2 or K562 Day 2) to predict this fixed weak label, without access to other contexts during training. Generalization is then evaluated under three settings: (i) in-domain (HEK293FT Day 2 $\rightarrow$ HEK293FT Day 2), (ii) cross-domain (K562 Day 2 $\rightarrow$ HEK293FT Day 2), and (iii) temporal (HEK293FT Day 2 $\rightarrow$ HEK293FT Day 7). By holding the supervision signal fixed and varying only the feature context, this design isolates changes in the feature–supervision relationship. While partial transfer is observed across cell lines, temporal generalization fails, revealing supervision drift, where the conditional relationship $P(y\mid x,c)$ changes across contexts.

Setting	Training features	Test features	Target
In-domain	HEK293FT Day 2	HEK293FT Day 2	fixed $\mathbf{y}$
Cross-domain	K562 Day 2	HEK293FT Day 2	fixed $\mathbf{y}$
Temporal	HEK293FT Day 2	HEK293FT Day 7	fixed $\mathbf{y}$

Table 1: Data assignments for the three evaluation settings. The weak supervision target

\mathbf{y}

is the same fixed transcript-level vector in all cases.

Figure 2: In-domain predictability under weak supervision (HEK293FT, Day 2). Performance is reported using $R^{2}$ , mean absolute error (MAE; lower is better), and Spearman rank correlation $\rho$ . Ridge regression and a depth-capped random forest achieve comparable in-domain performance, indicating that transcript-level features contain recoverable relative signal under indirect supervision.

Figure 3: Cross-domain generalization from K562 to HEK293FT at Day 2. Performance is reported using

R^{2}

(solid bars) and Spearman rank correlation

\rho

(hatched bars). Both ridge regression (

R^{2}=0.331

\rho=0.399

) and a depth-capped random forest (

R^{2}=0.310

\rho=0.364

) retain a substantial portion of their in-domain predictive signal, indicating partial cross-cell-line transferability under weak supervision.

Figure 4: Temporal generalization heatmap from Day 2 to Day 7 in HEK293FT. Performance is reported using $R^{2}$ and Spearman rank correlation $\rho$ (higher is better). All model families considered, including ridge regression, random forest (depth-capped), and XGBoost, exhibit severe performance degradation, indicating consistent failure to transfer predictive signals across time under weak supervision.

Context	Feature	Spearman Corr.	$n$
K562_D2	logTPM	$-0.282$	251,955
K562_D2	rank_pct_within_sample	$-0.282$	251,955
HEK293FT_D2	logTPM	$-0.468$	251,955
HEK293FT_D2	rank_pct_within_sample	$-0.468$	251,955
HEK293FT_D7	logTPM	$-0.001$	251,955
HEK293FT_D7	rank_pct_within_sample	$-0.001$	251,955

Table 2: Feature–label Spearman correlations across biological contexts. Correlations are computed between transcript-level features (logTPM and within-sample rank percentiles) and weak supervision signals in each context. Feature relevance is consistent across cell lines at Day 2 but collapses at Day 7, indicating a breakdown in the feature–supervision relationship under temporal shift.

Feature	Context Pair	$\|\Delta$ Corr. $\|$
logTPM	K562_D2 vs HEK293FT_D2	0.186
logTPM	HEK293FT_D2 vs HEK293FT_D7	0.468
rank_pct_within_sample	K562_D2 vs HEK293FT_D2	0.186
rank_pct_within_sample	HEK293FT_D2 vs HEK293FT_D7	0.468

Table 3: Feature stability across structured distribution shifts, measured as the absolute difference in feature–label Spearman correlation between context pairs. Feature relevance remains relatively stable under cross-domain shift (K562 Day 2 vs. HEK293FT Day 2) but exhibits substantially larger instability under temporal shift (HEK293FT Day 2 vs. Day 7), suggesting supervision drift rather than feature-specific effects.

Setting	Model	$R^{2}$	Spearman $\rho$
In-domain (HEK D2)	Ridge	0.051	0.205
In-domain (HEK D2)	RandomForest	0.081	0.131
Cross-domain (K562 $\rightarrow$ HEK)	Ridge	-0.227	-0.205
Cross-domain (K562 $\rightarrow$ HEK)	RandomForest	-0.363	-0.033
Temporal (D2 $\rightarrow$ D7)	Ridge	0.006	0.064
Temporal (D2 $\rightarrow$ D7)	RandomForest	-0.118	-0.029

Table 4: Performance using externally recomputed weak labels. The qualitative behavior remains unchanged: positive in-domain performance, substantial degradation under cross-domain transfer, and collapse under temporal transfer.

Model	$\rho(\text{ShiftScore},\text{Performance})$	Context Pairs
Ridge	-0.239	6
RandomForest	-0.478	6

Table 5: Correlation between supervision drift magnitude and predictive performance. Larger shifts in feature–label association correspond to worse generalization.

Setting	Model Variant	$R^{2}$	Spearman $\rho$
In-domain	Plain	0.359	0.444
In-domain	Context-aware	0.359	0.444
Cross-domain	Plain	0.331	0.399
Cross-domain	Context-aware	0.331	0.399
Temporal	Plain	-0.145	0.008
Temporal	Context-aware	-0.145	0.008

Table 6: Context-aware modeling does not improve temporal generalization.

Figure 5: Feature ablation study across evaluation scenarios. Performance is shown for expression-based features (logTPM only), rank-based features (rank only), and their combination (both), evaluated under in-domain, cross-domain, and temporal settings. Left: coefficient of determination (

R^{2}

). Middle: mean absolute error (MAE; lower is better). Right: Spearman rank correlation (

\rho

). For comparisons between logTPM-only and rank-only feature sets, the most informative metrics are

R^{2}

and MAE, since Spearman correlation is invariant to monotone transformations within a context. Across all feature representations, temporal generalization fails, indicating that supervision instability, not feature choice alone, limits transfer over time.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Evaluation Setting	Features	$R^{2}$	MAE	Spearman $\rho$
In-domain (HEK293FT D2)	logTPM	0.377	0.171	0.386
Cross-domain (K562 $\rightarrow$ HEK293FT)	logTPM	0.340	0.170	0.393
Temporal (HEK293FT D2 $\rightarrow$ D7)	logTPM	-0.155	0.225	0.056

Appendix

Table 7: XGBoost baseline results under weak supervision. Despite improved in-domain and cross-domain performance relative to linear models, temporal generalization remains poor, indicating that increased model capacity does not resolve temporal distribution shift.

Setting	Model	$R^{2}$	Spearman $\rho$
In-domain (HEK D2)	Ridge	0.051	0.205
In-domain (HEK D2)	RandomForest	0.081	0.131
Cross-domain (K562 $\rightarrow$ HEK)	Ridge	-0.227	-0.205
Cross-domain (K562 $\rightarrow$ HEK)	RandomForest	-0.363	-0.033
Temporal (D2 $\rightarrow$ D7)	Ridge	0.006	0.064
Temporal (D2 $\rightarrow$ D7)	RandomForest	-0.118	-0.029

Appendix

Table 8: Performance under externally recomputed weak labels. The same qualitative failure pattern persists.

Model	$\rho(\text{ShiftScore},\text{Performance})$	Context Pairs
Ridge	-0.239	6
RandomForest	-0.478	6

Appendix

Table 9: Correlation between shift score and predictive performance. Larger supervision drift corresponds to worse generalization.

Setting	Model Variant	$R^{2}$	Spearman $\rho$
In-domain	Plain	0.359	0.444
In-domain	Context-aware	0.359	0.444
Cross-domain	Plain	0.331	0.399
Cross-domain	Context-aware	0.331	0.399
Temporal	Plain	-0.145	0.008
Temporal	Context-aware	-0.145	0.008

Appendix

Table 10: Context-aware modeling does not improve temporal generalization.

Feature Set	Context Pair	SHAP Rank Corr.	Note
logTPM	K562 D2 $\rightarrow$ HEK293FT D2	–	Single feature
logTPM	HEK293FT D2 $\rightarrow$ HEK293FT D7	–	Single feature
Rank pct.	K562 D2 $\rightarrow$ HEK293FT D2	–	Single feature
Rank pct.	HEK293FT D2 $\rightarrow$ HEK293FT D7	–	Single feature
Both	K562 D2 $\rightarrow$ HEK293FT D2	High	Stable ordering
Both	HEK293FT D2 $\rightarrow$ HEK293FT D7	Moderate	Temporal instability

Appendix

Table 11: SHAP rank correlation diagnostics across contexts. Rank-based agreement is undefined for single-feature models. For the combined feature representation (logTPM + within-sample rank percentile), SHAP enables rank-based comparison, revealing stable importance ordering under cross-domain shift and partial instability under temporal shift.

Sample	Percent Mapped (%)	Mapped Reads	Processed Reads	Notes
HEK293FT D2	87.18	431M	495M	High-quality mapping
K562 D2	86.41	376M	435M	High-quality mapping
HEK293FT D7	71.66	52.6M	73.4M	Moderate mapping
Long-term (SRX26635950)	0.02	39K	169M	Failed mapping
Long-term (SRX27451361)	0.00	0	34M	Failed mapping

Appendix

Table 12: Salmon mapping summary across samples. Long-term samples exhibit near-zero mapping rates and were excluded from downstream modeling, while short-term samples demonstrate consistently high alignment quality.

Distribution Shift Robustness

Distribution shift robustness studies how to keep machine-learning predictions reliable when deployment data differ from training data.

See all 280 papers

May 25, 2026

When In-Distribution Gains Fail: Evaluating Weak-to-Strong Reward Models under Preference Shift

Weak-to-strong (W2S) generalization is a promising framework for scalable oversight, yet existing evaluations often test students under matched train-test distributions. Therefore, we study W2S preference learning under zero-shot distribution shift and find that strong students trained on weak preference labels can appear successful in-distribution while failing to transfer across preference datasets. We provide evidence for a representational failure mode in which weak-supervised fine-tuning can pull the strong model toward source-domain features instead of maintaining broadly transferable preference representations. To mitigate this, we propose Representation Anchoring (Anchor), a simple yet effective regularizer that constrains excessive drift from the pretrained strong model's representation space during fine-tuning, while still allowing task-relevant adaptation. Across preference domains, datasets, and model families, Anchor consistently improves out-of-distribution transfer while maintaining competitive in-distribution performance. Together, our evaluation protocol, transfer-aware metrics, and method expose hidden brittleness in current W2S reward modeling and provide a practical path toward more robust preference transfer.

Khoi Le, Tri Cao, Phong Nguyen +5

National University of Singapore · VinUniversity · Nanyang Technological University

Pairwise Preference Learning Reward Modeling Representation Learning OOD Generalization Weak-to-Strong Generalization Preference Learning

Ask PDF

May 29, 2026

Weak Critics Make Strong Learners: On-Policy Critique Distillation for Scalable Oversight

As large language models become stronger, weak supervisors may fail to provide reliable labels, preferences, or final judgments for complex outputs, limiting both weak-to-strong generalization and scalable oversight. We study a more tractable form of weak supervision: using a weak model as a critic rather than as a labeler or judge. Instead of solving the task or selecting the correct answer, the weak critic only needs to provide a non-misleading revision direction that helps the strong model better use its own knowledge. We call this setting weak-critic strong oversight. We first show that weak critiques can improve frozen strong models at inference time, and that critique quality is key to this improvement. We then propose progressive on-policy critique distillation (OPCD), which filters high-quality critiques and distills critic-guided behavior into the strong model through adaptive self-teacher signals. Experiments on reasoning and alignment benchmarks show that our method improves strong models over training epochs, suggesting an effective path for scalable oversight with weak supervision.

Can Jin, Jiakang Li, Rui Wu +2

Department of Computer Science, Rutgers University, Piscataway, United States.

Scalable Oversight Weak-to-Strong Generalization Weakly Supervised Learning Language Model Distillation On-Policy Distillation Human Oversight of AI

Ask PDF

Nov 28, 2025

A Unified and Stable Risk Minimization Framework for Weakly Supervised Learning with Theoretical Guarantees

Weakly supervised learning has emerged as a practical alternative to fully supervised learning when complete and accurate labels are costly or infeasible to acquire. However, many existing methods are tailored to specific supervision patterns -- such as positive-unlabeled (PU), unlabeled-unlabeled (UU), complementary-label (CLL), partial-label (PLL), or similarity-unlabeled annotations -- and rely on post-hoc corrections to mitigate instability induced by indirect supervision. We propose a principled, unified framework that bypasses such post-hoc adjustments by directly formulating a stable surrogate risk grounded in the structure of weakly supervised data. The formulation naturally subsumes diverse settings -- including PU, UU, CLL, PLL, multi-class unlabeled, and tuple-based learning -- under a single optimization objective. We further establish a non-asymptotic generalization bound via Rademacher complexity that clarifies how supervision structure, model capacity, and sample size jointly govern performance. Beyond this, we analyze the effect of class-prior misspecification on the bound, deriving explicit terms that quantify its impact, and we study identifiability, giving sufficient conditions -- most notably via supervision stratification across groups -- under which the target risk is recoverable. Extensive experiments show consistent gains across class priors, dataset scales, and class counts -- without heuristic stabilization -- while exhibiting robustness to overfitting.

Miao Zhang, Junpeng Li, Changchun Hua +1

Engineering Research Center of the Ministry of Education for Intelligent Control System and Intelligent Equipment, Yanshan University, Qinhuangdao, China

Weakly Supervised Learning Statistical Learning Theory Positive-Unlabeled Learning Generalization Bounds

Ask PDF

Abstract

Distribution Shift Robustness

Explore similar work

When In-Distribution Gains Fail: Evaluating Weak-to-Strong Reward Models under Preference Shift

Weak Critics Make Strong Learners: On-Policy Critique Distillation for Scalable Oversight

A Unified and Stable Risk Minimization Framework for Weakly Supervised Learning with Theoretical Guarantees