On the Power of Source Screening for Learning Shared Feature Extractors

Data collection is a critical component of modern statistical and machine learning pipelines, particularly when data must be gathered from multiple heterogeneous sources to study a target population of interest. In many use cases, such as medical studies or political polling, different sources incur different sampling costs. Observations often have associated group identities - for example, health markers, demographics, or political affiliations - and the relative composition of these groups may differ substantially, both among the source populations and between sources and target population. In this work, we study multi-source data collection under a fixed budget, focusing on the estimation of population means and group-conditional means. We show that naive data collection strategies (e.g. attempting to "match" the target distribution) or relying on standard estimators (e.g. sample mean) can be highly suboptimal. Instead, we develop a sampling plan which maximizes the effective sample size - the total sample size divided by $D_{χ^2}(q\mid\mid\overline{p}) + 1$, where $q$ is the target distribution, $\overline{p}$ is the aggregated source distribution, and $D_{χ^2}$ is the $χ^2$-divergence. We pair this sampling plan with a classical post-stratification estimator and upper bound its risk. We provide matching lower bounds, establishing that our approach achieves the budgeted minimax optimal risk. Our techniques also extend to prediction problems when minimizing the excess risk, providing a principled approach to multi-source learning with costly and heterogeneous data sources.

Spectral features can remain optimal under strongly uneven task preferences when they retain the directions most useful to the tasks. In a local-task model, expected probing gain depends on the task prior only through its covariance $Λ$: with $B$ recording dependence between views, $k$ spectral features span the leading eigenspace of $BB^\top$, while task-optimal features span that of $BΛB^\top$. We derive alignment-dependent regret bounds, matching worst-case lower bounds for a flat leading spectrum, and bounds using the leading $2k$ directions with a spectral-tail term; a spectral gap makes the bound quadratic in small anisotropy. We also bound the imbalance from random preference patterns and from averaging independent tasks with an isotropic population covariance. With known $Λ$, changing one term of the spectral contrastive loss selects task-optimal features; with a labelled task bank, we propose a diagnostic and test correction without retraining. Correction reduces synthetic held-out regret from $0.858$ to $0.003$ with $2000$ tasks when less task-relevant directions dominate, and on CIFAR-100 lowers empirical regret by $0.031$--$0.053$ on held-out fine-label tasks using the same images. With a small task bank, however, correction can hurt.

Mid-training has become an important stage in modern LLM development, using large-scale curated mixtures to strengthen capabilities before final post-training. Its data selection problem is distinct: the data are optimized under a pretraining-style objective at near-pretraining scale, but are curated toward downstream capabilities and drawn from heterogeneous sources with different formats and training roles. As a result, effective selection requires both scalability and source-adaptive semantic criteria. Existing model-based methods scale well, but provide only implicit quality signals. Semantic selection methods offer stronger judgments, but usually assume fixed rubrics or standardized data formats. To address this mismatch, we propose MIRA, a source-aware filtering framework based on self-anchored rubric discovery. The key idea is to make rubric construction part of data selection: MIRA first discovers what should be evaluated for each source group, then distills those judgments into scalable student scorers for full-corpus filtering. On code-oriented mid-training with 21 sources and 5 source groups, MIRA outperforms selection baselines across nine code benchmarks and matches the full-corpus run while using only half the tokens.