Help Me Help You: The Aggregate Value of Source and Target Data in Transfer Learning

Steve Hanneke; Samory Kpotufe

stat.MLAug 29, 2024

Help Me Help You: The Aggregate Value of Source and Target Data in Transfer Learning

Authors: Steve Hanneke, Samory Kpotufe

Organizations: Purdue University · Columbia University

Abstract

Transfer learning, in regression or classification, concerns problems where one aims to use imperfect data from a source distribution P to improve prediction w.r.t. a target distribution Q. While there is now much theoretical understanding of the unsupervised setting of this problem (where no labeled target data is available), the supervised setting with labeled target data remains under-studied. We identify and characterize an interesting dichotomy in the rates achievable in the supervised setting: there is a weak transfer regime where optimal rates are determined by information in either the source or target dataset but not both, and a strong transfer regime where much faster target rates are achievable, accounting for complementary information in source and target data. Furthermore, while previous theoretical tools for the unsupervised setting can be extended to capture the weak transfer regime, we argue that they cannot capture the strong regime due to limitations inherent in the various notions of relatedness they rely on. Instead, we provide an appropriate refinement of relatedness notions---termed weak and strong moduli of transfer---that reveal the two regimes and lead to a nontrivial characterization of which pairs of distributions P and Q adhere to which regime. Finally, the above dichotomy reveals a new categorization of existing algorithmic approaches in terms of their optimality or lack-thereof across supervised regimes, and we provide a generic approach that remains nearly optimal in all regimes without prior distributional knowledge.

Figures & tables

Figure 1: The weak modulus $\delta(\epsilon)$ is illustrated in terms of risk minimizers $h^{\!*}_{P},h^{\!*}_{Q}$ . The modulus is defined by the smallest $\mathcal{H}_{Q}(\cdot)$ (ball centered at $h^{\!*}_{Q}$ ) containing $\mathcal{H}_{P}(\epsilon)$ (ellipsoid centered at $h^{\!*}_{P}$ ).

Figure 2: Strong Modulus. The two ellipsoids centered at $h^{\!*}_{P}$ ’s together represent $\mathcal{H}_{P}(\epsilon_{2})$ , while the dotted ball centered at $h^{\!*}_{Q}$ represents $\mathcal{H}_{Q}(\epsilon_{1})$ . The worst-case $Q$ -risk in the inner ellipsoid determines $\delta(\epsilon_{1},\epsilon_{2})$ .

Figure 3: A confidence set $\hat{\mathcal{H}}_{\mu}$ is illustrated here with solid gray boundary; the ball of radius $\epsilon_{\mu}$ centered at $h^{\!*}_{\mu}$ represents $\mathcal{H}_{\mu}(\epsilon_{\mu})$ , while the inner ball represents $\mathcal{H}_{\mu}(\epsilon_{\mu}/C)$ .

Figure 4: Suppose $\mathcal{H}$ consists of one-sided thresholds on the line, $Y=h^{\!*}_{\mu}(x)$ for $\mu\equiv P$ or $Q$ . It’s easy to see that $\mathcal{E}_{Q}(h)$ (mass of disagreement regions) goes up as $h$ approaches $h^{\!*}_{P}$ from the left, and goes down otherwise. The problem therefore has no gap between moduli.

Figure 5: Suppose $P$ is supported in the gray region so that $h^{\!*}_{P,1},h^{\!*}_{P,2}$ are optimal under $P$ for classifying square vs circle . Notice that $h^{\!*}_{P}$ however relies on spurious features; a small amount of data from $Q$ , some falling in the upper-left quadrant, allows rejecting $h^{\!*}_{P,2}$ .

Figure 6: Possible graphs of modulus $\delta(\cdot,\cdot)$ . Shown are possible cross-sections at some values $\epsilon_{2}>\epsilon_{2}^{\prime}$ .

Transfer Learning

Transfer learning studies how models reuse knowledge across tasks, domains, or modalities when target data or computation are limited.

See all 417 papers

Limits of Transfer Learning
Jun 23, 2020Jake Williams, Abel Tadesse, Tyler Sam +2Transfer Learning
Harvey Mudd College, Claremont, CA 91711, USA · Claremont McKenna College, Claremont, CA 91711, USA · Pomona College, Claremont, CA 91711, USA
Sample Complexity of Transfer Learning: An Optimal Transport Approach
May 19, 2026Haoyang Cao, Xin Guo, Wenpin Tang +1Sample Complexity Transfer Learning
Department of Applied Mathematics and Statistics, John Hopkins University, Baltimore, MD, USA · Department of Industrial Engineering and Operations Research, University of California, Berkeley, CA, USA · Department of Industrial Engineering and Operations Research, Columbia University, New York, NY, USA +1
Minimax-optimal Scaling Laws in Ridge Regression via Transfer Learning
Oct 16, 2026Yalda Shabanzadeh, Diyuan Wu, Marco MondelliTransfer Learning Ridge Regression
Institute of Science and Technology Austria (ISTA).

Jun 23, 2020J/K move · Enter open · S save

Limits of Transfer Learning

Jake Williams, Abel Tadesse, Tyler Sam, Huey Sun+1

Harvey Mudd College, Claremont, CA 91711, USA · Claremont McKenna College, Claremont, CA 91711, USA · Pomona College, Claremont, CA 91711, USA

Transfer Learning

Open paper AskPDF arXiv

Transfer learning involves taking information and insight from one problem domain and applying it to a new problem domain. Although widely used in practice, theory for transfer learning remains less well-developed. To address this, we prove several novel results related to transfer learning, showing the need to carefully select which sets of information to transfer and the need for dependence between transferred information and target problems. Furthermore, we prove how the degree of probabilistic change in an algorithm using transfer learning places an upper bound on the amount of improvement possible. These results build on the algorithmic search framework for machine learning, allowing the results to apply to a wide range of learning problems using transfer.

Abstract

Transfer Learning

Explore similar work

Limits of Transfer Learning

Sample Complexity of Transfer Learning: An Optimal Transport Approach

Minimax-optimal Scaling Laws in Ridge Regression via Transfer Learning