Help Me Help You: The Aggregate Value of Source and Target Data in Transfer Learning

Steve Hanneke; Samory Kpotufe

stat.MLAug 29, 2024

Help Me Help You: The Aggregate Value of Source and Target Data in Transfer Learning

Authors: Steve Hanneke, Samory Kpotufe

Organizations: Purdue University · Columbia University

Abstract

Transfer learning, in regression or classification, concerns problems where one aims to use imperfect data from a source distribution P to improve prediction w.r.t. a target distribution Q. While there is now much theoretical understanding of the unsupervised setting of this problem (where no labeled target data is available), the supervised setting with labeled target data remains under-studied. We identify and characterize an interesting dichotomy in the rates achievable in the supervised setting: there is a weak transfer regime where optimal rates are determined by information in either the source or target dataset but not both, and a strong transfer regime where much faster target rates are achievable, accounting for complementary information in source and target data. Furthermore, while previous theoretical tools for the unsupervised setting can be extended to capture the weak transfer regime, we argue that they cannot capture the strong regime due to limitations inherent in the various notions of relatedness they rely on. Instead, we provide an appropriate refinement of relatedness notions---termed weak and strong moduli of transfer---that reveal the two regimes and lead to a nontrivial characterization of which pairs of distributions P and Q adhere to which regime. Finally, the above dichotomy reveals a new categorization of existing algorithmic approaches in terms of their optimality or lack-thereof across supervised regimes, and we provide a generic approach that remains nearly optimal in all regimes without prior distributional knowledge.

Figures & tables

Figure 1: The weak modulus $\delta(\epsilon)$ is illustrated in terms of risk minimizers $h^{\!*}_{P},h^{\!*}_{Q}$ . The modulus is defined by the smallest $\mathcal{H}_{Q}(\cdot)$ (ball centered at $h^{\!*}_{Q}$ ) containing $\mathcal{H}_{P}(\epsilon)$ (ellipsoid centered at $h^{\!*}_{P}$ ).

Figure 2: Strong Modulus. The two ellipsoids centered at $h^{\!*}_{P}$ ’s together represent $\mathcal{H}_{P}(\epsilon_{2})$ , while the dotted ball centered at $h^{\!*}_{Q}$ represents $\mathcal{H}_{Q}(\epsilon_{1})$ . The worst-case $Q$ -risk in the inner ellipsoid determines $\delta(\epsilon_{1},\epsilon_{2})$ .

Figure 3: A confidence set $\hat{\mathcal{H}}_{\mu}$ is illustrated here with solid gray boundary; the ball of radius $\epsilon_{\mu}$ centered at $h^{\!*}_{\mu}$ represents $\mathcal{H}_{\mu}(\epsilon_{\mu})$ , while the inner ball represents $\mathcal{H}_{\mu}(\epsilon_{\mu}/C)$ .

Figure 4: Suppose $\mathcal{H}$ consists of one-sided thresholds on the line, $Y=h^{\!*}_{\mu}(x)$ for $\mu\equiv P$ or $Q$ . It’s easy to see that $\mathcal{E}_{Q}(h)$ (mass of disagreement regions) goes up as $h$ approaches $h^{\!*}_{P}$ from the left, and goes down otherwise. The problem therefore has no gap between moduli.

Figure 5: Suppose $P$ is supported in the gray region so that $h^{\!*}_{P,1},h^{\!*}_{P,2}$ are optimal under $P$ for classifying square vs circle . Notice that $h^{\!*}_{P}$ however relies on spurious features; a small amount of data from $Q$ , some falling in the upper-left quadrant, allows rejecting $h^{\!*}_{P,2}$ .

Figure 6: Possible graphs of modulus $\delta(\cdot,\cdot)$ . Shown are possible cross-sections at some values $\epsilon_{2}>\epsilon_{2}^{\prime}$ .

Transfer Learning

Transfer learning studies how models reuse knowledge across tasks, domains, or modalities when target data or computation are limited.

See all 416 papers

Jun 23, 2020

Limits of Transfer Learning

Transfer learning involves taking information and insight from one problem domain and applying it to a new problem domain. Although widely used in practice, theory for transfer learning remains less well-developed. To address this, we prove several novel results related to transfer learning, showing the need to carefully select which sets of information to transfer and the need for dependence between transferred information and target problems. Furthermore, we prove how the degree of probabilistic change in an algorithm using transfer learning places an upper bound on the amount of improvement possible. These results build on the algorithmic search framework for machine learning, allowing the results to apply to a wide range of learning problems using transfer.

Jake Williams, Abel Tadesse, Tyler Sam +2

Harvey Mudd College, Claremont, CA 91711, USA · Claremont McKenna College, Claremont, CA 91711, USA · Pomona College, Claremont, CA 91711, USA

Transfer Learning

Ask PDF

May 19, 2026

Sample Complexity of Transfer Learning: An Optimal Transport Approach

Transfer learning is an essential technique for many machine learning/AI models of complex structures such as large language models and generative AI. The essence of transfer learning is to leverage knowledge from resolved source tasks for a new target task, especially when the sample size

m

of the training data for the latter is low. In this work, we rigorously analyze the potential benefit of transfer learning in terms of sample efficiency. Specifically, taking an optimal transport viewpoint of transfer learning, we find that when the data dimension

d

is higher than

3

, the sample complexity for transfer learning is

O(m^{-(α+1)/d})

, with

α

indicating the smoothness of the data distribution, as opposed to the

O(m^{-p/d})

sample complexity for direct learning with

p

indicating the smoothness of the optimal target model. Our finding theoretically supports a better sample efficiency for transfer learning, when the target task is optimizing over a family of not-so-smooth models (i.e., highly complex networks with the possible use of non-smooth activation functions). Using image classification as an example, we numerically demonstrate the sample efficiency for transfer learning, that is, in the data hungry regime, the model performance can be significantly improved by transfer learning.

Haoyang Cao, Xin Guo, Wenpin Tang +1

Department of Applied Mathematics and Statistics, John Hopkins University, Baltimore, MD, USA · Department of Industrial Engineering and Operations Research, University of California, Berkeley, CA, USA · Department of Industrial Engineering and Operations Research, Columbia University, New York, NY, USA +1

Sample Complexity Transfer Learning Statistical Learning Theory Data-Efficient Learning

Ask PDF

Oct 16, 2026

Minimax-optimal Scaling Laws in Ridge Regression via Transfer Learning

Pretrained models are often reused across downstream tasks, motivating the question of whether this transfer learning procedure can provide a substantial performance improvement, namely, in terms of the scaling law. We address this question in the prototypical setting of high-dimensional regression, where (i) a pretrained source model is trained via ridge regression, and (ii) a fine-tuned target model is regularized via the

L_2

distance from the source estimator. First, we establish a deterministic equivalent for the risk of this two-step procedure under both model and covariance shift. Next, by specializing the deterministic equivalent to a shared regression function and commuting power-law covariance spectra, we identify a regime where the resulting scaling law achieves the minimax-optimal exponent. Notably, in this regime, the optimal source regularization depends on source data only, indicating its universality across all targets. Furthermore, we show that the same optimal rate is achieved by performing the fine-tuning step via one-pass stochastic gradient descent. We support our theoretical analysis via experiments on both synthetic settings and language models.

Yalda Shabanzadeh, Diyuan Wu, Marco Mondelli

Institute of Science and Technology Austria (ISTA).

Transfer Learning Ridge Regression High-Dimensional Linear Regression Minimax Estimation

Ask PDF

Abstract

Transfer Learning

Explore similar work

Limits of Transfer Learning

Sample Complexity of Transfer Learning: An Optimal Transport Approach

Minimax-optimal Scaling Laws in Ridge Regression via Transfer Learning