We study transfer learning for a linear regression task using several least-squares pretrained models that can be overparameterized. We formulate the target learning task as optimization that minimizes squared errors on the target dataset with penalty on the distance of the learned model from the pretrained models. We analytically formulate the test error of the learned target model and provide the corresponding empirical evaluations. Our results elucidate when using more pretrained models can improve transfer learning. Specifically, if the pretrained models are overparameterized, using sufficiently many of them is important for beneficial transfer learning. However, the learning may be compromised by overparameterization bias of pretrained models, i.e., the minimum ℓ2-norm solution's restriction to a small subspace spanned by the training examples in the high-dimensional parameter space. We propose a simple debiasing via multiplicative correction factor that can reduce the overparameterization bias and leverage more pretrained models to learn a target predictor.
Figures & tables
Figure 1: Test error in the general case of Theorem 4.4 . Here, Hj corresponds to energy preserving subspace of dimension 2d (see Appendix E.1 ), and the assumed task relation is Hj=Id . Target covariance is Σx=Id (left subfig.) and exponential decay (Σx)il=0.5∣i−l∣ (right subfig.) See Appendix Q.1 for more experiments.
Figure 2: Test error in the simple case of Theorem 5.2 . Hj=Hj=Id . See Appendix Q.2 for more experiments.
Figure 3: Test error under debiasing. In Fig. 3(a) , Hj=Id , and in Fig. 3(b) Hj is a subspace projection of dimension 43 (for more details see Appendix E.1 ). The assumed task relation is Hj=ρjId . Target covariance is Σx=Id in both figures. For more figures see Appendix Q.3 .
Figure 4: Bias-variance decomposition of the test error. Dashed and dotted lines denote the bias and variance terms, respectively. Without debiasing uses Hj=Id ; with debiasing uses Hj=ρjId . See Appendix M for a detailed discussion and more results.
Figure 5: Difference in target test error between transfer learning with and without debiasing . Negative values denote beneficial debiasing. Here, Hj=Id ; see Fig. Q.4 for other task relations.
Figure 6: Empirical tuning of the shrinkage factor. The figure shows the expected test error (left) and the empirically selected factor ρ~ (right) as a function of the source parametrization γsrc . The cyan dotted line represents the isotropic baseline ρ=1/γsrc . Here, Hj corresponds to energy-preserving subspace (dimension 43d ), the assumed task relation is Hj=ρ~Id and the covariance is (Σx)il=(Σz)il=0.5∣i−l∣ . For other settings, see Fig. Q.5 .
Algorithm 1 Transfer learning with overparameterization debiasing for unknown task relations under isotropic source assumption
Figure L.1 : Difference in expected test error between transfer learning with and without debiasing in the source overparametrized regime. The error difference is examined for a varying source task overparameterization level and for several noise levels σξ2 , ση2 . A negative error difference implies that debiasing is beneficial. In Figs. 1(a) and 1(b) , the true relation is Hj=Id . The assumed relations are Hj=Id (without debiasing) and Hj=dn~jId (with debiasing). In Figs. 1(c) and 1(d) , the true relation matrices are circulant with κc=1,000 . For Fig. 1(c) , we use the matched assumptions: Hj=Hj (without debiasing) and Hj=dn~jHj (with debiasing). Conversely, for Fig. 1(d) , the assumed relation matrices are identical to those in Figs. 1(a) and 1(b) . (More figures with different noises and task relations can be seen in Appendix ).
Figure L.2 : Comparison between different assumed relation matrices under ill-condition of true relation matrix. In all figures Hj is a circulant matrix with condition number κc=1000 and σξ2=ση2=0.1 , γtgt=4 . Left Column displays using the identity matrix or the scaled identity matrix for without and with debiasing, respectively, and the right column displays using the true relation matrix or the scaled true relation matrix for without and with debiasing, respectively. First row displays without debiasing and the second is with. The test error difference between 3(a) - 3(b) and 3(c) - 3(d) is provided in Fig. M.3 .
Figure L.3 : Comparison between different assumed relation matrices under ill-condition of true relation matrix. In all figures Hj is a circulant matrix with condition number κc=1000 and σξ2=ση2=0.1 , γtgt=4 . Left Column displays using the identity matrix or the scaled identity matrix for without and with debiasing, respectively, and the right column displays using the true relation matrix or the scaled true relation matrix for without and with debiasing, respectively. First row displays without debiasing and the second is with. The test error difference between 3(a) - 3(b) and 3(c) - 3(d) is provided in Fig. M.3 .
Figure M.1 : Bias-variance decomposition of the expected test error of Fig. L.3 . The test error difference between 3(a) - 3(b) and 3(c) - 3(d) is provided in Fig. M.3 .
Figure M.2 : Bias-variance decomposition of the expected test error of Fig. L.3 . The test error difference between 3(a) - 3(b) and 3(c) - 3(d) is provided in Fig. M.3 .
Figure M.3 : Expected test error difference between using the true relation matrix Hj to using the identity matrix Id as Hj in transfer learning without debiasing in M.3 , and using ρjHj and ρjId when debiasing in M.3 corresponding to the figures in L.3 . Positive difference implies that using Id is better than using the true H .
Algorithm 2 Transfer learning with a validation-tuned debiasing factor for unknown anisotropic sources
Figure Q.1 : Test error in the general case of Theorem 4.4 . The results are shown for: (a) Hj corresponds to energy preserving subspace of dimension 43d with Hj=Id ; (b) Hj corresponds to subspace of dimension 43d with Hj=Id ; (c) Hj is circulant with κc=1,000 in the well-specified case, i.e. Hj=Hj ; and (d) Hj corresponds to energy preserving subspace of dimension 2d with Hj=Id .
Figure Q.2 : Test error in the simple case of Theorem 5.2 . In all figures Hj=Hj=Id
Figure Q.3 : Test error under debiasing. In Q.4 and Q.4 the relation matrices Hj are projection matrices to a random subspace of dimension 43 of the true data dimension, in Q.4 and Q.4 the relation matrices are energy-preserving subspace projection of dimension 21 (for further explanation see Appendix E.1.2 , E.1.2 ). The assumed relations are Hj=Id with Σx=Id except Q.4 where the Σx is exponential decay covariance matrix (see Appendix E.2.1 ). The solid lines are the numerical calculation of Theorem 4.4 and the circle markers are the corresponding empirical evaluations.
Figure Q.4 : Difference in target test error between transfer learning with and without debiasing , the task relation mentioned is for with out debiasing: (a) Hj=Hj=Id and ; (b) Hj corresponds to subspace of dimension 2d with Hj=Id ; (c) Hj corresponds to energy preserving subspace of dimension 2d with Hj=Id ; and (d) Hj is circulant with κc=1,000 in the well-specified case, i.e. Hj=Hj .
Figure Q.5 : Empirical tuning of the shrinkage factor. Each pair of figures shows the expected test error (left) and the empirically selected factor ρ~ (right) as a function of the source parametrization γsrc . The cyan dotted line represents the isotropic baseline ρ=1/γsrc . In Figure 5(a) , Hj corresponds to a circulant matrix with κc=1000 ; in Figure 5(b) , Hj is the identity matrix. In both figures, the assumed task relation is Hj=ρ~Id .
Pretrained models are often reused across downstream tasks, motivating the question of whether this transfer learning procedure can provide a substantial performance improvement, namely, in terms of the scaling law. We address this question in the prototypical setting of high-dimensional regression, where (i) a pretrained source model is trained via ridge regression, and (ii) a fine-tuned target model is regularized via the L2 distance from the source estimator. First, we establish a deterministic equivalent for the risk of this two-step procedure under both model and covariance shift. Next, by specializing the deterministic equivalent to a shared regression function and commuting power-law covariance spectra, we identify a regime where the resulting scaling law achieves the minimax-optimal exponent. Notably, in this regime, the optimal source regularization depends on source data only, indicating its universality across all targets. Furthermore, we show that the same optimal rate is achieved by performing the fine-tuning step via one-pass stochastic gradient descent. We support our theoretical analysis via experiments on both synthetic settings and language models.
Yalda Shabanzadeh, Diyuan Wu, Marco Mondelli
Institute of Science and Technology Austria (ISTA).
In transfer learning, the learner leverages auxiliary data to improve generalization on a main task. However, the precise theoretical understanding of when and how auxiliary data help remains incomplete. We provide new insights on this issue in two canonical linear settings: ordinary least squares regression and under-parameterized linear neural networks. For linear regression, we derive exact closed-form expressions for the expected generalization error with bias-variance decomposition, yielding necessary and sufficient conditions for auxiliary tasks to improve generalization on the main task. We also derive globally optimal task weights as outputs of solvable optimization programs, with consistency guarantees for empirical estimates. For linear neural networks with shared representations of width q≤K, where K is the number of auxiliary tasks, we derive a non-asymptotic expectation bound on the generalization error, yielding the first non-vacuous sufficient condition for beneficial auxiliary learning in this setting, as well as principled directions for task weight curation. We achieve this by proving a new column-wise low-rank perturbation bound for random matrices, which improves upon existing bounds by preserving fine-grained column structures. Our results are verified on synthetic data simulated with controlled parameters.
The last decade of progress in machine learning (ML), especially the deep learning era, has raised a number of scientific questions that challenge the longstanding dogma of the field. One of the most important riddles was the good empirical generalization of overparameterized models. Overparameterized models are highly complex with respect to the size of the training dataset, which enables them to perfectly fit (i.e., interpolate) even noisy training data. Such interpolation of noisy data is traditionally associated with detrimental overfitting, and yet a wide range of interpolating models -- from simple linear models to deep neural networks -- have been observed to generalize remarkably well on fresh test data. Indeed, the discovery of the double descent phenomenon has revealed that highly overparameterized models can improve over the best underparameterized model in test performance. Understanding learning in this overparameterized regime required new theory and foundational empirical studies, even for the simplest case of the linear model. The underpinnings of this understanding have been laid in foundational analyses of overparameterized linear regression and related statistical learning tasks, mostly published between 2018 and 2022, which resulted in precise analytic characterizations of double descent. This paper provides an overview of the theory of overparameterized ML (henceforth abbreviated as TOPML) by focusing on explaining the most foundational findings through a statistical signal processing perspective. We emphasize the unique aspects that define the TOPML research area as a subfield of modern ML theory and outline interesting open frontiers that remain.
Yehuda Dar, Vidya Muthukumar, Richard G. Baraniuk
Faculty of Computer and Information Science, Ben-Gurion University · Schools of Electrical and Computer Engineering, and Industrial and Systems Engineering, Georgia Institute of Technology · Electrical and Computer Engineering Department, Rice University