Transfer Learning of Linear Regression with Multiple Pretrained Models: Benefiting from More Pretrained Models via Overparameterization Debiasing

Daniel Boharon; Yehuda Dar

cs.LGFeb 18, 2026

Transfer Learning of Linear Regression with Multiple Pretrained Models: Benefiting from More Pretrained Models via Overparameterization Debiasing

Authors: Daniel Boharon, Yehuda Dar

Organizations: Faculty of Computer and Information Science, Ben-Gurion University.

Abstract

We study transfer learning for a linear regression task using several least-squares pretrained models that can be overparameterized. We formulate the target learning task as optimization that minimizes squared errors on the target dataset with penalty on the distance of the learned model from the pretrained models. We analytically formulate the test error of the learned target model and provide the corresponding empirical evaluations. Our results elucidate when using more pretrained models can improve transfer learning. Specifically, if the pretrained models are overparameterized, using sufficiently many of them is important for beneficial transfer learning. However, the learning may be compromised by overparameterization bias of pretrained models, i.e., the minimum $\ell_2$ -norm solution's restriction to a small subspace spanned by the training examples in the high-dimensional parameter space. We propose a simple debiasing via multiplicative correction factor that can reduce the overparameterization bias and leverage more pretrained models to learn a target predictor.

Figures & tables

Panel of Figure 1 — Figure 1: Test error in the general case of Theorem 4.4 . Here, $\mathbf{H}_{j}$ corresponds to energy preserving subspace of dimension $\frac{d}{2}$ (see Appendix E.1 ), and the assumed task relation is $\widetilde{\mathbf{H}}_{j}=\mathbf{I}_{d}$ . Target covariance is $\mathbf{\Sigma}_{\mathbf{x}}=\mathbf{I}_{d}$ (left subfig.) and exponential decay $(\mathbf{\Sigma}_{\mathbf{x}})_{il}=0.5^{|i-l|}$ (right subfig.) See Appendix Q.1 for more experiments.

Panel of Figure 2 — Figure 2: Test error in the simple case of Theorem 5.2 . $\mathbf{H}_{j}=\widetilde{\mathbf{H}}_{j}=\mathbf{I}_{d}$ . See Appendix Q.2 for more experiments.

Panel of Figure 3 — Figure 3: Test error under debiasing. In Fig. 3(a) , $\mathbf{H}_{j}=\mathbf{I}_{d}$ , and in Fig. 3(b) $\mathbf{H}_{j}$ is a subspace projection of dimension $\frac{3}{4}$ (for more details see Appendix E.1 ). The assumed task relation is $\widetilde{\mathbf{H}}_{j}=\rho_{j}\mathbf{I}_{d}$ . Target covariance is $\mathbf{\Sigma}_{\mathbf{x}}=\mathbf{I}_{d}$ in both figures. For more figures see Appendix Q.3 .

Panel of Figure 4 — Figure 4: Bias-variance decomposition of the test error. Dashed and dotted lines denote the bias and variance terms, respectively. Without debiasing uses $\widetilde{\mathbf{H}}_{j}=\mathbf{I}_{d}$ ; with debiasing uses $\widetilde{\mathbf{H}}_{j}=\rho_{j}\mathbf{I}_{d}$ . See Appendix M for a detailed discussion and more results.

Panel of Figure 5 — Figure 5: Difference in target test error between transfer learning with and without debiasing . Negative values denote beneficial debiasing. Here, $\mathbf{H}_{j}=\mathbf{I}_{d}$ ; see Fig. Q.4 for other task relations.

Panel of Figure 6 — Figure 6: Empirical tuning of the shrinkage factor. The figure shows the expected test error (left) and the empirically selected factor $\tilde{\rho}$ (right) as a function of the source parametrization $\gamma_{\mathrm{src}}$ . The cyan dotted line represents the isotropic baseline $\rho=1/\gamma_{\mathrm{src}}$ . Here, $\mathbf{H}_{j}$ corresponds to energy-preserving subspace (dimension $\frac{3d}{4}$ ), the assumed task relation is $\widetilde{\mathbf{H}}_{j}=\tilde{\rho}\,\mathbf{I}_{d}$ and the covariance is $(\mathbf{\Sigma_{\mathbf{x}}})_{il}=(\mathbf{\Sigma_{\mathbf{z}}})_{il}=0.5^{|i-l|}$ . For other settings, see Fig. Q.5 .

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

\widehat{\boldsymbol{\beta}}_{\mathrm{TL}}=\argmin_{\mathbf{b}\in\mathbb{R}^{d}}\left\|{{\mathbf{y}-\mathbf{X}\mathbf{b}}}\right\|_{2}^{2}+n\alpha_{\mathrm{TL}}\sum_{j=1}^{m}\left\|{{\widetilde{\mathbf{H}}_{j}\mathbf{b}-\widehat{\boldsymbol{\theta}}_{j}}}\right\|_{2}^{2}

(L.83)

Appendix

Algorithm 1 Transfer learning with overparameterization debiasing for unknown task relations under isotropic source assumption

Panel of Figure 8 — Figure L.1 : Difference in expected test error between transfer learning with and without debiasing in the source overparametrized regime. The error difference is examined for a varying source task overparameterization level and for several noise levels $\sigma_{\xi}^{2}$ , $\sigma_{\eta}^{2}$ . A negative error difference implies that debiasing is beneficial. In Figs. 1(a) and 1(b) , the true relation is $\mathbf{H}_{j}=\mathbf{I}_{d}$ . The assumed relations are $\mathbf{\widetilde{H}}_{j}=\mathbf{I}_{d}$ (without debiasing) and $\mathbf{\widetilde{H}}_{j}=\frac{\tilde{n}_{j}}{d}\mathbf{I}_{d}$ (with debiasing). In Figs. 1(c) and 1(d) , the true relation matrices are circulant with $\kappa_{\mathrm{c}}=1,000$ . For Fig. 1(c) , we use the matched assumptions: $\mathbf{\widetilde{H}}_{j}=\mathbf{H}_{j}$ (without debiasing) and $\mathbf{\widetilde{H}}_{j}=\frac{\tilde{n}_{j}}{d}\mathbf{H}_{j}$ (with debiasing). Conversely, for Fig. 1(d) , the assumed relation matrices are identical to those in Figs. 1(a) and 1(b) . (More figures with different noises and task relations can be seen in Appendix ).

Panel of Figure 9 — Figure L.2 : Comparison between different assumed relation matrices under ill-condition of true relation matrix. In all figures $\mathbf{H}_{j}$ is a circulant matrix with condition number $\kappa_{\mathrm{c}}=1000$ and $\sigma_{\xi}^{2}=\sigma_{\eta}^{2}=0.1$ , $\gamma_{\mathrm{tgt}}=4$ . Left Column displays using the identity matrix or the scaled identity matrix for without and with debiasing, respectively, and the right column displays using the true relation matrix or the scaled true relation matrix for without and with debiasing, respectively. First row displays without debiasing and the second is with. The test error difference between 3(a) - 3(b) and 3(c) - 3(d) is provided in Fig. M.3 .

Panel of Figure 10 — Figure L.3 : Comparison between different assumed relation matrices under ill-condition of true relation matrix. In all figures $\mathbf{H}_{j}$ is a circulant matrix with condition number $\kappa_{\mathrm{c}}=1000$ and $\sigma_{\xi}^{2}=\sigma_{\eta}^{2}=0.1$ , $\gamma_{\mathrm{tgt}}=4$ . Left Column displays using the identity matrix or the scaled identity matrix for without and with debiasing, respectively, and the right column displays using the true relation matrix or the scaled true relation matrix for without and with debiasing, respectively. First row displays without debiasing and the second is with. The test error difference between 3(a) - 3(b) and 3(c) - 3(d) is provided in Fig. M.3 .

Panel of Figure 11 — Figure M.1 : Bias-variance decomposition of the expected test error of Fig. L.3 . The test error difference between 3(a) - 3(b) and 3(c) - 3(d) is provided in Fig. M.3 .

Panel of Figure 12 — Figure M.2 : Bias-variance decomposition of the expected test error of Fig. L.3 . The test error difference between 3(a) - 3(b) and 3(c) - 3(d) is provided in Fig. M.3 .

Panel of Figure 13 — Figure M.3 : Expected test error difference between using the true relation matrix $\mathbf{H}_{j}$ to using the identity matrix $\mathbf{I}_{d}$ as $\widetilde{\mathbf{H}}_{j}$ in transfer learning without debiasing in M.3 , and using $\rho_{j}\mathbf{H}_{j}$ and $\rho_{j}\mathbf{I}_{d}$ when debiasing in M.3 corresponding to the figures in L.3 . Positive difference implies that using $\mathbf{I}_{d}$ is better than using the true $\mathbf{H}$ .

\widehat{\boldsymbol{\beta}}_{\alpha,\tilde{\rho}}=\argmin_{\mathbf{b}\in\mathbb{R}^{d}}\left\|{{\mathbf{y}-\mathbf{X}\mathbf{b}}}\right\|_{2}^{2}+n\alpha\sum_{j=1}^{m}\left\|{{\widetilde{\mathbf{H}}_{j}\mathbf{b}-\widehat{\boldsymbol{\theta}}_{j}}}\right\|_{2}^{2}

Appendix

Algorithm 2 Transfer learning with a validation-tuned debiasing factor for unknown anisotropic sources

Panel of Figure 15 — Figure Q.1 : Test error in the general case of Theorem 4.4 . The results are shown for: (a) $\mathbf{H}_{j}$ corresponds to energy preserving subspace of dimension $\frac{3d}{4}$ with $\widetilde{\mathbf{H}}_{j}=\mathbf{I}_{d}$ ; (b) $\mathbf{H}_{j}$ corresponds to subspace of dimension $\frac{3d}{4}$ with $\widetilde{\mathbf{H}}_{j}=\mathbf{I}_{d}$ ; (c) $\mathbf{H}_{j}$ is circulant with $\kappa_{\mathrm{c}}=1,000$ in the well-specified case, i.e. $\mathbf{H}_{j}=\widetilde{\mathbf{H}}_{j}$ ; and (d) $\mathbf{H}_{j}$ corresponds to energy preserving subspace of dimension $\frac{d}{2}$ with $\widetilde{\mathbf{H}}_{j}=\mathbf{I}_{d}$ .

Figure Q.2 : Test error in the simple case of Theorem 5.2 . In all figures

\mathbf{H}_{j}=\widetilde{\mathbf{H}}_{j}=\mathbf{I}_{d}

Panel of Figure 17 — Figure Q.3 : Test error under debiasing. In Q.4 and Q.4 the relation matrices $\mathbf{H}_{j}$ are projection matrices to a random subspace of dimension $\frac{3}{4}$ of the true data dimension, in Q.4 and Q.4 the relation matrices are energy-preserving subspace projection of dimension $\frac{1}{2}$ (for further explanation see Appendix E.1.2 , E.1.2 ). The assumed relations are $\widetilde{\mathbf{H}}_{j}=\mathbf{I}_{d}$ with $\mathbf{\Sigma}_{\mathbf{x}}=\mathbf{I}_{d}$ except Q.4 where the $\mathbf{\Sigma}_{\mathbf{x}}$ is exponential decay covariance matrix (see Appendix E.2.1 ). The solid lines are the numerical calculation of Theorem 4.4 and the circle markers are the corresponding empirical evaluations.

Panel of Figure 18 — Figure Q.4 : Difference in target test error between transfer learning with and without debiasing , the task relation mentioned is for with out debiasing: (a) $\mathbf{H}_{j}=\widetilde{\mathbf{H}}_{j}=\mathbf{I}_{d}$ and ; (b) $\mathbf{H}_{j}$ corresponds to subspace of dimension $\frac{d}{2}$ with $\widetilde{\mathbf{H}}_{j}=\mathbf{I}_{d}$ ; (c) $\mathbf{H}_{j}$ corresponds to energy preserving subspace of dimension $\frac{d}{2}$ with $\widetilde{\mathbf{H}}_{j}=\mathbf{I}_{d}$ ; and (d) $\mathbf{H}_{j}$ is circulant with $\kappa_{\mathrm{c}}=1,000$ in the well-specified case, i.e. $\mathbf{H}_{j}=\widetilde{\mathbf{H}}_{j}$ .

Panel of Figure 19 — Figure Q.5 : Empirical tuning of the shrinkage factor. Each pair of figures shows the expected test error (left) and the empirically selected factor $\tilde{\rho}$ (right) as a function of the source parametrization $\gamma_{\mathrm{src}}$ . The cyan dotted line represents the isotropic baseline $\rho=1/\gamma_{\mathrm{src}}$ . In Figure 5(a) , $\mathbf{H}_{j}$ corresponds to a circulant matrix with $\kappa_{c}=1000$ ; in Figure 5(b) , $\mathbf{H}_{j}$ is the identity matrix. In both figures, the assumed task relation is $\widetilde{\mathbf{H}}_{j}=\tilde{\rho}\,\mathbf{I}_{d}$ .

Overparameterization

See all 22 papers

Minimax-optimal Scaling Laws in Ridge Regression via Transfer Learning
Oct 16, 2026Yalda Shabanzadeh, Diyuan Wu, Marco MondelliTransfer Learning Ridge Regression
Institute of Science and Technology Austria (ISTA).
Expectation Error Bounds for Transfer Learning in Linear Regression and Linear Neural Networks
Mar 30, 2026Meitong Liu, Christopher Jung, Rui Li +2Transfer Learning Generalization Bounds
University of Illinois Urbana-Champaign · Meta
A Farewell to the Bias-Variance Tradeoff? An Overview of the Theory of Overparameterized Machine Learning
Sep 6, 2021Yehuda Dar, Vidya Muthukumar, Richard G. BaraniukDouble Descent Neural Network Generalization
Faculty of Computer and Information Science, Ben-Gurion University · Schools of Electrical and Computer Engineering, and Industrial and Systems Engineering, Georgia Institute of Technology · Electrical and Computer Engineering Department, Rice University

Oct 16, 2026J/K move · Enter open · S save

Minimax-optimal Scaling Laws in Ridge Regression via Transfer Learning

Yalda Shabanzadeh, Diyuan Wu, Marco Mondelli

Institute of Science and Technology Austria (ISTA).

Transfer Learning Ridge Regression High-Dimensional Linear Regression Minimax Estimation

Open paper AskPDF arXiv

Pretrained models are often reused across downstream tasks, motivating the question of whether this transfer learning procedure can provide a substantial performance improvement, namely, in terms of the scaling law. We address this question in the prototypical setting of high-dimensional regression, where (i) a pretrained source model is trained via ridge regression, and (ii) a fine-tuned target model is regularized via the

L_2

distance from the source estimator. First, we establish a deterministic equivalent for the risk of this two-step procedure under both model and covariance shift. Next, by specializing the deterministic equivalent to a shared regression function and commuting power-law covariance spectra, we identify a regime where the resulting scaling law achieves the minimax-optimal exponent. Notably, in this regime, the optimal source regularization depends on source data only, indicating its universality across all targets. Furthermore, we show that the same optimal rate is achieved by performing the fine-tuning step via one-pass stochastic gradient descent. We support our theoretical analysis via experiments on both synthetic settings and language models.

Abstract

Overparameterization

Explore similar work

Minimax-optimal Scaling Laws in Ridge Regression via Transfer Learning

Expectation Error Bounds for Transfer Learning in Linear Regression and Linear Neural Networks

A Farewell to the Bias-Variance Tradeoff? An Overview of the Theory of Overparameterized Machine Learning