cs.LGOct 16, 2026

Minimax-optimal Scaling Laws in Ridge Regression via Transfer Learning

Authors: Yalda Shabanzadeh, Diyuan Wu, Marco Mondelli

Organizations: Institute of Science and Technology Austria (ISTA).

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Pretrained models are often reused across downstream tasks, motivating the question of whether this transfer learning procedure can provide a substantial performance improvement, namely, in terms of the scaling law. We address this question in the prototypical setting of high-dimensional regression, where (i) a pretrained source model is trained via ridge regression, and (ii) a fine-tuned target model is regularized via the L2L_2 distance from the source estimator. First, we establish a deterministic equivalent for the risk of this two-step procedure under both model and covariance shift. Next, by specializing the deterministic equivalent to a shared regression function and commuting power-law covariance spectra, we identify a regime where the resulting scaling law achieves the minimax-optimal exponent. Notably, in this regime, the optimal source regularization depends on source data only, indicating its universality across all targets. Furthermore, we show that the same optimal rate is achieved by performing the fine-tuning step via one-pass stochastic gradient descent. We support our theoretical analysis via experiments on both synthetic settings and language models.

Figures & tables

Figure 1: Accuracy of theoretical predictions across various regimes for a=2a=2 , b=6b=6 , rtr=0.6r_{\texttt{tr}}=0.6 , d=50,000d=50{,}000 . Solid lines show the deterministic equivalent, markers show Gaussian simulations (mean ±\pm s.e., 5050 seeds), and dotted segments show the predicted slopes. Left: Bias is the bottleneck in the risk, and the optimal scaling law is reached by choosing the correct λsr\lambda_{\texttt{sr}} . Middle: Variance is the bottleneck in the risk, and the optimal λtr\lambda_{\texttt{tr}} balances V‾tr=V‾sr\overline{{\mathcal{V}}}_{\texttt{tr}}=\overline{{\mathcal{V}}}_{\texttt{sr}} . Right: Comparison between optimally tuned target-only, source-only, and transfer learning risks.
Figure 2: Scaling law exponents of optimally regularized ridge and minimax rates, as a function of pretraining data size. Solid lines show the exponents from Theorem 3 . The region γc<γn<1\gamma_{c}<\gamma_{n}<1 is where transfer learning attains the minimax rate obtained by employing both datasets, and it improves over estimators using either target-only or source-only data. The gray band is the minimax rate, i.e., the best achievable worst-case exponent any estimator can reach using both source and target data.
Figure 3: Transfer learning in language models. Target test loss as a function of target sample size. Different columns correspond to different values of the source sample size ( γ=0\gamma=0 in the left-most plot and then γ=0.75\gamma=0.75 , γ=1\gamma=1 , and γ=1.25\gamma=1.25 ). Different rows correspond to different number of passes on the data (single pass in the first row, and multiple passes with number of epochs chosen from validation in the second row). Access to more source data produces a significant advantage of FT under short target training (one-pass), while the advantage becomes weaker with additional target optimization (multi-pass). We denote the local exponent by Γ^=−ln⁡L(2n)−ln⁡L(n)ln⁡2\widehat{\Gamma}=-\frac{\ln L(2n)-\ln L(n)}{\ln 2} , see ( 129 ) for details. Green shading shows the ranges where Γ^FT>max⁡(Γ^TO,Γ^SO)\hat{\Gamma}_{FT}>\max(\widehat{\Gamma}_{TO},\widehat{\Gamma}_{SO}) , and purple hatching marks where Γ^FTSP>max⁡(Γ^TO,Γ^SO)\widehat{\Gamma}_{FTSP}>\max(\widehat{\Gamma}_{TO},\widehat{\Gamma}_{SO}) . FTSP refers to our implementation of L2-SP ( Li et al., 2018 ) , see Appendix E.2 for details.
Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Deterministic equivalent on MNIST PCA features. Inputs are the top pp principal components of the centered training images, and outputs follow a linear model with noise standard deviation 0.20.2 and normalized correlation between βsr{\bm{\beta}}_{\texttt{sr}} and βtr{\bm{\beta}}_{\texttt{tr}} equal to 0.80.8 . Source and target inputs are drawn with replacement from the training images. We use λsr=0.5λtr\lambda_{\rm sr}=0.5\lambda_{\rm tr} . The left column fixes nsr=ntr=300n_{\rm sr}=n_{\rm tr}=300 and varies λtr∈{10−1,5×10−1,1}\lambda_{\rm tr}\in\{10^{-1},5\times 10^{-1},1\} ; the right column fixes λtr=10−1\lambda_{\rm tr}=10^{-1} and varies nsr=ntr∈{300,500,700}n_{\rm sr}=n_{\rm tr}\in\{300,500,700\} . Top: solid curves show the deterministic equivalent and markers show the mean target excess risk over 100100 training draws with ±1\pm 1 s.e. bars. The risk increases up to p=np=n and then flattens. For n=300n=300 and λtr=10−1\lambda_{\rm tr}=10^{-1} , there is a peak at p=np=n . Discrepancies are larger at weaker regularization. Bottom: ratios of empirical risk to the deterministic equivalent for image features and Gaussian features with the same covariance spectrum. The Gaussian control stays within 3.4%3.4\% of the prediction, while the image features deviate more.
Figure 5: Deterministic equivalent on CIFAR-10 PCA features. We use the same setup as in Figure 4 , using the same image pool for PCA and for drawing 20,00020{,}000 training samples. Source and target inputs are drawn with replacement from this pool. The left column fixes nsr=ntr=300n_{\rm sr}=n_{\rm tr}=300 and varies λtr∈{10−1,5×10−1,1}\lambda_{\rm tr}\in\{10^{-1},5\times 10^{-1},1\} ; the right column fixes λtr=10−1\lambda_{\rm tr}=10^{-1} and varies nsr=ntr∈{300,500,700}n_{\rm sr}=n_{\rm tr}\in\{300,500,700\} . Top: solid curves show the deterministic equivalent and markers show the mean target excess risk over 100100 training draws with ±1\pm 1 s.e. bars. The curves show a peak near p=np=n , which shifts with nn , with larger errors under weaker regularization. Bottom: ratios of empirical risk to the deterministic equivalent for image features and Gaussian features with the same covariance spectrum. The Gaussian control stays within 3.5%3.5\% of the prediction, while the image features deviate more.
More in this topic

Transfer Learning

Transfer learning studies how models reuse knowledge across tasks, domains, or modalities when target data or computation are limited.

See all 404 papers

Explore similar work

CardsList

  1. Optimal Unconstrained Self-Distillation in Ridge Regression: Strict Improvements, Precise Asymptotics, and One-Shot Tuning

    Feb 19, 2026Hien Dang, Pratik Patil, Alessandro RinaldoSelf-DistillationStatistical Learning Theory

    Department of Statistics and Data Sciences, University of Texas, Austin, TX 78712, USA.

  2. Transfer Learning of Linear Regression with Multiple Pretrained Models: Benefiting from More Pretrained Models via Overparameterization Debiasing

    Feb 18, 2026Daniel Boharon, Yehuda DarOverparameterizationTransfer Learning

    Faculty of Computer and Information Science, Ben-Gurion University.

  3. When do data mixtures improve scaling laws? Insights from high-dimensional regression

    Sep 29, 2026Diyuan Wu, Lehan Chen, Theodor Misiakiewicz +1High-Dimensional Linear RegressionData Mixture Optimization

    Institute of Science and Technology Austria (ISTA). · Department of Applied and Computational Mathematics, Yale University. · Department of Statistics and Data Science, Yale University.