Statistical Benefits of Fine-Tuning from Pretrained Initialization in Diagonal Linear Networks

Alexandre Declèves; Etienne Boursier; Nicolas Flammarion

stat.MLSep 28, 2026

Statistical Benefits of Fine-Tuning from Pretrained Initialization in Diagonal Linear Networks

Authors: Alexandre Declèves, Etienne Boursier, Nicolas Flammarion

Organizations: TML, EPFL, Lausanne, Switzerland · INRIA, LMO, Université Paris-Saclay, Orsay, France

Abstract

Adapting pretrained models to downstream tasks with limited data has become a central paradigm in modern deep learning. Yet, despite its widespread practical success, how fine-tuning leverages information from pretraining remains poorly understood theoretically. We study fine-tuning from pretrained weights through the lens of sparse linear regression and two-layer diagonal linear networks. In our setting, pretraining provides information through the support (and signs) of the initialization predictor, which may contain coordinates relevant to the downstream task. We show how pretrained information reshapes the implicit bias and training dynamics, and can thereby reduce the sample complexity of recovering the target parameters and support. In particular, for a clean initialization with correctly inherited signs, we show that the required sample size is comparable to that of a weighted Lasso estimator that explicitly exploits the pretrained support through a suitably chosen regularizer. Our results thus show how information encoded in pretrained weights can be implicitly exploited by gradient-based fine-tuning, reducing the amount of data needed to recover a downstream task.

Figures & tables

Panel of Figure 1 — Figure 1: Schematic dual trajectories under non-zero initialization. Missing and wrongly signed true coordinates travel dual distances $1$ and $2$ , respectively, before entering with the correct sign.

	Early-stopped S2S	Weighted Lasso
Sample size	$n\gtrsim s+f_{0}+m^{2}+mf_{0}+m\log(d)$	$n\gtrsim m_{\rm miss}\log(d)+(s-m_{\rm miss})\log(s_{\rm init})$
Support guarantee	$S^{\star}\subseteq S_{\rm final}\subseteq S^{\star}\cup F_{0}$	$\operatorname{supp}(\widehat{\beta}^{\rm WL})=S^{\star}$

Table 1: Noisy recovery guarantees under the respective assumptions: Theorem 3 with

B=m+f_{0}

for S2S, and Proposition 1 with

\sigma/\min_{i\in S^{\star}}|\beta_{i}^{\star}|=O(1)

for weighted Lasso. Numerical constants and confidence dependence are suppressed.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 2 — Figure 2: Early stopping along S2S trajectories on two independent instances. Left: test loss for $d=300$ , $n_{\rm train}=220$ , $n_{\rm test}=1200$ , $s=40$ , and $\sigma=0.09$ . The test loss reaches its oracle minimum at saddle $k=33$ and then increases sharply as the trajectory continues. Right: null-gradient stopping rule for $d=32$ , $n=220$ , $s=6$ , and $\sigma=0.05$ . The four missing true coordinates are proposed before any null coordinate; the first null proposal falls below the stopping threshold and is therefore rejected.

Figure 3: Empirical probability of exact support recovery as a function of the number

m

of missing true coordinates for a clean initialization (

d=260

n=140

s=20

\sigma=0.005

). Probabilities are estimated over

200

independent trials. S2S succeeds with high probability when few true coordinates remain to be learned, while its recovery probability decreases sharply as

m

increases. Weighted Lasso degrades more gradually.

Explore similar work

A Theory of How Pretraining Shapes Inductive Bias in Fine-Tuning
Feb 23, 2026Nicolas Anguita, Francesco Locatello, Andrew M. Saxe +4Model Fine-Tuning Feature Learning
Department of Engineering, University of Cambridge · Institute of Science and Technology, Austria (ISTA) · Gatsby Computational Neuroscience Unit and Sainsbury Wellcome Centre, UCL +1
Two-Timescale Fine-tuning Provably Learns New Features for Two-Layer ReLU Networks
Sep 28, 2026Etienne Boursier, Nicolas FlammarionFeature Learning Model Fine-Tuning
INRIA, LMO, Universit´e Paris-Saclay, Orsay, France · EPFL, Lausanne, Switzerland
Pretraining Induces a Reusable Spectral Basis for Downstream Task Adaptation
May 8, 2026Junjie Yu, Yue Wang, Zihan Deng +3Pretraining Model Fine-Tuning
Department of Biomedical Engineering, Southern University of Science and Technology · Department of Psychology, The University of Hong Kong

Feb 23, 2026, cs.LGJ/K move · Enter open · S save

A Theory of How Pretraining Shapes Inductive Bias in Fine-Tuning

Nicolas Anguita, Francesco Locatello, Andrew M. Saxe, Marco Mondelli+3

Department of Engineering, University of Cambridge · Institute of Science and Technology, Austria (ISTA) · Gatsby Computational Neuroscience Unit and Sainsbury Wellcome Centre, UCL+1

Model Fine-Tuning Feature Learning Inductive Bias Theory

Open paper AskPDF arXiv

Pretraining and fine-tuning are central stages in modern machine learning systems. In practice, feature learning plays an important role across both stages: deep neural networks learn a broad range of useful features during pretraining and further refine those features during fine-tuning. However, an end-to-end theoretical understanding of how choices of initialization impact the ability to reuse and refine features during fine-tuning has remained elusive. Here we develop an analytical theory of the pretraining fine-tuning pipeline in diagonal linear networks, deriving exact expressions for the generalization error as a function of initialization parameters and task statistics. We find that different initialization choices place the network into four distinct fine-tuning regimes that are distinguished by their ability to support feature learning and reuse and therefore by the task statistics for which they are beneficial. In particular, a smaller initialization scale in earlier layers enables the network to both reuse and refine its features, leading to superior generalization on fine-tuning tasks that rely on a subset of pretraining features. We demonstrate empirically that the same initialization parameters impact generalization in ResNets trained on CIFAR-100 and SVHN as well as Transformers trained on modular arithmetic tasks. Overall, our results demonstrate an alytically how data and network initialization interact to shape fine-tuning generalization, highlighting an important role for the relative scale of initialization across different layers in enabling continued feature learning during fine-tuning.

Abstract

Explore similar work

A Theory of How Pretraining Shapes Inductive Bias in Fine-Tuning

Two-Timescale Fine-tuning Provably Learns New Features for Two-Layer ReLU Networks

Pretraining Induces a Reusable Spectral Basis for Downstream Task Adaptation