Expectation Error Bounds for Transfer Learning in Linear Regression and Linear Neural Networks

Meitong Liu; Christopher Jung; Rui Li; Xue Feng; Han Zhao

cs.LGMar 30, 2026

Expectation Error Bounds for Transfer Learning in Linear Regression and Linear Neural Networks

Authors: Meitong Liu, Christopher Jung, Rui Li, Xue Feng, Han Zhao

Organizations: University of Illinois Urbana-Champaign · Meta

Abstract

In transfer learning, the learner leverages auxiliary data to improve generalization on a main task. However, the precise theoretical understanding of when and how auxiliary data help remains incomplete. We provide new insights on this issue in two canonical linear settings: ordinary least squares regression and under-parameterized linear neural networks. For linear regression, we derive exact closed-form expressions for the expected generalization error with bias-variance decomposition, yielding necessary and sufficient conditions for auxiliary tasks to improve generalization on the main task. We also derive globally optimal task weights as outputs of solvable optimization programs, with consistency guarantees for empirical estimates. For linear neural networks with shared representations of width $q \leq K$ , where $K$ is the number of auxiliary tasks, we derive a non-asymptotic expectation bound on the generalization error, yielding the first non-vacuous sufficient condition for beneficial auxiliary learning in this setting, as well as principled directions for task weight curation. We achieve this by proving a new column-wise low-rank perturbation bound for random matrices, which improves upon existing bounds by preserving fine-grained column structures. Our results are verified on synthetic data simulated with controlled parameters.

Figures & tables

Source	Object	Assumption	Bound	Possible to be smaller than $\\|Z\\|$ or $\\|Ze_{j}\\|$ ?
EYM ( Tran et al., 2025 )	$\\|[M]_{q}-[S]_{q}\\|$	-	$2(\sigma_{q+1}(S)+\\|Z\\|)$	No
TVV ( Tran et al., 2025 )	$\\|[M]_{q}-[S]_{q}\\|$	$r\geq 4$	$\frac{6r}{r-2}\\|Z\\|(\frac{\sigma_{q}(S)}{\delta_{q}(S)}+\log\frac{6\sigma_{1}(S)}{\delta_{q}(S)})$	No
Ours ( \cref th:4.2)	$\\|([M]_{q}-[S]_{q})e_{j}\\|$	$r>1$	$\frac{1}{r-1}(\\|Se_{j}\\|+\\|Ze_{j}\\|)+\\|P_{U}Ze_{j}\\|$	Yes

Table 1: Comparison of low-rank perturbation bounds. Here,

\delta_{q}(S)=\sigma_{q}(S)-\sigma_{q+1}(S)

r=\delta_{q}(S)/\|Z\|

, and

P_{U}=U_{:,q}U_{:,q}^{\top}

, where

U_{:,q}

contains the top

q

left singular vectors of

S

Panel of Figure 2 — (a) Simulation results of for both linear regression and linear neural networks.

Transfer Learning

Transfer learning studies how models reuse knowledge across tasks, domains, or modalities when target data or computation are limited.

See all 406 papers

An Analytical Theory of Auxiliary Learning
Sep 24, 2026Federico Milanesio, Alessandro Ingrosso, Matteo OsellaStatistical Physics of Learning Neural Network Generalization
Department of Physics and INFN University of Turin Via Giuria 1, 10125 Turin, Italy · Donders Centre for Neuroscience Radboud University Heyendaalseweg 135, 6525 AJ Nijmegen, The Netherlands
A Generalization Bound for Nearly-Linear Networks
Jul 9, 2024Eugene GolikovNeural Network Generalization Generalization Bounds
Chair of Statistical Field Theory École Polytechnique Fédérale de Lausanne (EPFL)
Transfer Learning of Linear Regression with Multiple Pretrained Models: Benefiting from More Pretrained Models via Overparameterization Debiasing
Feb 18, 2026Daniel Boharon, Yehuda DarOverparameterization Transfer Learning
Faculty of Computer and Information Science, Ben-Gurion University.

Sep 24, 2026J/K move · Enter open · S save

An Analytical Theory of Auxiliary Learning

Federico Milanesio, Alessandro Ingrosso, Matteo Osella

Department of Physics and INFN University of Turin Via Giuria 1, 10125 Turin, Italy · Donders Centre for Neuroscience Radboud University Heyendaalseweg 135, 6525 AJ Nijmegen, The Netherlands

Statistical Physics of Learning Neural Network Generalization Neural Network Training Dynamics

Open paper AskPDF arXiv

Auxiliary learning is an optimization paradigm in which a neural network's performance on a target task is improved by jointly training it on additional tasks. However, the mechanisms behind this improvement remain poorly understood. We study this problem using a teacher-student framework and derive a closed system of differential equations describing the dynamics of online stochastic gradient descent in the large-input limit. For linear networks, we obtain a closed-form expression for the generalization error to leading order in the learning rate, quantifying how task correlations and label noise determine the benefit of auxiliary learning. For non-linear activation functions, we develop a fluctuation-dissipation analytical theory that establishes a general relation linking the main and auxiliary errors to the corresponding single-task error. Numerical experiments support the theoretical predictions and show how auxiliary tasks improve generalization by balancing the forcing dynamics towards the optimal solution with gradient noise.

Abstract

Transfer Learning

Explore similar work

An Analytical Theory of Auxiliary Learning

A Generalization Bound for Nearly-Linear Networks

Transfer Learning of Linear Regression with Multiple Pretrained Models: Benefiting from More Pretrained Models via Overparameterization Debiasing