Learning a shared subspace in multitask learning is an important statistical problem, yet exact rates remain poorly understood in many of the formal settings considered in the literature. To address this gap, which arises from limitations of existing upper-bound techniques, we propose a general multitask lemma. The lemma provides sufficient conditions for fast subspace estimation in generic multitask settings, in the form of geometric conditions on the task regression functions and their estimators. We subsequently show that the multitask lemma yields sharp minimax rates---previously unknown---for linear regression and classification in the regimes considered, and rates matching up to logarithmic factors for additive-noise regression.
Figures & tables
Model
Taskwise construction
Rate
Optimality status
Linear regression with well-conditioned designs
Whitened least-squares estimator
R⋆
Matching upper and lower bounds for n≳d , k≳logT , and T≤ecn ; full conditions in Corollary 3.7
Binary classification with Gaussian design
First-moment estimator
R⋆
Matching upper and lower bounds for n≳k and T≲decn ; full conditions in Corollary 3.11
Additive-noise regression with Gaussian design
First-moment estimator
R⋆ up to explicit logarithmic factors
Upper and lower bounds matching up to logarithmic factors for n≳k and T≲decn ; full conditions in Corollary 3.13
Table 1: Principal model-specific results. The benchmark R⋆ is defined in ( 4 ). Displayed restrictions concern the minimax comparisons; the upper bounds hold under the assumptions of the corresponding upper-bound theorems.
Figure 1: The two main rate phenomena in the Gaussian linear model. Left: rare-task design with d=20 , n=400 , and T/n varying from 1/8 to 4 . Both rate curves are calibrated at the first grid point to compare parameter dependence. Right: phase transition with d=5 , n=1000 , T=50000 , and nλ2 varying from 10−2 to 102 . Dashed and dash-dotted lines are empirical log–log fits below and above nλ2=1 , respectively.
Figure 2: Task-spectrum dependence in logistic classification and the benefit of whitening in linear regression. (a) Logistic first-moment recovery with d=20 , n=200 , and T=4000 , varying λ1 at two fixed values of λ2 . (b) Raw versus whitened OLS with d=20 , n=40 , and (λ1,λ2)=(0.3,0.1) , as T varies from 200 to 204800 under non-isotropic fixed designs. The dashed line is the raw population loss. Each point averages 80 independent Monte Carlo repetitions; error bars are pointwise 95% Monte Carlo intervals for the mean sinΘ loss.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Model
Taskwise summary and population mean
Effective variance
Relation to the benchmark
Linear regression
Least squares; conditionally, E[θt∣Xt]=βt
σξ2/n
Exact when Xt⊤Xt=nId ; whitening gives the heterogeneous analogue for well-conditioned designs.
Classification
βt=n−1∑iYt,iXt,i , with Eβt=atβt
1/n , up to link constants
Gaussian symmetry makes the population cross-block vanish, and at preserves the task covariance up to fixed factors.
Additive-noise regression
The same first-moment summary, after truncation for the upper bound
1/n , inflated by the truncation level
The resulting logarithmic inflation multiplies the two benchmark terms.
Appendix
Table 2: The Gaussian mean-vector benchmark in the three multitask applications.
We study the multi-task linear regression problem in the presence of contaminated tasks. We address the setting where the unknown parameters of a majority of tasks are close in the ℓ2-norm, while a fraction of tasks are arbitrary outliers. Existing theoretical frameworks for this problem rely heavily on the assumption that the empirical second moment of each task has a minimum eigenvalue bounded away from zero (order Ω(1)). Crucially, this assumption fails in many high-dimensional scenarios, rendering prior guarantees vacuous. To overcome this limitation, we propose an estimator based on matrix-weighted norm regularization. We also introduce a relative balancedness condition, quantified by a balancedness constant, that compares each task's second moment with the average inlier geometry and relaxes the need for taskwise second-moment lower bounds. In favorable regimes with moderate balancedness, our prediction MSE bounds match the rate of Duan and Wang (2023) under substantially weaker spectral assumptions; the resulting task-overall MSE is minimax optimal up to logarithmic factors. Furthermore, we demonstrate that our estimator enjoys a safety guarantee: when the relevant balancedness constant is large or infinite, or when tasks are unrelated, the method performs no worse than independent task learning.
Multi-task learning (MTL) has emerged as a pivotal paradigm in machine learning by leveraging shared structures across multiple related tasks. Despite its empirical success, the development of likelihood-based efficiently solvable algorithms--even for shared linear representations--remains largely underdeveloped, primarily due to the non-convex structure intrinsic to matrix factorization. This paper introduces a first-order algorithm that jointly learns a shared representation and task-specific parameters, with guaranteed efficiency. Notably, it converges in O(1) iterations and attains a near-optimal estimation error of O(dk/(TN)), improving over existing likelihood-based methods by a factor of k, where d, k, T, N denote input dimension, representation dimension, task count, and samples per task, respectively. Our results justify that likelihood-based first-order methods can efficiently solve the MTL problem.
Shihong Ding, Fangyu Du, Cong Fang
State Key Lab of General AI, School of Intelligence Science and Technology, Peking University, China · School of Mathematical Sciences, Peking University, China · Institute for Artificial Intelligence, Peking University, China
Multi-task learning is effective for related applications, but its performance can deteriorate when the target sample size is small. Transfer learning can borrow strength from related studies; yet, many existing methods rely on restrictive bounded-difference assumptions between the source and target models. We propose SMART, a spectral transfer method for multi-task linear regression that instead assumes spectral similarity: the target left and right singular subspaces lie within the corresponding source subspaces and are sparsely aligned with the source singular bases. Such an assumption is natural when studies share latent structures and enables transfer beyond the bounded-difference settings. SMART estimates the target coefficient matrix through structured regularization that incorporates spectral information from a source study. Importantly, it requires only a fitted source model rather than the raw source data, making it useful when data sharing is limited. Although the optimization problem is nonconvex, we develop a practical ADMM-based algorithm. We establish general, non-asymptotic error bounds and a minimax lower bound in the noiseless-source regime. Under additional regularity conditions, these results yield near-minimax Frobenius error rates up to logarithmic factors. Simulations confirm improved estimation accuracy and robustness to negative transfer, and analysis of multi-modal single-cell data demonstrates better predictive performance. The Python implementation of SMART, along with the code to reproduce all experiments in this paper, is publicly available at https://github.com/boxinz17/smart.
Boxin Zhao, Mladen Kolar, Jinchi Lv
University of Chicago · University of Southern California