The Geometry of Multitask Learning: Sharp Rates for Shared Subspace Estimation
Organizations: Department of Statistics and Data Science, University of Pennsylvania · Department of Statistics, Columbia University
Abstract
Learning a shared subspace in multitask learning is an important statistical problem, yet exact rates remain poorly understood in many of the formal settings considered in the literature. To address this gap, which arises from limitations of existing upper-bound techniques, we propose a general multitask lemma. The lemma provides sufficient conditions for fast subspace estimation in generic multitask settings, in the form of geometric conditions on the task regression functions and their estimators. We subsequently show that the multitask lemma yields sharp minimax rates---previously unknown---for linear regression and classification in the regimes considered, and rates matching up to logarithmic factors for additive-noise regression.
Figures & tables
| Model | Taskwise construction | Rate | Optimality status |
|---|---|---|---|
| Linear regression with well-conditioned designs | Whitened least-squares estimator | Matching upper and lower bounds for , , and ; full conditions in Corollary 3.7 | |
| Binary classification with Gaussian design | First-moment estimator | Matching upper and lower bounds for and ; full conditions in Corollary 3.11 | |
| Additive-noise regression with Gaussian design | First-moment estimator | up to explicit logarithmic factors | Upper and lower bounds matching up to logarithmic factors for and ; full conditions in Corollary 3.13 |
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
| Model | Taskwise summary and population mean | Effective variance | Relation to the benchmark |
|---|---|---|---|
| Linear regression | Least squares; conditionally, | Exact when ; whitening gives the heterogeneous analogue for well-conditioned designs. | |
| Classification | , with | , up to link constants | Gaussian symmetry makes the population cross-block vanish, and preserves the task covariance up to fixed factors. |
| Additive-noise regression | The same first-moment summary, after truncation for the upper bound | , inflated by the truncation level | The resulting logarithmic inflation multiplies the two benchmark terms. |
Multi-Task Learning
Multi-task learning trains models to solve related tasks together by sharing representations, parameters, or supervision.
See all 417 papers