Trace-Norm Geometry of Composed Multitask Predictors

Mahdi Mohammadigohari

cs.LGAug 25, 2026

Trace-Norm Geometry of Composed Multitask Predictors

Authors: Mahdi Mohammadigohari

Organizations: Faculty of Engineering, Free University of Bozen–Bolzano, Via Bruno Buozzi 1, 39100 Bolzano, Italy

Abstract

Multitask predictors apply task-specific functions to a shared representation. Penalizing the representation and task functions separately can assign different costs to equivalent predictors and charge for directions the tasks do not use. We study the trace norm of the complete task predictors in a fixed input reproducing kernel Hilbert space, weighted by task sample sizes. For a finite-dimensional intermediate function space, this norm is the infimum of the product of the representation operator norm and task trace norm over invertible coordinate changes. Task inner products determine this norm; standard duality gives the exact empirical Rademacher complexity of its norm ball. We estimate the feature Gram matrix used in fitting. Under bounded leverage, one estimate controls the penalty for every task head and shared linear combination of fixed features, even when the Gram is singular and the coefficients are chosen after estimation. This yields comparisons of penalized empirical objectives and a generalization bound under an estimated norm constraint, with the stated loss and sampling assumptions. Controlled experiments show that signal alignment can favor either penalty. A SARCOS comparison finds close agreement between exact and estimated penalties on overlapping training and evaluation observations.

Figures & tables

Figure 1: Regularization and Gram estimation with fixed nonlinear features. (a) Excess error for the separated penalty minus excess error for the composed penalty, with descriptive

95\%

Student-

t

intervals over

16

paired seeds in each cell. Positive values favor the composed penalty. (b) Median operator norm of the error in the whitened

L^{2}

feature Gram matrix over

32

independent repetitions; shading shows the

10

th–

90

th percentiles. Within each repetition, larger reference samples contain the smaller samples. Orthonormalizing the features gives an equivalent implementation of the composed penalty.

$n$	Exact Gram	Estimated Gram	Separated	Estimated minus exact mean [95% interval]
64	0.65904	0.65873	0.66392	$-0.000309\;[-0.000709,0.000092]$
256	0.57404	0.57407	0.57534	$0.000033\;[-0.000242,0.000307]$

Table 1: SARCOS: mean standardized MSE over all 4,449 evaluation rows, including observations used in preparation, training or validation, averaged across seven torques and five paired seeds. The final column gives estimated minus exact mean error and its descriptive 95% paired interval across training seeds, using the recorded paired differences rather than the rounded means.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 2 — Figure 2: Deterministic mechanism checks. (a) Equivalent factorizations of one predictor keep joint geometry fixed while the separated product grows. (b) Fixed marginal spectra and a fixed separated product do not determine joint geometry. (c) Cancelling operator pairs keep the composed map fixed while layer-norm products grow exponentially

$\rho$	Truth directions	Composed	Separated	Gain [ $95\%$ interval]
1	Largest	0.13503	0.13503	$0\ [0,0]$
1	Smallest	0.13315	0.13315	$0\ [0,0]$
1	Mixed	0.13748	0.13748	$0\ [0,0]$
4	Largest	0.13503	0.08591	$-0.04912\ [-0.06114,-0.03710]$
4	Smallest	0.13315	0.18519	$0.05204\ [0.04039,0.06369]$
4	Mixed	0.13748	0.17608	$0.03860\ [0.02971,0.04749]$

Appendix

Table 2: Exact population excess and paired gains in all

9

prespecified composition cells. Each mean uses

16

paired seeds.

Quantity	Replication	Structural transfer
Input dimension / tasks	$8/12$	$10/15$
Ranks	$1,2,4$	$1,3,5$
Imbalance ratios $\rho$	$1,8,32$	$1,4,16,64$
Minimum train count	$24$	$20$
Validation / test per task	$64/256$	$80/320$
Noise standard deviation	$0.75$	$0.8$

Appendix

Table 3: Data and metric setup. Training counts are nearest integers on a geometric progression from the minimum count to

\rho

times that count, with exact endpoints. Population excess MSE is

\lVert F-F^{\star}\rVert_{\mathrm{HS}}^{2}/T

, without the optimization loss’s factor

1/2

. Low-resource excess averages squared column errors over the first

\max(1,\lfloor T/4\rfloor)

indices obtained by sorting the counts; the implementation uses NumPy’s default sorting order for ties.

Parameter	Replication	Transfer
Shared spectral endpoints	$4,\ 0.25$	$3.5,\ 0.2$
Task spectral endpoints	$1.5,\ 0.4$	$1.4,\ 0.35$
Basis / generator / right-basis seeds	$117/223/329$	$811/823/829$
Global seed	$20261117$	$20261203$
Data seeds	$101,102,103$	$201,202,203$
Evaluated angles	$0.3,0.7,1.1,1.5$	$0.25,0.65,1.05,1.45$

Appendix

Table 4: Fixed geometry and scenario parameters in the supplied suite configurations. Dimensions, counts and metric definitions appear in Table 3 . Angles are parameters of the orthogonal path

\exp(\theta A)Q_{0}

Panel of Figure 3 — Figure 3: Paired effects over $252$ evaluation scenarios. Points show mean effects and bars show $95\%$ grouped-bootstrap intervals; positive values favor weighted joint nuclear. (a) Population excess; (b) least-sampled-quartile excess

Panel of Figure 4 — Figure 4: Correct-count gains by suite, rank, and imbalance. Effects are shifted-count excess minus correct-count excess. They vanish at balance and are positive in all $15$ imbalanced cells for both metrics. (a) Population excess; (b) least-sampled-quartile excess

Suite	Baseline	Population effect	Low-resource effect	Pairs
Replication	Unweighted nuclear	$0.00217$ $[0.00046,0.00434]$	$-0.00502$ $[-0.00755,-0.00258]$	$108$
Transfer	Unweighted nuclear	$0.01174$ $[0.00714,0.01717]$	$0.00181$ $[-0.00145,0.00516]$	$144$
Replication	Shifted counts	$0.00762$ $[0.00330,0.01302]$	$0.02548$ $[0.01206,0.04158]$	$108$
Transfer	Shifted counts	$0.01304$ $[0.00718,0.01986]$	$0.03071$ $[0.01568,0.04827]$	$144$
Replication	Weighted Frobenius	$0.04059$ $[0.03163,0.05071]$	$0.06266$ $[0.05397,0.07025]$	$108$
Transfer	Weighted Frobenius	$0.04056$ $[0.03504,0.04658]$	$0.03800$ $[0.03036,0.04556]$	$144$

Appendix

Table 5: Suite-level paired effects. Positive values favor weighted joint nuclear regularization.

Criterion	Threshold	Result
Unweighted population	Mean $\geq 0.001$ ; pooled lower endpoint and suite means positive	$0.00764$ ; lower $0.00465$ ; both means positive
Shifted-count population	Mean $\geq 0.004$ ; pooled/suite lower endpoints positive; $\geq 13$ positive cells	$0.01072$ ; lower $0.00686$ ; $15/15$ cells
Shifted-count low resource	Mean $\geq 0.010$ ; analogous interval and cell criteria	$0.02847$ ; lower $0.01805$ ; $15/15$ cells
Weighted Frobenius	Both means $\geq 0.020$ and lower endpoints positive	$0.04057/0.04857$ ; both intervals positive
Independent ridge	Same as Frobenius	$0.05580/0.05415$ ; both intervals positive
Balanced-count null	Maximum absolute cell mean $\leq 10^{-12}$	$0$

Appendix

Table 6: Protocol criteria and experimental results. Numerical diagnostics are detailed below.

Multi-Task Learning

Multi-task learning trains models to solve related tasks together by sharing representations, parameters, or supervision.

See all 392 papers

Multi-task Linear Regression without Eigenvalue Lower Bounds: Adaptivity, Robustness, and Safety
May 16, 2026Seok-Jin KimRobust Regression Multi-Task Learning
Columbia IEOR
Understanding and Enforcing Weight Disentanglement in Task Arithmetic
Apr 18, 2026Shangge Liu, Yuehan Yin, Lei Wang +5Task Vectors Weight-Space Analysis
State Key Laboratory for Novel Software Technology, Nanjing University, China · University of Wollongong, Australia · Nanyang Technological University, Singapore
The Geometry of Multi-Task Grokking: Transverse Instability, Superposition, and Weight Decay Phase Structure
Feb 19, 2026Yongzhong XuGrokking Neural Network Training Dynamics

May 16, 2026J/K move · Enter open · S save

Multi-task Linear Regression without Eigenvalue Lower Bounds: Adaptivity, Robustness, and Safety

Seok-Jin Kim

Columbia IEOR

Robust Regression Multi-Task Learning Statistical Learning Theory Robust Estimation

Open paper AskPDF arXiv

We study the multi-task linear regression problem in the presence of contaminated tasks. We address the setting where the unknown parameters of a majority of tasks are close in the

\ell_2

-norm, while a fraction of tasks are arbitrary outliers. Existing theoretical frameworks for this problem rely heavily on the assumption that the empirical second moment of each task has a minimum eigenvalue bounded away from zero (order

Ω(1)

). Crucially, this assumption fails in many high-dimensional scenarios, rendering prior guarantees vacuous. To overcome this limitation, we propose an estimator based on matrix-weighted norm regularization. We also introduce a relative balancedness condition, quantified by a balancedness constant, that compares each task's second moment with the average inlier geometry and relaxes the need for taskwise second-moment lower bounds. In favorable regimes with moderate balancedness, our prediction MSE bounds match the rate of Duan and Wang (2023) under substantially weaker spectral assumptions; the resulting task-overall MSE is minimax optimal up to logarithmic factors. Furthermore, we demonstrate that our estimator enjoys a safety guarantee: when the relevant balancedness constant is large or infinite, or when tasks are unrelated, the method performs no worse than independent task learning.

Abstract

Multi-Task Learning

Explore similar work

Multi-task Linear Regression without Eigenvalue Lower Bounds: Adaptivity, Robustness, and Safety

Understanding and Enforcing Weight Disentanglement in Task Arithmetic

The Geometry of Multi-Task Grokking: Transverse Instability, Superposition, and Weight Decay Phase Structure