An Analytical Theory of Auxiliary Learning

Federico Milanesio; Alessandro Ingrosso; Matteo Osella

cs.LGSep 24, 2026

An Analytical Theory of Auxiliary Learning

Authors: Federico Milanesio, Alessandro Ingrosso, Matteo Osella

Organizations: Department of Physics and INFN University of Turin Via Giuria 1, 10125 Turin, Italy · Donders Centre for Neuroscience Radboud University Heyendaalseweg 135, 6525 AJ Nijmegen, The Netherlands

Abstract

Auxiliary learning is an optimization paradigm in which a neural network's performance on a target task is improved by jointly training it on additional tasks. However, the mechanisms behind this improvement remain poorly understood. We study this problem using a teacher-student framework and derive a closed system of differential equations describing the dynamics of online stochastic gradient descent in the large-input limit. For linear networks, we obtain a closed-form expression for the generalization error to leading order in the learning rate, quantifying how task correlations and label noise determine the benefit of auxiliary learning. For non-linear activation functions, we develop a fluctuation-dissipation analytical theory that establishes a general relation linking the main and auxiliary errors to the corresponding single-task error. Numerical experiments support the theoretical predictions and show how auxiliary tasks improve generalization by balancing the forcing dynamics towards the optimal solution with gradient noise.

Figures & tables

Figure 1: Schema of the multi-index teacher-student setup.

Figure 2: Performance of linear models. A) Sample learning trajectory obtained with ODEs (continuous line) and averages over ten runs of SGD (dots) when $\alpha=0.25$ and $\beta=0.75$ . The dashed line is the theoretical loss at convergence, and different colors/shapes correspond to different correlations $\rho$ . B) Relative error $\epsilon_{\text{main}}/\epsilon_{0}$ for a linear model with four different $\rho$ trained with online gradient descent, $\alpha=1$ , and different values of $\beta$ . The dashed lines show the theoretical solutions, the continuous lines show the results of numerically solving the mean-field ODEs, the dots show simulation results for models trained with online SGD, averaged over 10000 epochs after convergence across 10 simulations, with the vertical bars denoting the corresponding confidence intervals. Parameters for all simulations are $\mu=0.01$ , $\sigma=0.1$ , $K=4$ , $N=784$ .

Figure 3: Performance of models with different activations. Relative error $\epsilon_{\text{main}}/\epsilon_{0}$ for a model with A) erf activation ( $K=4$ ) and B) ReLU activation ( $K=2$ ), with different $\rho$ trained with online gradient descent, $\alpha=1$ , and different $\beta$ s. The dashed lines show the theoretical solutions, the continuous lines show the results of numerically solving the mean-field ODEs, the dots show simulation results for models trained with online SGD, averaged over 10000 epochs after convergence across 5 simulations for erf and 10 for ReLU, with the vertical bars denoting the corresponding confidence intervals. Results for $\mu=0.01$ , $\sigma=0.1$ , $N=784$ .

Figure 4: Best improvement (theory). Best relative loss against optimal $\beta^{*}$ computed for the model with A) erf and B) ReLU activation, $\alpha=1$ . We uniformly sample ten $\bm{v}$ , $\bm{u}$ with norm $|\bm{v}|=|\bm{u}|=\sqrt{K}$ and given correlation $\rho$ . Results obtained by numerically solving the corresponding Lyapunov equation for $\mu=0.01$ , $\sigma=0.1$ , $K=4$ .

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure S1: Optimal generalization for different noise scales. Leading term of the generalization error on the main task of the two-output linear model computed at the optimal $\beta^{*}$ as a function of noise scale $\delta=\sigma_{\text{aux}}/\sigma$ and different $\rho$ , with $\alpha=1$ .

Figure S2: Higher-order effects in $\mu$ . Relative error $|\epsilon^{\text{ode}}-\epsilon^{\text{theory}}|/\epsilon^{\text{theory}}=\Delta\epsilon/\epsilon^{\text{theory}}$ for different $\mu$ , with $\epsilon^{\text{theory}}$ obtained by numerically solving the corresponding Lyapunov equation, and $\epsilon^{\text{ode}}$ by integrating the ODEs. Average over 10 random choices of $\bm{u}$ , with erf activation, $\alpha=1$ , $\beta=0.5$ , $\sigma=0.1$ , $K=4$ , $\bm{v}=[1,1,1,1]$ , and correlation between tasks $\rho=0.6$ .

Figure S3: Convergence of different initializations. Comparison of how often ODE integration of different initializations of $W$ , $W^{*}$ , $\bm{v}$ , $\bm{u}$ converges to the theoretical $\epsilon^{\text{theory}}$ , when $\beta=\sqrt{2}-1$ and when $\beta=0$ . We sample entries independently from a standard normal distribution. Every row is the same teacher initialization of $\bm{v}$ , $\bm{u}$ , and $W^{*}$ ; every column is the same initialization of $W$ . We plot 30 random initializations of $W$ for each of 30 teachers, with erf activation, $\alpha=1$ , $\mu=0.01$ , $\sigma=0.1$ , $K=4$ . Convergence when the relative error is less than 5%.

		$\beta=0$
		converges	does not converge
$\beta=\sqrt{2}-1$	converges	451	374
$\beta=\sqrt{2}-1$	does not converge	6	69

Appendix

Table S1: Convergence of different initializations. Paired counts for the runs in Figure S3 . McNemar exact test,

p<10^{-10}

Figure S4: Performance of models with $\alpha=1-\beta$ . Relative error $\epsilon_{\text{main}}/\epsilon_{0}$ for a model with A) linear, B) erf activation, and C) ReLU activation, and different $\rho$ values trained with online gradient descent, $\alpha=1-\beta$ , and different $\beta$ values. The dashed lines show the theoretical solutions, the continuous lines show the results of numerically solving the mean-field ODEs, the dots show simulation results for models trained with online SGD, averaged over 10000 epochs after convergence across 5 simulations for linear and erf and 10 for ReLU, with the vertical bars denoting the corresponding confidence intervals. $\mu=0.01$ , $\sigma=0.1$ , $N=784$ , $K=2$ for ReLU and $K=4$ for linear and erf.

Explore similar work

A Theoretical Analysis of Generalization Dynamics in Neural Networks under Gradient Descent with Weight Decay
Sep 7, 2026Yuqing Wang, Ioannis G. Kevrekidis, Mikhail BelkinTwo-Layer Neural Networks Generalization Bounds
Johns Hopkins University · University of California San Diego
Learning Through Noise: Why Subliminal Learning Works and When It Fails
May 22, 2026Vincent C. Brockers, Roman D. Ventzke, Valentin Neuhaus +2Singular Learning Theory
Max Planck Institute for Dynamics and Self-Organization, Göttingen · Faculty of Physics, Institute for the Dynamics of Complex Systems, University of Göttingen
The Dynamics of Generalization in Deep Learning
Apr 23, 2025Rubing Yang, Pratik ChaudhariGradient Descent Overparameterization
Department of Applied Mathematics and Computational Sciences, University of Pennsylvania Philadelphia, PA, 19104, USA · Department of Electrical and Systems Engineering, University of Pennsylvania Philadelphia, PA, 19104, USA

Sep 7, 2026, cs.LGJ/K move · Enter open · S save

A Theoretical Analysis of Generalization Dynamics in Neural Networks under Gradient Descent with Weight Decay

Yuqing Wang, Ioannis G. Kevrekidis, Mikhail Belkin

Johns Hopkins University · University of California San Diego

Two-Layer Neural Networks Generalization Bounds Gradient Descent Weight Decay Neural Network

Open paper AskPDF arXiv

Understanding generalization remains a central challenge in machine learning because it requires jointly considering data, architecture, and training dynamics. In this paper, we develop a theoretical framework that characterizes how these factors jointly shape generalization performance throughout training. More precisely, we study a broad class of neural networks trained under the

\ell^2

loss by gradient descent (GD) with weight decay, and prove the convergence of GD to a neighbourhood of the global minimizers of the empirical loss. By partitioning the space based on the input data, we then decompose the population error into data error, optimization error, and prediction variation error, and bound them separately. In particular, for the prediction variation error, which measures the oscillations of the learned function, we propose (local) approximate homogeneity and derive explicit cellwise and layerwise bounds for its evolution along the training trajectory. These bounds yield two important implications: a necessary condition of improved generalization explains differences in layerwise generalization behavior; a sufficient condition describes delayed generalization and provides a theoretical characterization of grokking.

Abstract

Explore similar work

A Theoretical Analysis of Generalization Dynamics in Neural Networks under Gradient Descent with Weight Decay

Learning Through Noise: Why Subliminal Learning Works and When It Fails

The Dynamics of Generalization in Deep Learning