Multitask Interactions in Recurrent Dynamics across 100 Cognitive Tasks
Authors: Sílvia García-Hernández, Leema Hamid, Manuel Brenner
Organizations: Ernst Strüngmann Institute (ESI) of the Max Planck Society, Germany · Goethe University (GU) Frankfurt, University Hospital, Department for Psychiatry, Psychosomatic Medicine and Psychotherapy, Germany · GU Frankfurt, Cooperative Brain Imaging Center, Germany
The brain continually learns tasks with very different computational demands. Recurrent neural networks (RNNs) have been used as dynamical surrogates for studying task solutions, with recent work extending this approach to joint training on multitask batteries. However, it remains unclear how tasks interact as these batteries grow in scale and heterogeneity: under what conditions do they transfer or interfere with one another, and how do these effects relate to task complexity and model structure? We study this question by simultaneously training hierarchical Almost-Linear RNNs (AL-RNNs) on 100 tasks spanning a wide range of cognitive demands. AL-RNNs allow us to vary the amount of nonlinearity in the recurrence, while hierarchical task features provide a low-dimensional interface for task-specific adaptation within shared dynamics. On this task battery, performance improves with both nonlinear capacity and task-feature dimensionality, with the joint hierarchical model outperforming both individually trained models and a single unstructured joint model. Many tasks are individually solvable with linear dynamics, yet cannot be solved together by a shared linear model. Through hierarchical task features, task-specific linear region usage, and task geometry, the trained model provides several related but distinct measures of task similarity that capture how computations are separated and combined within the shared backbone. Across a series of experiments, we relate these representations to individual task complexity, joint solvability, pairwise transfer learning, and leave-one-out adaptation. Together, our experiments characterize task interactions in multitask recurrent learning while demonstrating both the potential and the challenges of reusable dynamical backbones for cognitive computation.
Figures & tables
Figure 1: Hierarchical task conditioning and task-similarity spaces. A : Every task receives low-dimensional task features that modulate group-level parameters, producing task-conditioned AL-RNNs within a shared recurrent backbone. B : The learned model supports complementary task representations based on task features, piecewise-linear region usage, and state geometry.
Model
M
P
Nfeat
Test accuracy (%)
Tasks solved
Parameters
Nonlinear models
LSTM
32
–
16
96.3 ± 0.1
94
87K
GRU
32
–
16
95.6 ± 0.7
94
66K
Vanilla RNN
64
–
16
93.8 ± 0.4
90
81K
Hierarchical AL-RNN
64
32
16
96.5 ± 0.3
94
82K
Flat joint
234
16
–
80.9 ± 4.3
65
81K
Table 1: Performance of the best nonlinear model for each configuration and the corresponding linear models, with matched parameter counts and 200 samples per task. A task counts as solved if its test accuracy reaches at least 80% in at least 3 of the 5 seeds. † 52 tasks are solved by at least one of the 5 seeds of the linear individual model; we refer to these as linear tasks.
Figure 2: A : Test accuracy of the hierarchical model (AL-RNN, M=64 ) across task-embedding dimension Nfeat and nonlinear units P , averaged across 5 seeds. B : Epochs to reach >80% test accuracy per task at P=1 and P=16 for individual models ( M=64 ), the hierarchical model ( Nfeat=16 ) and a parameter-matched flat model ( M=234 ), averaged over the seeds that reached threshold. Grey: never reached. Purple names: tasks solved by a fully linear individual model.
Figure 3: Task-similarity spaces learned by the hierarchical AL-RNN for A task features, B geometry, and C linear region usage, for Nfeat=16 and P=16 . Tasks are ordered by clustering the hierarchical task features (clustering details and task labels reported in Appx. A.2 ).
Figure 4: A : Example of three tasks (HeadDirection, NoiseCleaner+int and PerceptualDM+int) close in the task similarity spaces ( P=16,Nfeat=16 ) leveraging shared attractor structure. Top: Network states during the integration/delay phase, projected onto the first two PCs of HeadDirection’s state space. The ring attractor is approximated by piecewise linear subregions (see also Fig. 5 ). Points of the 12 most visited regions, coloured by the linear region they occupy (grey for all others), appear at similar positions on all three rings. Bottom: 30 example trials per task, coloured by trial progress (dark = trial start, yellow = trial end). B : Five tasks with their respective task modifiers in the hierarchical feature space cluster together. C : Pro and Anti variants of the same tasks are placed in opposing linear subregions. ++ denotes +int+seq modifier.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Response type
Correct if
Angular
∣θ−θ∗∣<36∘
Binary
output and target on same side of 0.5
Scalar / vector
readout error <0.35σ
K -way categorical
argmax over readout channels matches target
Sequence production
∣θt−θt∗∣<36∘ at each step
Flip-flop
correct sign on every bit channel at each step
Appendix
Table 2: Trial-level accuracy criteria by response type. θ denotes the angle decoded from the (cosθ,sinθ) output pair; σ the standard deviation of the task’s target distribution. Single-response tasks are evaluated at the last scored timestep; sequential tasks are scored as the fraction of scored timesteps meeting the criterion.
15∘
20∘
25∘
30∘
36∘
40∘
45∘
50∘
Δ50−15
P
1
72.9
76.5
79.4
81.6
83.7
84.9
86.1
87.1
14.2
2
78.0
82.2
85.1
87.3
89.2
90.1
91.1
91.8
13.8
4
81.6
85.5
88.2
90.1
91.7
92.5
93.3
93.8
12.2
8
85.0
89.0
91.5
93.1
94.4
95.0
95.5
95.9
10.9
16
89.0
92.4
94.1
95.2
95.8
96.2
96.4
96.6
7.6
Appendix
Table 3: Robustness of mean battery accuracy to the angular accuracy threshold, averaged across 5 seeds. Values show mean test accuracy (%) for models with Nfeat=16 . The 36∘ threshold is used in the main text. Although absolute accuracy increases with the tolerance, the relative ordering of nonlinear capacities is largely preserved. The final column reports the change between the most and least permissive thresholds.
Figure 5: Hierarchical AL-RNN ( P=16,Nfeat=16 ) trained on the HeadDirection task. A : Probability that each ReLU unit is active, P(unit>0) , as a function of heading (one concentric ring per unit, ordered by the onset of its active arc). Each unit switches on over a single contiguous arc, partitioning the circle into sectors whose boundaries tile the ring attractor (Fig. 4 ). An exactly continuous attractor would require a one-dimensional manifold of fixed points with marginal stability ( ∣λ∣=1 ) everywhere along it. This is a measure-zero condition that is not learnable by gradient descent in practice. The network converges instead to a robust piecewise-linear approximation, where local dynamics within each region are either near-marginal (rotation-like, carrying the heading forward) or, in a few regions, weakly contracting toward an embedded fixed point (3 stable nodes, 2 saddles), producing slow drift toward these discrete resting headings. This tiling is fine enough to preserve near-perfect heading decoding ( R2≈1 from the full state, R2=0.97 from the bitcode alone) and the resulting drift remains small within typical trial lengths ( ∼ 40 timesteps). B : Task accuracy as a function of the angular error threshold, for different numbers of nonlinear units, averaged across 5 seeds. As in our other threshold-ablation analyses (Table 3 ), the choice of accuracy threshold affects how well different architectures are distinguished, where progressively tighter thresholds reveal a graded improvement in ring-attractor approximation quality with increasing nonlinearity.
Figure 6: Test accuracy (%) as a function of nonlinearity P and training sample size per task, for A individual models (hidden size M=64 ) trained separately on each of 100 tasks and B a single flat model (hidden size M=234 ) jointly trained on all 100 tasks. Values are means over 5 seeds.
Figure 7: Individually linear tasks require nonlinear capacity when learned jointly. We constructed subsets of T tasks for which every task was solved by an individually trained linear AL-RNN, and trained joint models with varying numbers of nonlinear units. A: Each cell reports the fraction of six runs that solved all T tasks, values in parentheses show the mean fraction of tasks solved. B : Minimum number of nonlinear units required to solve every task in a subset reliably. The dotted line shows the idealized lower bound log2T obtained by assigning each task a distinct nonlinear region. The larger number required in practice likely reflects the additional constraint of shared recurrent parameterization in the AL-RNN.
Figure 8: Four example tasks, each shown with one of its closest neighbours by geometric overlap. Tasks placed close together also tend to produce trajectories of similar shape.
Nfeat=8
Nfeat=16
Similarity space / pair
P=8
P=16
P=32
P=8
P=16
P=32
Within-space reproducibility
Hierarchical features
0.56±0.06
0.63±0.06
0.64±0.06
0.76±0.02
0.78±0.04
0.82±0.01
Task geometry
0.67±0.04
0.74±0.04
0.72±0.04
0.77±0.03
0.78±0.02
0.82±0.02
Nonlinear usage
0.66±0.06
0.78±0.05
0.85±0.05
0.71±0.07
0.76±0.03
0.81±0.05
Between-space similarity
Appendix
Table 4: Robustness and overlap of task-similarity spaces across random seeds. Entries report mean seed-pair Pearson correlations, with Δr=(rmax−rmin)/2 denoting half the observed range. The upper panel measures reproducibility within each similarity space, while the lower panel measures correlations between spaces. The robustness of a task representation is partly coupled to task performance, since poor performance constrains the feature vectors and dynamical activity less. On the other hand, particularly for the nonlinearity usage, we found that increasing capacity enforces less sharing between model components, and tasks are separated into increasingly distinct regions, reducing off-diagonal similarity and making the nonlinear usage matrix approach an identity-like structure. Accordingly, the cross-space agreement was generally highest at intermediate nonlinear capacity, consistent with sparse nonlinearity providing task differentiation without over-separating tasks into distinct linear regions.
Model
Geometry
Task features
LSTM seed consistency
0.81
0.76
GRU seed consistency
0.78
0.74
AL-RNN–LSTM feature similarity
0.53
AL-RNN–GRU feature similarity
0.57
Appendix
Table 5: Seed reproducibility and cross-architecture agreement for task representations across 5 seeds. The nonlinear feature space is only feasible for the AL-RNN, and hence omitted here. Models with settings as in Table 1 ( M=64 , Nfeat=16 ).
Figure 9: Geometric task similarity matrices for the hierarchical ( A ) and flat models ( B ) with M=64 latent dimensions and the same task order in both. C : Cumulative variance explained via the first 15 principal components. The hierarchical model compresses activity into fewer dimensions, retaining substantially more overlap among tasks.
Figure 10: A : Linear region usage for a hierarchical AL-RNN with P=2 , leading to 22=4 linear subregions. B : Pro-Anti task pairs are separated in nonlinear space. Each row shows the frequency with which a task visits each linear region of the AL-RNN. Pro and anti variants share some regions but also exhibit distinct usage patterns, indicating that the model reuses shared nonlinear structure while separating computations with opposing response rules.
Figure 11: Isolation in the hierarchical AL-RNN ( Nfeat=16,P=16 ) feature spaces (geometry, task features and linear region usage from left to right), plotted against two different measures of source quality. A : Source quality is computed from the individual-model transfer matrix: one network per task ( P=16 ), fine-tuned onto every other task. Red points are the 10 worst sources when considering the first 100 training epochs (E100). B : Source quality computed from the leave-one-out experiments, where the task feature for the target task was fine-tuned initialized from the source task’s task feature.
Task
Description
Readout
DelayPro
A direction cue is presented, followed by a delay phase. After delay, the network reports it once the fixation cue is removed [ Yang et al., 2019 ] .
angular
DelayAnti
As DelayPro, but the reported direction is opposite to the cued one [ Yang et al., 2019 , Munoz and Everling, 2004 ] .
angular
RTGo
The stimulus direction is reported as soon as it appears, with no intervening delay [ Yang et al., 2019 ] .
angular
RTAnti
As RTGo, but reporting the direction opposite the stimulus [ Yang et al., 2019 , Munoz and Everling, 2004 ] .
angular
PerceptualDM
Noisy directional evidence is presented at varying coherence and the net direction must be integrated out of the noise within a given tolerance level [ Britten et al., 1992 ] .
angular
NoiseCleaner
A true direction must be estimated from a stream of noisy angular samples within a given tolerance.
When does a shared representation become insufficient for jointly supporting multiple tasks? This question lies at the heart of multi-task learning models. Although task interference in shared representations has been widely observed, its onset, measurable form, and capacity boundary remain poorly understood. In this paper, we revisit this problem through the lens of Bell's theorem. We show that if four binary task outcomes can be jointly represented by a common shared representation, their correlations must satisfy the classical CHSH bound ∣S∣≤2. We introduce NCnet, a controlled multi-task model for studying how representational capacity affects task compatibility. Experiments reveal a clear capacity-dependent transition. Once nontrivial task fitting emerges, constrained capacity can yield ∣S∣>2. As capacity further increases, these exceedances progressively disappear and S returns to the bounded region. This systematic pattern provides evidence that limited representational capacity is an important source of the observed incompatibility. We observe similar transitions across CNNs, PLMs, and LLMs, suggesting that the phenomenon extends beyond the controlled NCnet setting. We refer to this capacity-dependent transition as the Bell Boundary, providing a measurable diagnostic for representation bottlenecks, task incompatibility, and capacity allocation in multi-task models.
How is computation organized and reused across tasks and time in a trained recurrent network? Most analyses emphasize the geometry of neural activity, dynamical motifs, or local perturbation growth. We instead study the network's global first-order perturbation response. The finite-horizon Green's operator maps perturbations at each source along a trajectory to their downstream state-space responses and therefore directly represents perturbation routing. Simple reductions of this operator provide task-to-task and time-to-time views of the same computation, while matrix-free products make these views accessible without constructing the full operator. In a flexible multitask recurrent network, task reductions reveal structured reuse of known computational motifs, while temporal reductions reveal causal pathways and how they emerge during training. Our main point is simple: the Green's operator provides a global response geometry for mapping the organization of learned dynamical computation.
James Hazelden
Applied Mathematics, University of Washington · The Allen Institute Seattle, WA, USA
In multi-task learning (MTL) negative transfer is often considered as an optimization artifact, but it can also be viewed as a consequence of limited shared capacity and weak task redundancy. We investigate this effect through a Capacity--Redundancy (CR) identity that decomposes the sum of per-task predictive informations into joint predictive information that includes label redundancy defined via total correlation (TC), and a residual coupling term that quantifies interference left unresolved by the shared representation. Additionally, we show two key results: (i) a clustering-gap decomposition that gives a necessary and sufficient condition for clustered sharing to outperform global sharing, and (ii) a gradient--TC bridge in a Gaussian multi-task model that formally justifies gradient cosine similarity as a proxy for redundancy ordering. Empirically, we estimate the residual coupling Δ from validation residual correlations, showing that clustered LoRA substantially reduces Δ, outperforms size-matched random partitions, and results in statistically significant gains with multi-seed confidence intervals.