cs.LGOct 24, 2026
SaveMultitask Interactions in Recurrent Dynamics across 100 Cognitive Tasks
Organizations: Ernst Strüngmann Institute (ESI) of the Max Planck Society, Germany · Goethe University (GU) Frankfurt, University Hospital, Department for Psychiatry, Psychosomatic Medicine and Psychotherapy, Germany · GU Frankfurt, Cooperative Brain Imaging Center, Germany
Abstract
The brain continually learns tasks with very different computational demands. Recurrent neural networks (RNNs) have been used as dynamical surrogates for studying task solutions, with recent work extending this approach to joint training on multitask batteries. However, it remains unclear how tasks interact as these batteries grow in scale and heterogeneity: under what conditions do they transfer or interfere with one another, and how do these effects relate to task complexity and model structure? We study this question by simultaneously training hierarchical Almost-Linear RNNs (AL-RNNs) on 100 tasks spanning a wide range of cognitive demands. AL-RNNs allow us to vary the amount of nonlinearity in the recurrence, while hierarchical task features provide a low-dimensional interface for task-specific adaptation within shared dynamics. On this task battery, performance improves with both nonlinear capacity and task-feature dimensionality, with the joint hierarchical model outperforming both individually trained models and a single unstructured joint model. Many tasks are individually solvable with linear dynamics, yet cannot be solved together by a shared linear model. Through hierarchical task features, task-specific linear region usage, and task geometry, the trained model provides several related but distinct measures of task similarity that capture how computations are separated and combined within the shared backbone. Across a series of experiments, we relate these representations to individual task complexity, joint solvability, pairwise transfer learning, and leave-one-out adaptation. Together, our experiments characterize task interactions in multitask recurrent learning while demonstrating both the potential and the challenges of reusable dynamical backbones for cognitive computation.
Figures & tables
Figure 1: Hierarchical task conditioning and task-similarity spaces. A : Every task receives low-dimensional task features that modulate group-level parameters, producing task-conditioned AL-RNNs within a shared recurrent backbone. B : The learned model supports complementary task representations based on task features, piecewise-linear region usage, and state geometry.
| Model | Test accuracy (%) | Tasks solved | Parameters | |||
| Nonlinear models | ||||||
| LSTM | 32 | – | 16 | 96.3 0.1 | 94 | 87K |
| GRU | 32 | – | 16 | 95.6 0.7 | 94 | 66K |
| Vanilla RNN | 64 | – | 16 | 93.8 0.4 | 90 | 81K |
| Hierarchical AL-RNN | 64 | 32 | 16 | 96.5 0.3 | 94 | 82K |
| Flat joint | 234 | 16 | – | 80.9 4.3 | 65 | 81K |
Table 1: Performance of the best nonlinear model for each configuration and the corresponding linear models, with matched parameter counts and samples per task. A task counts as solved if its test accuracy reaches at least 80% in at least 3 of the 5 seeds. † 52 tasks are solved by at least one of the 5 seeds of the linear individual model; we refer to these as linear tasks.
Figure 2: A : Test accuracy of the hierarchical model (AL-RNN, ) across task-embedding dimension and nonlinear units , averaged across 5 seeds. B : Epochs to reach test accuracy per task at and for individual models ( ), the hierarchical model ( ) and a parameter-matched flat model ( ), averaged over the seeds that reached threshold. Grey: never reached. Purple names: tasks solved by a fully linear individual model.
Figure 3: Task-similarity spaces learned by the hierarchical AL-RNN for A task features, B geometry, and C linear region usage, for and . Tasks are ordered by clustering the hierarchical task features (clustering details and task labels reported in Appx. A.2 ).
Figure 4: A : Example of three tasks (HeadDirection, NoiseCleaner+int and PerceptualDM+int) close in the task similarity spaces ( ) leveraging shared attractor structure. Top: Network states during the integration/delay phase, projected onto the first two PCs of HeadDirection’s state space. The ring attractor is approximated by piecewise linear subregions (see also Fig. 5 ). Points of the 12 most visited regions, coloured by the linear region they occupy (grey for all others), appear at similar positions on all three rings. Bottom: 30 example trials per task, coloured by trial progress (dark = trial start, yellow = trial end). B : Five tasks with their respective task modifiers in the hierarchical feature space cluster together. C : Pro and Anti variants of the same tasks are placed in opposing linear subregions. ++ denotes +int+seq modifier.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
| Response type | Correct if |
|---|---|
| Angular | |
| Binary | output and target on same side of |
| Scalar / vector | readout error |
| -way categorical | argmax over readout channels matches target |
| Sequence production | at each step |
| Flip-flop | correct sign on every bit channel at each step |
Table 2: Trial-level accuracy criteria by response type. denotes the angle decoded from the output pair; the standard deviation of the task’s target distribution. Single-response tasks are evaluated at the last scored timestep; sequential tasks are scored as the fraction of scored timesteps meeting the criterion.
| P | |||||||||
|---|---|---|---|---|---|---|---|---|---|
| 1 | 72.9 | 76.5 | 79.4 | 81.6 | 83.7 | 84.9 | 86.1 | 87.1 | 14.2 |
| 2 | 78.0 | 82.2 | 85.1 | 87.3 | 89.2 | 90.1 | 91.1 | 91.8 | 13.8 |
| 4 | 81.6 | 85.5 | 88.2 | 90.1 | 91.7 | 92.5 | 93.3 | 93.8 | 12.2 |
| 8 | 85.0 | 89.0 | 91.5 | 93.1 | 94.4 | 95.0 | 95.5 | 95.9 | 10.9 |
| 16 | 89.0 | 92.4 | 94.1 | 95.2 | 95.8 | 96.2 | 96.4 | 96.6 | 7.6 |
Table 3: Robustness of mean battery accuracy to the angular accuracy threshold, averaged across 5 seeds. Values show mean test accuracy (%) for models with . The threshold is used in the main text. Although absolute accuracy increases with the tolerance, the relative ordering of nonlinear capacities is largely preserved. The final column reports the change between the most and least permissive thresholds.
Figure 5: Hierarchical AL-RNN ( ) trained on the HeadDirection task. A : Probability that each ReLU unit is active, , as a function of heading (one concentric ring per unit, ordered by the onset of its active arc). Each unit switches on over a single contiguous arc, partitioning the circle into sectors whose boundaries tile the ring attractor (Fig. 4 ). An exactly continuous attractor would require a one-dimensional manifold of fixed points with marginal stability ( ) everywhere along it. This is a measure-zero condition that is not learnable by gradient descent in practice. The network converges instead to a robust piecewise-linear approximation, where local dynamics within each region are either near-marginal (rotation-like, carrying the heading forward) or, in a few regions, weakly contracting toward an embedded fixed point (3 stable nodes, 2 saddles), producing slow drift toward these discrete resting headings. This tiling is fine enough to preserve near-perfect heading decoding ( from the full state, from the bitcode alone) and the resulting drift remains small within typical trial lengths ( 40 timesteps). B : Task accuracy as a function of the angular error threshold, for different numbers of nonlinear units, averaged across 5 seeds. As in our other threshold-ablation analyses (Table 3 ), the choice of accuracy threshold affects how well different architectures are distinguished, where progressively tighter thresholds reveal a graded improvement in ring-attractor approximation quality with increasing nonlinearity.
Figure 6: Test accuracy (%) as a function of nonlinearity and training sample size per task, for A individual models (hidden size ) trained separately on each of 100 tasks and B a single flat model (hidden size ) jointly trained on all 100 tasks. Values are means over 5 seeds.
Figure 7: Individually linear tasks require nonlinear capacity when learned jointly. We constructed subsets of tasks for which every task was solved by an individually trained linear AL-RNN, and trained joint models with varying numbers of nonlinear units. A: Each cell reports the fraction of six runs that solved all tasks, values in parentheses show the mean fraction of tasks solved. B : Minimum number of nonlinear units required to solve every task in a subset reliably. The dotted line shows the idealized lower bound obtained by assigning each task a distinct nonlinear region. The larger number required in practice likely reflects the additional constraint of shared recurrent parameterization in the AL-RNN.
Figure 8: Four example tasks, each shown with one of its closest neighbours by geometric overlap. Tasks placed close together also tend to produce trajectories of similar shape.
| Similarity space / pair | ||||||
|---|---|---|---|---|---|---|
| Within-space reproducibility | ||||||
| Hierarchical features | ||||||
| Task geometry | ||||||
| Nonlinear usage | ||||||
| Between-space similarity | ||||||
Table 4: Robustness and overlap of task-similarity spaces across random seeds. Entries report mean seed-pair Pearson correlations, with denoting half the observed range. The upper panel measures reproducibility within each similarity space, while the lower panel measures correlations between spaces. The robustness of a task representation is partly coupled to task performance, since poor performance constrains the feature vectors and dynamical activity less. On the other hand, particularly for the nonlinearity usage, we found that increasing capacity enforces less sharing between model components, and tasks are separated into increasingly distinct regions, reducing off-diagonal similarity and making the nonlinear usage matrix approach an identity-like structure. Accordingly, the cross-space agreement was generally highest at intermediate nonlinear capacity, consistent with sparse nonlinearity providing task differentiation without over-separating tasks into distinct linear regions.
| Model | Geometry | Task features |
|---|---|---|
| LSTM seed consistency | 0.81 | 0.76 |
| GRU seed consistency | 0.78 | 0.74 |
| AL-RNN–LSTM feature similarity | 0.53 | |
| AL-RNN–GRU feature similarity | 0.57 | |
Table 5: Seed reproducibility and cross-architecture agreement for task representations across 5 seeds. The nonlinear feature space is only feasible for the AL-RNN, and hence omitted here. Models with settings as in Table 1 ( , ).
Figure 9: Geometric task similarity matrices for the hierarchical ( A ) and flat models ( B ) with latent dimensions and the same task order in both. C : Cumulative variance explained via the first 15 principal components. The hierarchical model compresses activity into fewer dimensions, retaining substantially more overlap among tasks.
Figure 10: A : Linear region usage for a hierarchical AL-RNN with , leading to linear subregions. B : Pro-Anti task pairs are separated in nonlinear space. Each row shows the frequency with which a task visits each linear region of the AL-RNN. Pro and anti variants share some regions but also exhibit distinct usage patterns, indicating that the model reuses shared nonlinear structure while separating computations with opposing response rules.
Figure 11: Isolation in the hierarchical AL-RNN ( ) feature spaces (geometry, task features and linear region usage from left to right), plotted against two different measures of source quality. A : Source quality is computed from the individual-model transfer matrix: one network per task ( ), fine-tuned onto every other task. Red points are the 10 worst sources when considering the first 100 training epochs (E100). B : Source quality computed from the leave-one-out experiments, where the task feature for the target task was fine-tuned initialized from the source task’s task feature.
| Task | Description | Readout |
|---|---|---|
| DelayPro | A direction cue is presented, followed by a delay phase. After delay, the network reports it once the fixation cue is removed [ Yang et al., 2019 ] . | angular |
| DelayAnti | As DelayPro, but the reported direction is opposite to the cued one [ Yang et al., 2019 , Munoz and Everling, 2004 ] . | angular |
| RTGo | The stimulus direction is reported as soon as it appears, with no intervening delay [ Yang et al., 2019 ] . | angular |
| RTAnti | As RTGo, but reporting the direction opposite the stimulus [ Yang et al., 2019 , Munoz and Everling, 2004 ] . | angular |
| PerceptualDM | Noisy directional evidence is presented at varying coherence and the net direction must be integrated out of the noise within a given tolerance level [ Britten et al., 1992 ] . | angular |
| NoiseCleaner | A true direction must be estimated from a stream of noisy angular samples within a given tolerance. | angular |
Table 6: The paradigms of the task battery
More in this topic
Multi-Task Learning
Multi-task learning trains models to solve related tasks together by sharing representations, parameters, or supervision.
See all 394 papers