One of the fundamental differences between biological neural networks and artificial neural networks (ANNs) is that biological neurons obey Dale's principle: each neuron projects either exclusively excitatory or exclusively inhibitory outputs. While this constraint is central to biological circuits, previous attempts to impose Dale's principle in ANNs have often led to training instability or degraded performance. In this work, we introduce EISep RNN, a recurrent neural network architecture that implements Dale's principle through a learnable excitatory-inhibitory separation. Across a series of learning settings, including multi-task learning, continual learning, and reinforcement learning, EISep achieves more stable training dynamics, competitive or improved performance, and sparser, more modular connectivity than vanilla RNNs and previous Dale-constrained models. EISep also exhibits stable and informative latent dynamics. These results establish Dale's principle as a biologically grounded and computationally effective inductive bias for recurrent neural networks.
Figures & tables
Figure 1: A. In Simple RNN, each neuron can project positive (red) and negative (blue) weights. B. In EISep RNN, each neuron can only project positive weights (excitatory), or negative weights (inhibitory). This sign is learnable (left to right) during training.
Model
Initialization
E-I sign
Simple
Gaussian
–
ColEI ( Li et al., 2023a )
Exponential
fixed
DBPN ( Balwani et al., 2025 )
Uniform
fixed
DANN ( Cornford et al., 2020 )
Exponential
fixed
EISep (ours)
Rect. Gauss.
learnable
Table 1: Comparison of initialization methods and E-I sign learnability across Dale-constrained ANNs. Rect. Gauss. denotes a rectified Gaussian distribution.
Figure 2: Multi-task cognitive benchmark consisting of 20 cognitive tasks. The same legend at the right of D is used throughout the figure. A. EISep consistently achieves higher accuracy than prior Dale-constrained baselines and matches Simple across tasks. B. EISep achieves the highest average performance. C. At high learning rates, EISep maintains higher test accuracy (left), lower gradient norms (middle), and tighter recurrent balance (right) than Simple. D. EISep exhibits lower decoding loss (left), lower gradient norms (middle), and more stable spectral radius trajectories (right) throughout training, indicating more stable dynamics. All models use the same learning rate 10−2 except for C .
Figure 3: Case study on the 2AFC task. Neurons are grouped by selectivity using the signal-detection index d′ . A. EISep exhibits a clear block structure in the recurrent weight W , with within-group excitation and cross-group inhibition. B. Averaged connectivity within neuron groups further reveals strong self-excitation and mutual inhibition. C-D. Simple lacks clear separation between intra-group and inter-group connectivity. E. The emergent EISep connectivity resembles the conceptual winner-take-all circuit commonly used in perceptual decision-making models ( Wong and Wang, 2006 ) .
Figure 4: Population analysis on the multi-task cognitive benchmark. Neurons are partitioned into groups using two clustering methods. EISep achieves the highest modularity under ( A. ) directed spectral clustering and ( B. ) lifted spectral clustering, indicating more modular circuit organization. C. EISep also achieves substantially higher recurrent weight sparsity than baseline models.
Figure 5: Continual learning on four sequentially presented cognitive tasks repeated over 10 rounds. A. EISep achieves higher performance and stronger resistance to catastrophic forgetting than Simple across learning rates. B. EISep exhibits high gradient similarity for similar tasks ( Go vs. Delay Go ) and near-zero similarity for opposing tasks ( Go vs. Anti ), indicating reduced gradient interference. C. EISep shows structured correlations between task-induced weight updates, with positive correlations for similar tasks and negative correlations for opposing tasks. D. EISep’s performance shifts from Go to Anti as transfer strength α increases.
Figure 6: Reinforcement learning benchmarks using recurrent A2C agents. A. Four actor–critic variants combining Simple and EISep recurrent modules. B. On the discrete Lunar Lander task, EISep achieves higher rewards than Simple. C. On the more challenging HalfCheetah task, EISep again outperforms Simple, demonstrating improved optimization stability and control performance across discrete and continuous RL settings.
Figure 7: Hidden dynamics and ablation analysis. A. EISep exhibits smooth and diverse hidden dynamics during Lunar Lander control. B. Simple shows high-frequency repetitive jittering, suggesting low-dimensional limit-cycle collapse. C. EISep achieves substantially higher normalized mutual information between HMM latent states and manually annotated environment phases, indicating that its hidden dynamics encode richer environment information. D-F. Across multi-task learning ( D ), continual learning ( E ), and reinforcement learning ( F ), EISep consistently matches or outperforms ablation models, highlighting the importance of initialization method, learnable signs, biologically grounded E-I separation, and the ReLU parameterization.
Appendix figures & tables35 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Under different hidden sizes, the performance in multi-task learning ( A ), continual learning ( B ) and reinforcement learning ( C ).
Figure 9: Under different temperature for Gumbel-Softmax sampling, the performance in multi-task learning ( A ), continual learning ( B ), and reinforcement learning ( C ).
Figure 10: Under different E-I logit entropy loss coefficients, the performance in multi-task learning ( A ), continual learning ( B ) and reinforcement learning ( C )).
Figure 11: Under activation functions, the performance in multi-task learning ( A ) and continual learning ( B ).
Model
Default initialization
EISep-like initialization
EISep
0.9699±0.0025
0.9699±0.0025
Simple
0.9654±0.0089
0.9705±0.0024
DANN-SGD
0.8344±0.0215
0.6276±0.0217
DANN-Adam
0.7503±0.0558
0.6528±0.0523
ColEI-SGD
0.7095±0.0545
0.7065±0.0397
ColEI-Adam
0.6108±0.0918
0.6230±0.0980
Appendix
Table 2: Performance under default and EISep-like initialization.
Model
Final spectral radius
Accuracy
EISep
2.7762±0.1374
0.9745±0.0024
Simple
6.9175±0.5926
0.9705±0.0024
DANN-SGD
1.0200±0.0210
0.8539±0.0118
DANN-Adam
14.4388±4.4922
0.8203±0.0355
ColEI-SGD
0.9769±0.0281
0.6734±0.0618
ColEI-Adam
5.8978±0.7504
0.7823±0.1355
Appendix
Table 3: Control with matched initial spectral radius. All models have the same initial spectral radius, 0.6715±0.0093 , measured across all 5 seeds. Final spectral radius and performance are reported as mean ± SEM over valid terminal runs: 5 seeds for every model except ColEI-Adam (3 valid seeds).
Figure 12: Under different training orders, the performance in continual learning for all baseline models ( A, C ) and ablation models ( B, D ).
Figure 13: The performance of EISep-ReInit in multi-task learning ( A ), continual learning ( B )
Figure 14: Under different truncation proportions, the performance in multi-task learning ( A ), continual learning ( B ), and reinforcement learning ( C )
Figure 15: Under different L1 coefficients, the sparsity and performance in multi-task learning ( A, B ), continual learning ( C, D ), and reinforcement learning ( E, F ). In RL, EISep(A+C) is compared with Simple, whose actor and critic are both L1-regularized with the same coefficients.
Figure 16: Training metrics under different learning rates in the cognitive task suite, including ( A. ) loss, ( B. ) gradient norm, ( C. ) spectral radius, and ( D. ) recurrent balance.
Figure 17: Performance and stable zones for different baselines vs. learning rates in the cognitive task suite.
Figure 18: In continual learning, a full task sequence is composed of four tasks, and the sequence is repeated for 10 times, resulting in 40 task trials in total. A-D. Training curve in continual learning for different models and learning rates. For a two-channel stimulus, the two “go” tasks require selecting the stronger channel, while the two “anti” tasks require the weaker one. The two “delay” variants insert a delay period between stimulus and response. E. Average performance across all tasks over training.
Figure 19: Loss in continual learning for tasks in Fig. 5 . A. Loss curve of EISep with learning rate 1×10−2 . B. Loss curve of Simple with learning rate 5×10−3 . C. Loss curve of Simple with learning rate 1×10−2 .
Figure 20: A–G. Training curves of different models with their best learning rates. H. Average performance across tasks. I. Final-round performance averaged over four trials. J. The sparsity of all models across trials.
Figure 21: Performance in general tasks, including the adding problem ( A ), sequential MNIST ( B ) and sorting problem ( C ).
Figure 22: The EI logit entropy loss during the training stage for EISep. The fast convergence of the entropy objective indicates that the neuron signs also converge fast.
Figure 23: The EI logit matrix M during the entire training stage, recorded at every 150 steps.
Figure 24: The performance of masked models and their relative errors with the original models. A . Performance of 7 masked models on 20 cognitive tasks. B . The performance differences of 7 masked models on 20 cognitive tasks. C . Performance of 7 models averaged across tasks. D . The performance differences of 7 models averaged across tasks.
Figure 25: Performance of other Dale’s networks in the Lunar Lander task.
Figure 26: Performance of EISep FFN on the Lunar Lander task.
Figure 27: Performance for EISep LSTM with shared E-I identities across gates (left) or with independent identities across gates (right).
Figure 28: Performance for EISep GRU on HalfCheetah.
Figure 29: Sparsity averaged across all learning rates and seeds in the A. Lunar Lander task, and B. HalfCheetah task.
Figure 30: Extended visualizations of the HMM fitting results for both EISep(A+C) and Simple agents. A. A raw clip (length=100, beginning of the first episode) of hidden state activities for all 64 units in the EISep critic RNN. B. HMM transition matrix across all hidden labels. C. HMM labels (up) and manually annotated phases (bottom) for a clip (length=200, beginning of the first episode). The first 100 timesteps are aligned with the timesteps in A . D-F. Same as A-C for Simple critic RNN. G.-I. Same as A-C for EISep(A+C) actor RNN. J.-L. Same as A-C for Simple actor RNN.
Figure 31: Different metrics of the HMM results of EISep. A. The HMM log-likelihood on the train/test dataset for the critic/actor of the two agents(EISep/Simple). B . Stickiness of the transition matrix. C . Entropy of the transition matrix. D . Switch rate of HMM hidden states. E . Normalized mutual information between HMM hidden states and handmade labels.
Figure 32: Schematic comparison of EISep and its ablation variants. Abs-EISep replaces the ReLU recurrent-weight filter with an absolute-value filter; Fixed-EISep removes learnable neuron signs; EntryEI relaxes neuron-level E–I separation into entry-wise sign masks; EISep(N1/N2) preserves the EISep parameterization but changes the initialization scheme; and Cstr-Simple tests whether EISep-style initialization alone explains the observed gains.
Figure 33: HMM metrics for EntryEI(A+C), including ( A ) log-likelihood, ( B ) stickiness of transition matrices, ( C ) entropy of transition matrices, ( D ) switch rate of HMM labels, and ( E ) normalized mutual information between HMM labels and handmade labels. Same as Fig. 31
Figure 34: Performance in the Lunar Lander task for different ablation models. The performance of Cstr-Simple has been shown in Sec. 6 .
Figure 35: Metrics on the cognitive task suite for different ablation models. We compare Abs-EISep, Fixed-EISep, and EntryEI against the EISep and Simple baselines. A . Task-wise performance of the 5 models across 20 cognitive tasks. B . Performance averaged across all tasks for each model. C . Recurrent weight sparsity averaged across tasks for different models. D . Training dynamics over the full training process, including decoding loss (left), gradient norm (middle), and spectral radius of the recurrent weight matrix (right).
Figure 36: Performance and stable zones for different ablation models vs. learning rates in the cognitive task suite.
Figure 37: The spectral radius of different models at initialization ( n=1000 , Error bar indicates standard error).
Figure 38: A–H. Training curves of different ablation models with their best learning rates in continual learning. I. Average performance across tasks. J. Final-round performance averaged over four trials. K. The sparsity of all models across trials.
Biological neural circuits obey Dale's principle: each neuron's synapses are uniformly excitatory or inhibitory. Artificial networks that respect this constraint must coordinate separate excitatory and inhibitory populations, fundamentally changing how credit is assigned during learning. Several biologically plausible learning rules avoid backpropagation's weight transport requirement, but it has been difficult to achieve strong performance under Dale's principle beyond MNIST. Error Diffusion (ED) was originally proposed in a dual-stream excitatory/inhibitory architecture, where learning is driven by routing global error signals to all layers without transporting transposed forward weights or relying on random feedback matrices. Whether such a rule can scale under Dale's principle across both supervised classification and reinforcement learning remains unknown. Here, we introduce modulo error routing to extend Error Diffusion beyond binary classification, and show that a dual-stream excitatory/inhibitory architecture trained with this method achieves 96.7% on MNIST and establishes a 61.7% baseline on CIFAR-10, demonstrating that representation learning is possible even when strictly enforcing Dale's principle. For the classification setting, we introduce three domain-specific innovations: layer-specific sigmoid widths, batch-centered class error signals, and asymmetric initialization, and ablation analysis reveals that their relative importance reverses between MNIST and CIFAR-10, exposing task-dependent credit-assignment bottlenecks invisible to single-benchmark evaluation. In reinforcement learning, we integrate ED with Proximal Policy Optimization (PPO) and evaluate it on continuous-control tasks in Google Brax and on Craftax, an open-ended exploration task. We show that ED-PPO achieves competitive performance relative to Direct Feedback Alignment, a backpropagation-free baseline.
Biological neural networks are noisy: their dynamics involve difficult-to-predict fluctuations which are likely due not just to chaos, but also to a variety of low-level biophysical processes. Despite the omnipresence of intrinsic noise, its theoretical consequences are incompletely understood, especially in the context of learning. For example, does noise meaningfully shape representation learning, or is it better viewed as a nuisance factor whose presence or absence changes little about what solution is found and when? Here, we argue that intrinsic noise can significantly modulate representation learning in recurrent neural networks (RNNs), since it can strongly break solution-space degeneracies and in some cases control whether a non-degenerate solution exists at all. We show that this is true for an analytically tractable model of noisy RNN learning with a linear activation function and white noise inputs. We find that, in the so-called rich regime (where weights are initialized with small values), noise strongly modulates the directions in which the read-in, readout, and recurrent weights align. Moreover, we find that noise produces a kind of phase transition: for sufficiently high noise, a non-degenerate solution is not even optimal if parameter scales are limited (e.g., via regularization). These results extend recent analytic work on RNN representation learning and show that noise can act as a fundamental determinant of representations.
John J. Vastola, Kanaka Rajan
Department of Neurobiology, Harvard Medical School · Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University
Recurrent and deep neural networks (RNNs/DNNs) are cornerstone architectures in machine learning. Remarkably, RNNs differ from DNNs only by weight sharing, as can be shown through unrolling in time. How does this structural similarity fit with the distinct functional properties these networks exhibit? To address this question, we here develop a unified mean-field theory for RNNs and DNNs in terms of representational kernels, describing fully trained networks in the feature learning (μP) regime. This theory casts training as Bayesian inference over sequences and patterns, directly revealing the functional implications induced by the RNNs' weight sharing. In DNN-typical tasks, we identify a phase transition when the learning signal overcomes the noise due to randomness in the weights: below this threshold, RNNs and DNNs behave identically; above it, only RNNs develop correlated representations across timesteps. For sequential tasks, the RNNs' weight sharing furthermore induces an inductive bias that aids generalization by interpolating unsupervised time steps. Overall, our theory offers a way to connect architectural structure to functional biases.
Jan P. Bauer, Kirsten Fischer, Moritz Helias +1
Gatsby Computational Neuroscience Unit, UCL, London, UK · IAS-6, Forschungszentrum Juelich, Juelich, Germany