Leveraging Dale's Principle as an Inductive Bias in Recurrent Neural Networks

Jiyi Wang; Chenxiao Yang; Yujia Zhao; Jingzhao Zhang; Lu Mi

cs.LGOct 25, 2026

Leveraging Dale's Principle as an Inductive Bias in Recurrent Neural Networks

Authors: Jiyi Wang, Chenxiao Yang, Yujia Zhao, Jingzhao Zhang, Lu Mi

Organizations: Shanghai Qi Zhi Institute, Shanghai, China · Tsinghua University, Beijing, China

Abstract

One of the fundamental differences between biological neural networks and artificial neural networks (ANNs) is that biological neurons obey Dale's principle: each neuron projects either exclusively excitatory or exclusively inhibitory outputs. While this constraint is central to biological circuits, previous attempts to impose Dale's principle in ANNs have often led to training instability or degraded performance. In this work, we introduce EISep RNN, a recurrent neural network architecture that implements Dale's principle through a learnable excitatory-inhibitory separation. Across a series of learning settings, including multi-task learning, continual learning, and reinforcement learning, EISep achieves more stable training dynamics, competitive or improved performance, and sparser, more modular connectivity than vanilla RNNs and previous Dale-constrained models. EISep also exhibits stable and informative latent dynamics. These results establish Dale's principle as a biologically grounded and computationally effective inductive bias for recurrent neural networks.

Figures & tables

Figure 1: A. In Simple RNN, each neuron can project positive (red) and negative (blue) weights. B. In EISep RNN, each neuron can only project positive weights (excitatory), or negative weights (inhibitory). This sign is learnable (left to right) during training.

Model	Initialization	E-I sign
Simple	Gaussian	–
ColEI ( Li et al., 2023a )	Exponential	fixed
DBPN ( Balwani et al., 2025 )	Uniform	fixed
DANN ( Cornford et al., 2020 )	Exponential	fixed
EISep (ours)	Rect. Gauss.	learnable

Table 1: Comparison of initialization methods and E-I sign learnability across Dale-constrained ANNs. Rect. Gauss. denotes a rectified Gaussian distribution.

Figure 2: Multi-task cognitive benchmark consisting of 20 cognitive tasks. The same legend at the right of D is used throughout the figure. A. EISep consistently achieves higher accuracy than prior Dale-constrained baselines and matches Simple across tasks. B. EISep achieves the highest average performance. C. At high learning rates, EISep maintains higher test accuracy (left), lower gradient norms (middle), and tighter recurrent balance (right) than Simple. D. EISep exhibits lower decoding loss (left), lower gradient norms (middle), and more stable spectral radius trajectories (right) throughout training, indicating more stable dynamics. All models use the same learning rate $10^{-2}$ except for C .

Figure 3: Case study on the 2AFC task. Neurons are grouped by selectivity using the signal-detection index $d^{\prime}$ . A. EISep exhibits a clear block structure in the recurrent weight $W$ , with within-group excitation and cross-group inhibition. B. Averaged connectivity within neuron groups further reveals strong self-excitation and mutual inhibition. C-D. Simple lacks clear separation between intra-group and inter-group connectivity. E. The emergent EISep connectivity resembles the conceptual winner-take-all circuit commonly used in perceptual decision-making models ( Wong and Wang, 2006 ) .

Figure 4: Population analysis on the multi-task cognitive benchmark. Neurons are partitioned into groups using two clustering methods. EISep achieves the highest modularity under ( A. ) directed spectral clustering and ( B. ) lifted spectral clustering, indicating more modular circuit organization. C. EISep also achieves substantially higher recurrent weight sparsity than baseline models.

Figure 5: Continual learning on four sequentially presented cognitive tasks repeated over 10 rounds. A. EISep achieves higher performance and stronger resistance to catastrophic forgetting than Simple across learning rates. B. EISep exhibits high gradient similarity for similar tasks ( Go vs. Delay Go ) and near-zero similarity for opposing tasks ( Go vs. Anti ), indicating reduced gradient interference. C. EISep shows structured correlations between task-induced weight updates, with positive correlations for similar tasks and negative correlations for opposing tasks. D. EISep’s performance shifts from Go to Anti as transfer strength $\alpha$ increases.

Figure 6: Reinforcement learning benchmarks using recurrent A2C agents. A. Four actor–critic variants combining Simple and EISep recurrent modules. B. On the discrete Lunar Lander task, EISep achieves higher rewards than Simple. C. On the more challenging HalfCheetah task, EISep again outperforms Simple, demonstrating improved optimization stability and control performance across discrete and continuous RL settings.

Figure 7: Hidden dynamics and ablation analysis. A. EISep exhibits smooth and diverse hidden dynamics during Lunar Lander control. B. Simple shows high-frequency repetitive jittering, suggesting low-dimensional limit-cycle collapse. C. EISep achieves substantially higher normalized mutual information between HMM latent states and manually annotated environment phases, indicating that its hidden dynamics encode richer environment information. D-F. Across multi-task learning ( D ), continual learning ( E ), and reinforcement learning ( F ), EISep consistently matches or outperforms ablation models, highlighting the importance of initialization method, learnable signs, biologically grounded E-I separation, and the $\mathrm{ReLU}$ parameterization.

Appendix figures & tables35 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Under different hidden sizes, the performance in multi-task learning ( A ), continual learning ( B ) and reinforcement learning ( C ).

Figure 9: Under different temperature for Gumbel-Softmax sampling, the performance in multi-task learning ( A ), continual learning ( B ), and reinforcement learning ( C ).

Figure 10: Under different E-I logit entropy loss coefficients, the performance in multi-task learning ( A ), continual learning ( B ) and reinforcement learning ( C )).

Figure 11: Under activation functions, the performance in multi-task learning ( A ) and continual learning ( B ).

Model	Default initialization	EISep-like initialization
EISep	$0.9699\pm 0.0025$	$0.9699\pm 0.0025$
Simple	$0.9654\pm 0.0089$	$0.9705\pm 0.0024$
DANN-SGD	$0.8344\pm 0.0215$	$0.6276\pm 0.0217$
DANN-Adam	$0.7503\pm 0.0558$	$0.6528\pm 0.0523$
ColEI-SGD	$0.7095\pm 0.0545$	$0.7065\pm 0.0397$
ColEI-Adam	$0.6108\pm 0.0918$	$0.6230\pm 0.0980$

Appendix

Table 2: Performance under default and EISep-like initialization.

Model	Final spectral radius	Accuracy
EISep	$2.7762\pm 0.1374$	$0.9745\pm 0.0024$
Simple	$6.9175\pm 0.5926$	$0.9705\pm 0.0024$
DANN-SGD	$1.0200\pm 0.0210$	$0.8539\pm 0.0118$
DANN-Adam	$14.4388\pm 4.4922$	$0.8203\pm 0.0355$
ColEI-SGD	$0.9769\pm 0.0281$	$0.6734\pm 0.0618$
ColEI-Adam	$5.8978\pm 0.7504$	$0.7823\pm 0.1355$

Appendix

Table 3: Control with matched initial spectral radius. All models have the same initial spectral radius,

0.6715\pm 0.0093

, measured across all 5 seeds. Final spectral radius and performance are reported as mean

\pm

SEM over valid terminal runs: 5 seeds for every model except ColEI-Adam (3 valid seeds).

Figure 12: Under different training orders, the performance in continual learning for all baseline models ( A, C ) and ablation models ( B, D ).

Figure 13: The performance of EISep-ReInit in multi-task learning ( A ), continual learning ( B )

Figure 14: Under different truncation proportions, the performance in multi-task learning ( A ), continual learning ( B ), and reinforcement learning ( C )

Figure 15: Under different L1 coefficients, the sparsity and performance in multi-task learning ( A, B ), continual learning ( C, D ), and reinforcement learning ( E, F ). In RL, EISep(A+C) is compared with Simple, whose actor and critic are both L1-regularized with the same coefficients.

Figure 16: Training metrics under different learning rates in the cognitive task suite, including ( A. ) loss, ( B. ) gradient norm, ( C. ) spectral radius, and ( D. ) recurrent balance.

Figure 17: Performance and stable zones for different baselines vs. learning rates in the cognitive task suite.

Figure 18: In continual learning, a full task sequence is composed of four tasks, and the sequence is repeated for 10 times, resulting in 40 task trials in total. A-D. Training curve in continual learning for different models and learning rates. For a two-channel stimulus, the two “go” tasks require selecting the stronger channel, while the two “anti” tasks require the weaker one. The two “delay” variants insert a delay period between stimulus and response. E. Average performance across all tasks over training.

Figure 19: Loss in continual learning for tasks in Fig. 5 . A. Loss curve of EISep with learning rate $1\times 10^{-2}$ . B. Loss curve of Simple with learning rate $5\times 10^{-3}$ . C. Loss curve of Simple with learning rate $1\times 10^{-2}$ .

Figure 20: A–G. Training curves of different models with their best learning rates. H. Average performance across tasks. I. Final-round performance averaged over four trials. J. The sparsity of all models across trials.

Figure 21: Performance in general tasks, including the adding problem ( A ), sequential MNIST ( B ) and sorting problem ( C ).

Figure 22: The EI logit entropy loss during the training stage for EISep. The fast convergence of the entropy objective indicates that the neuron signs also converge fast.

Figure 23: The EI logit matrix $M$ during the entire training stage, recorded at every 150 steps.

Figure 24: The performance of masked models and their relative errors with the original models. A . Performance of 7 masked models on 20 cognitive tasks. B . The performance differences of 7 masked models on 20 cognitive tasks. C . Performance of 7 models averaged across tasks. D . The performance differences of 7 models averaged across tasks.

Figure 25: Performance of other Dale’s networks in the Lunar Lander task.

lr(C) $\backslash$ lr(A)	$\mathbf{3\times 10^{-4}}$	$\mathbf{1\times 10^{-3}}$	$\mathbf{3\times 10^{-3}}$
$\mathbf{1.5\times 10^{-3}}$	$\mathbf{19.90\pm 5.21}$ $-54.30\pm 7.56$ $69.99\pm 7.51$ $-75.01\pm 11.42$	$14.60\pm 2.97$ $7.04\pm 3.08$ $66.74\pm 7.52$ $11.57\pm 2.24$	$1.19\pm 5.62$ $-5.25\pm 2.53$ $5.57\pm 11.14$ $31.98\pm 3.61$
$\mathbf{5\times 10^{-3}}$	$10.63\pm 3.28$ $-54.26\pm 10.73$ $\mathbf{102.57\pm 5.79}$ $-58.65\pm 8.92$	$-6.09\pm 2.68$ $\mathbf{10.13\pm 3.94}$ $101.52\pm 10.22$ $45.41\pm 7.85$	$-17.44\pm 6.36$ $-0.69\pm 2.07$ $61.33\pm 7.55$ $\mathbf{133.91\pm 11.46}$
$\mathbf{1.5\times 10^{-2}}$	$2.98\pm 2.75$ $-77.64\pm 12.96$ $70.89\pm 7.52$ $-25.56\pm 6.63$	$0.04\pm 2.07$ $9.75\pm 3.19$ $93.81\pm 7.89$ $37.88\pm 7.43$	$14.09\pm 5.11$ $-29.33\pm 3.74$ $14.11\pm 9.87$ $52.23\pm 9.68$

Appendix

Table 4: Learning-rate sweep on LunarLander-v3.

lr(C) $\backslash$ lr(A)	$\mathbf{3\times 10^{-4}}$	$\mathbf{1\times 10^{-3}}$	$\mathbf{3\times 10^{-3}}$
$\mathbf{1.5\times 10^{-3}}$	$1387.94\pm 7.32$ $1379.76\pm 23.80$ $1510.74\pm 17.22$ $1359.10\pm 14.13$	$1503.11\pm 7.33$ $1570.41\pm 22.11$ $1550.80\pm 25.58$ $1623.82\pm 26.35$	$1581.90\pm 8.66$ $1659.70\pm 37.18$ $1781.78\pm 46.34$ $1585.68\pm 8.68$
$\mathbf{5\times 10^{-3}}$	$1530.66\pm 5.55$ $1426.88\pm 24.39$ $3089.99\pm 56.89$ $2865.65\pm 92.40$	$1646.83\pm 18.94$ $1676.68\pm 36.92$ $2297.24\pm 89.07$ $2439.39\pm 102.15$	$1644.64\pm 13.33$ $1552.41\pm 20.62$ $2266.02\pm 68.28$ $1971.96\pm 73.05$
$\mathbf{1.5\times 10^{-2}}$	$1592.10\pm 6.12$ $1384.61\pm 23.61$ $2870.01\pm 128.35$ $2542.09\pm 138.04$	$1670.19\pm 5.71$ $\mathbf{2019.97\pm 60.97}$ $3591.95\pm 116.64$ $3258.02\pm 133.81$	$\mathbf{1687.00\pm 20.22}$ $1693.77\pm 25.94$ $\mathbf{4332.40\pm 64.58}$ $\mathbf{3870.66\pm 132.16}$

Appendix

Table 5: Learning-rate sweep on HalfCheetah-v5.

Figure 26: Performance of EISep FFN on the Lunar Lander task.

Figure 27: Performance for EISep LSTM with shared E-I identities across gates (left) or with independent identities across gates (right).

Figure 28: Performance for EISep GRU on HalfCheetah.

Figure 29: Sparsity averaged across all learning rates and seeds in the A. Lunar Lander task, and B. HalfCheetah task.

Figure 30: Extended visualizations of the HMM fitting results for both EISep(A+C) and Simple agents. A. A raw clip (length=100, beginning of the first episode) of hidden state activities for all 64 units in the EISep critic RNN. B. HMM transition matrix across all hidden labels. C. HMM labels (up) and manually annotated phases (bottom) for a clip (length=200, beginning of the first episode). The first 100 timesteps are aligned with the timesteps in A . D-F. Same as A-C for Simple critic RNN. G.-I. Same as A-C for EISep(A+C) actor RNN. J.-L. Same as A-C for Simple actor RNN.

Figure 31: Different metrics of the HMM results of EISep. A. The HMM log-likelihood on the train/test dataset for the critic/actor of the two agents(EISep/Simple). B . Stickiness of the transition matrix. C . Entropy of the transition matrix. D . Switch rate of HMM hidden states. E . Normalized mutual information between HMM hidden states and handmade labels.

Panel of Figure 32 — Figure 32: Schematic comparison of EISep and its ablation variants. Abs-EISep replaces the ReLU recurrent-weight filter with an absolute-value filter; Fixed-EISep removes learnable neuron signs; EntryEI relaxes neuron-level E–I separation into entry-wise sign masks; EISep(N1/N2) preserves the EISep parameterization but changes the initialization scheme; and Cstr-Simple tests whether EISep-style initialization alone explains the observed gains.

Figure 33: HMM metrics for EntryEI(A+C), including ( A ) log-likelihood, ( B ) stickiness of transition matrices, ( C ) entropy of transition matrices, ( D ) switch rate of HMM labels, and ( E ) normalized mutual information between HMM labels and handmade labels. Same as Fig. 31

Figure 34: Performance in the Lunar Lander task for different ablation models. The performance of Cstr-Simple has been shown in Sec. 6 .

Figure 35: Metrics on the cognitive task suite for different ablation models. We compare Abs-EISep, Fixed-EISep, and EntryEI against the EISep and Simple baselines. A . Task-wise performance of the 5 models across 20 cognitive tasks. B . Performance averaged across all tasks for each model. C . Recurrent weight sparsity averaged across tasks for different models. D . Training dynamics over the full training process, including decoding loss (left), gradient norm (middle), and spectral radius of the recurrent weight matrix (right).

Figure 36: Performance and stable zones for different ablation models vs. learning rates in the cognitive task suite.

Figure 37: The spectral radius of different models at initialization (

n=1000

, Error bar indicates standard error).

Figure 38: A–H. Training curves of different ablation models with their best learning rates in continual learning. I. Average performance across tasks. J. Final-round performance averaged over four trials. K. The sparsity of all models across trials.

Recurrent Neural Networks

Recurrent neural networks process sequences by updating a persistent hidden state, enabling temporal modeling, memory, and streaming computation.

See all 309 papers

Diffusing Blame: Task-Dependent Credit Assignment in Biologically Plausible Dual-Stream Networks
Jun 30, 2026Yutaro Yamada, Luca Grillotti, Rujikorn Charakorn +3Credit Assignment in RL Biologically Plausible Learning
Sakana AI
A theory of noisy linear recurrent neural network learning dynamics
Oct 26, 2026John J. Vastola, Kanaka RajanRecurrent Neural Networks Neural Network Training Dynamics
Department of Neurobiology, Harvard Medical School · Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University
A unified theory of feature learning in RNNs and DNNs
Feb 17, 2026Jan P. Bauer, Kirsten Fischer, Moritz Helias +1Recurrent Neural Networks Representation Learning
Gatsby Computational Neuroscience Unit, UCL, London, UK · IAS-6, Forschungszentrum Juelich, Juelich, Germany

Jun 30, 2026J/K move · Enter open · S save

Diffusing Blame: Task-Dependent Credit Assignment in Biologically Plausible Dual-Stream Networks

Yutaro Yamada, Luca Grillotti, Rujikorn Charakorn, Sebastian Risi+2

Sakana AI

Credit Assignment in RL Biologically Plausible Learning

Open paper AskPDF arXiv

Biological neural circuits obey Dale's principle: each neuron's synapses are uniformly excitatory or inhibitory. Artificial networks that respect this constraint must coordinate separate excitatory and inhibitory populations, fundamentally changing how credit is assigned during learning. Several biologically plausible learning rules avoid backpropagation's weight transport requirement, but it has been difficult to achieve strong performance under Dale's principle beyond MNIST. Error Diffusion (ED) was originally proposed in a dual-stream excitatory/inhibitory architecture, where learning is driven by routing global error signals to all layers without transporting transposed forward weights or relying on random feedback matrices. Whether such a rule can scale under Dale's principle across both supervised classification and reinforcement learning remains unknown. Here, we introduce modulo error routing to extend Error Diffusion beyond binary classification, and show that a dual-stream excitatory/inhibitory architecture trained with this method achieves 96.7% on MNIST and establishes a 61.7% baseline on CIFAR-10, demonstrating that representation learning is possible even when strictly enforcing Dale's principle. For the classification setting, we introduce three domain-specific innovations: layer-specific sigmoid widths, batch-centered class error signals, and asymmetric initialization, and ablation analysis reveals that their relative importance reverses between MNIST and CIFAR-10, exposing task-dependent credit-assignment bottlenecks invisible to single-benchmark evaluation. In reinforcement learning, we integrate ED with Proximal Policy Optimization (PPO) and evaluate it on continuous-control tasks in Google Brax and on Craftax, an open-ended exploration task. We show that ED-PPO achieves competitive performance relative to Direct Feedback Alignment, a backpropagation-free baseline.

Abstract

Recurrent Neural Networks

Explore similar work

Diffusing Blame: Task-Dependent Credit Assignment in Biologically Plausible Dual-Stream Networks

A theory of noisy linear recurrent neural network learning dynamics

A unified theory of feature learning in RNNs and DNNs