Inhibitory normalization of error signals improves learning in neural circuits
Authors: Roy Henha Eyono, Daniel Levenstein, Arna Ghosh, Jonathan Cornford, Blake Richards
Organizations: Mila-Quebec AI Institute · School of Computer Science, McGill University · Yale University · Google, Paradigms of Intelligence Team · Leeds University · Department of Neurology & Neurosurgery, McGill University · Montreal Neurological Institute, McGill University · CIFAR Learning in Machines & Brains Program
Normalization is a critical operation in neural circuits. In the brain, there is evidence that normalization is implemented via inhibitory interneurons and allows neural populations to adjust to changes in the distribution of their inputs. In artificial neural networks (ANNs), normalization is used to improve learning in tasks that involve complex input distributions. However, it is unclear whether inhibition-mediated normalization in biological neural circuits also improves learning. Here, we explore this possibility using ANNs with separate excitatory and inhibitory populations trained on an image recognition task with variable luminosity. We find that inhibition-mediated normalization does not improve learning if normalization is applied only during inference. However, when this normalization is extended to include back-propagated errors, performance improves significantly. These results suggest that if inhibition-mediated normalization improves learning in the brain, it additionally requires the normalization of learning signals.
Figures & tables
Figure 1: Schematic of the Excitatory-Inhibitory (EI) network with Layer Normalization and the perceptual invariance task. a: Feedforward EI network architecture. Gray units represents the Excitatory (E) population , and purple unit represents the Inhibitory (I) population . Outgoing synaptic weights share the same sign ( E,+ or I,− ). b: To test perceptual invariance, a shift is applied to each individual image in the FashionMNIST dataset during both training and testing. For every image, a unique constant Δ is sampled within the threshold ∣Δ∣<ϵ . The figure displays three example augmentations to illustrate how the shift varies across the allowable range.
Figure 2: Layer normalization (LN) improves perceptual invariance in Excitatory-Inhibitory (EI) networks. a: Test accuracy (Acc %) comparison of EI networks with LN (x-axis) to those without LN (y-axis). Data points represent performance across 30 hyperparameter combinations (layer widths and E/I learning rates) and four luminosity ranges ( ϵ=0,0.25,0.5,0.75 ). Points below the dashed diagonal line indicate cases where networks with LN performed better. b: Top-10 test accuracy comparison of an EI network with LN against an Excitatory-only (E-only) network, also with LN. The box plots summarize the distribution across the same 30 hyperparameter combinations reported in panel a .
Figure 3: Inhibitory populations learn to implement layer normalization of excitatory activity. a: Schematic showing how the inhibitory circuit (purple) is trained locally via the LI-Norm loss (purple lines) to normalize excitatory activity. Excitatory to excitatory weights are updated only by the task loss LTask (dotted left arrow). Forward Pass: Inhibition performs subtractive ( − ) and divisive ( ÷ ) modulation. Backward Pass: Inhibitory gradients enforce layer-normalized excitatory statistics. b: Box-and-whisker plots of the first and second moments of excitatory activations. Each plot compares three conditions: No-Norm, Subtractive-only I-Norm (sub), and I-Norm (as depicted in a). Each box plot summarizes model results aggregated across the sampled range of ϵ luminosity augmentations.
Figure 4: I-Norm networks struggle to recapitulate the performance of LN in EI networks. a: Test accuracy comparison between LN (x-axis) and I-Norm (y-axis) across hyperparameter and luminosity ranges ( ϵ=0,0.25,0.5,0.75 ). Each point represents a single network training run. Points falling below the dashed diagonal line indicate cases where LN achieved higher test accuracy than I-Norm. b: Layer-wise alignment between I-Norm and LN. We quantify alignment as the cosine similarity between I-Norm and LayerNorm (LN) across all network layers for outputs (top) and gradients (bottom). All results correspond to the highest luminosity range ( ϵ=0.75 ).
Figure 5: Hard-coded LN gradients in I-Norm networks restore LN performance in EI networks. a: Schematic illustrating the Backward Pass of the I-Norm network incorporating, GradNorm , from equation 2 . The GradNorm operation is applied to the backward signal ( δ ) to enforce the LN gradient. b: Average cosine similarity across all network layers. The top and bottom panels show the similarity between LN and I-Norm ( with GradNorm ) for outputs and gradients, respectively. c: Test accuracy (Acc %) comparison. LN network performance (x-axis) versus I-Norm network with GradNorm (y-axis). Data is shown across 30 hyperparameter initializations and four luminosity ranges ( ϵ=0,0.25,0.5,0.75 ). Points clustered along the dashed diagonal line indicate a strong match in performance between the two models.
Figure 6: Gradient centering, not scaling or decorrelating, drives LN performance recovery in I-Norm networks. a: Box plots of test accuracy comparing Scale , Decorrelate , Center , and Full LN gradient components applied to the I-Norm network. The red dashed line ( LN+ ) shows the baseline performance of the hard-coded LN network. Results are separated by luminosity range ( ϵ=0,0.75 ). Significance is indicated comparing components to the LN+ baseline. Red tildes ( ∼ ) denote outliers performing near random chance, omitted to preserve plot scaling. b: The panels contrast the LN gradient with specific gradient components trained on an I-Norm network: Center (top, orange), Decorrelate (middle, teal), and Scale (bottom, purple).
Figure 7: Lateral inhibition with fixed synaptic weights implements gradient centering. a: Schematic of the I-Norm network with a lateral inhibition pathway. The inhibitory unit (red) pools and transforms excitatory activity (gray) using fixed, random connections to compute the centering term (mean). b: Test accuracies (Acc %) of the lateral inhibition solution, explicit gradient centering, and the full layer normalization gradient applied to the I-Norm network. The red dashed line ( LN+ ) shows the baseline performance of the explicit layer normalization network. Results are averaged across hyperparameter and luminosity ranges ( ϵ=0,0.75 ). Significance is shown relative to the LN+ baseline.
Parameter
Value/Range
Description
Training Parameters
Epochs
50
Number of training epochs
Batch size
32
Mini-batch size
Dataset
FashionMNIST
Source dataset
Learning Rates
Excitatory LR ( ηexc )
10−3 to 10−1
Log-uniform sampling
Table 1: Hyperparameters used in the experimental evaluation
Biologically plausible learning models aim to explain how neural circuits can implement effective learning under the constraints of real neurons. Although significant progress has been made, a major remaining challenge is that existing models often allow neurons or synapses to represent mixed-sign values, both positive and negative, in violation of a basic aspect of cortical circuitry -- Dale's constraint: biological neurons are either excitatory or inhibitory, but not both, and synapses cannot change sign. In this work, we address this discrepancy by introducing a biologically motivated neural architecture in which both neural activations and learning signals are represented by non-negative activity, and synapses have fixed sign, while still supporting backpropagation-like learning. Our approach uses two complementary interacting non-negative channels to represent positive and negative contributions, inspired by evidence of on-off representations in the brain. These channels are implemented through a simple neural circuit motif, which is repeated throughout the network in both bottom-up and top-down pathways. Combined with a local Hebbian learning rule, the resulting model propagates learning signals and updates weights using only local interactions between neurons. We show theoretically that our learning scheme can exactly recover the backpropagation update despite relying solely on non-negative error signals. Empirically, beyond satisfying stronger biological constraints, the on-off architecture learns efficient representations, yielding substantial gains over comparable vanilla networks on the Tiny ImageNet benchmark. These results demonstrate that effective learning can emerge from biologically plausible mechanisms without requiring mixed-sign signals, providing a step toward more realistic models of neural computation.
Biological neural circuits obey Dale's principle: each neuron's synapses are uniformly excitatory or inhibitory. Artificial networks that respect this constraint must coordinate separate excitatory and inhibitory populations, fundamentally changing how credit is assigned during learning. Several biologically plausible learning rules avoid backpropagation's weight transport requirement, but it has been difficult to achieve strong performance under Dale's principle beyond MNIST. Error Diffusion (ED) was originally proposed in a dual-stream excitatory/inhibitory architecture, where learning is driven by routing global error signals to all layers without transporting transposed forward weights or relying on random feedback matrices. Whether such a rule can scale under Dale's principle across both supervised classification and reinforcement learning remains unknown. Here, we introduce modulo error routing to extend Error Diffusion beyond binary classification, and show that a dual-stream excitatory/inhibitory architecture trained with this method achieves 96.7% on MNIST and establishes a 61.7% baseline on CIFAR-10, demonstrating that representation learning is possible even when strictly enforcing Dale's principle. For the classification setting, we introduce three domain-specific innovations: layer-specific sigmoid widths, batch-centered class error signals, and asymmetric initialization, and ablation analysis reveals that their relative importance reverses between MNIST and CIFAR-10, exposing task-dependent credit-assignment bottlenecks invisible to single-benchmark evaluation. In reinforcement learning, we integrate ED with Proximal Policy Optimization (PPO) and evaluate it on continuous-control tasks in Google Brax and on Craftax, an open-ended exploration task. We show that ED-PPO achieves competitive performance relative to Direct Feedback Alignment, a backpropagation-free baseline.
Delayed loss spikes have been reported in neural-network training, but existing theory mainly explains earlier non-monotone behavior caused by overly large fixed learning rates. We study one stylized hypothesis: normalization can postpone instability by gradually increasing the effective learning rate during otherwise stable descent. To test this hypothesis at theorem level, we analyze batch-normalized linear models. Our flagship result concerns whitened square-loss linear regression, where we derive explicit no-rising-edge and delayed-onset conditions, bound the waiting time to directional onset, and show that the rising edge self-stabilizes within finitely many iterations. Combined with a square-loss decomposition, this yields a concrete delayed-spike mechanism in the whitened regime. For logistic regression, under highly restrictive active-margin assumptions, we prove only a supporting finite-horizon directional precursor in a knife-edge regime, with an optional appendix-only loss lower bound under an extra non-degeneracy condition. The paper should therefore be read as a stylized mechanism study rather than a general explanation of neural-network loss spikes. Within that scope, the results isolate one concrete delayed-instability pathway induced by batch normalization.
Peifeng Gao, Wenyi Fang, Yang Zheng +1
School of Computing and Data Science, The University of Hong Kong . · RAMS Technologies Lab, Huawei Technologies Ltd.