Inhibitory normalization of error signals improves learning in neural circuits

Roy Henha Eyono; Daniel Levenstein; Arna Ghosh; Jonathan Cornford; Blake Richards

q-bio.NCMar 18, 2026

Inhibitory normalization of error signals improves learning in neural circuits

Authors: Roy Henha Eyono, Daniel Levenstein, Arna Ghosh, Jonathan Cornford, Blake Richards

Organizations: Mila-Quebec AI Institute · School of Computer Science, McGill University · Yale University · Google, Paradigms of Intelligence Team ·

Abstract

Normalization is a critical operation in neural circuits. In the brain, there is evidence that normalization is implemented via inhibitory interneurons and allows neural populations to adjust to changes in the distribution of their inputs. In artificial neural networks (ANNs), normalization is used to improve learning in tasks that involve complex input distributions. However, it is unclear whether inhibition-mediated normalization in biological neural circuits also improves learning. Here, we explore this possibility using ANNs with separate excitatory and inhibitory populations trained on an image recognition task with variable luminosity. We find that inhibition-mediated normalization does not improve learning if normalization is applied only during inference. However, when this normalization is extended to include back-propagated errors, performance improves significantly. These results suggest that if inhibition-mediated normalization improves learning in the brain, it additionally requires the normalization of learning signals.

Figures & tables

Figure 1: Schematic of the Excitatory-Inhibitory (EI) network with Layer Normalization and the perceptual invariance task. a: Feedforward EI network architecture. Gray units represents the Excitatory (E) population , and purple unit represents the Inhibitory (I) population . Outgoing synaptic weights share the same sign ( $\text{E},+$ or $\text{I},-$ ). b: To test perceptual invariance, a shift is applied to each individual image in the FashionMNIST dataset during both training and testing. For every image, a unique constant $\Delta$ is sampled within the threshold $|\Delta|<\epsilon$ . The figure displays three example augmentations to illustrate how the shift varies across the allowable range.

Figure 2: Layer normalization (LN) improves perceptual invariance in Excitatory-Inhibitory (EI) networks. a: Test accuracy (Acc %) comparison of EI networks with LN (x-axis) to those without LN (y-axis). Data points represent performance across 30 hyperparameter combinations (layer widths and E/I learning rates) and four luminosity ranges (

\epsilon=0,0.25,0.5,0.75

). Points below the dashed diagonal line indicate cases where networks with LN performed better. b: Top-10 test accuracy comparison of an EI network with LN against an Excitatory-only (E-only) network, also with LN. The box plots summarize the distribution across the same 30 hyperparameter combinations reported in panel

\mathbf{a}

Figure 3: Inhibitory populations learn to implement layer normalization of excitatory activity. a: Schematic showing how the inhibitory circuit (purple) is trained locally via the

\mathcal{L}_{\text{I-Norm}}

loss (purple lines) to normalize excitatory activity. Excitatory to excitatory weights are updated only by the task loss

\mathcal{L}_{\text{Task}}

(dotted left arrow). Forward Pass: Inhibition performs subtractive (

-

) and divisive (

\div

) modulation. Backward Pass: Inhibitory gradients enforce layer-normalized excitatory statistics. b: Box-and-whisker plots of the first and second moments of excitatory activations. Each plot compares three conditions: No-Norm, Subtractive-only I-Norm (sub), and I-Norm (as depicted in a). Each box plot summarizes model results aggregated across the sampled range of

\epsilon

luminosity augmentations.

Figure 4: I-Norm networks struggle to recapitulate the performance of LN in EI networks. a: Test accuracy comparison between LN (x-axis) and I-Norm (y-axis) across hyperparameter and luminosity ranges (

\epsilon=0,0.25,0.5,0.75

). Each point represents a single network training run. Points falling below the dashed diagonal line indicate cases where LN achieved higher test accuracy than I-Norm. b: Layer-wise alignment between I-Norm and LN. We quantify alignment as the cosine similarity between I-Norm and LayerNorm (LN) across all network layers for outputs (top) and gradients (bottom). All results correspond to the highest luminosity range (

\epsilon=0.75

Figure 5: Hard-coded LN gradients in I-Norm networks restore LN performance in EI networks. a: Schematic illustrating the Backward Pass of the I-Norm network incorporating, GradNorm , from equation 2 . The GradNorm operation is applied to the backward signal (

\delta

) to enforce the LN gradient. b: Average cosine similarity across all network layers. The top and bottom panels show the similarity between LN and I-Norm ( with GradNorm ) for outputs and gradients, respectively. c: Test accuracy (Acc %) comparison. LN network performance (x-axis) versus I-Norm network with GradNorm (y-axis). Data is shown across 30 hyperparameter initializations and four luminosity ranges (

\epsilon=0,0.25,0.5,0.75

). Points clustered along the dashed diagonal line indicate a strong match in performance between the two models.

Figure 6: Gradient centering, not scaling or decorrelating, drives LN performance recovery in I-Norm networks. a: Box plots of test accuracy comparing Scale , Decorrelate , Center , and Full LN gradient components applied to the I-Norm network. The red dashed line (

\mathbf{LN^{+}}

) shows the baseline performance of the hard-coded LN network. Results are separated by luminosity range (

\epsilon=0,0.75

). Significance is indicated comparing components to the

\text{LN}^{+}

baseline. Red tildes (

\sim

) denote outliers performing near random chance, omitted to preserve plot scaling. b: The panels contrast the LN gradient with specific gradient components trained on an I-Norm network: Center (top, orange), Decorrelate (middle, teal), and Scale (bottom, purple).

Figure 7: Lateral inhibition with fixed synaptic weights implements gradient centering. a: Schematic of the I-Norm network with a lateral inhibition pathway. The inhibitory unit (red) pools and transforms excitatory activity (gray) using fixed, random connections to compute the centering term (mean). b: Test accuracies (Acc %) of the lateral inhibition solution, explicit gradient centering, and the full layer normalization gradient applied to the I-Norm network. The red dashed line (

\mathbf{LN^{+}}

) shows the baseline performance of the explicit layer normalization network. Results are averaged across hyperparameter and luminosity ranges (

\epsilon=0,0.75

). Significance is shown relative to the LN+ baseline.

Parameter	Value/Range	Description
Training Parameters
Epochs	50	Number of training epochs
Batch size	32	Mini-batch size
Dataset	FashionMNIST	Source dataset
Learning Rates
Excitatory LR ( $\eta_{exc}$ )	$10^{-3}$ to $10^{-1}$	Log-uniform sampling

Table 1: Hyperparameters used in the experimental evaluation

Biologically Plausible Learning

Biologically plausible learning studies how neural networks can learn using mechanisms and signals consistent with biological brains.

See all 26 papers

Learning in Deep Networks under Dale's Constraint
Aug 7, 2026Roy Abel, Shimon UllmanBackpropagation Biologically Plausible Learning
Weizmann Institute of Science
Diffusing Blame: Task-Dependent Credit Assignment in Biologically Plausible Dual-Stream Networks
Jun 30, 2026Yutaro Yamada, Luca Grillotti, Rujikorn Charakorn +3Credit Assignment in RL Biologically Plausible Learning
Sakana AI
A Mechanism Study of Delayed Loss Spikes in Batch-Normalized Linear Models
Apr 18, 2026Peifeng Gao, Wenyi Fang, Yang Zheng +1Neural Network Optimization Neural Network Training Dynamics
School of Computing and Data Science, The University of Hong Kong . · RAMS Technologies Lab, Huawei Technologies Ltd.

Aug 7, 2026J/K move · Enter open · S save

Learning in Deep Networks under Dale's Constraint

Roy Abel, Shimon Ullman

Weizmann Institute of Science

Backpropagation Biologically Plausible Learning Neural Network Training Dynamics

Open paper AskPDF arXiv

Biologically plausible learning models aim to explain how neural circuits can implement effective learning under the constraints of real neurons. Although significant progress has been made, a major remaining challenge is that existing models often allow neurons or synapses to represent mixed-sign values, both positive and negative, in violation of a basic aspect of cortical circuitry -- Dale's constraint: biological neurons are either excitatory or inhibitory, but not both, and synapses cannot change sign. In this work, we address this discrepancy by introducing a biologically motivated neural architecture in which both neural activations and learning signals are represented by non-negative activity, and synapses have fixed sign, while still supporting backpropagation-like learning. Our approach uses two complementary interacting non-negative channels to represent positive and negative contributions, inspired by evidence of on-off representations in the brain. These channels are implemented through a simple neural circuit motif, which is repeated throughout the network in both bottom-up and top-down pathways. Combined with a local Hebbian learning rule, the resulting model propagates learning signals and updates weights using only local interactions between neurons. We show theoretically that our learning scheme can exactly recover the backpropagation update despite relying solely on non-negative error signals. Empirically, beyond satisfying stronger biological constraints, the on-off architecture learns efficient representations, yielding substantial gains over comparable vanilla networks on the Tiny ImageNet benchmark. These results demonstrate that effective learning can emerge from biologically plausible mechanisms without requiring mixed-sign signals, providing a step toward more realistic models of neural computation.

Abstract

Biologically Plausible Learning

Explore similar work

Learning in Deep Networks under Dale's Constraint

Diffusing Blame: Task-Dependent Credit Assignment in Biologically Plausible Dual-Stream Networks

A Mechanism Study of Delayed Loss Spikes in Batch-Normalized Linear Models