cs.LGDec 6, 2024
SaveRobust and efficient enriched representation with intrinsic neuronal heterogeneity
Abstract
Heterogeneity is a ubiquitous property of many biological systems with profound computational implications. Integrating this heterogeneity into artificial computing systems has recently gained attention. While optimizing neuronal and synaptic heterogeneity for a specific task can achieve state-of-the-art performance, such top-down optimization is biologically implausible, prone to catastrophic forgetting, and both data- and energy-intensive. In contrast, biological organisms possess inherent, task-unspecific, stable heterogeneity and can perform many tasks with minimal metabolic cost. Inspired by this, we study task-agnostic heterogeneity in recurrent neural networks solving hundreds of diverse temporal tasks. Heterogeneity is introduced as intrinsic disorder in neuronal membrane time constants. Our results show that intrinsic heterogeneity markedly improves overall performance and robustness in both rate- and spike-coded networks. Small heterogeneous networks, even without recurrence, match the performance of large, highly recurrent, homogeneous networks at a fraction of the cost and complexity. We suggest that, despite being task-agnostic, heterogeneity enhances representational capacity by diversifying neuronal responses, demonstrated on two machine-learning and one neuroscience benchmark. Our work highlights the utility of intrinsic task-agnostic heterogeneity for versatile, energy-efficient computing--a highly relevant property for the neuromorphic community, where device-to-device variability could be exploited to offset limited training budgets and architectural constraints.
Figures & tables
Figure 1: Method summary . (A) The reservoir computing paradigm. The multi - dimensional input drives a recurrent network, giving rise to the neural state . This state is then read out linearly to approximate arbitrary target functions or tasks . indexes each task in a collection of tasks. Note that all target functions use the same state but different readouts . (B) A schematic representation of (top) rate - based (leaky integrator, LI) and (bottom) spiking (leaky integrate - and - fire, LIF) neuronal codes. The state of neurons in the LI implementation is a continuous time series, whereas neurons in the LIF implementation spike discretely. (C) Neuronal heterogeneity is introduced by assigning different membrane time constants to the neurons, drawn from a distribution with a given mean and variance. The variance determines the heterogeneity level. (D) The task family targeted by our approach includes tasks such as memory recall, input prediction, input processing, and their combinations. (E) (Left) The output of networks with various heterogeneity levels for an exemplary task ( ). (Right) Performance of each network measured using the coefficient of determination, a scalar score that aggregates the normalized mismatch between the ground truth and the output over an unobserved test interval. A value of 1 indicates a perfect match, whereas 0 corresponds to a chance - level prediction.
Figure 2: Intrinsically heterogeneous networks process time series more accurately. (A1) The chaotic trajectory of the Lorenz system is used as a three-dimensional input to create a collection of 882 tasks. (A2) (top) Mutual similarity and (bottom) complexity distribution of the tasks, showing they are mostly dissimilar and span a broad spectrum of complexities. (A3) Cross - sections of the performance profile for , and various values are shown for the (top) spiking and (bottom) rate - based neurons. (A4) Direct score comparison between heterogeneous (y - axis) and corresponding homogeneous (x - axis) networks for each task (each dot). Colors indicate the degree of heterogeneity. The diagonal dotted line marks equal performance, while the vertical and horizontal dashed lines denote the chance - level score. The right - hand panel corresponds to the spiking implementation and the left - hand panel to the rate - based implementation. Panels (A3) and (A4) share the same color code. The other rows show similar experiments with different inputs: (B) Mackey - Glass input (three - dimensional; two components chaotic, one periodic), (C) Order - 30 NARMA input (one - dimensional, chaotic and non - smooth), and (D) Absolute value of a sinusoid as input (one - dimensional, smooth and predictable). For 1D inputs, the first column shows the 3D Poincare sections, and 294 tasks are generated.
Figure 3: Neuronal heterogeneity renders networks robust. (A) (Top) Schematic of the network and its inputs. Purple and green traces represent the deterministic input and stochastic noise that are projected onto the network at its center. (Bottom) Tasks are divided into three complexity tiers (indicated by the gradient), and performance is averaged within each tier for clarity. (B–E) (Top) Schematic of the varied hyperparameters; see panel (A) for reference. (Bottom) Impact of varying network size (B), noise intensity (C), feedforward gain (D), and recurrent gain (E) on the performance of rate - based RNNs. All panels use the same color code. The plateaus in panel (D) arise from saturation of neuronal activity. The uniform decline of the score at large recurrent - gain values in panel (E) reflects a transition to chaos. Analogous results for spiking networks are shown in Fig. S3. In all panels dots and shaded area represent the mean and 95% confidence interval, calculated over tasks (of a fixed complexity tier) and three non-overlapping training sets (see Sec. 4.6 for details).
Figure 4: Heterogeneity lowers the cost at each performance level. At every performance bin, the network with minimal energy consumption ( ) is the most efficient. In all systems, heterogeneity substantially reduces the total minimal cost (note the logarithmic scale of the y - axis). This cost saving primarily results from using small but performant networks. The definition of cost for each hardware system reflects its specific characteristics. We used (A) the total number of floating - point operations (FLOPs) and the memory footprint for rate - based networks, and (B) the total expenditure of adenosine triphosphate (ATP) molecules in biological spiking networks. (C) For the neuromorphic hardware system SpiNNaker 2, the total energy consumption (in joules) was estimated based on direct measurements of their power usage. A network with an equivalent firing rate was simulated for 100 steps, and the energy expenditure was linearly extrapolated to match the number of time steps used in the CPU simulations (cf. Section A). Note that the traces associated with lower heterogeneity are partially occluded by the one of high heterogeneity level.
Figure 5: The heterogeneity profile substantially affects task versatility in networks. (A) At any given value of , heterogeneity improves performance. By spreading neuronal time constants over a broader interval, heterogeneity increases the likelihood of covering the regions required by the tasks. (B) The performance depends on the heterogeneity profile. Skewed distributions (log - normal and gamma) result in higher performance levels. The empirical mean and variance were identical for all profiles. The shaded area in (A) and (B) correspond to indicate 95% confidence interval across all tasks.
Figure 6: Heterogeneity increases temporal alignment between tasks and the network’s state (A) Any temporal task can be regarded as a high - dimensional vector in an appropriate vector space. The network states form a basis for that space. The extent to which the basis can span determines the ultimate performance of a network for that task. Performance is perfect only if the task lies in the subspace spanned by the state (full state - task alignment). A partial overlap results in suboptimal performance (partial state - task alignment). (top) A low - dimensional (2D) span; (bottom) a high - dimensional (3D) span. Only the latter can fully represent the task . (C) Distribution of task - state alignment for networks with different degrees of heterogeneity for (left) LIF and (right) LI models. (Top) Total overlap histogram. (Bottom) Cumulative overlap density. Higher scores in heterogeneous networks are corroborated by a systematic shift toward higher task - state overlap across tasks. However, note that higher alignment does not necessarily mean a higher - dimensional space spanned by the network. Direct assessment of the participation ratio (PR) indicates that while the state’s dimensionality increases with heterogeneity in the LI model, it decreases in LIF networks. Neural states are decorrelated before computing alignment. Colors indicate different heterogeneity levels, as in Fig. 2 .
Figure 7: Heterogeneity improves versatility in real-world problems. Each system is composed only of fully decoupled rate neurons with non-linearity to isolate manifestation of heterogeneous time constants. Heterogeneous time constants are distributed uniformly from 1 timestep to the maximal length of all trials . The last 10 timesteps of neuronal trajectory are used a feature vectors of a multinomial logistic regression, with minimal regularization. (A) Mean scores and their standard deviations (error bars) across realizations. Bars heights and the error bars indicate respectively the mean and standard deviation of 10 independent realizations of feed-forward weights and time constants. (B) Score differences between the heterogeneous and homogeneous networks across 10 trials. Each dot represents on trial. The gray line, shows average and standard deviations across trials. Columns : from left to right: Speech commands (limited to samples with top 10 most frequent labels); Permuted sequential MNIST; Contextual decision-making task (with 6 choice) with stochastic delay and stochastic stimulation period
Appendix figures & tables30 assets
Supplementary material from the paper’s appendix.
Appendix
Figure S.1: Matching a spiking neuron to its rate - based counterpart . Normalized firing rates of the spiking neuron (blue) and the rate neuron (black) are shown together with their difference, , as a function of the input current . The LIF neuron parameters are set to , , , , and . is multiplied by two in the plot to highlight the slow convergence at large values.
Figure S.2: High performance requires better state–task alignment . (A) Temporal (right) and frequency (left) representations of a simple target function composed of two sinusoids. This target is to be reconstructed linearly by two candidate networks. (B) The first network consists of four independent neurons, two of which oscillate at frequencies that match those of the target function (bold lines). Consequently, a linear combination of the network’s state can perfectly reconstruct the target. (C) The second network contains many more neurons but lacks units with the frequencies required by the target function (marked with red circles). Despite its higher dimensionality than the first network, it fails to reconstruct the simple task.
Figure S.3: Neuronal heterogeneity renders networks robust. As in Fig. 3 , but for networks of LIF neurons. (A) Schematic of the network and the deterministic and noise (top) and the task complexity (bottom) . Panels depict the influence of (B) network size, (C) noise intensity, (D) feedforward gain, and (E) recurrent gain. The shaded area indicates the 95% confidence interval across tasks within the corresponding complexity tier. Unlike rate - based networks, the scores do not drop to the chance - level value in panel (E) because LIF networks remain in a non - chaotic regime, operating at firing rates far below their maximum. See Fig. S.20 G.
Figure S.4: Effect of population size and synaptic weights. (A) (Top) Schematic of the recurrent connections in the networks. (Bottom) Task - complexity tiers. (B–D) (Top) Schematic of the varied hyperparameter: connection probability (B), fraction of excitatory neurons (C), and weight dispersion (D), which controls the degree to which Dale’s law is respected. (Bottom) When excitatory–inhibitory balance is preserved, none of these changes affect the performance of rate - based networks at any level of heterogeneity. The shaded areas in all panels indicate the 95% confidence interval across tasks within the corresponding complexity tier.
Figure S.5: Effect of population and weights . Same as Fig. S.4 , but for spiking networks. (A) Schematic overview of the network and the task complexity. The impact on performance due to variation of (B) recurrent sparsity, (C) E-I ratio, and (D) recurrent weight dispersion.
Figure S.6: Variation of score distribution with network size. Performance distributions for (A) rate - based and (B) spiking networks across (rows) tasks of different complexities. The mean of each distribution corresponds to the levels shown in Figures 3 B and S.3 B, respectively. The color code is the same as in the main text.
Figure S.7: Variation of score distribution versus noise intensity. Performance distributions for (A) rate - based networks and (B) spiking networks across (rows) tasks of different complexities. The mean of each distribution corresponds to the levels shown in Figures 3 C and S.3 C, respectively. The color code is the same as in the main text.
Figure S.8: Variation of score distribution versus input gain. Performance distributions for (A) rate - based networks and (B) spiking networks across (rows) tasks of different complexities. The average of each distribution corresponds to the levels shown in Figures 3 D and S.3 D, respectively. The color code is the same as in the main text.
Figure S.9: Variation of score distribution versus recurrent gain. Performance distributions for (A) rate - based networks and (B) spiking networks across (rows) tasks of different complexities. The mean of each distribution corresponds to the levels shown in Figures 3 E and S.3 E, respectively. The color code is the same as in the main text.
Figure S.10: Variation of score distribution versus connection probability. Performance distributions for (A) rate - based networks and (B) spiking networks across (rows) tasks of different complexities. The mean of each distribution corresponds to the levels shown in Figures S.4 B and S.5 B, respectively. The color code is the same as in the main text.
Figure S.11: Variation of score distribution versus excitation-inhibition ratio. Performance distributions for (A) rate - based networks and (B) spiking networks across (rows) tasks of different complexities. The mean of each distribution corresponds to the levels shown in Figures S.4 C and S.5 C, respectively. The color code is the same as in the main text.
Figure S.12: Variation of score distribution versus recurrent weight dispersion. Performance distributions for (A) rate - based networks and (B) spiking networks across (rows) tasks of different complexities. The mean of each distribution corresponds to the levels shown in Figures S.4 D and S.5 D, respectively. The color code is the same as in the main text.
Figure S.13: Performance uncertainty in rate networks. Performance standard deviation of rate networks, computed from three readouts obtained with fully independent samples. The data are broken down by task - complexity tier ( columns ) and by the hyperparameter values shown: (A) network size, (B) noise intensity, (C) feedforward gain, (D) recurrent gain, (E) connection probability, (F) fraction of excitatory neurons, and (G) weight dispersion. The narrow range of standard deviations across all panels indicates that the error in the average performance is small, and three independent trials are sufficient. Note also the lower error observed in heterogeneous networks. The shaded areas indicates the 95% confidence interval across tasks within the corresponding complexity tier.
Figure S.14: Performance error in spiking networks. As in Fig. S.13 , but for spiking networks. The error bars are computed from three independent readouts. As before, heterogeneous networks exhibit lower variability. Panels indicate the influence on standard deviation as a function of (A) network size, (B) noise intensity, (C) feedforward gain, (D) recurrent gain, (E) recurrent sparsity, (F) E-I ratio, and (G) weight dispersion.
Figure S.15: Performance variation for all tasks in large LI network (A) Tasks (x-axis) are sorted based on their mean score on the homogeneous network. The darker lines and the shaded area respectively indicate the average and standard deviation obtained over 10 different network realizations. (Top to bottom) Different average time constants. (B) Same as (A) but only for the homogeneous and highly heterogeneous network, shown for better visibility. Networks size is 250.
Figure S.16: Performance variation for all tasks in large LIF network As Fig. S.15 but for spiking neurons. (A) All heterogeneity levels; (B) Only minimal and maximal heterogeneity levels for better visibility.
Figure S.17: Performance variation for all tasks in small LI network As Fig. S.15 but for network size of 50. (A) All heterogeneity levels; (B) Only minimal and maximal heterogeneity levels for better visibility.
Figure S.18: Performance variation for all tasks in small LI network As Fig. S.16 but for network size of 50. (A) All heterogeneity levels; (B) Only minimal and maximal heterogeneity levels for better visibility.
Figure S.19: Energy and performance variation of rate networks with heterogeneity. Task - averaged (A) performance , (B) energy , and (C) efficiency across different heterogeneity levels, together with (D–F) their ratios relative to the corresponding values in a homogeneous network. (Rows) each varied hyperparameter. Note that, because the score , we first map it monotonically to the interval via before computing efficiency and the ratio metrics, thereby avoiding negative values or division by zero.
Figure S.20: Energy and performance variation of spiking networks with heterogeneity . As in Fig. S.19 , but for spiking networks. (G) The additional column shows the average firing rate . Its correlation with indicates that the energy increase in heterogeneous networks is primarily due to higher neuronal activity. Note that in all scenarios, the networks fire with a rate far below the maximum possible firing rate of Hz. Consequently, even at the maximal recurrence level simulated ( ), spiking networks stay effectively input - driven, which also explains the lack of performance variation with in Fig. S.3 E.
Figure S.21: The speech command setup . (Top) Input stream to the agent. Except for the first channel (starting from the bottom), which encodes the “fixation” cue, the other 80 channels contain the log - Mel content of the audio files and can be regarded as frequency bands. The triangles at the top indicate the start of a trial. (Middle) Time - aligned neuronal voltages; note the resetting effect at the start of each trial. (Bottom) Activation map of neurons over time.
Figure S.22: The permuted sequential MNIST setup . Same as Fig. S.21 but for the psMNIST task with only one input channel. Note that here, as opposed to the other tasks, the fixation signal is provided as within the sole signal channel (as a negative value). Panels indicate the temporal variation of (top) the stimulus, (middle) neurons’ voltage, and (bottom) their activation.
Figure S.23: The CDM setup with stochastic delay period . Same as Fig. S.21 but for CDM task. The first channel of the input stream (starting from the bottom) encodes the “fixation” cue as in SC task. The next 6 channels indicate the observed stimulus of the first modality (e.g., color) at distinct angular location (60 degrees apart), followed by analogous signal for modality 2 (motion) in the subsequent 6 channels. The last channel indicate the context (motion or color). Panels indicate the temporal variation of (top) the stimulus, (middle) neurons’ voltage, and (bottom) their activation.
Figure S.24: The CDM setup with stochastic stimulus period . Same as Fig. S.21 but for fixed delay period and stochastic stimulus period. Panels indicate the temporal variation of (top) the stimulus, (middle) neurons’ voltage, and (bottom) their activation.
Figure S.25: (A) Speech command task; (B) Permuted sequential MNIST task. A higher value leads to weaker regularization. For the SC and psMNIST tasks, the top row ( “Go” pooling) with regularization values of, respectively, and , are shown in the main text. Network setup and the number of realizations are identical to the ones described in the main text.
Figure S.26: (A) CDM task with stochastic stimulation period and fixed delay; (B) CDM task with stochastic delay period and fixed stimulation time.
Figure S.27: CDM task with two options. The performance of both networks is almost perfect for all pooling-regularization combinations. This is due to the simplicity of the task. Note the 50% chance level as opposed to in Fig. S.26 for a setup with 6 choices.
Figure S.28: Energy-score without Dale’s law . Similar to Figure 4 , but size-dependent simulation horizon and weights randomly ascribed to different synapses, thereby violating Dale’s law. The traces associated with higher heterogeneity levels are partially occluded by those of lower heterogeneity levels.
Figure S.29: Score-energy signatures in rate networks . Seventy-two different state conditions (3 levels of , 4 heterogeneity levels, and 6 network sizes) yield a distinct score for each task. Each subpanel shows the (score-energy) pair for either a single task (top row) across different complexity tiers (columns) or the average (score-energy) pair across all tasks within each tier (bottom row) . Different panels display results for different values: (A) , (B) , and (C) .
Figure S.30: Score-energy signatures in spiking networks . As in Fig. S.29 , but for spiking networks. Note the different slopes for different heterogeneity levels.
More in this topic
Recurrent Neural Networks
Recurrent neural networks process sequences by updating a persistent hidden state, enabling temporal modeling, memory, and streaming computation.
See all 309 papers