Organizations: Siebel School of Computing and Data Science University of Illinois Urbana-Champaign Champaign, IL, USA · Computer Science and Mathematics Division Oak Ridge National Laboratory Oak Ridge, TN, USA
Biological neural circuits are both modular and recurrent, with dense communities sparsely linked to one another. Spiking neural networks (SNNs) for reinforcement learning are usually feed-forward, and it is unclear how to obtain functional modules in a recurrent SNN or what they buy. In this work, we introduce a class of spiking networks, called Community Recurrent SNNs (CoRe SNNs): recurrent networks partitioned into communities with fixed roles. A controller community receives the stimulus, each action has its own community, and latent communities carry internal state. Spiking activity is then a walk between communities, and the action is the community the walk reaches. All learning is local and three-factor, and the recurrent communities are closer to cortical microcircuits than a feed-forward actor. We compare three regimes for obtaining the communities: dynamic modularity, in which a new learning rule we introduce, Map-STDP, encourages modular structure during learning; partial modularity, in which a stochastic block model (SBM) initialization enforces it partly; and full modularity, in which sealing enforces it fully. On CartPole, across three spiking neuron models, Map-STDP improves learning over the same network without it matches full modularity (sealed communities) on two of the three models, and makes the observed spike walk more compressible. Replacing the feed-forward actor of a TD-STDP actor-critic with recurrent action communities improves on the feed-forward network on CartPole and LunarLander, and matches it on Acrobot. How strongly to enforce modularity depends on the task: full modularity helps most on Acrobot, while on LunarLander the dynamic, partial and full regimes perform alike. Overall, these recurrent modular networks can match or even improve upon feed-forward approaches, with improvements in bio-plausibility and energy efficiency.
Figures & tables
Fig. 2: (a) Trained CartPole networks on the flow engine, one seed (2000), from the same initial network. C is the controller, L1–L2 the latent modules and A1–A2 the action modules. Edges of at least 2% of the maximum weight are drawn, with width and opacity proportional to weight. Below each network: the share of weight within modules and across modules (excluding controller routing), and LW , the map equation (bits) of W’s own walk. (b) AUC, the mean return over 1500 training episodes, for each engine and regime; 20 seeds per condition. (c) Effective synaptic operations per decision (spikes × outgoing synapses above the weight floor, averaged over 50 held-out stimuli with W frozen) against Lobs , for the final networks of the three spiking engines. Color: engine; shape: regime. Dashed lines are least-squares fits of log10 effective SOPs on Lobs ; ρ is Spearman’s. Post-hoc, exploratory analysis. (d) Lobs , the held-out description length (bits) of the engine’s observed walk, at episode 1500. In (b) and (d), boxes show the median and interquartile range, whiskers the full range, and dots individual runs.
Lesion
Removal
Latent
Engine
Regime
Intact
L
L1
L2
L
path only
Flow
dynamic
462
−3
−77
−117
−206
−455
partial
414
−18
+1
0
−287
−418
full
481
0†
0†
0†
−254
−478
Branching
dynamic
199
+47
+18
+16
−118
−167
partial
124
+18
+12
+8
−65
−105
TABLE I: Change in return from the intact baseline when the latents are ablated (median over the 20 models in each row). † Identical to intact on every model. Intact: mean return of the intact models.
Fig. 3: TD-STDP actor-critic backbone. The actor network generates actions based on the current state, while the critic network evaluates the actions. The TD-STDP learning rule updates the weights based on the prediction error and the eligibility trace. In panels (a-c), the mean return over training episodes is shown for each environment and arm; 20 seeds per arm. Lunar Lander shows mean return for the last 100 episodes.
Fig. 4: Inter-module connectivity effects in the TD-STDP actor-critic backbone. The figure illustrates how the connectivity between different modules influences the performance and learning dynamics of the network. The Spearman coefficients are shown to the right of each plot, indicating the strength and direction of the relationship between inter-module connectivity and performance.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Engine
Regime
AUC
Return
Frozen
Lobs (bits)
Solved
pAUC
pLobs
Flow
Dynamic
356.9 ± 96.7
442.4 ± 103.5
463.9 ± 94.3
4.54 ± 0.01
15/20
–
–
Partial
335.9 ± 64.9
431.6 ± 105.6
412.8 ± 122.8
6.00 ± 0.14
20/20
0.17
<10−6
Full
388.3 ± 68.8
475.9 ± 80.3
481.4 ± 80.1
4.34 ± 0.01
20/20
1.0
<10−6
Branching
Dynamic
204.1 ± 14.1
213.2 ± 50.4
198.9 ± 90.2
6.69 ± 0.03
0/20
–
–
Partial
116.8 ± 37.5
128.7 ± 68.1
126.5 ± 108.8
7.06 ± 0.14
0/20
<10−6
<10−6
Full
201.4 ± 39.0
176.2 ± 67.9
177.5 ± 78.3
6.75 ± 0.04
3/20
1.0
5.9×10−5
Appendix
TABLE II: CartPole, model v4 with the observed-walk rule: dynamic ( λ=0.05 ), partial ( λ=0 ) and full (sealed: no cross-module synapses, λ=0 ) modularity, on four engines. Mean ± s.d. over 20 seeds (2000–2019), 1500 episodes per run. AUC: mean return over all episodes. Return: mean of the last 100 episodes. Frozen: mean return of 20 episodes with W frozen at the end. Lobs : held-out description length (bits) of the engine’s observed walk at episode 1500. Solved: runs whose trailing-100 mean reaches 475. p : two-sided Mann–Whitney, Holm-adjusted over the four engines, of each arm against the map-term-on arm of the same engine.
Fig. 5: Energy use and description length across the inter-module connectivity sweep in the TD-STDP actor-critic backbone, for (a) CartPole, (b) Acrobot and (c) LunarLander. Each point is one final network (7 values of pcross× 2 map-term settings × 20 seeds, n=140 ); color gives pcross and shape the map-term setting (dynamic with the term on, partial with it off, full at pcross=0 with it off). Columns 1–3: effective synaptic operations (SOPs) per decision against Lobs and LW , and Lobs against LW , with the Spearman ρ over all points above each panel. Column 4: Spearman ρ within each pcross value ( n=20 per setting), with bootstrap intervals.
SOPs vs
Lobs vs
Return vs
Group
n
Lobs
LW
LW
Lobs
LW
Map-STDP v4, CartPole
Branching
60
+0.61∗
+0.71∗
+0.49∗
−0.49∗
−0.19
[+0.35,+0.77]
[+0.52,+0.81]
[+0.21,+0.67]
[−0.70,−0.24]
[−0.44,+0.08]
QIF
60
+0.95∗
+0.93∗
+0.92∗
−0.09
−0.11
[+0.90,+0.97]
[+0.87,+0.96]
[+0.86,+0.95]
[−0.35,+0.18]
[−0.37,+0.17]
Appendix
TABLE III: Spearman correlation between energy use, random-walk description length and performance of the final networks: effective synaptic operations (SOPs) per decision, the map equation of the observed walk ( Lobs ) and of the walk on the trained weights ( LW ), and the mean return of the frozen final network. ρ with its bootstrap 95% interval (2000 resamples over seeds) below; ∗ p<0.05 . Map-STDP v4: per spiking engine, pooled over the three modularity regimes. TD-STDP: all regimes pooled, then within each regime. All settings: each variable ranked within its setting (three v4 engines, three TD-STDP tasks) and pooled; bootstrap stratified by setting. —: every frozen TD-STDP CartPole network reaches the 500 ceiling, so its return is constant; that setting is left out of the all-settings return columns ( n=300 ).
Task
Dynamic
Full
Partial
CartPole
3.3
9.5
9.5
Acrobot
7.1
9.6
8.3
LunarLander
15.6
15.6
6.2
Appendix
TABLE IV: Effective fan-out (recurrent synapses per neuron above 2wmin ) of the final networks, mean over seeds.
Task
Mean cross-column weight
Within-column synapses
Within-column share
CartPole
0.094→0.001
9.5→9.5
0.91→1.00
Acrobot
0.084→0.187
9.5→6.9
0.83→0.62
LunarLander
0.048→0.140
15.6→4.5
0.77→0.31
Appendix
TABLE V: Recurrent weights of the partial arm (map term off) from the start to the end of training, mean over seeds. Within-column synapses: per neuron, above 2wmin . Within-column share: fraction of the total recurrent weight inside columns.
Spiking neural networks process information through discrete, time-dependent neural events, aiming for efficient computation and biologically inspired learning.
The Norwegian University of Life Sciences, Elizabeth Stephansens v. 15, 1433 Ås, Norway · Østfold University of Applied Sciences, B R A Veien 4, 1757 Halden, Norway