Bio-Inspired Auditory Processing: Coupling Parallel Resonate-and-Fire Dynamics with Ternary Recurrent Networks for Speech Recognition
Authors: Pavlos Poulos, Katelyn Sadorf, Themos Stafylakis, Costas A. Anastassiou
Organizations: Department of Informatics, Athens University of Economics and Business, Greece · Archimedes AI, Athena Research Center, Greece · California Institute of Technology, USA · Cedars-Sinai Medical Center, USA
Spiking Neural Networks (SNNs) promise energy-efficient inference through sparse, event-driven computation, yet remain constrained on continuous, context-dependent tasks such as automatic speech recognition (ASR). A key bottleneck is the Leaky Integrate-and-Fire (LIF) neuron, whose membrane decay erases information across pauses and phonetic boundaries while it must both extract features and retain long-range context. Inspired by cortical separation of nonlinear filtering and somatic integration, we introduce the Spiking Ternary GRU (stGRU): a ternary-weight gated recurrent unit performs temporal filtering, while downstream Resonate-and-Fire (RF) neurons generate the binary spike stream driving the next block's event pathway. This makes binary event transmission a native training constraint rather than a separate deployment-time activation quantization step; together with ternary weights, learned projections avoid dense floating-point matrix MACs. Trained end-to-end on LibriSpeech-960, Small, Medium, and Large stGRUs reach 8.09%, 7.04%, and 6.44±0.17% WER on test-clean without an external language model. To our knowledge, these are the first matrix-MAC-free spiking networks without a learned ANN sequence decoder to achieve single-digit WER on LibriSpeech. The architecture also achieves 10.88% test CER on Aishell-1 (9.28% on dev) when trained from scratch. Internally, trained networks develop a depth-graded hierarchy: shallow layers act as fast acoustic filters, while deep layers form slower semantic integrators whose latent representations segregate into part-of-speech manifolds and word-level spike events. Deep-layer firing rates follow lognormal distributions with a sparsity gradient, mirroring hierarchical processing observed in human cortex in vivo. Our biologically grounded architecture combines competitive ASR with favorable modeled compute and memory cost.
Figures & tables
Figure 1: Acoustic alignment, internal population activity, and semantic manifold emergence in stGRU. (Top) Input acoustic waveform (LibriSpeech test utterance) aligned with ground-truth word boundaries (color-shaded spans) and internal spiking representations across physical time (top axis) and discrete network timesteps (bottom axis). The central heatmap displays the Z -scored activity of 128 superneurons (co-active neural clusters sorted via Rastermap Stringer et al. (2025) ), with darker pixels indicating higher population firing. Red labels beneath the raster mark subword BPE token emissions predicted by the CTC readout layer (leading underscores denote new word onsets; consecutive identical tokens are collapsed into a single emission; CTC blanks omitted for clarity). (Bottom) 2D UMAP projections of latent representations across network depth (layers 0, 6, 10, and 14). Internal states progressively disentangle into distinct semantic and syntactic clusters, color-coded by part of speech: verbs (red), nouns (blue), and pronouns (green).
Figure 2: Architecture and efficiency comparisons. (a) stGRU block architecture. (b) LibriSpeech test-clean WER versus estimated arithmetic energy over the 19,440 s evaluation set. (c) WER versus theoretical parameter storage in decimal MB. The Large stGRU WER is a six-seed mean; energy estimates correspond to the reported profiling run.
Architecture
Params (M)
Weight Type
Memory
Energy rate (mJ/s)
WER (%)
(MB)
(mJ/s)
test-clean
test-other
Continuous Artificial Neural Networks (ANNs)
Moritz et al. Moritz et al. (2019)
70.0
FP32
280.0†
-
5.4
16.1
Park et al. Park et al. (2019)
360.0
FP32
1440.0†
-
2.8
6.8
Whisper-Small Radford et al. (2022)
244.0
FP16
488.0†
-
3.1
7.4
ResNet-BiGRU Reza et al. (2022)
33.0
FP32
132.8†
-
15.6
-
Table 1: Comprehensive Architectural Comparison on LibriSpeech-960 ASR. All reported WER results evaluate the standalone acoustic model without an external language model. The Large RF WER is mean ± standard deviation over six seeds; other in-house rows and arithmetic-energy rates correspond to seed 3407.
Figure 3: Depth-graded coding emerges in trained Spiking-Ternary networks. Row 1: PSTH of word-selective neurons at shallow (L0–5), uncertainty (L11), and commitment (L14) depths. Each panel pair shows responses to a target word (“this,” blue) against acoustically similar competitors (gray), with the right sub-panel split by outcome (green: high-conf correct; orange: low-conf correct; red: incorrect). Row 2: Per-layer counts of outcome-selective neurons (matched-AUC, FDR q<0.001 ); correct-preferring neurons are sub-classified by high- vs. low-confidence via median CTC log-probability. Selectivity grows from L0–L5 to L11–L14 ( <50 vs. >250 neurons), and the bipartite partition reverses across depth: incorrect-preferring neurons peak at L11 ( n=194 ), high-confidence-correct-preferring at L14 ( n=254 ). Row 3: Population sparsity (per-word silent fraction) across encoder depth, with shaded IQR. Row 4: Population firing rate distributions for L9–L14 split by outcome (brackets: pairwise Mann–Whitney U-test with Bonferroni correction; ns : panels with at least one non-significant comparison). Row 5: Per-neuron firing rate histograms for L9–L14 with Gaussian fit overlaid.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Linguistic Stage
Channels
Kernel Size
Number of Layers
Small
Medium
Large
Stage 1 (Phonemes)
1024
11
1
2
3
Stage 2 (Syllables)
1024
21
1
2
3
Stage 3 (Words)
1024
39
1
2
3
Stage 4 (Sentences)
1024
51
1
2
3
Stage 5 (Extended Context)
1024
63
–
–
3
Appendix
Table S1: Detailed architectural configurations for the Small, Medium, and Large stGRU variants. The receptive field expands progressively across stages to capture hierarchical linguistic features.
Component
Hyperparameter
Value / Initialization
PRF Dynamics
Membrane Time Constant ( τ )
Target 1.0 (with N(0,0.05) jitter on τ−1 )
Resonant Frequency ( θ )
Linear spread ∈[0,π/2] + N(0,0.02) jitter
Integration Step ( dt )
Target 1.0 + N(0,0.05) jitter
Spike Threshold ( Vthr )
Target 1.0 + N(0,0.05) jitter
Optimization
SRCA Peak Learning Rate ( ηmax )
10−3
SRCA Valley Learning Rate ( ηmin )
10−5
Appendix
Table S2: Key hyperparameters for PRF neuronal dynamics, optimization, tokenization, and CTC decoding .
Figure S1: Semi-Resonant Cosine Annealing (SRCA) learning rate schedule with initial holding period ( Shold=150 ), followed by 2.75 cosine cycles with exponential peak decay ( γ=0.75 ) and initial phase offset ( ϕstart=0.75 ).
Architecture
Estimated Arithmetic Energy (Joules over 19,440 s)
Estimated Arithmetic-Energy Rate (mJ/s)
Dense Matrix MACs
Sub-Sampler
Encoder
Decoder
Total
Sub-Sampler
Encoder
Decoder
Total
Continuous Artificial Neural Networks (ANNs)
Transformer 112M Hou et al. (2024)
53.13
117.01
1.16
171.31
2.73
6.01
0.06
8.80
Yes
Transformer++ 112M Hou et al. (2024)
0.18
167.73
1.16
169.08
0.009
8.623
0.06
8.69
Yes
Conformer 136M Hou et al. (2024)
53.13
165.19
1.16
219.49
2.73
8.49
0.06
11.28
Yes
Hybrid Spiking Neural Networks (SNNs) & Dense ANN Decoder
Appendix
Table S3: Estimated arithmetic-energy decomposition on LibriSpeech test-clean (5.40 hours; 19,440 s). For the continuous ANN comparisons, the downstream column is a modeled FP32 CTC readout; for IML-Spikeformer it is a six-layer ANN decoder; for stGRU it is active-weight accumulation in an FP32 CTC readout driven by binary spikes; see Appendix D.1 ). Estimates omit normalization, activation functions, residual scaling, memory traffic, and decoding/search.
Table S4: Stage-by-Stage Dynamic Energy Profiling across Model Scales and Architectures for LibriSpeech and Aishell-1 .
Architecture
Params (M)
Weight Type
Memory
Energy rate
CER (%)
(MB)
(mJ/s)
dev
test
Continuous Artificial Neural Networks (ANNs)
VGG-BiLSTM Ravanelli et al. (2022)
93.30
FP32
373.0†
–
9.7
10.7
Binary S4D ∗ Stan and Rhodes (2023)
21.00
FP32
84.0†
–
10.3
11.9
Transformer ∗ Stan and Rhodes (2023)
61.90
FP32
247.6†
–
5.1
5.6
Hybrid Spiking Neural Networks (SNNs) & Dense ANN Decoder/LM
Appendix
Table S5: Comprehensive Architectural Comparison on Aishell-1 Mandarin ASR. Evaluated on Character Error Rate (CER %). Memory sizes are theoretical footprints based on parameter counts and precision formats.
Test Set
4-gram LM
Seed 1
Seed 10
Seed 100
Seed 1000
Seed 10000
Seed 3407
Mean ± Std
test-clean
Without
6.53
6.60
6.33
6.46
6.58
6.14
6.44±0.17
With
6.01
6.12
5.86
5.98
6.09
5.66
5.95±0.16
test-other
Without
17.49
17.74
17.18
17.74
17.55
17.28
17.49±0.23
With
16.54
16.77
16.28
16.75
16.58
16.30
16.53±0.21
Appendix
Table S6: Large Model WER (%) on LibriSpeech across 6 Random Seeds .
Scale
Architecture Variant
Precision
Params (M)
Memory (MB)
Error Rate (%)
Energy rate (mJ/s)
test-clean / Dev
test-other / Test
Part I: LibriSpeech-960 Evaluation (WER %)
Small
Linear → RF
FP32 ‡
65.39
70.79
19.46
35.40
1.31
Conv1D → RF
FP32 ‡
65.51
71.28
14.96
30.39
1.63
MLGRU → ReLU
FP32
128.33
513.32
14.46
28.75
21.87
MLGRU → RF
Ternary
128.35
70.98
15.95
30.87
1.74
Appendix
Table S7: Component-Wise Ablation: Model Scale, Architecture, and Weight Precision across LibriSpeech and Aishell-1 . Isolates individual contributions of low-precision weights, sequence gating, and discrete spiking activations. Reported numbers correspond to seed 3407.
Model
Scaling Mode
WER (%)
Sparsity (%)
Energy rate
test-clean
test-other
Activation
Weight
(mJ/s)
Small
Naive ( X(Wγ)⊤ )
8.15
20.92
12.93
41.53
9.78
Proposed ( (XW⊤)γ )
8.09
20.79
12.84
41.76
1.95
None ( XW⊤ )
9.44
22.61
14.44
39.81
2.21
Large
Naive ( X(Wγ)⊤ )
6.44
17.43
17.36
38.57
17.74
Proposed ( (XW⊤)γ )
6.14
17.28
17.08
39.15
3.42
Appendix
Table S8: Placement of the weight scale in Small and Large stGRUs. All modes share input normalization. Naive folds γ into the weights; Proposed applies it after ternary accumulation; None omits it. Proposed has substantially lower modeled arithmetic-energy rate than Naive, while omitting γ worsens test-clean WER at both scales. Energy rates use the stated 45 nm operation costs and exclude an external language model.
Model Size
Neuron Type
Training Time (10 epochs)
Small
LIF
8h 53m
Small
PRF
2h 40m ( 3.3× speedup)
Medium
LIF
14h 35m
Medium
PRF
4h 44m ( 3.1× speedup)
Large
LIF
25h 00m
Large
PRF
7h 21m ( 3.4× speedup)
Appendix
Table S9: Empirical Training Throughput for 10 Epochs on LibriSpeech-960 : PRF vs. LIF on 8 AMD Instinct MI250X GCDs.
Figure S2: Depth-dependent change in firing-rate fits (L9–L14). Top: log-axis firing-rate histograms with lognormal (red) and Gaussian (blue) fits. Bottom: ΔR2=Rln2−RN2 ; the lognormal fit has its largest advantage at L11 ( +0.091 ), while the Gaussian fit is slightly better at L14 ( −0.011 ).
Figure S3: Outcome-selective neuron counts (L9–L14) for full vs. attempted-emission-only trial pools, MLGRU vs. no-MLGRU (Large variants) . Selective neuron counts per layer, split by outcome preference (incorrect-preferring, red; correct-preferring, green). Left: Full MLGRU (Conv1D → MLGRU → RF). Empty-emission rate on incorrect words is 13.82%; restricting to attempted emissions reduces aggregate selective count by only 8.1%. Right: Ablated variant without MLGRU (Conv1D → RF). Empty-emission rate is 57.76%; restricting to attempted emissions collapses aggregate selective count by 95.5%, supporting a role for MLGRU gating for the emergent error-monitoring property.
The mismatch between continuous acoustic signals and discrete event-driven processing remains a fundamental bottleneck for neuromorphic speech processing. Current systems typically rely on fixed spike encoders, forcing downstream Spiking Neural Networks (SNNs) to compensate for non-adaptive input representations. To address this, we present a learnable residual speech-to-spike encoder jointly trained end-to-end with a Recurrent Leaky Integrate-and-Fire (R-LIF) backbone. We validate this approach on the Google Speech Commands v2 (GSC-v2) benchmark, achieving up to 94.97% accuracy. Notably, the learned encoder remains highly parameter-efficient with a compact 35k-parameter variant that reaches 89.8%, matching or exceeding prior baselines that require an order of magnitude more parameters. Our encoder-focused analysis, including linear probing and gradient-residual inspection, indicates that the encoder does not target faithful signal reconstruction but instead learns task-aligned spike representations that enhance class separability. Finally, we benchmark bio-inspired, hardware-friendly credit assignment by comparing Direct Feedback Alignment (DFA) with surrogate-gradient BPTT under identical architectures and training conditions. We find that DFA reaches 91.5% accuracy, quantifying the performance trade-off of bio-inspired learning rules for modern neuromorphic audio.
Spiking Neural Networks offer event-driven, energy-efficient computation for speech command recognition, yet most existing methods follow fixed computation paths, allocating redundant computation to easy samples while providing limited insight into how evidence evolves during inference. To address these challenges, we introduce SpikCTU, a compact spiking network with adaptive recurrent computation for speech command recognition. Specifically, a hybrid spiking embedding module preserves continuous input information alongside spike features to construct a fixed temporal memory. A shared Continuous Thinking Unit (CTU) repeatedly reads this memory using keys derived from spikes and continuous values, and updates a binary recurrent state with lightweight stage adapters. Furthermore, combining a fixed base prediction with accumulated corrections at each step enables learned aggregation over all steps or confidence-based early exit using only the predictions available so far. Experiments on three speech command recognition benchmarks show that SpikCTU achieves competitive accuracy with only 0.16--0.17M parameters, and the early exit strategy can substantially further reduce computational cost without compromising accuracy. In addition, trajectory and attention analyses reveal that recurrent steps enable error correction and progressively refine predictions. Code is available at https://github.com/Rivflyyy/spikctu.
Li Jin, Jiang Wenbin, Chen Tao +1
Hangzhou Dianzi University · Olympus Hedge Fund Investment Co., Ltd. · Shanghai Jiao Tong University
Deep learning has greatly advanced automatic speech recognition (ASR), enabling widespread deployment on edge devices such as smartphones and smart home systems. However, the computational and energy demands of deep neural networks pose significant challenges for such resource-constrained deployments, introducing latency and limiting real-time interaction. Neuromorphic computing offers a promising solution by introducing activation sparsity through spiking neural networks (SNNs) and event-driven neural networks, converting dense operations into sparse computations. However, a study that evaluates the hardware benefits of different neuromorphic strategies remains lacking for ASR. This paper explores spiking and event-driven neuromorphic neural networks to improve activation sparsity in the state-of-the-art SpeechMamba model for ASR. We introduce an event-driven SpeechMamba with FATReLU activation, achieving over 60% activation sparsity with less than 1% accuracy degradation on LibriSpeech. We also propose a spiking SpeechMamba that attains over 70% sparsity while using 30% fewer parameters than comparable SNNs. Finally, we develop a cycle-accurate event-driven simulator enabling flexible algorithm-hardware co-exploration, which helps us identify computational bottlenecks and yields over 10% additional efficiency improvements.
Tauseef Ahmed, Tao Sun, Jeronimo Castrillon +2
Department of Advanced Computing Sciences, Maastricht University, Netherlands · Hardware-Efficient AI Team, IMEC, Netherlands · Chair for Compiler Construction, TU Dresden, Germany +1