SpikCTU: A Compact Spiking Network with Adaptive Recurrent Computation for Speech Command Recognition

Li Jin; Jiang Wenbin; Chen Tao; Wen Fei

cs.SDOct 10, 2026

SpikCTU: A Compact Spiking Network with Adaptive Recurrent Computation for Speech Command Recognition

Authors: Li Jin, Jiang Wenbin, Chen Tao, Wen Fei

Organizations: Hangzhou Dianzi University · Olympus Hedge Fund Investment Co., Ltd. · Shanghai Jiao Tong University

Abstract

Spiking Neural Networks offer event-driven, energy-efficient computation for speech command recognition, yet most existing methods follow fixed computation paths, allocating redundant computation to easy samples while providing limited insight into how evidence evolves during inference. To address these challenges, we introduce SpikCTU, a compact spiking network with adaptive recurrent computation for speech command recognition. Specifically, a hybrid spiking embedding module preserves continuous input information alongside spike features to construct a fixed temporal memory. A shared Continuous Thinking Unit (CTU) repeatedly reads this memory using keys derived from spikes and continuous values, and updates a binary recurrent state with lightweight stage adapters. Furthermore, combining a fixed base prediction with accumulated corrections at each step enables learned aggregation over all steps or confidence-based early exit using only the predictions available so far. Experiments on three speech command recognition benchmarks show that SpikCTU achieves competitive accuracy with only 0.16--0.17M parameters, and the early exit strategy can substantially further reduce computational cost without compromising accuracy. In addition, trajectory and attention analyses reveal that recurrent steps enable error correction and progressively refine predictions. Code is available at https://github.com/Rivflyyy/spikctu.

Figures & tables

Figure 1: SpikCTU separates a fixed memory that preserves information from shared recurrent computation over a binary spike state. Each step emits evidence tied to the current prediction; inference either exits using current evidence or aggregates all steps. Grey paths denote logically skipped steps.

Figure 2: One recurrent CTU step. A query from the binary state

s_{t-1}

reads fixed spike keys

K

and continuous values

V

with multi-head attention. The shared refiner updates

s_{t}

with recurrent gain and a stage adapter. A shared classifier adds the running correction mean to base evidence

b

, producing

\ell_{t}

and a causal confidence signal.

Figure 3: Diagnostics of the two learned mechanisms in the CTU. (a) Binary state activity varies across steps, indicating that the recurrent state is updated rather than replayed. (b) Aggregation assigns weights that vary across steps and tasks.

Data	Model	Params (M)	Acc. (%)	Energy (mJ)
SHD	DCLS-Delays (2L) ( Hammouamri et al., 2024 )	0.20	95.07	–
	d-cAdLIF ( Deckers et al., 2024 )	0.08	94.85	–
	SpikeSCR (1L) ( Wang et al., 2026a )	0.26	95.60	0.0051
	Pfa-SNN ( Sun et al., 2025 )	0.20	96.26	–
	EventProp+Delays ( Mészáros et al., 2025 )	–	93.20	–
	SpikCommander (1L) ( Wang et al., 2026b )	0.19	96.41	0.0039

Table 1: Accuracy, trainable parameters, and theoretical 45-nm proxy energy. A denotes ANN. Bold and underlined values rank first and second within each dataset. “–” denotes unavailable results. Baseline sources are given in Appendix A.4 , and multi-seed results for SpikCTU in Appendix A.3 .

Configuration	SHD ( $D{=}2$ )		SSC ( $D{=}8$ )		GSC ( $D{=}4$ )
Configuration	Acc. (%)	Params (K)	Acc. (%)	Params (K)	Acc. (%)	Params (K)
Full	96.60	164.23	81.74	174.04	95.96	169.43
w/o temporal composition	94.30	160.19	78.99	170.01	94.38	165.39
w/ continuous K/V memory	95.14	164.23	81.28	174.04	95.85	169.43
w/ spike K/V memory	95.49	164.23	81.12	174.04	95.75	169.43
w/o stage adapters	94.83	161.92	81.24	164.82	95.85	164.82

Table 2: Cross-task component ablation. Each row removes or replaces one component of the full model on the first row. Bold and underline mark the best and second-best accuracy per task. † The full SHD model trains without

L_{\mathrm{ref}}

, so this row enables it on SHD.

Figure 4: Prediction changes during recurrent computation. (a) Trajectory composition and correction types. Detailed counts are given in Appendix C.1 . (b) Top-1 change across steps of the same model. (c) Top-1 gap to full inference under

h{=}0.9

early exit.

Figure 5: Sample-level attention across recurrent steps on SSC. Columns show predictions that (a) remain correct, (b) change from wrong to correct, and (c) change from correct to wrong. Top: attention maps. Bottom: certainty, aggregation weight, and correctness.

Data	$D$	$h$	Acc. (%)	Mean steps	Reduction	Gain (pp)
SHD	2	0.01	95.63	1.00	50.00%	+0.00
SSC	8	0.84	81.42	3.28	59.04%	+0.81
GSC	4	0.91	95.99	1.32	66.97%	+1.21

Table 3: Adaptive computation with thresholds selected on validation data: accuracy, mean evaluated steps, and gain over a fixed-depth result at the same average compute.

Data	$\mathsf{T}=50$	$\mathsf{T}=100$	$\mathsf{T}=250$	$\Delta$
SHD	95.01	96.60	94.66	$-1.94$
SSC	79.78	81.74	82.53	$+0.79$
GSC	95.25	95.96	95.44	$-0.52$

Table 4: Top-1 accuracy (%) at different sensory resolutions.

\Delta=\mathrm{Acc}_{250}-\mathrm{Acc}_{100}

, in percentage points.

Figure 6: Top-1 accuracy versus recurrent CTU depth on SHD, SSC, and GSC. Each point is trained and evaluated at the indicated number of steps. The different optima show that additional recurrent computation does not always improve accuracy.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Data	Classes	$D$	Adapters	Epochs	LR	WD	Batch	$\lambda_{\mathrm{ref}}$
SHD	20	2	1	500	0.010	0.010	256	0.00
SSC	35	8	4	300	0.005	0.020	256	0.05
GSC	35	4	2	300	0.002	0.005	256	0.05

Appendix

Table 5: Frozen task configurations. LR and WD denote the initial learning rate and weight decay, respectively.

Data	Params (M)	Seed 312	Seed 326	Seed 12306	Seed 11408	Seed 42	Mean $\pm$ s.d.
SHD	0.16	96.60	95.80	95.27	95.10	94.79	$95.51\pm 0.71$
SSC	0.17	81.74	81.42	81.38	81.25	81.18	$81.39\pm 0.22$
GSC	0.17	95.96	95.80	95.76	95.60	95.80	$95.78\pm 0.13$

Appendix

Table 6: Five independent runs. Values are top-1 accuracy (%) under the official evaluation protocol; boldface and underlining indicate the best and second-best results, respectively, for each task. Tied second-best values are underlined.

Data	$D=1$	$D=2$	$D=4$	$D=8$	$D=16$
SHD	95.49	96.60	95.36	95.05	92.45
SSC	79.98	80.68	81.22	81.74	81.46
GSC	95.47	95.51	95.96	95.94	95.43

Appendix

Table 7: Recurrent-depth sensitivity (top-1 accuracy, %). Boldface and underlining indicate the best and second-best results, respectively, for each task.

Figure 7: Sensory time-step sensitivity. Open markers show task-specific optima.

Data	$\mathsf{T}=10$	$\mathsf{T}=20$	$\mathsf{T}=40$	$\mathsf{T}=50$	$\mathsf{T}=100$	$\mathsf{T}=200$	$\mathsf{T}=250$
SHD	87.94	92.18	94.79	95.01	96.60	94.43	94.66
SSC	66.75	75.05	79.23	79.78	81.74	82.17	82.53
GSC	85.31	93.32	95.24	95.25	95.96	95.68	95.44

Appendix

Table 8: Sensory time-step sensitivity (top-1 accuracy, %). Boldface and underlining indicate the best and second-best results, respectively, for each task.

Data	Depths (easy/medium/hard)	Accuracy (%)	Mean steps
SHD	$1/1/1$	95.63	1.00
SSC	$1/3/5$	81.12	3.06
GSC	$1/1/2$	95.61	1.35

Appendix

Table 9: Difficulty-aware allocation. Tertiles of normalized Step 1 entropy define the easy, medium, and hard groups; depths that do not decrease across these groups are selected on calibration data subject to the primary budget for early exit.

Figure 8: Complete confidence-threshold scan. Accuracy is relative to full adaptive inference; the right panels show the steps actually evaluated under each threshold and the fraction reaching full task depth. Reported thresholds are selected on calibration rather than from these evaluation curves.

Figure 9: Dynamics of the recurrent binary spike state. Firing probabilities for each neuron and step, population firing and flip rates, and pairwise Hamming distances show that the binary state vectors evolve across steps. Neurons are independently sorted per task for visualization.

Figure 10: Representative trajectories for samples that remain correct, end correctly after initially being wrong, and change from correct to wrong. Examples illustrate path shape only; prevalence claims are based on the counts reported in Fig. 4 .

Figure 11: Population attention entropy, consecutive Jensen–Shannon divergence, center shift, and relation between migration and target margin change. Shading denotes standard error over test samples.

Figure 12: Temporal occlusion of the input. Masking the highest-attention bins is substantially more damaging than masking matched random bins or the lowest-attention bins; the dashed line is unmasked accuracy.

Data	$D$	SOPs (G)	MACs (G)	Total ops (G)	Energy (mJ)
SHD	2	0.0014	0.0028	0.0041	0.0140
SSC	8	0.0016	0.0031	0.0047	0.0158
GSC	4	0.0006	0.0049	0.0055	0.0231

Appendix

Table 11: Operation counts at the full depth selected for each task and 45-nm-process proxy energy at

\mathsf{T}=100

Data	Task-selected full $D$	Accuracy (%)	SOPs (G)	Energy (mJ)	Energy reduction (%)
SHD	2	95.63	0.0013	0.0139	0.76%
SSC	8	75.22	0.0015	0.0150	4.90%
GSC	4	94.38	0.0005	0.0227	1.50%

Appendix

Table 12: Theoretical lower bound obtained by running one step; tokenizer and memory construction remain included.

Spiking Neural Networks

Spiking neural networks process information through discrete, time-dependent neural events, aiming for efficient computation and biologically inspired learning.

See all 351 papers

Adaptive Speech-to-Spike Encoding for Spiking Neural Networks
Jun 17, 2026Taharim Rahman Anon, Jakaria Islam EmonKeyword Spotting Speech Processing
1PI LLC, Sapporo, Hokkaido, Japan
Bio-Inspired Auditory Processing: Coupling Parallel Resonate-and-Fire Dynamics with Ternary Recurrent Networks for Speech Recognition
Oct 21, 2026Pavlos Poulos, Katelyn Sadorf, Themos Stafylakis +1Automatic Speech Recognition Spiking Neural Networks
Department of Informatics, Athens University of Economics and Business, Greece · Archimedes AI, Athena Research Center, Greece · California Institute of Technology, USA +1
Latency and accuracy tradeoffs in Spiking Neural Networks
Sep 28, 2026Zhanglu Yan, Zixuan Zhu, Kaiwen Tang +3Efficient Neural Network Inference Automatic Speech Recognition
Department of computer science, National University of Singapore · Shanghai Advanced Research Institute, Chinese Academy of Science · University of Chinese Academy of Sciences, Beijing +1

Jun 17, 2026J/K move · Enter open · S save

Adaptive Speech-to-Spike Encoding for Spiking Neural Networks

Taharim Rahman Anon, Jakaria Islam Emon

1PI LLC, Sapporo, Hokkaido, Japan

Keyword Spotting Speech Processing Spiking Neural Networks Neuromorphic Computing Spike-Based Signal Encoding

Open paper AskPDF arXiv

The mismatch between continuous acoustic signals and discrete event-driven processing remains a fundamental bottleneck for neuromorphic speech processing. Current systems typically rely on fixed spike encoders, forcing downstream Spiking Neural Networks (SNNs) to compensate for non-adaptive input representations. To address this, we present a learnable residual speech-to-spike encoder jointly trained end-to-end with a Recurrent Leaky Integrate-and-Fire (R-LIF) backbone. We validate this approach on the Google Speech Commands v2 (GSC-v2) benchmark, achieving up to 94.97% accuracy. Notably, the learned encoder remains highly parameter-efficient with a compact 35k-parameter variant that reaches 89.8%, matching or exceeding prior baselines that require an order of magnitude more parameters. Our encoder-focused analysis, including linear probing and gradient-residual inspection, indicates that the encoder does not target faithful signal reconstruction but instead learns task-aligned spike representations that enhance class separability. Finally, we benchmark bio-inspired, hardware-friendly credit assignment by comparing Direct Feedback Alignment (DFA) with surrogate-gradient BPTT under identical architectures and training conditions. We find that DFA reaches 91.5% accuracy, quantifying the performance trade-off of bio-inspired learning rules for modern neuromorphic audio.

Abstract

Spiking Neural Networks

Explore similar work

Adaptive Speech-to-Spike Encoding for Spiking Neural Networks

Bio-Inspired Auditory Processing: Coupling Parallel Resonate-and-Fire Dynamics with Ternary Recurrent Networks for Speech Recognition

Latency and accuracy tradeoffs in Spiking Neural Networks