Hardware-accelerated graph neural networks: an alternative approach for neuromorphic event-based audio classification and keyword spotting on SoC FPGA

Kamil Jeziorek; Piotr Wzorek; Krzysztof Blachut; Hiroshi Nakano; Manon Dampfhoffer; Thomas Mesquida; Hiroaki Nishi; Thomas Dalgaty

Figure 1. In this work, we propose an event-based keyword spotting system in which speech signals are converted into asynchronous events by an artificial cochlea and represented as spectro-temporal event-graphs. These are processed by a GCN–RNN model deployed on a SoC FPGA, enabling low-power, low-latency, and efficient keyword spotting. The figure illustrates an event-based keyword spotting system. Speech signals are converted into asynchronous events by an artificial cochlea and represented as spectro-temporal event-graphs. These are processed by a~~GCN–RNN model deployed on a~~SoC FPGA, enabling low-power, low-latency, and efficient keyword detection.

Figure 2. Overview of our hardware-accelerated event-graph neural network implementation. Events from the artificial cochlea are first read and preprocessed in the processing system (PS), and subsequently processed by an asynchronous graph neural network in the programmable logic (PL). Modules marked with dotted lines operate on an event-by-event basis. For the classification task, we employ a global average pooling layer, which updates an accumulator over the entire sample and returns the result to a fully connected classifier (MLP). For keyword spotting, the data is divided into smaller time windows (

\Delta

t), aggregated using a graph max pooling operation, and processed sequentially by two MLPs and a GRU, after which confidence and class scores are computed. In both tasks, we employ the same feature extractor, consisting of a graph generator followed by four graph convolution layers. The figure presents the hardware-accelerated event-graph neural network implementation. Events from the artificial cochlea are converted into event-graphs, processed by graph convolution layers, and shared between two tasks: classification and keyword spotting. Classification uses global average pooling followed by an MLP, while keyword spotting divides data into time windows, applies graph max pooling, and processes it with MLPs and a~GRU to compute confidence and class scores.

Figure 3. Diagram of the graph generation hardware module, which takes events from FIFO as input, and outputs the vertex features along with the list of its edges. The blue colour indicates the time (measured in clock cycles) required for the individual steps. The figure shows the hardware graph generation module. Events are read from a~FIFO buffer, searched across channels and temporal windows, and stored in BRAM. Valid edges are identified through temporal search, while features are computed using an accumulator and divider. The output consists of vertex features and the corresponding edge list, with blue labels indicating clock cycle costs of each step.

Figure 4. Diagram of the graph convolution hardware module. The arrows indicate the dataflow in the module, while the blue text indicates the time (measured in clock cycles) required for the individual steps. The figure illustrates the hardware architecture of the graph convolution (GraphConv) module. Events and their edge lists are processed using stored features in two-port BRAM. Features are quantised and combined with edge differences, then multiplied with weights from memory. The results undergo ReLU activation and max pooling to generate output features. The diagram also shows clock cycle costs for read and write operations, highlighting the efficiency of the hardware design.

Figure 5. Diagram of the head of the hardware KWS module. The key part of this module is vector multiplication shared between consecutive MLPs – its utilisation is controlled with the state machine. The figure shows the head of the hardware keyword spotting module (GruHead). Input features, arriving every 10 ms, are processed by linear layers and vector multiplications with weights stored in BRAM. A~~GRU cell updates hidden states using lookup-table-based activation functions (sigmoid and tanh), controlled by a~~state machine. The outputs are passed through linear layers for classification and confidence score estimation, enabling efficient keyword spotting.

Figure 6. Examples of spiking event stream with automatically extracted keyword boundaries. Top: raw spiking events; bottom: smoothed histogram with detected active range. The figure presents examples of spiking event streams with extracted keyword boundaries. The top row shows raw spiking events over time, while the bottom row shows corresponding smoothed histograms. The highlighted red regions indicate the automatically detected active keyword ranges.

Quantity	Symbol/name	Default/example
Sample window	$T$	1 s (based on the sample)
Histogram bin width	$\Delta t$	10 ms
Gaussian kernel size (in bins)	$k$	7
High threshold	$T_{\text{high}}=\mu+\alpha\sigma$	$\alpha=0.5$
Low threshold	$T_{\text{low}}=\beta T_{\text{high}}$	$\beta=0.2$
Cooldown	cooldown_steps , $\;\tau_{\text{cool}}=\texttt{cooldown\_steps}\cdot\Delta t$	5 bins

Table 1. Hyperparameters for

\Delta t

-binned KWS label extraction.

$r_{ch}$	skip step	top-1 acc. float	top-1 acc. quantised	$r_{ch}$	skip step	top-1 acc. float	top-1 acc. quantised
100	1	90.95 %	-	30	10	85.43 %	84.23 %
100	5	91.37 %	90.10 %	50	10	88.25 %	87.60 %
100	10	92.74 %	92.30 %	200	10	91.23 %	90.05 %
100	15	90.63 %	89.22 %	250	10	90.47 %	89.72 %
100	20	90.05 %	88.91 %	300	10	89.66 %	88.62 %

Table 2. The impact of search radius

r_{ch}

and skip step parameters on top-1 accuracy. The best results, highlighted in bold , were obtained for moderate parameters.

Figure 7. Model complexity analysis in term of FLOPs. The figures illustrate not only the significant impact of model size on computational complexity, but also the parameters of the graph generator. The figure shows model complexity analysis in terms of FLOPs. (a) The left plot presents average FLOPs per event for different graph generation configurations, showing how FLOPs decrease with increasing skip channels and vary with channel radius. (b) The right plot compares average FLOPs per event across convolution layers for different model sizes, highlighting the impact of convolution depth and output size on computational cost.

Name	Conv1	Conv2	Conv3	Conv4	FC	Params [k]	SHD top-1 acc. [%]		SSC-11 top-1 acc. [%]		SSC-35 top-1 acc. [%]
Name	Conv1	Conv2	Conv3	Conv4	FC	Params [k]	float	quantised	float	quantised	float	quantised
tiny	8	16	32	64	64	8.6	89.22	88.78	78.93	78.36	66.97	66.31
small	16	32	64	64	64	12.9	90.54	90.98	80.63	80.16	67.49	67.21
base	64	64	64	64	64	18.9	92.74	92.30	81.89	81.45	68.17	67.92
big	128	128	128	128	128	70.5	93.93	93.31	83.11	82.79	69.54	69.36
large	256	256	256	256	256	272.0	94.64	94.45	84.73	84.32	71.02	70.91

Table 3. Layer configurations (number of output features) with number of parameters and top-1 accuracy. Increasing the number of parameters improves classification results, demonstrating the scalability of our approach.

Table 4. Comparison of different operations in graph pooling for the SHD dataset:

Dataset	Author(s)	Method	Recurrent	Hardware	#Params	top-1 acc. [%]	Year
SHD	Cramer et al. ( Cramer et al., 2022 )	Recurrent SNN	✓	✗	-	83.2	2020
	Yin et al. ( Yin et al., 2021 )	Adaptive SRNN	✓	✗	-	90.4	2021
	Yao et al. ( Yao et al., 2021 )	TA-SNN	✗	✗	-	91.1	2021
	Yu et al. ( Yu et al., 2022 )	STSC-SNN	✗	✗	2.1M	92.4	2022
	Dampfhoffer et al. ( Dampfhoffer et al., 2022 )	Cuba-LIF	✓	✗	139k	87.8	2022
	Bittar et al. ( Bittar and Garner, 2022 )	Recurrent SNN	✓	✗	3.9M	94.6	2022

Table 5. Comparison of accuracy and model size on the SHD and SSC datasets for the classification task. The symbol | indicates that only two models are present, while “–” denotes several intermediate values.

Model	Device	Logic cells	LUT	FF	BRAM	DSP	Freq. [MHz]	Latency [ $\mu$ s]	Power [W]	Acc. [%]
Matinizadeh ( Matinizadeh et al., 2024 )	Zynq ZCU104	-	149,760	92,160	75	-	100	-	1.629 1	87.80
Carpegna ( Carpegna et al., 2024 )	Zynq Z7-20	18,268	-	-	51	-	100	540	0.43	72.99
Our (base)	Zynq ZCU104	-	81,567	47,699	70	318	200	8.07 (179)	0.99 (3.73)	92.30
Our (tiny)	Zynq ZCU104	-	34,474	23,713	28	106	200	4.01 (175)	0.78 (3.52)	88.78
Our (2VecMuls)	Zynq ZCU104	-	41,925	30,198	70	164	200	15.2 (186)	0.89 (3.63)	92.30
Our (PL-head)	Zynq ZCU104	-	91,630	51,492	84.5	330	200	8.45	1.08	92.12

Table 6. Comparison of different variants of our architecture with state-of-the-art hardware implementations. The values in parentheses take into account the PS part (classification head) – apart from our last model (PL-head), which was fully realised in the programmable logic part of the SoC FPGA. The values marked with “-” were not mentioned in the respective papers. The resource utilisation for ( Matinizadeh et al., 2024 ) was reported only in terms of percentages; thus, the absolute values were estimated based on the maximum resources available on the ZCU104 board.

Dataset	#Params	Float		Quantised
Dataset	#Params	$Acc_{K}$ [%]	$Acc_{K,\Delta}$ [%]	$Acc_{K}$ [%]	$Acc_{K,\Delta}$ [%]
SHD	60.34k	90.11	87.19	88.69	85.91
SSC-11	59.77k	76.21	73.66	73.49	71.30
SSC-35	61.52k	66.80	64.45	64.74	63.12

Table 7. Results of the floating-point (FP32) and quantised (8-bit) models for the keyword spotting task on the SHD and SSC datasets.

	FE + pool	GRU head	Total
LUT	99,940	25,190	125,130
FF	55,182	27,190	82,372
BRAM	58.5	17	75.5
DSP	108	32	140
Latency [ $\mu$ s]	8.48	2.05	10.53
Average power [W]	1.03	0.15	1.18

Table 8. Resource utilisation, latency and power consumption for KWS model as described in Section 4.4 . We report the metrics for entire architecture as well as graph generation + feature extraction + pooling (FE + pool) and GRU head separately.

Figure 8. Latency comparison of tiny, base, and keyword spotting models on Jetson Orin NX (CPU/GPU) and ZCU104 FPGA. The figure shows a~latency comparison of tiny, base, and keyword spotting models on Jetson Orin NX (CPU/GPU) and ZCU104 FPGA. (a) Tiny model classification, (b) base model classification, and (c) keyword spotting are evaluated. The stacked bars indicate contributions from pre-event generation, event processing, graph feature extraction, and model forward pass. Results highlight that the FPGA achieves significantly lower latency compared to CPU and GPU implementations.

Figure 9. Power consumption of tiny, base, and keyword spotting models on Jetson Orin NX (CPU/GPU) and ZCU104 FPGA. The figure presents the power consumption of tiny, base, and keyword spotting models on Jetson Orin NX (CPU/GPU) and ZCU104 FPGA. CPU and GPU implementations show significantly higher power usage, especially in MAXN modes, while the FPGA achieves the lowest consumption (~1.1 W) across all model types, demonstrating its energy efficiency.

Hardware-accelerated graph neural networks: an alternative approach for neuromorphic event-based audio classification and keyword spotting on SoC FPGA

Abstract

Figures & tables

Explore similar work

End-to-End Keyword Spotting on FPGA Using Graph Neural Networks with a Neuromorphic Auditory Sensor

A Neuromorphic Trigger for Efficient Audio Event Detection

Spiking and Event-driven Neuromorphic Mamba Models for Efficient Speech Recognition