Hardware-accelerated graph neural networks: an alternative approach for neuromorphic event-based audio classification and keyword spotting on SoC FPGA
Authors: Kamil Jeziorek, Piotr Wzorek, Krzysztof Blachut, Hiroshi Nakano, Manon Dampfhoffer, Thomas Mesquida, Hiroaki Nishi, Thomas Dalgaty, +1 more
Organizations: The AGH University of Krakow, Poland · Graduate School of Science and Technology, Keio University, Japan · CEA-List, Université Grenoble Alpes, France
As the volume of data recorded by embedded edge sensors increases, particularly from neuromorphic devices producing discrete event streams, there is a growing need for hardware-aware neural architectures that enable efficient, low-latency, and energy-conscious local processing. We present an FPGA implementation of event-graph neural networks for audio processing. We utilise an artificial cochlea that converts time-series signals into sparse event data, reducing memory and computation costs. Our architecture was implemented on a SoC FPGA and evaluated on two open-source datasets. For classification task, our baseline floating-point model achieves 92.7% accuracy on SHD dataset - only 2.4% below the state of the art - while requiring over 10x and 67x fewer parameters. On SSC, our models achieve 66.9-71.0% accuracy. Compared to FPGA-based spiking neural networks, our quantised model reaches 92.3% accuracy, outperforming them by up to 19.3% while reducing resource usage and latency. For SSC, we report the first hardware-accelerated evaluation. We further demonstrate the first end-to-end FPGA implementation of event-audio keyword spotting, combining graph convolutional layers with recurrent sequence modelling. The system achieves up to 95% word-end detection accuracy, with only 10.53 microsecond latency and 1.18 W power consumption, establishing a strong benchmark for energy-efficient event-driven KWS.
Figures & tables
Figure 1. In this work, we propose an event-based keyword spotting system in which speech signals are converted into asynchronous events by an artificial cochlea and represented as spectro-temporal event-graphs. These are processed by a GCN–RNN model deployed on a SoC FPGA, enabling low-power, low-latency, and efficient keyword spotting. The figure illustrates an event-based keyword spotting system. Speech signals are converted into asynchronous events by an artificial cochlea and represented as spectro-temporal event-graphs. These are processed by aGCN–RNN model deployed on aSoC FPGA, enabling low-power, low-latency, and efficient keyword detection.
Figure 2. Overview of our hardware-accelerated event-graph neural network implementation. Events from the artificial cochlea are first read and preprocessed in the processing system (PS), and subsequently processed by an asynchronous graph neural network in the programmable logic (PL). Modules marked with dotted lines operate on an event-by-event basis. For the classification task, we employ a global average pooling layer, which updates an accumulator over the entire sample and returns the result to a fully connected classifier (MLP). For keyword spotting, the data is divided into smaller time windows ( Δ t), aggregated using a graph max pooling operation, and processed sequentially by two MLPs and a GRU, after which confidence and class scores are computed. In both tasks, we employ the same feature extractor, consisting of a graph generator followed by four graph convolution layers. The figure presents the hardware-accelerated event-graph neural network implementation. Events from the artificial cochlea are converted into event-graphs, processed by graph convolution layers, and shared between two tasks: classification and keyword spotting. Classification uses global average pooling followed by an MLP, while keyword spotting divides data into time windows, applies graph max pooling, and processes it with MLPs and a~GRU to compute confidence and class scores.
Figure 3. Diagram of the graph generation hardware module, which takes events from FIFO as input, and outputs the vertex features along with the list of its edges. The blue colour indicates the time (measured in clock cycles) required for the individual steps. The figure shows the hardware graph generation module. Events are read from a~FIFO buffer, searched across channels and temporal windows, and stored in BRAM. Valid edges are identified through temporal search, while features are computed using an accumulator and divider. The output consists of vertex features and the corresponding edge list, with blue labels indicating clock cycle costs of each step.
Figure 4. Diagram of the graph convolution hardware module. The arrows indicate the dataflow in the module, while the blue text indicates the time (measured in clock cycles) required for the individual steps. The figure illustrates the hardware architecture of the graph convolution (GraphConv) module. Events and their edge lists are processed using stored features in two-port BRAM. Features are quantised and combined with edge differences, then multiplied with weights from memory. The results undergo ReLU activation and max pooling to generate output features. The diagram also shows clock cycle costs for read and write operations, highlighting the efficiency of the hardware design.
Figure 5. Diagram of the head of the hardware KWS module. The key part of this module is vector multiplication shared between consecutive MLPs – its utilisation is controlled with the state machine. The figure shows the head of the hardware keyword spotting module (GruHead). Input features, arriving every 10 ms, are processed by linear layers and vector multiplications with weights stored in BRAM. AGRU cell updates hidden states using lookup-table-based activation functions (sigmoid and tanh), controlled by astate machine. The outputs are passed through linear layers for classification and confidence score estimation, enabling efficient keyword spotting.
Figure 6. Examples of spiking event stream with automatically extracted keyword boundaries. Top: raw spiking events; bottom: smoothed histogram with detected active range. The figure presents examples of spiking event streams with extracted keyword boundaries. The top row shows raw spiking events over time, while the bottom row shows corresponding smoothed histograms. The highlighted red regions indicate the automatically detected active keyword ranges.
Quantity
Symbol/name
Default/example
Sample window
T
1 s (based on the sample)
Histogram bin width
Δt
10 ms
Gaussian kernel size (in bins)
k
7
High threshold
Thigh=μ+ασ
α=0.5
Low threshold
Tlow=βThigh
β=0.2
Cooldown
cooldown_steps , τcool=cooldown_steps⋅Δt
5 bins
Table 1. Hyperparameters for Δt -binned KWS label extraction.
rch
skip step
top-1 acc. float
top-1 acc. quantised
rch
skip step
top-1 acc. float
top-1 acc. quantised
100
1
90.95 %
-
30
10
85.43 %
84.23 %
100
5
91.37 %
90.10 %
50
10
88.25 %
87.60 %
100
10
92.74 %
92.30 %
200
10
91.23 %
90.05 %
100
15
90.63 %
89.22 %
250
10
90.47 %
89.72 %
100
20
90.05 %
88.91 %
300
10
89.66 %
88.62 %
Table 2. The impact of search radius rch and skip step parameters on top-1 accuracy. The best results, highlighted in bold , were obtained for moderate parameters.
Figure 7. Model complexity analysis in term of FLOPs. The figures illustrate not only the significant impact of model size on computational complexity, but also the parameters of the graph generator. The figure shows model complexity analysis in terms of FLOPs. (a) The left plot presents average FLOPs per event for different graph generation configurations, showing how FLOPs decrease with increasing skip channels and vary with channel radius. (b) The right plot compares average FLOPs per event across convolution layers for different model sizes, highlighting the impact of convolution depth and output size on computational cost.
Name
Conv1
Conv2
Conv3
Conv4
FC
Params [k]
SHD top-1 acc. [%]
SSC-11 top-1 acc. [%]
SSC-35 top-1 acc. [%]
float
quantised
float
quantised
float
quantised
tiny
8
16
32
64
64
8.6
89.22
88.78
78.93
78.36
66.97
66.31
small
16
32
64
64
64
12.9
90.54
90.98
80.63
80.16
67.49
67.21
base
64
64
64
64
64
18.9
92.74
92.30
81.89
81.45
68.17
67.92
big
128
128
128
128
128
70.5
93.93
93.31
83.11
82.79
69.54
69.36
large
256
256
256
256
256
272.0
94.64
94.45
84.73
84.32
71.02
70.91
Table 3. Layer configurations (number of output features) with number of parameters and top-1 accuracy. Increasing the number of parameters improves classification results, demonstrating the scalability of our approach.
Table 4. Comparison of different operations in graph pooling for the SHD dataset:
Dataset
Author(s)
Method
Recurrent
Hardware
#Params
top-1 acc. [%]
Year
SHD
Cramer et al. ( Cramer et al., 2022 )
Recurrent SNN
✓
✗
-
83.2
2020
Yin et al. ( Yin et al., 2021 )
Adaptive SRNN
✓
✗
-
90.4
2021
Yao et al. ( Yao et al., 2021 )
TA-SNN
✗
✗
-
91.1
2021
Yu et al. ( Yu et al., 2022 )
STSC-SNN
✗
✗
2.1M
92.4
2022
Dampfhoffer et al. ( Dampfhoffer et al., 2022 )
Cuba-LIF
✓
✗
139k
87.8
2022
Bittar et al. ( Bittar and Garner, 2022 )
Recurrent SNN
✓
✗
3.9M
94.6
2022
Table 5. Comparison of accuracy and model size on the SHD and SSC datasets for the classification task. The symbol | indicates that only two models are present, while “–” denotes several intermediate values.
Model
Device
Logic cells
LUT
FF
BRAM
DSP
Freq. [MHz]
Latency [ μ s]
Power [W]
Acc. [%]
Matinizadeh ( Matinizadeh et al., 2024 )
Zynq ZCU104
-
149,760
92,160
75
-
100
-
1.629 1
87.80
Carpegna ( Carpegna et al., 2024 )
Zynq Z7-20
18,268
-
-
51
-
100
540
0.43
72.99
Our (base)
Zynq ZCU104
-
81,567
47,699
70
318
200
8.07 (179)
0.99 (3.73)
92.30
Our (tiny)
Zynq ZCU104
-
34,474
23,713
28
106
200
4.01 (175)
0.78 (3.52)
88.78
Our (2VecMuls)
Zynq ZCU104
-
41,925
30,198
70
164
200
15.2 (186)
0.89 (3.63)
92.30
Our (PL-head)
Zynq ZCU104
-
91,630
51,492
84.5
330
200
8.45
1.08
92.12
Table 6. Comparison of different variants of our architecture with state-of-the-art hardware implementations. The values in parentheses take into account the PS part (classification head) – apart from our last model (PL-head), which was fully realised in the programmable logic part of the SoC FPGA. The values marked with “-” were not mentioned in the respective papers. The resource utilisation for ( Matinizadeh et al., 2024 ) was reported only in terms of percentages; thus, the absolute values were estimated based on the maximum resources available on the ZCU104 board.
Dataset
#Params
Float
Quantised
AccK [%]
AccK,Δ [%]
AccK [%]
AccK,Δ [%]
SHD
60.34k
90.11
87.19
88.69
85.91
SSC-11
59.77k
76.21
73.66
73.49
71.30
SSC-35
61.52k
66.80
64.45
64.74
63.12
Table 7. Results of the floating-point (FP32) and quantised (8-bit) models for the keyword spotting task on the SHD and SSC datasets.
FE + pool
GRU head
Total
LUT
99,940
25,190
125,130
FF
55,182
27,190
82,372
BRAM
58.5
17
75.5
DSP
108
32
140
Latency [ μ s]
8.48
2.05
10.53
Average power [W]
1.03
0.15
1.18
Table 8. Resource utilisation, latency and power consumption for KWS model as described in Section 4.4 . We report the metrics for entire architecture as well as graph generation + feature extraction + pooling (FE + pool) and GRU head separately.
Figure 8. Latency comparison of tiny, base, and keyword spotting models on Jetson Orin NX (CPU/GPU) and ZCU104 FPGA. The figure shows a~latency comparison of tiny, base, and keyword spotting models on Jetson Orin NX (CPU/GPU) and ZCU104 FPGA. (a) Tiny model classification, (b) base model classification, and (c) keyword spotting are evaluated. The stacked bars indicate contributions from pre-event generation, event processing, graph feature extraction, and model forward pass. Results highlight that the FPGA achieves significantly lower latency compared to CPU and GPU implementations.
Figure 9. Power consumption of tiny, base, and keyword spotting models on Jetson Orin NX (CPU/GPU) and ZCU104 FPGA. The figure presents the power consumption of tiny, base, and keyword spotting models on Jetson Orin NX (CPU/GPU) and ZCU104 FPGA. CPU and GPU implementations show significantly higher power usage, especially in MAXN modes, while the FPGA achieves the lowest consumption (~1.1 W) across all model types, demonstrating its energy efficiency.
With the rapid growth of mobile robotics and embedded intelligence, there is an increasing demand for efficient on-device data processing on edge platforms. A promising research direction is the use of neuromorphic sensors inspired by human sensory systems, which generate sparse, event-based data encoding changes in the environment. In this work, we present the first end-to-end FPGA implementation of a keyword spotting system that integrates a Neuromorphic Auditory Sensor (NAS) and a graph neural network (GNN) on a single FPGA device, enabling real-time processing of raw audio data. The proposed architecture eliminates conventional signal preprocessing and operates directly on event-based audio streams. Leveraging a compute-near-memory network architecture, the system achieves efficient inference with low latency and low power consumption. Experimental results demonstrate an accuracy of 87.43% after quantization on the Google Speech Commands v2 dataset processed through the neuromorphic sensor, with end-to-end latency below 35 us and average power consumption of 1.12 W. The processed datasets, software models, and hardware modules are available at https://github.com/vision-agh/NAS-GNN-KWS.
Wiktor Matykiewicz, Piotr Wzorek, Kamil Jeziorek +4
AGH University of Krakow, Poland · Embedded Vision Systems Group, Computer Vision Laboratory · Robotics and Technology of Computers Lab., ETSII, EPS, SCORE, I3US, Universidad de Sevilla
Efficient processing of continuous audio streams remains a key challenge for real-time and resource-constrained systems. This paper introduces a neuromorphic trigger for audio event detection, based on a spiking neural network (SNN) that selectively gates input to downstream models. The proposed neuromorphic trigger acts as a flexible low-cost front-end, identifying salient audio segments and enabling these to be processed by a more computationally intensive model for tasks such as classification. The trigger is implemented as a lightweight fully connected SNN using a close-open filter for postprocessing, and is evaluated on two representative tasks: Anomalous Sound Detection (ASD) and Sound Event Detection (SED). For ASD, the trigger achieves a one-second segment-based F1 score of 0.97 on a class-agnostic form of the URBAN-SED dataset, demonstrating high reliability in identifying relevant audio regions. For SED, the trigger is combined with the Dang classifier on the DCASE 2017 Challenge Task 2 dataset, showing a potential 42.6× reduction in FLOPs while reducing the lower bound of the event-based error rate from 0.41 to 0.25. These results highlight the potential of neuromorphic triggers as real-time, energy-efficient front-end filters, enabling substantial reductions in computational cost.
Deep learning has greatly advanced automatic speech recognition (ASR), enabling widespread deployment on edge devices such as smartphones and smart home systems. However, the computational and energy demands of deep neural networks pose significant challenges for such resource-constrained deployments, introducing latency and limiting real-time interaction. Neuromorphic computing offers a promising solution by introducing activation sparsity through spiking neural networks (SNNs) and event-driven neural networks, converting dense operations into sparse computations. However, a study that evaluates the hardware benefits of different neuromorphic strategies remains lacking for ASR. This paper explores spiking and event-driven neuromorphic neural networks to improve activation sparsity in the state-of-the-art SpeechMamba model for ASR. We introduce an event-driven SpeechMamba with FATReLU activation, achieving over 60% activation sparsity with less than 1% accuracy degradation on LibriSpeech. We also propose a spiking SpeechMamba that attains over 70% sparsity while using 30% fewer parameters than comparable SNNs. Finally, we develop a cycle-accurate event-driven simulator enabling flexible algorithm-hardware co-exploration, which helps us identify computational bottlenecks and yields over 10% additional efficiency improvements.
Tauseef Ahmed, Tao Sun, Jeronimo Castrillon +2
Department of Advanced Computing Sciences, Maastricht University, Netherlands · Hardware-Efficient AI Team, IMEC, Netherlands · Chair for Compiler Construction, TU Dresden, Germany +1