Organizations: School of Information and Communication Engineering, Beijing University of Posts and Telecommunications, Beijing, China · International School, Beijing University of Posts and Telecommunications, Beijing, China · School of Economics and Management, Beijing University of Posts and Telecommunications, Beijing, China
Skeleton-based action recognition is crucial for multimedia applications but heavily relies on power-hungry Artificial Neural Networks (ANNs), limiting their deployment on resource-constrained edge devices. Spiking Neural Networks (SNNs) provide an energy-efficient alternative; however, existing spiking models for skeleton data often compromise the intrinsic sparsity of SNNs by resorting to dense matrix aggregations, heavy multimodal fusion modules, or non-sparse frequency domain transformations. Furthermore, they severely suffer from the short-term amnesia of spiking neurons. In this paper, we propose the Spiking State-Space Topology Transformer (S3T-Former), which, to the best of our knowledge, is the first purely spike-driven Transformer architecture specifically designed for energy-efficient skeleton action recognition. Rather than relying on heavy fusion overhead, we formulate a Multi-Stream Anatomical Spiking Embedding (M-ASE) that acts as a generalized kinematic differential operator, elegantly transforming multimodal skeleton features into heterogeneous, highly sparse event streams. To achieve true topological and temporal sparsity, we introduce Lateral Spiking Topology Routing (LSTR) for on-demand conditional spike propagation, and a Spiking State-Space (S3) Engine to systematically capture long-range temporal dynamics without non-sparse spectral workarounds. Extensive experiments on multiple large-scale datasets demonstrate that S3T-Former achieves highly competitive accuracy while theoretically reducing energy consumption compared to classic ANNs, establishing a new state-of-the-art for energy-efficient neuromorphic action recognition.
Figures & tables
Figure 1. Overall architecture of the proposed Spiking State-Space Topology Transformer (S3T-Former). The input skeletal coordinates are first processed by the Multi-Stream Anatomical Spiking Embedding ( M-ASE ) to generate rich, multi-order (identity, spatial, and temporal) event streams. The core S3T Block employs a Spiking State-Space Topology Attention ( S3T-Attn ) module and a Spiking MLP . A non-spiking IF Integrator ( Uth=∞ ) serves as a lossless membrane potential readout, optimized directly with Temporal Efficient Training (TET) loss.
Figure 2. Lateral Spiking Topology Routing (LSTR). It decouples spatial anatomy into multi-head pathways, executing zero-MAC spatial feature broadcasts via conditional sparse additions.
Figure 3. Spiking State-Space (S3) Engine. It constructs a linear-complexity temporal memory pool to integrate global spatio-temporal context without dense multiplications.
Model
Mod.
T
Param.
FLOPs
NTU RGB+D
NTU RGB+D 120
NW-UCLA
(M)
(G)
Xs(%)
Xv(%)
Xs(%)
Xt(%)
(%)
Part-aware LSTM ( Du et al., 2015 ) [ANN-RNN]
J
-
-
-
62.9
70.3
25.5
26.3
-
ST-GCN ( Yan et al., 2018 ) [ANN-GCN]
J
-
3.10
3.48
81.5
88.3
70.7
73.2
-
2S-AGCN ( Shi et al., 2019 ) [ANN-GCN]
J+B
-
3.48
37.32
88.5
95.1
82.5
84.3
-
Shift-GCN ( Cheng et al., 2020 ) [ANN-GCN]
All
-
-
10.00
90.7
96.5
85.3
86.6
94.6
MS-G3D ( Liu et al., 2020 ) [ANN-GCN]
J+B
-
3.19
48.88
91.5
96.2
86.9
88.4
-
Table 1. Comparison with prior ANN- and SNN-based models on NTU RGB+D, NTU RGB+D 120, and NW-UCLA datasets. ‘Mod.’ denotes input modalities (J: Joint, B: Bone, JM: Joint Motion, BM: Bone Motion). The network type and architecture paradigm are denoted in brackets (e.g., [ANN-GCN], [SNN-Trans.]).
Step
Components
Param.
FLOPs
Acc
UR
S3
LS
MA
AQ
(M)
(G)
(%)
1 (Baseline)
4.77
1.91
76.32
2
✓
4.77
1.91
77.15
3
✓
✓
4.77
1.91
78.68
4
✓
✓
✓
4.78
1.91
80.24
5
✓
✓
✓
✓
4.78
1.91
82.77
Table 2. Progressive core build-up ablation study of S3T-Former. UR: U-Readout, S3: S3-Engine, LS: LSTR, MA: M-ASE, AQ: ATG-QKV. FLOPs are reported for a single person.
Routing Strategy
Heads
Factor
Acc
H
γ
(%)
Dense (No Prior)
-
-
78.68
Physical Only (Static)
-
-
79.12
LSTR (Hybrid)
Shared ( H=1 )
0.5
81.45
LSTR (Hybrid)
Split ( H=4 )
0.1
81.75
LSTR (Hybrid)
Split ( H=4 )
1.0
81.20
Table 3. Comprehensive ablation of LSTR routing strategies, multi-head configurations ( H ), and scaling factors ( γ ).
Temporal Memory
Decay Factor
Readout Type
Acc
λ
(%)
None
-
Spike-rate
76.32
None
-
U-Readout
77.15
S3-Engine
Learnable
Spike-rate
78.68
S3-Engine
Fixed (0.5)
U-Readout
79.42
S3-Engine
Linear
U-Readout
81.18
Table 4. Ablation of S3-Engine and output readout strategies.
Figure 4. Visualization of the dynamic topology matrix Adyn(h) (Eq. 7 ) across different network depths for head h=8 .
Figure 5. Target vs. Competitor dynamics. U-Readout avoids the staircase effect of Spike Counting.
Figure 6. Top-5 class progression. U-Readout provides smoother, more distinguishable trajectories.
Figure 7. Visualization of spatio-temporal activation maps across three distinct action classes.
Figure 8. t-SNE visualization of learned feature distributions for visually confusing action classes.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Block
ATG-QKV Generation
Spatial-Temporal Routing(LSTR+S3)
Spiking FFN (MLP)
Q
K
V
Topo Buffer
Attn Out
MLP 1 (Exp.)
MLP 2 (Proj.)
1
27.49
53.00
44.24
10.09
28.15
4.20
20.60
2
21.43
49.31
27.71
2.95
23.17
2.45
17.62
3
24.77
56.07
24.19
1.86
25.90
1.90
20.53
4
24.26
53.65
22.42
1.32
22.93
1.56
28.19
5
19.77
55.84
19.13
1.31
30.01
1.38
34.24
Appendix
Table 5. Layer-wise average spike firing rates ( Fr , %) of the standard S3T-Former ( T=16 ). MLP 1 and MLP 2 denote the channel expansion ( 4× ) and projection layers within the Spiking FFN, respectively. The most parameter-heavy expansion modules (Topo Buffer and MLP 1) exhibit extreme deep-layer sparsity.
Figure 9. Full confusion matrix of S3T-Former on the NTU RGB+D 60 benchmark. The red dashed lines explicitly highlight the top three most confusing action classes: Class 10 (Reading), Class 11 (Writing), and Class 28 (Playing with phone/tablet). These failures highlight the inherent difficulty of distinguishing highly localized micro-motions within an aggressively sparse, event-driven SNN framework.
Figure 10. Layer-wise evolution of spiking heatmaps. Red indicates active firing, while blue denotes zero-energy dormancy. This illustrates how the network funnels dense low-level kinematics into ultra-sparse semantic endpoints at deeper layers.
Figure 11. Per-class Top-1 accuracy (%) comparison between the Vanilla SNN baseline and our S3T-Former on the NTU RGB+D 60 dataset. The left section highlights the Top-10 classes with the most significant absolute accuracy improvements, while the right section shows the Top-5 classes with slight declines or minimal gains. S3T-Former demonstrates overwhelming superiority on complex, fine-grained actions while maintaining extreme robustness on saturated macro-motions.
Spiking Neural Networks (SNNs), particularly Spiking Transformers, offer energy-efficient processing of event-based sensor data for healthcare applications. Yet current architectures are rigid: they are trained and deployed as static networks with fixed parameter counts and computational graphs. This limits deployment on neuromorphic hardware such as Loihi and SpiNNaker, where on-chip constraints often require smaller models that trade accuracy for feasibility. We introduce the Elastic Spiking Transformer, a runtime-adaptive architecture that brings elasticity into the spiking paradigm. Inspired by Matryoshka-style representation learning, it embeds nested elasticity in the Feature Extractor, Spiking Self-Attention, and Feed-Forward blocks. Through granularity-aware weight sharing, a single universal model can dynamically slice network width and attention heads at inference time without retraining. This design provides two key advantages for SNNs. First, it allows the model to adjust its parameter footprint to different hardware memory budgets. Second, reducing active neurons also lowers spike firing rates, yielding proportional reductions in synaptic operations, an energy benefit not directly available in standard artificial neural networks. We evaluate the approach on CIFAR10/100, CIFAR10-DVS, and the EHWGesture clinical gesture understanding dataset. Results show that one Elastic Spiking Transformer spans a broad range of complexity-accuracy trade-offs, matching or surpassing independently trained baselines while supporting adaptive, real-time gesture recognition on resource-constrained edge devices.
Alberto Ancilotto, Gianluca Amprimo, Stefano Di Carlo +1
Spiking Neural Networks (SNNs) offer notable advantages in biological plausibility and energy efficiency, making them promising candidates for building low-power Transformers. However, existing Spiking Transformers largely adhere to a passive reactive paradigm, which struggles to focus on task-relevant information and incurs substantial computational overhead when processing redundant visual data. To overcome this fundamental yet underexplored limitation, we propose SAFformer, a novel Spiking Transformer architecture based on an active predictive filtering paradigm. Inspired by the brain's predictive coding mechanism, SAFformer actively suppresses predictable signals and focuses on salient visual features. Extensive experiments show that SAFformer establishes new state-of-the-art performance on CIFAR-10/100 and CIFAR10-DVS. Remarkably, on ImageNet-1K, it achieves 80.44% Top-1 accuracy with only 26.58M parameters and an energy consumption of 5.88 mJ, demonstrating an exceptional balance between accuracy and efficiency.
Zequan Xie, Weiming Zeng, Yunhua Chen +3
School of Computer Science and Technology, Guangdong University of Technology, Guangzhou, China · Faculty of Science, Hong Kong Baptist University, China · School of Electronic Information, Wuhan University, China
Integrating Spiking Neural Networks (SNNs) with Transformer architectures offers a promising pathway to balance energy efficiency and performance, particularly for edge vision applications. However, existing Spiking Transformers face two critical challenges: (i) a substantial performance gap compared to their Artificial Neural Networks (ANNs) counterparts and (ii) high memory overhead during inference. Through theoretical analysis, we attribute both limitations to the Spiking Self-Attention (SSA) mechanism: the lack of locality bias and the need to store large attention matrices. Inspired by the localized receptive fields (LRF) and membrane-potential dynamics of biological visual neurons, we propose LRF-Dyn, which uses spiking neurons with localized receptive fields to compute attention while reducing memory requirements. Specifically, we introduce a LRF method into SSA to assign higher weights to neighboring regions, strengthening local modeling and improving performance. Building on this, we approximate the resulting attention computation via charge-fire-reset dynamics, eliminating explicit attention-matrix storage and reducing inference-time memory. Extensive experiments on visual tasks confirm that our method reduces memory overhead while delivering significant performance improvements. These results establish it as a key unit for achieving energy-efficient Spiking Transformers.
Dehao Zhang, Fukai Guo, Shuai Wang +6
University of Electronic Science and Technology of China · Shenzhen Loop Area Institute · The Chinese University of Hong Kong (Shenzhen)