S3T-Former: A Purely Spike-Driven State-Space Topology Transformer for Skeleton Action Recognition

Naichuan Zheng; Hailun Xia; Zepeng Sun; Weiyi Li; Yujia Wang

cs.CVMar 18, 2026

S3T-Former: A Purely Spike-Driven State-Space Topology Transformer for Skeleton Action Recognition

Authors: Naichuan Zheng, Hailun Xia, Zepeng Sun, Weiyi Li, Yujia Wang

Organizations: School of Information and Communication Engineering, Beijing University of Posts and Telecommunications, Beijing, China · International School, Beijing University of Posts and Telecommunications, Beijing, China · School of Economics and Management, Beijing University of Posts and Telecommunications, Beijing, China

Abstract

Skeleton-based action recognition is crucial for multimedia applications but heavily relies on power-hungry Artificial Neural Networks (ANNs), limiting their deployment on resource-constrained edge devices. Spiking Neural Networks (SNNs) provide an energy-efficient alternative; however, existing spiking models for skeleton data often compromise the intrinsic sparsity of SNNs by resorting to dense matrix aggregations, heavy multimodal fusion modules, or non-sparse frequency domain transformations. Furthermore, they severely suffer from the short-term amnesia of spiking neurons. In this paper, we propose the Spiking State-Space Topology Transformer (S3T-Former), which, to the best of our knowledge, is the first purely spike-driven Transformer architecture specifically designed for energy-efficient skeleton action recognition. Rather than relying on heavy fusion overhead, we formulate a Multi-Stream Anatomical Spiking Embedding (M-ASE) that acts as a generalized kinematic differential operator, elegantly transforming multimodal skeleton features into heterogeneous, highly sparse event streams. To achieve true topological and temporal sparsity, we introduce Lateral Spiking Topology Routing (LSTR) for on-demand conditional spike propagation, and a Spiking State-Space (S3) Engine to systematically capture long-range temporal dynamics without non-sparse spectral workarounds. Extensive experiments on multiple large-scale datasets demonstrate that S3T-Former achieves highly competitive accuracy while theoretically reducing energy consumption compared to classic ANNs, establishing a new state-of-the-art for energy-efficient neuromorphic action recognition.

Figures & tables

Figure 1. Overall architecture of the proposed Spiking State-Space Topology Transformer (S3T-Former). The input skeletal coordinates are first processed by the Multi-Stream Anatomical Spiking Embedding ( M-ASE ) to generate rich, multi-order (identity, spatial, and temporal) event streams. The core S3T Block employs a Spiking State-Space Topology Attention ( S3T-Attn ) module and a Spiking MLP . A non-spiking IF Integrator ( $U_{th}=\infty$ ) serves as a lossless membrane potential readout, optimized directly with Temporal Efficient Training (TET) loss.

Figure 2. Lateral Spiking Topology Routing (LSTR). It decouples spatial anatomy into multi-head pathways, executing zero-MAC spatial feature broadcasts via conditional sparse additions.

Figure 3. Spiking State-Space (S3) Engine. It constructs a linear-complexity temporal memory pool to integrate global spatio-temporal context without dense multiplications.

Model	Mod.	$T$	Param.	FLOPs	NTU RGB+D		NTU RGB+D 120		NW-UCLA
			(M)	(G)	Xs(%)	Xv(%)	Xs(%)	Xt(%)	(%)
Part-aware LSTM ( Du et al., 2015 ) [ANN-RNN]	J	-	-	-	62.9	70.3	25.5	26.3	-
ST-GCN ( Yan et al., 2018 ) [ANN-GCN]	J	-	3.10	3.48	81.5	88.3	70.7	73.2	-
2S-AGCN ( Shi et al., 2019 ) [ANN-GCN]	J+B	-	3.48	37.32	88.5	95.1	82.5	84.3	-
Shift-GCN ( Cheng et al., 2020 ) [ANN-GCN]	All	-	-	10.00	90.7	96.5	85.3	86.6	94.6
MS-G3D ( Liu et al., 2020 ) [ANN-GCN]	J+B	-	3.19	48.88	91.5	96.2	86.9	88.4	-

Table 1. Comparison with prior ANN- and SNN-based models on NTU RGB+D, NTU RGB+D 120, and NW-UCLA datasets. ‘Mod.’ denotes input modalities (J: Joint, B: Bone, JM: Joint Motion, BM: Bone Motion). The network type and architecture paradigm are denoted in brackets (e.g., [ANN-GCN], [SNN-Trans.]).

Step	Components					Param.	FLOPs	Acc
Step	UR	S3	LS	MA	AQ	(M)	(G)	(%)
1 (Baseline)						4.77	1.91	76.32
2	✓					4.77	1.91	77.15
3	✓	✓				4.77	1.91	78.68
4	✓	✓	✓			4.78	1.91	80.24
5	✓	✓	✓	✓		4.78	1.91	82.77

Table 2. Progressive core build-up ablation study of S3T-Former. UR: U-Readout, S3: S3-Engine, LS: LSTR, MA: M-ASE, AQ: ATG-QKV. FLOPs are reported for a single person.

Routing Strategy	Heads	Factor	Acc
	$H$	$\gamma$	(%)
Dense (No Prior)	-	-	78.68
Physical Only (Static)	-	-	79.12
LSTR (Hybrid)	Shared ( $H=1$ )	0.5	81.45
LSTR (Hybrid)	Split ( $H=4$ )	0.1	81.75
LSTR (Hybrid)	Split ( $H=4$ )	1.0	81.20

Table 3. Comprehensive ablation of LSTR routing strategies, multi-head configurations (

H

), and scaling factors (

\gamma

Temporal Memory	Decay Factor	Readout Type	Acc
	$\lambda$		(%)
None	-	Spike-rate	76.32
None	-	U-Readout	77.15
S3-Engine	Learnable	Spike-rate	78.68
S3-Engine	Fixed (0.5)	U-Readout	79.42
S3-Engine	Linear	U-Readout	81.18

Table 4. Ablation of S3-Engine and output readout strategies.

Panel of Figure 4 — Figure 4. Visualization of the dynamic topology matrix $\mathbf{A}_{dyn}^{(h)}$ (Eq. 7 ) across different network depths for head $h=8$ .

Panel of Figure 5 — Figure 5. Target vs. Competitor dynamics. U-Readout avoids the staircase effect of Spike Counting.

Panel of Figure 6 — Figure 6. Top-5 class progression. U-Readout provides smoother, more distinguishable trajectories.

Figure 7. Visualization of spatio-temporal activation maps across three distinct action classes.

Figure 8. t-SNE visualization of learned feature distributions for visually confusing action classes.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Block	ATG-QKV Generation			Spatial-Temporal Routing(LSTR+S3)		Spiking FFN (MLP)
Block	$Q$	$K$	$V$	Topo Buffer	Attn Out	MLP 1 (Exp.)	MLP 2 (Proj.)
1	27.49	53.00	44.24	10.09	28.15	4.20	20.60
2	21.43	49.31	27.71	2.95	23.17	2.45	17.62
3	24.77	56.07	24.19	1.86	25.90	1.90	20.53
4	24.26	53.65	22.42	1.32	22.93	1.56	28.19
5	19.77	55.84	19.13	1.31	30.01	1.38	34.24

Appendix

Table 5. Layer-wise average spike firing rates (

Fr

, %) of the standard S3T-Former (

T=16

). MLP 1 and MLP 2 denote the channel expansion (

4\times

) and projection layers within the Spiking FFN, respectively. The most parameter-heavy expansion modules (Topo Buffer and MLP 1) exhibit extreme deep-layer sparsity.

Figure 9. Full confusion matrix of S3T-Former on the NTU RGB+D 60 benchmark. The red dashed lines explicitly highlight the top three most confusing action classes: Class 10 (Reading), Class 11 (Writing), and Class 28 (Playing with phone/tablet). These failures highlight the inherent difficulty of distinguishing highly localized micro-motions within an aggressively sparse, event-driven SNN framework.

Panel of Figure 10 — Figure 10. Layer-wise evolution of spiking heatmaps. Red indicates active firing, while blue denotes zero-energy dormancy. This illustrates how the network funnels dense low-level kinematics into ultra-sparse semantic endpoints at deeper layers.

Figure 11. Per-class Top-1 accuracy (%) comparison between the Vanilla SNN baseline and our S3T-Former on the NTU RGB+D 60 dataset. The left section highlights the Top-10 classes with the most significant absolute accuracy improvements, while the right section shows the Top-5 classes with slight declines or minimal gains. S3T-Former demonstrates overwhelming superiority on complex, fine-grained actions while maintaining extreme robustness on saturated macro-motions.

Explore similar work

Elastic Spiking Transformers for Efficient Gesture Understanding
May 4, 2026Alberto Ancilotto, Gianluca Amprimo, Stefano Di Carlo +1Efficient Neural Network Inference Spiking Transformers
Fondazione Bruno Kessler · Politecnico di Torino
SAFformer:Improving Spiking Transformer via Active Predictive Filtering
May 8, 2026Zequan Xie, Weiming Zeng, Yunhua Chen +3Spiking Transformers Efficient ViTs
School of Computer Science and Technology, Guangdong University of Technology, Guangzhou, China · Faculty of Science, Hong Kong Baptist University, China · School of Electronic Information, Wuhan University, China
Neural Dynamics Self-Attention for Spiking Transformers
Mar 9, 2026Dehao Zhang, Fukai Guo, Shuai Wang +6Spiking Transformers Spiking Neural Networks
University of Electronic Science and Technology of China · Shenzhen Loop Area Institute · The Chinese University of Hong Kong (Shenzhen)

May 4, 2026J/K move · Enter open · S save

Elastic Spiking Transformers for Efficient Gesture Understanding

Alberto Ancilotto, Gianluca Amprimo, Stefano Di Carlo, Elisabetta Farella

Fondazione Bruno Kessler · Politecnico di Torino

Efficient Neural Network Inference Spiking Transformers Spiking Neural Networks Gesture Recognition Neural Network Compression

Open paper AskPDF arXiv

Spiking Neural Networks (SNNs), particularly Spiking Transformers, offer energy-efficient processing of event-based sensor data for healthcare applications. Yet current architectures are rigid: they are trained and deployed as static networks with fixed parameter counts and computational graphs. This limits deployment on neuromorphic hardware such as Loihi and SpiNNaker, where on-chip constraints often require smaller models that trade accuracy for feasibility. We introduce the Elastic Spiking Transformer, a runtime-adaptive architecture that brings elasticity into the spiking paradigm. Inspired by Matryoshka-style representation learning, it embeds nested elasticity in the Feature Extractor, Spiking Self-Attention, and Feed-Forward blocks. Through granularity-aware weight sharing, a single universal model can dynamically slice network width and attention heads at inference time without retraining. This design provides two key advantages for SNNs. First, it allows the model to adjust its parameter footprint to different hardware memory budgets. Second, reducing active neurons also lowers spike firing rates, yielding proportional reductions in synaptic operations, an energy benefit not directly available in standard artificial neural networks. We evaluate the approach on CIFAR10/100, CIFAR10-DVS, and the EHWGesture clinical gesture understanding dataset. Results show that one Elastic Spiking Transformer spans a broad range of complexity-accuracy trade-offs, matching or surpassing independently trained baselines while supporting adaptive, real-time gesture recognition on resource-constrained edge devices.

Abstract

Explore similar work

Elastic Spiking Transformers for Efficient Gesture Understanding

SAFformer:Improving Spiking Transformer via Active Predictive Filtering

Neural Dynamics Self-Attention for Spiking Transformers