Neural Dynamics Self-Attention for Spiking Transformers

Dehao Zhang; Fukai Guo; Shuai Wang; Jingya Wang; Jieyuan Zhang; Yimeng Shan; Malu Zhang; Yang Yang

cs.NEMar 9, 2026

Neural Dynamics Self-Attention for Spiking Transformers

Authors: Dehao Zhang, Fukai Guo, Shuai Wang, Jingya Wang, Jieyuan Zhang, Yimeng Shan, Malu Zhang, Yang Yang, +1 more

Organizations: University of Electronic Science and Technology of China · Shenzhen Loop Area Institute · The Chinese University of Hong Kong (Shenzhen)

Abstract

Integrating Spiking Neural Networks (SNNs) with Transformer architectures offers a promising pathway to balance energy efficiency and performance, particularly for edge vision applications. However, existing Spiking Transformers face two critical challenges: (i) a substantial performance gap compared to their Artificial Neural Networks (ANNs) counterparts and (ii) high memory overhead during inference. Through theoretical analysis, we attribute both limitations to the Spiking Self-Attention (SSA) mechanism: the lack of locality bias and the need to store large attention matrices. Inspired by the localized receptive fields (LRF) and membrane-potential dynamics of biological visual neurons, we propose LRF-Dyn, which uses spiking neurons with localized receptive fields to compute attention while reducing memory requirements. Specifically, we introduce a LRF method into SSA to assign higher weights to neighboring regions, strengthening local modeling and improving performance. Building on this, we approximate the resulting attention computation via charge-fire-reset dynamics, eliminating explicit attention-matrix storage and reducing inference-time memory. Extensive experiments on visual tasks confirm that our method reduces memory overhead while delivering significant performance improvements. These results establish it as a key unit for achieving energy-efficient Spiking Transformers.

Figures & tables

Figure 1: (a) Limited Local Modeling Capability: For a given n-th query (blue), VSA captures only limited and local relation. (b) High Memory Requirements: SSA requires explicit storage of their associated attention scores ( $\mathbf{QK}$ or $\mathbf{KV}$ ), leading to substantial computational overhead.

Figure 2: Mismatch between VSA and SSA attention scores: (a) and (b) show the average attention scores at different Manhattan distances, with VSA demonstrating stronger local modeling capabilities. (c) and (d) illustrate the distribution of attention scores, with VSA exhibiting lower entropy.

Figure 3: (a) Cognitive processes in biological vision, which exhibit local receptive field properties realized through multi-dendritic neurons. (b) The proposed LRF method together with the dynamic processes of dendritic neurons. (c) The implementation of LRF-SSA and LRF-Dyn.

Figure 4: Visual results for image recognition and semantic segmentation. Both LRF-SSA and LRF-Dyn produce sparser attention scores and achieve finer-grained segmentation results.

Method	Architecture	SR.	Param.(M)	Acc.(%)
SEWResNet ( Fang et al., 2021 )	SEW-ResNet-34	-	21.79	67.04
MSResNet ( Hu et al., 2024 )	MS-ResNet-34	-	21.80	69.42
SDT-V1 ( Yao et al., 2023 )	Spike-driven-8-512	$\mathcal{O}\mathrm{(Nd)}$	29.68	74.57
SDT-V2 ( Yao et al., 2024b )	Meta-SpikeFormer-384	$\mathcal{O}\mathrm{(d^{2})}$	15.08	74.10
SDT-V2 ( Yao et al., 2024b )	Meta-SpikeFormer-512	$\mathcal{O}\mathrm{(d^{2})}$	55.35	79.70
Spikformer ( Zhou et al., 2023b )	Spikformer-8-512	$\mathcal{O}\mathrm{(d^{2})}$	29.68	73.38

Table 1: Comparison with Similar Methods on ImageNet-1K.

Model	Para. (M)	Attn	T	MIoU(%)
ResNet	15.5	✗	1	32.9
( Yu et al., 2022 )	28.5	✗	1	36.7
PVT	28.2	✓	1	39.8
( Wang et al., 2021 )	48.0	✓	1	41.6
SDT-V3	5.1 + 1.4	✓	4	33.6
( Yao et al., 2025 )	18.99 + 1.4 †	✓	4	41.3

Table 2: Performance of segmentation.

Figure 5: (a) Visualization of effective receptive field for different methods, where both LRF-SSA and LRF-Dyn demonstrate strong locality. (b) Comparative analysis of memory usage, accuracy, and parameter efficiency. The results show that LRF-Dyn maintains performance comparable to LRF-SSA while substantially reducing memory requirements during inference.

Method	w/o LRF	$\Omega\leq 1$	$\Omega\leq 3$	$\Omega\leq 5$
LRF-SSA	77.86	78.26	78.52	78.64
LRF-Dyn	77.78	78.16	78.50	78.57
Caused SSA †	74.30	75.30	76.20	76.50

Table 3: Ablation Experiment.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Comparison of convergence performance across different models.

Explore similar work

Breaking Global Self-Attention Bottlenecks in Transformer-based Spiking Neural Networks with Local Structure-Aware Self-Attention
May 12, 2026Lingdong Li, Hangming Zhang, Qiang YuSpiking Transformers Efficient ViTs
School of Future Technology, Tianjin Key Laboratory of Cognitive Computing and Application, Tianjin University, Tianjin 300354, China · School of Artificial Intelligence, Tianjin Key Laboratory of Cognitive Computing and Application, College of Intelligence and Computing, Tianjin University, Tianjin 300354, China
SAFformer:Improving Spiking Transformer via Active Predictive Filtering
May 8, 2026Zequan Xie, Weiming Zeng, Yunhua Chen +3Spiking Transformers Efficient ViTs
School of Computer Science and Technology, Guangdong University of Technology, Guangzhou, China · Faculty of Science, Hong Kong Baptist University, China · School of Electronic Information, Wuhan University, China
Contrastive Attention Mitigates Spectral Bias in Spiking Transformers
Oct 1, 2026Xiaoli Liu, Malu Zhang, Yang YangSpiking Transformers Self-Attention
School of Computer Science and Engineering, University of Electronic Science and Technology of China

May 12, 2026J/K move · Enter open · S save

Breaking Global Self-Attention Bottlenecks in Transformer-based Spiking Neural Networks with Local Structure-Aware Self-Attention

Lingdong Li, Hangming Zhang, Qiang Yu

School of Future Technology, Tianjin Key Laboratory of Cognitive Computing and Application, Tianjin University, Tianjin 300354, China · School of Artificial Intelligence, Tianjin Key Laboratory of Cognitive Computing and Application, College of Intelligence and Computing, Tianjin University, Tianjin 300354, China

Spiking Transformers Efficient ViTs Spiking Neural Networks Local Attention Attention Mechanisms Image Classification

Open paper AskPDF arXiv

Transformer-based Spiking Neural Networks (SNNs) integrate SNNs with global self-attention and have demonstrated impressive performance. However, existing Transformer-based SNNs suffer from two fundamental limitations. First, they typically employ max pooling layers to reduce the size of feature maps, but the max pooling captures only the strongest response and fails to comprehensively preserve representative regional features. Second, the global self-attention involves all global feature interactions, resulting in computational redundancy and quadratic computational complexity, thus conflicting with the sparse and energy-efficient characteristics of SNNs. To address these challenges, we develop Local Structure-Aware Spiking Transformer (LSFormer), a novel Transformer-based Spiking Neural Network that incorporates Spiking Response Pooling (SPooling) and Local Structure-Aware Spiking Self-Attention (LS-SSA). For the first time, our LSFormer leverages a local dilated window mechanism to capture both local details and long-range dependencies. Experimental results demonstrate that our LSFormer achieves state-of-the-art performance compared to existing advanced Transformer-based SNNs. Notably, on the more challenging static dataset Tiny-ImageNet and neuromorphic dataset N-CALTECH101, LSFormer substantially outperforms state-of-the-art baselines by 4.3% and 8.6% in top-1 classification accuracy, respectively. These results highlight the potential of LSFormer to advance energy-efficient spiking models toward practical deployment in large-scale vision applications.

Abstract

Explore similar work

Breaking Global Self-Attention Bottlenecks in Transformer-based Spiking Neural Networks with Local Structure-Aware Self-Attention

SAFformer:Improving Spiking Transformer via Active Predictive Filtering

Contrastive Attention Mitigates Spectral Bias in Spiking Transformers