Spiking Local Interaction and Adaptive Complementary Fusion for Spiking Transformer

Dongcheng Zhao; Sicheng Shen; Zhenyu Yang; Zhiyuan Li; Jinyan Yu; Yongjian Wang; Tiechui Yao; Wenli Zhang

cs.NEAug 11, 2026

Spiking Local Interaction and Adaptive Complementary Fusion for Spiking Transformer

Authors: Dongcheng Zhao, Sicheng Shen, Zhenyu Yang, Zhiyuan Li, Jinyan Yu, Yongjian Wang, Tiechui Yao, Wenli Zhang,

Organizations: Center for Excellence in Brain Science and Intelligence Technology, State Key Laboratory of Brain Cognition and Brain-inspired Intelligence Technology, Institute of Neuroscience, Chinese Academy of Sciences · China Electric Power Research Institute Co., Ltd. (CEPRI)

Abstract

Spiking Transformers model token interactions primarily through spiking self-attention (SSA). However, binary query and key representations map continuous similarities to sparse and discrete relation responses, which may suppress weak relations and limit the propagation of local spatial context. To address this limitation, we introduce Spiking Local Interaction (SLI) and Adaptive Complementary Fusion (ACF). SLI establishes an attention-independent pathway for direct information exchange among neighboring spiking tokens using lightweight depthwise--pointwise transformations. ACF integrates SSA and SLI through layer-specific, channel-wise coefficients that adaptively balance their contributions at different network depths. The proposed design preserves the original attention formulation and can be incorporated into different Spiking Transformer architectures with modest parameter overhead. Experiments on ImageNet-1K, CIFAR-10, CIFAR-100, CIFAR10-DVS, and ADE20K show consistent improvements across image classification, event-based recognition, and semantic segmentation. In particular, QKFormer with SLI and ACF achieves $84.37\%$ Top-1 accuracy on ImageNet-1K and $37.5\%$ mIoU on ADE20K, where the segmentation model is trained without ImageNet pretraining. Ablation studies and qualitative analyses further indicate that SSA and SLI capture complementary interaction patterns and that learnable fusion consistently outperforms fixed weighting.

Figures & tables

Figure 1: Architecture of the proposed spiking Transformer with Adaptive Complementary Fusion (ACF). Each block combines spiking self-attention (SSA) with a parallel Spiking Local Interaction (SLI) pathway. SSA constructs input-dependent token relations, while SLI aggregates local spatial context independently of QK co-activation. Their responses are adaptively calibrated by learnable channel-wise coefficients, and the fused representation is propagated to subsequent blocks.

Figure 2: Sparse and discrete relation responses in spiking attention. (a) Comparison of vanilla self-attention and spiking self-attention. (b) Attention-response non-zero ratios across eight blocks of Spikingformer and QKFormer.

Methods	Type	Architecture	Params (M)	Time Steps	Top-1 Acc. (%)
ViT ( Dosovitskiy et al. 2020 )	ANN	ViT-B/16	86.59	1	77.90
DeiT ( Touvron et al. 2021 )	ANN	DeiT-B	86.59	1	81.80
DeiT ( Touvron et al. 2021 )	ANN	DeiT-B †	86.59	1	83.10
Spikformer ( Zhou et al. 2022 )	SNN	Spikformer-8-512	29.68	4	73.38
Spikformer ( Zhou et al. 2022 )	SNN	Spikformer-8-768	66.34	4	74.81
SDT ( Yao et al. 2023 )	SNN	SDT-8-512	29.68	4	74.57

Table 1: ImageNet-1K classification results. ∗ denotes our reproduction under the original experimental settings, and † denotes an input resolution of

288\times 288

; all other models use

224\times 224

. HST denotes Hierarchical Spiking Transformer.

Method	CIFAR10			CIFAR100			CIFAR10-DVS
Method	Params (M)	$T$	Acc. (%)	Params (M)	$T$	Acc. (%)	Params (M)	$T$	Acc. (%)
$\alpha$ -SSA ( Xiao et al. 2025 )	9.32	4	95.97	9.32	4	80.22	2.57	16	81.64
SacSSA ( Wang et al. 2025 )	5.57	4	96.1	5.57	4	80.1	1.52	16	82.3
$A^{2}OS^{2}$ ( Guo et al. 2025 )	10.23	4	96.42	10.23	4	79.90	-	-	-
STAtten+ ( Lee et al. 2025 )	-	4	95.35	-	4	80.20	-	16	83.9
SEMM ( Lee et al. 2025 )	-	4	96.16	-	4	80.24	-	16	82.1

Table 2: Comparison on CIFAR-10, CIFAR-100, and CIFAR10-DVS. The Spikingformer and QKFormer baselines in the lower block are reproduced under a unified benchmark for direct comparison with their ACF-enhanced counterparts.

Figure 3: Qualitative ADE20K results of Spikingformer and QKFormer with and without SLI and ACF.

Figure 4: Component ablation on CIFAR-100 using Spikingformer-4-384 and QKFormer-4-384.

Backbone	Mean Top-1 (%) $\uparrow$		Std. $\downarrow$
Backbone	Fixed	Learnable	Fixed	Learnable
QKFormer	81.64	81.74	0.33	0.08
Spikingformer	80.62	80.77	0.17	0.06

Table 4: Mean and standard deviation of Top-1 accuracy across five fusion-coefficient settings on CIFAR-100.

Figure 5: Channel-wise distributions of the learned $\gamma_{\mathrm{ssa}}$ and $\gamma_{\mathrm{sli}}$ across blocks, backbones, and datasets.

Figure 6: Grad-CAM++ visualizations of SSA, SLI, and their ACF-fused responses on ImageNet-1K.

Stage	Spatial resolution	Layer specification		8-512	10-512	8-768
Spikingformer (columnar)
Spiking Tokenizer	$H\times W$	Downsampling 1	Conv $3\times 3$	$64$	—	$96$
	$\frac{H}{2}\times\frac{W}{2}$	Downsampling 2	MP + Conv $3\times 3$	$128$	—	$192$
	$\frac{H}{4}\times\frac{W}{4}$	Downsampling 3	MP + Conv $3\times 3$	$256$	—	$384$
	$\frac{H}{8}\times\frac{W}{8}$	Downsampling 4	MP + Conv $3\times 3$	$512$	—	$768$
	$\frac{H}{16}\times\frac{W}{16}$	Downsampling 5	MP + Conv $3\times 3$	$512$	—	$768$

Table 5: Architecture specifications of the ImageNet-scale Spikingformer and QKFormer backbones with SLI and ACF. In each Transformer block, SLI operates in parallel with SSA or TSSA, and the two pathways are fused by ACF before the spiking MLP. MP denotes

3\times 3

max-pooling with stride

2

, and each QKFormer downsampling module includes a

1\times 1

stride-

2

convolutional shortcut.

Stage	Spatial resolution	Layer specification		Spikingformer	QKFormer
1	$\frac{H}{2}\times\frac{W}{2}$	Downsampling	Conv $7\times 7$ , stride $2$ , dim $32$
		Blocks	Conv-based SNN block	$\times 1\rightarrow P_{1}$	$\times 1\rightarrow P_{1}$
2	$\frac{H}{4}\times\frac{W}{4}$	Downsampling	Conv $3\times 3$ , stride $2$ , dim $64$
		Blocks	Conv-based SNN block	$\times 1\rightarrow P_{2}$	$\times 1\rightarrow P_{2}$
3	$\frac{H}{8}\times\frac{W}{8}$	Downsampling	Conv $3\times 3$ , stride $2$ , dim $128$
		Blocks	Conv-based SNN block	$\times 2\rightarrow P_{3}$	$\times 2\rightarrow P_{3}$

Table 6: Architecture specifications of the ADE20K segmentation backbones with SLI and ACF. In each Transformer SNN block, SLI operates in parallel with SSA or TSSA, and their outputs are fused by ACF before the spiking MLP.

P_{1}

–

P_{4}

denote the multi-scale features passed to the FPN neck. The downsampling modules follow SDT-V2 and contain convolutional shortcuts. PW and DW denote pointwise and depthwise convolutions, respectively.

Configuration	CIFAR-10/100	CIFAR10-DVS	ImageNet-1K
Input size	$32^{2}$	$128^{2}$	$224^{2}$
Time steps $T$	$4$	$16$	$4$
Epochs	$400$	$96$	$300$ / $200$
Warmup epochs	$20$	$10$	$20$ / $5$
Batch size	$64$	$16$	$24$ per GPU
Learning rate	$10^{-3}$	$10^{-3}$	$5\times 10^{-4}$

Table 7: Training configurations for the classification experiments. Values separated by “/” correspond to Spikingformer and QKFormer, respectively. The reported epoch counts exclude the subsequent

10

-epoch cooldown phase.

Spiking Transformers

Spiking Transformers combine attention-based architectures with event-driven spiking neurons to process information using sparse, temporally coded computation.

See all 42 papers

Rethinking Attention Locality in Spiking Transformers
Aug 9, 2026Zeqi Zheng, Zizheng Zhu, Yuping Yan +3Spiking Transformers Spiking Neural Networks
1Zhejiang University · 2Westlake University · 3Peking University
Breaking Global Self-Attention Bottlenecks in Transformer-based Spiking Neural Networks with Local Structure-Aware Self-Attention
May 12, 2026Lingdong Li, Hangming Zhang, Qiang YuSpiking Transformers Efficient ViTs
School of Future Technology, Tianjin Key Laboratory of Cognitive Computing and Application, Tianjin University, Tianjin 300354, China · School of Artificial Intelligence, Tianjin Key Laboratory of Cognitive Computing and Application, College of Intelligence and Computing, Tianjin University, Tianjin 300354, China
SAFformer:Improving Spiking Transformer via Active Predictive Filtering
May 8, 2026Zequan Xie, Weiming Zeng, Yunhua Chen +3Spiking Transformers Efficient ViTs
School of Computer Science and Technology, Guangdong University of Technology, Guangzhou, China · Faculty of Science, Hong Kong Baptist University, China · School of Electronic Information, Wuhan University, China

Aug 9, 2026J/K move · Enter open · S save

Rethinking Attention Locality in Spiking Transformers

Zeqi Zheng, Zizheng Zhu, Yuping Yan, Wenxuan Pan+2

1Zhejiang University · 2Westlake University · 3Peking University

Spiking Transformers Spiking Neural Networks Local Attention Attention Mechanisms

Open paper AskPDF arXiv

Spiking Transformers provide a promising paradigm for efficient visual processing with spike-driven computation, yet their Softmax-free Spiking Self-Attention (SSA) struggles to establish spatially localized token interactions. Although existing locality-enhanced SSA methods improve accuracy, it remains unclear whether they consistently induce spatial locality across layers and different Spiking Transformer architectures. Through Mean Attention Distance (MAD) analysis, we reveal that computational locality does not necessarily translate into spatial locality and show that uniformly applying the same locality enhancement overlooks architecture-dependent deployment requirements. Motivated by these observations, we propose Spatially Contiguous Local Attention with Boundary Continuity Pathway (SCLA-BCP). SCLA computes attention within non-overlapping regions of spatially adjacent tokens, while BCP facilitates cross-boundary information exchange through a lightweight convolutional pathway. Furthermore, we develop a hierarchical locality deployment strategy to effectively apply SCLA-BCP across the two major Spiking Transformer architectures. Extensive experiments on seven static and neuromorphic datasets covering classification, detection, and segmentation demonstrate consistent improvements with limited parameter and energy overhead. Notably, our approach improves mAP@50 by up to 9.50% on COCO 2017 and mIoU by up to 3.42% on ADE20K. Visualizations, MAD analysis, and ablation studies further validate its effectiveness.

Abstract

Spiking Transformers

Explore similar work

Rethinking Attention Locality in Spiking Transformers

Breaking Global Self-Attention Bottlenecks in Transformer-based Spiking Neural Networks with Local Structure-Aware Self-Attention

SAFformer:Improving Spiking Transformer via Active Predictive Filtering