Contrastive Attention Mitigates Spectral Bias in Spiking Transformers

Xiaoli Liu; Malu Zhang; Yang Yang

cs.AIOct 1, 2026

Contrastive Attention Mitigates Spectral Bias in Spiking Transformers

Authors: Xiaoli Liu, Malu Zhang, Yang Yang

Organizations: School of Computer Science and Engineering, University of Electronic Science and Technology of China

Abstract

Spiking Transformers merge the energy-efficiency of spiking neural networks (SNNs) with the representational power of self-attention, creating a promising architecture for high-performance, energy-efficient computation. However, a performance gap persists versus its counterparts in artificial neural networks (ANNs). Unlike prior works attributing this to binary activations, we reveal that both spiking neurons and spiking self-attention (SSA) act as low-pass filters through multiscale spectral analysis. This characteristic leads to the dissipation of high-frequency components. To address this issue, we propose the Spiking Contrastive Attention (SCA) paradigm, which draw inspiration from the edge-detection and differential sensing properties of biological visual system. By extracting contrast prototypes via global contrastive aggregation and applying local differential refinement, SCA effectively enhances high-frequency information. Extensive experiments show that SCA is a general module that consistently boosts Spiking Transformers across image classification, semantic segmentation, and event-based tracking. Furthermore, it achieves lower complexity, offering superior efficiency over original SSA. These results establish its potential as a fundamental building block for energy-efficient Spiking Transformers.

Figures & tables

Figure 1 : The description of spectral bias.

Figure 2 : (a) Vanilla Self-Attention (VSA). (b)-(d) Softmax-free Spiking Self-Attention (SSA, SDSA and Meta-SDSA). (e) Relative Logarithmic Amplitude of Fourier Transform Features.

Method	Type	Unified Form
SSA [ 67 ]	Attn	$\mathcal{SN}(\mathrm{BN}(\mathrm{Lin}(\mathcal{SN}(\mathbf{QK}^{\top}\mathbf{V}*s))))$
SSA [ 67 ]	$\mathcal{A}$	$\mathbf{QK}^{\top}$
SDSA [ 50 ]	Attn	$\mathcal{SN}(\sum_{c}(\mathbf{Q}_{S}\otimes\mathbf{K}_{S}))\otimes\mathbf{V}_{S}$
SDSA [ 50 ]	$\mathcal{A}$	$\mathrm{diag}(\mathcal{M}(\mathbf{Q},\mathbf{K}))$
Meta-SDSA [ 49 ]	Attn	$\mathcal{SN}_{s}(\mathbf{Q}_{S}(\mathbf{K}_{S}^{\top}\mathbf{V}_{S}))$
Meta-SDSA [ 49 ]	$\mathcal{A}$	$\mathbf{QK}^{\top}$

Table 1 : Unified forms of SSA from a GSP perspective.

Figure 3 : (a) Edge Detection and Biological Visual Cortex. (b) Spiking Contrastive Attention for Spiking Transformer.

Method	Type	Architecture	Time Step	Param (M)	Energy (mJ)	Top-1 Acc (%)
ViT [ 37 ]	ANN	ViT-L/16	1	304.3	80.96	79.70
PVT [ 39 ]	ANN	PVT-Large	1	61.4	45.08	81.70
Spikformer [ 67 ]	SNN	Spikformer-8-384	4	16.8	7.73	70.24
Spikformer [ 67 ]	SNN	Spikformer-8-512	4	29.7	11.57	73.38
Spikingformer [ 62 ]	SNN	Spikingformer-8-512	4	29.7	4.69	74.79
Spikingformer [ 62 ]	SNN	Spikingformer-8-768	4	66.4	13.68	75.85

Table 2 : Detailed comparison with other methods on ImageNet-1K.

Figure 4 : (a) Attention heatmap with SDT-V1+SCA on ImageNet. (b) Fourier spectrum of SDT-V1, SDT-V3, and QKFormer on ImageNet, along with their corresponding SCA-enhanced models.

Architecture	Param (M)	Step	MIoU (%)
ResNet-18 [ 55 ]	15.5	1	32.9
PVT-Tiny [ 39 ]	17.0	1	35.7
PVT-Small [ 39 ]	28.2	1	39.8
DeepLab-V3 [ 56 ]	68.1	1	42.7
SDT-V2 [ 49 ]	16.5	1	32.3
SDT-V2 [ 49 ]	16.5	4	33.6

Table 3 : Performance of segmentation.

Methods	Param. (M)	$T$	FELT [ 40 ]		FE108 [ 59 ]		VisEvent [ 41 ]
Methods	Param. (M)	$T$	AUC(%)	PR(%)	AUC(%)	PR(%)	AUC(%)	PR(%)
STARK [ 48 ]	28.23	1	39.6	51.7	57.4	89.2	34.1	46.8
ARTrack [ 45 ]	202.56	1	39.5	49.4	56.6	88.5	33.0	43.8
OSTrack 256 [ 52 ]	92.52	1	35.9	45.5	54.6	87.1	32.7	46.4
HIPTrack [ 2 ]	120.41	1	38.2	48.9	50.8	81.0	32.1	45.2
SNNTrack [ 60 ]	31.40	5	-	-	-	-	35.4	50.4

Table 4 : Performance on event-based tracking datasets.

Figure 5 : (a) Semantic segmentation results for ADE20K [ 61 ] (SDT-V3-L+SCA). (b) Visualization of tracking results comparing our approach with other SOTA methods on the VisEvent [ 41 ] dataset.

Figure 7 : Rank restoration. Refer to A.5.3

Hyper-parameter	SDT-V1	QKformer	SDT-V3
Timestep	4	4	4
Epochs	300	200	200
Resolution	224	224	224
Batch size	48	100	600
Optimizer	AdamW	AdamW	LAMB
Base Learning rate	1.5e-5	6e-4	6e-4

Table 6 : Comparison of Hyperparameters for Different Model Architectures

Figure 8 : Attention heatmap for SCA (Ours) with SDT-V1 architecture on ImageNet-1K.

Figure 9 : Visualization of semantic segmentation results on the ADE20K dataset.

Test Set Condition	QKFormer Baseline	QKFormer + SCA	$\mathbf{\Delta}$ (Gain)
Standard Image (Full spectrum)	$81.15\pm 0.18$	$82.34\pm 0.12$	$+1.19\%$
Low-Pass Filter (High-freq removed)	$77.12\pm 0.09$	$77.18\pm 0.13$	$+0.06\%$
High-Pass Filter (High-freq dominant)	$68.45\pm 0.21$	$\mathbf{74.80\pm 0.16}$	$\mathbf{+6.35\%}$

Table 7 : Performance comparison under different test-set conditions

Architecture	Params (M)	$T$	CIFAR-10 Acc. (%)	CIFAR-100 Acc. (%)	CIFAR-10 DVS Acc. (%)
SEW-ResNet [ 7 ]	1.19	4	-	-	64.80
SEW-ResNet [ 7 ]	1.19	8	-	-	70.20
SEW-ResNet (Ours)	1.19	4	-	-	67.79 ( $\uparrow$ 2.99)
SEW-ResNet (Ours)	1.19	8	-	-	72.66 ( $\uparrow$ 2.46)
MS-ResNet-18 [ 14 ]	11.22	4	94.40	75.06	-
MS-ResNet-18 (Ours)	11.22	4	96.37 ( $\uparrow$ 1.97)	77.15 ( $\uparrow$ 2.09)	-

Table 8 : Generality across Convolutional Architectures

Figure 10 : Throughput and latency comparison in baselines and our method.

Explore similar work

SAFformer:Improving Spiking Transformer via Active Predictive Filtering
May 8, 2026Zequan Xie, Weiming Zeng, Yunhua Chen +3Spiking Neural Networks Transformer Architectures
School of Computer Science and Technology, Guangdong University of Technology, Guangzhou, China · Faculty of Science, Hong Kong Baptist University, China · School of Electronic Information, Wuhan University, China
Breaking Global Self-Attention Bottlenecks in Transformer-based Spiking Neural Networks with Local Structure-Aware Self-Attention
May 12, 2026Lingdong Li, Hangming Zhang, Qiang YuSpiking Neural Networks Transformer Attention
School of Future Technology, Tianjin Key Laboratory of Cognitive Computing and Application, Tianjin University, Tianjin 300354, China · School of Artificial Intelligence, Tianjin Key Laboratory of Cognitive Computing and Application, College of Intelligence and Computing, Tianjin University, Tianjin 300354, China
Rethinking Attention Locality in Spiking Transformers
Aug 9, 2026Zeqi Zheng, Zizheng Zhu, Yuping Yan +3Transformer Attention Time-To-First-Spike
1Zhejiang University · 2Westlake University · 3Peking University

May 8, 2026, cs.CVJ/K move · Enter open · S save

SAFformer:Improving Spiking Transformer via Active Predictive Filtering

Zequan Xie, Weiming Zeng, Yunhua Chen, Sichang Ling+2

School of Computer Science and Technology, Guangdong University of Technology, Guangzhou, China · Faculty of Science, Hong Kong Baptist University, China · School of Electronic Information, Wuhan University, China

Spiking Neural Networks Transformer Architectures Bayesian Filtering

Open paper AskPDF arXiv

Spiking Neural Networks (SNNs) offer notable advantages in biological plausibility and energy efficiency, making them promising candidates for building low-power Transformers. However, existing Spiking Transformers largely adhere to a passive reactive paradigm, which struggles to focus on task-relevant information and incurs substantial computational overhead when processing redundant visual data. To overcome this fundamental yet underexplored limitation, we propose SAFformer, a novel Spiking Transformer architecture based on an active predictive filtering paradigm. Inspired by the brain's predictive coding mechanism, SAFformer actively suppresses predictable signals and focuses on salient visual features. Extensive experiments show that SAFformer establishes new state-of-the-art performance on CIFAR-10/100 and CIFAR10-DVS. Remarkably, on ImageNet-1K, it achieves 80.44% Top-1 accuracy with only 26.58M parameters and an energy consumption of 5.88 mJ, demonstrating an exceptional balance between accuracy and efficiency.

Abstract

Explore similar work

SAFformer:Improving Spiking Transformer via Active Predictive Filtering

Breaking Global Self-Attention Bottlenecks in Transformer-based Spiking Neural Networks with Local Structure-Aware Self-Attention

Rethinking Attention Locality in Spiking Transformers