TP-Spikformer: Token Pruned Spiking Transformer

Wenjie Wei; Xiaolong Zhou; Malu Zhang; Ammar Belatreche; Qian Sun; Yimeng Shan; Dehao Zhang; Zijian Zhou

cs.CVFeb 28, 2026

TP-Spikformer: Token Pruned Spiking Transformer

Authors: Wenjie Wei, Xiaolong Zhou, Malu Zhang, Ammar Belatreche, Qian Sun, Yimeng Shan, Dehao Zhang, Zijian Zhou,

Organizations: University of Electronic Science and Technology of China · Shenzhen Loop Area Institute · Northumbria University · The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen)

Abstract

Spiking neural networks (SNNs) offer an energy-efficient alternative to traditional neural networks due to their event-driven computing paradigm. However, recent advancements in spiking transformers have focused on improving accuracy with large-scale architectures, which require significant computational resources and limit deployment on resource-constrained devices. In this paper, we propose a simple yet effective token pruning method for spiking transformers, termed TP-Spikformer, that reduces storage and computational overhead while maintaining competitive performance. Specifically, we first introduce a heuristic spatiotemporal information-retaining criterion that comprehensively evaluates tokens' importance, assigning higher scores to informative tokens for retention and lower scores to uninformative ones for pruning. Based on this criterion, we propose an information-retaining token pruning framework that employs a block-level early stopping strategy for uninformative tokens, instead of removing them outright. This also helps preserve more information during token pruning. We demonstrate the effectiveness, efficiency and scalability of TP-Spikformer through extensive experiments across diverse architectures, including Spikformer, QKFormer and Spike-driven Transformer V1 and V3, and a range of tasks such as image classification, object detection, semantic segmentation and event-based object tracking. Particularly, TP-Spikformer performs well in a training-free manner. These results reveal its potential as an efficient and practical solution for deploying SNNs in real-world applications with limited computational resources.

Figures & tables

Figure 1: Visualization of token pruning across time step and block with our method. Experiments are conducted on SDT-V1-8-512, and white areas are pruned tokens.

Figure 2: The overall workflow of the proposed TP-Spikformer, including the information-retention token pruning framework ( top ) and the spatiotemporal information-retaining criterion ( bottom ).

Panel of Figure 3 — Figure 3: Visualization of ground truth and ours, showing its efficacy on diverse downstream tasks.

Method	$N_{avg}$	$T$	Param (M)	Thr (imgs/s)	MIoU (%)
( Yao et al., )	$\times 1$	4	16.5	59.6	33.6
	$\times 1$	1	58.9	37.6	34.8
	$\times 1$	4	58.9	36.8	35.3
( Yao et al., 2025 )	$\times 1$	2	11.0	82.7	31.9
	$\times 1$	4	11.0	74.5	40.1
	$\times 1$	8	11.0	70.4	41.4

Table 3: Segmentation result on ADE20K.

Method	$N_{avg}$	$T$	Param (M)	Thr (imgs/s)	mAP @0.5(%)
( Kim et al., 2020 )	-	3500	10.2	-	25.7
( Li et al., 2022 )	-	512	17.1	-	45.3
( Zhang et al., 2023 )	-	4	11.3	-	28.5
( Su et al., 2023 )	-	4	26.9	-	50.1
( Yao et al., )	$\times 1$	1	34.9	32.6	44.0
( Yao et al., 2025 )	$\times 1$	8	38.7	29.8	58.8

Table 4: Object detection results on COCO 2017.

Methods	Time step	$N_{avg}$	Power (mJ)	FE108		FELT		VisEvent
Methods	Time step	$N_{avg}$	Power (mJ)	AUC(%)	PR(%)	AUC(%)	PR(%)	AUC(%)	PR(%)
STARK Yan et al. (2021)	1	$\times 1$	58.88	57.4	89.2	39.6	51.7	34.1	46.8
SimTrack Chen et al. (2022)	1	$\times 1$	93.84	56.7	88.3	36.8	47.0	34.6	47.6
OSTrack 256 Ye et al. (2022)	1	$\times 1$	98.90	54.6	87.1	35.9	45.5	32.7	46.4
ARTrack 256 Wei et al. (2023)	1	$\times 1$	174.8	56.6	88.5	39.5	49.4	33.0	43.8
SeqTrack-B 256 Chen et al. (2023)	1	$\times 1$	302.7	53.5	85.5	33.0	42.0	28.6	43.3

Table 5: TP-Spikformer vs. advanced trackers on three event-based object tracking benchmarks.

Figure 4: Zero-finetuning accuracy preservation (top) and efficiency gains (bottom) of TP-Spikformer.

Figure 5: Visualization of spatial and temporal token scores in the 8th block of SDT-V1-8-768.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Block	CIFAR-10		CIFAR-100		DVS-CIFAR10
Block	0.25	0.20	0.60	0.55	0.78	0.50
1	0.76	0.56	0.76	0.76	0.78	0.76
2	0.14	0.14	0.76	0.76	0.78	0.25
3	0.06	0.06	0.76	0.56	-	-
4	0.06	0.06	0.14	0.14	-	-

Appendix

Table 7: Token preserving ratio on small datasets.

Block	SDT-V1			QKFormer			SDT-V3
Block	0.74	0.65	0.51	0.72	0.65	0.53	0.78	0.65	0.56
1	1	1	0.73	0.90	0.81	0.64	1	0.81	0.64
2	1	0.73	0.73	0.90	0.81	0.64	0.90	0.81	0.64
3	0.73	0.73	0.51	0.72	0.64	0.64	0.90	0.64	0.64
4	0.73	0.73	0.51	0.72	0.64	0.49	0.90	0.64	0.64
5	0.73	0.51	0.51	0.72	0.64	0.49	0.64	0.64	0.49

Appendix

Table 8: Token preserving ratio in each transformer block on Imagenet-1K.

Hyper-parameter	SDT-V1	QKFormer	SDT-V3
$N_{\text{avg}}$	0.74, 0.65, 0.51	0.72, 0.65, 0.51	0.78, 0.65, 0.56
$k$ in IRToP	3	3	3
Time step	4	4	4
Warmup epoch	None	None	None
Epoch	50	50	50
Resolution	224 $\times$ 224	224 $\times$ 224	224 $\times$ 224

Appendix

Table 9: Experimental setups on Imagenet-1K.

Model	$N_{avg}$	Searching space per token retention ratio	Number of combinations	Evaluation time per combination	Total Time (4*NVIDIA 4090)
SDT-V3	0.78	[1,0.90,0.81,0.72,0.64,0.56,0.49]	65	26s	28min 11s
SDT-V3	0.65	[0.81,0.72,0.64,0.56,0.49,0.42,0.36]	89	22s	32min 38s
SDT-V3	0.56	[0.81,0.72,0.64,0.56,0.49,0.42,0.36]	166	16s	44min 16s

Appendix

Table 10: Search details for the SDT-V3.

$k$ value	SDT-V1			QKFormer			SDT-V3
$k$ value	0.74	0.65	0.51	0.72	0.65	0.53	0.78	0.65	0.56
3	75.38	75.48	73.78	83.79	81.34	64.38	75.95	67.03	60.19
5	75.37	75.48	73.77	83.65	81.23	64.18	75.77	66.59	59.71
7	75.34	75.48	73.77	83.66	81.20	64.32	75.59	66.47	59.40

Appendix

Table 12: Performance of TP-Spifkormer with different

k

on ImageNet-1K without finetuning.

Model	Ratio	Final ratio $\alpha$	Fine-tuning accuracy under adaptive $\alpha$	Fine-tuning accuracy under fixed $\alpha$ =0.5
SDT-V1	0.51	0.3	74.23%	74.79%

Appendix

Table 13: Analysis of adaptive weighting of spatial and temporal scorers in IRToP.

Figure 6: Detailed visualization of spatial and temporal token scores of SDT-V1-8-768.

Figure 7: TP-Spikformer’s zero-finetuning accuracy preservation on segmentation and detection.

Model	Params (M)	Token preservation ratio	Total training time	GPU Memory (batch size=1280)	Acc.
SDT-V3	5.1	1	57h14min	77.49 GB	73.9%
TP-Spikformer	5.1	0.65	50h 49min	66.90 GB	73.6%

Appendix

Table 14: Effect of TP-Spikformer in the training process.

Model	SST-2	MRPC	RTE	MNLI	QNLI	QQP	CoLA	STS-B	Avg.
SpikeLM	87.0	85.7	69.0	77.1	85.3	83.9	38.8	84.9	76.5
TP-Spikformer	87.9	84.7	68.2	76.0	84.6	84.2	37.0	84.9	75.9

Appendix

Table 15: Validation of TP-Spikformer on the GLUE benchmark.

Model	Preserving Ratio	Bits	GPU Memory (GB)	Accuracy (%)
Q-SDT Qiu et al. (2025)	1	4-1	19.27	97.8%
Q-SDT+TP-Spikformer	0.54	4-1	15.98	96.9%

Appendix

Table 16: Potential of TP-Spikformer with quantization.

Spiking Transformers

Spiking Transformers combine attention-based architectures with event-driven spiking neurons to process information using sparse, temporally coded computation.

See all 42 papers

SAFformer:Improving Spiking Transformer via Active Predictive Filtering
May 8, 2026Zequan Xie, Weiming Zeng, Yunhua Chen +3Spiking Transformers Efficient ViTs
School of Computer Science and Technology, Guangdong University of Technology, Guangzhou, China · Faculty of Science, Hong Kong Baptist University, China · School of Electronic Information, Wuhan University, China
Breaking Global Self-Attention Bottlenecks in Transformer-based Spiking Neural Networks with Local Structure-Aware Self-Attention
May 12, 2026Lingdong Li, Hangming Zhang, Qiang YuSpiking Transformers Efficient ViTs
School of Future Technology, Tianjin Key Laboratory of Cognitive Computing and Application, Tianjin University, Tianjin 300354, China · School of Artificial Intelligence, Tianjin Key Laboratory of Cognitive Computing and Application, College of Intelligence and Computing, Tianjin University, Tianjin 300354, China
Uncertainty-Aware Token Importance Estimation in Spiking Transformers
May 10, 2026Wenxuan Liu, Zecheng Hao, Tong Bu +2Spiking Transformers Uncertainty Quantification
School of Computer Science, Peking University · School of Computer Science, Peking University. · Institute for Artificial Intelligence, Peking University +2

May 8, 2026J/K move · Enter open · S save

SAFformer:Improving Spiking Transformer via Active Predictive Filtering

Zequan Xie, Weiming Zeng, Yunhua Chen, Sichang Ling+2

School of Computer Science and Technology, Guangdong University of Technology, Guangzhou, China · Faculty of Science, Hong Kong Baptist University, China · School of Electronic Information, Wuhan University, China

Spiking Transformers Efficient ViTs Spiking Neural Networks Image Classification

Open paper AskPDF arXiv

Spiking Neural Networks (SNNs) offer notable advantages in biological plausibility and energy efficiency, making them promising candidates for building low-power Transformers. However, existing Spiking Transformers largely adhere to a passive reactive paradigm, which struggles to focus on task-relevant information and incurs substantial computational overhead when processing redundant visual data. To overcome this fundamental yet underexplored limitation, we propose SAFformer, a novel Spiking Transformer architecture based on an active predictive filtering paradigm. Inspired by the brain's predictive coding mechanism, SAFformer actively suppresses predictable signals and focuses on salient visual features. Extensive experiments show that SAFformer establishes new state-of-the-art performance on CIFAR-10/100 and CIFAR10-DVS. Remarkably, on ImageNet-1K, it achieves 80.44% Top-1 accuracy with only 26.58M parameters and an energy consumption of 5.88 mJ, demonstrating an exceptional balance between accuracy and efficiency.

Abstract

Spiking Transformers

Explore similar work

SAFformer:Improving Spiking Transformer via Active Predictive Filtering

Breaking Global Self-Attention Bottlenecks in Transformer-based Spiking Neural Networks with Local Structure-Aware Self-Attention

Uncertainty-Aware Token Importance Estimation in Spiking Transformers