Spiking Transformers

Follow this topic Ask about this topic

Spiking Transformers adapt attention-based models to spiking neural networks, where neurons communicate through discrete events over time, with the goal of reducing computation and energy use. Common approaches use leaky integrate-and-fire neurons trained with surrogate gradients, time-to-first-spike or binary temporal encodings, and spike-friendly attention; local attention, token pruning, and distillation further reduce cost or improve representations. Work spans image and event-based vision, language modeling, and robotics, with evaluations on benchmarks such as ImageNet, GLUE, and neuromorphic datasets, as well as emerging tests on specialized hardware. Many methods report competitive accuracy and lower estimated energy, but actual savings depend on task and hardware, and language-modeling quality and energy estimates remain important points of tension.

Latest papers

42 papers6 papers in the last four weeks, up 100% on the four weeks before. 0.0% of all new papers.

SAFformer:Improving Spiking Transformer via Active Predictive Filtering
May 8, 2026Zequan Xie, Weiming Zeng, Yunhua Chen +3Spiking Transformers Efficient ViTs
School of Computer Science and Technology, Guangdong University of Technology, Guangzhou, China · Faculty of Science, Hong Kong Baptist University, China · School of Electronic Information, Wuhan University, China
Elastic Spiking Transformers for Efficient Gesture Understanding
May 4, 2026Alberto Ancilotto, Gianluca Amprimo, Stefano Di Carlo +1Efficient Neural Network Inference Spiking Transformers
Fondazione Bruno Kessler · Politecnico di Torino
BSViT: A Burst Spiking Vision Transformer for Expressive and Efficient Visual Representation Learning
Apr 25, 2026Hongxiang Peng, Dewei Bai, Hong QuSpiking Transformers Efficient ViTs
School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu 610054, China
Closing the Theory-Practice Gap in Spiking Transformers via Effective Dimension
Apr 17, 2026Dongxin Guo, Jikun Wu, Siu Ming YiuSpiking Transformers Transformer Attention
*The University of Hong Kong, Hong Kong, China · †Brain Investing Limited, Hong Kong, China
BiSpikCLM: A Spiking Language Model integrating Softmax-Free Spiking Attention and Spike-Aware Alignment Distillation
Apr 14, 2026Sihang Guo, Chenlin Zhou, Jiaqi Wang +3Spiking Language Models Spiking Neural Networks
Harbin Institute of Technology, Shenzhen, China · School of Electronic and Computer Engineering, Shenzhen Graduate School, Peking University, Shenzhen, China · Peng Cheng Laboratory, Shenzhen, China
Winner-Take-All Spiking Transformer for Language Modeling
Apr 13, 2026Chenlin Zhou, Sihang Guo, Jiaqi Wang +6Spiking Transformers Spiking Language Models
School of Electronic and Computer Engineering, Shenzhen Graduate School, Peking University · Pengcheng Laboratory · Harbin Institute of Technology +1
Spike-driven Large Language Model
Apr 11, 2026Han Xu, Xuerui Qiu, Baiyu Chen +7Spiking Language Models Spiking Neural Networks
Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China · Brain-Inspired Models Group, Beijing Academy of Artificial Intelligence, Beijing 100862, China +7
Ge $^\text{2}$ mS-T: Multi-Dimensional Grouping for Ultra-High Energy Efficiency in Spiking Transformer
Apr 10, 2026Zecheng Hao, Shenghao Xie, Kang Chen +3Spiking Transformers Spiking Neural Networks
Peking University, School of Computer Science, State Key Laboratory for Multimedia Information Processing, Beijing, China · Peking University, School of Computer Science, Beijing Key Laboratory of Brain-inspired Spiking Large Models, Beijing, China
S3T-Former: A Purely Spike-Driven State-Space Topology Transformer for Skeleton Action Recognition
Mar 18, 2026Naichuan Zheng, Hailun Xia, Zepeng Sun +2Skeleton-Based Action Recognition Spiking Transformers
School of Information and Communication Engineering, Beijing University of Posts and Telecommunications, Beijing, China · International School, Beijing University of Posts and Telecommunications, Beijing, China · School of Economics and Management, Beijing University of Posts and Telecommunications, Beijing, China
CATFormer: When Continual Learning Meets Spiking Transformers With Dynamic Thresholds
Mar 16, 2026Vaishnavi Nagabhushana, Kartikay Agrawal, Ayon BorthakurClass-Incremental Learning Spiking Neural Networks
SustainAI Lab, MFSDS&AI, IIT Guwahati
Neural Dynamics Self-Attention for Spiking Transformers
Mar 9, 2026Dehao Zhang, Fukai Guo, Shuai Wang +6Spiking Transformers Spiking Neural Networks
University of Electronic Science and Technology of China · Shenzhen Loop Area Institute · The Chinese University of Hong Kong (Shenzhen)
TP-Spikformer: Token Pruned Spiking Transformer
Feb 28, 2026Wenjie Wei, Xiaolong Zhou, Malu Zhang +8Spiking Transformers Token Pruning
University of Electronic Science and Technology of China · Shenzhen Loop Area Institute · Northumbria University +1

May 8, 2026J/K move · Enter open · S save

SAFformer:Improving Spiking Transformer via Active Predictive Filtering

Zequan Xie, Weiming Zeng, Yunhua Chen, Sichang Ling+2

School of Computer Science and Technology, Guangdong University of Technology, Guangzhou, China · Faculty of Science, Hong Kong Baptist University, China · School of Electronic Information, Wuhan University, China

Spiking Transformers Efficient ViTs Spiking Neural Networks Image Classification

Open paper AskPDF arXiv

Spiking Neural Networks (SNNs) offer notable advantages in biological plausibility and energy efficiency, making them promising candidates for building low-power Transformers. However, existing Spiking Transformers largely adhere to a passive reactive paradigm, which struggles to focus on task-relevant information and incurs substantial computational overhead when processing redundant visual data. To overcome this fundamental yet underexplored limitation, we propose SAFformer, a novel Spiking Transformer architecture based on an active predictive filtering paradigm. Inspired by the brain's predictive coding mechanism, SAFformer actively suppresses predictable signals and focuses on salient visual features. Extensive experiments show that SAFformer establishes new state-of-the-art performance on CIFAR-10/100 and CIFAR10-DVS. Remarkably, on ImageNet-1K, it achieves 80.44% Top-1 accuracy with only 26.58M parameters and an energy consumption of 5.88 mJ, demonstrating an exceptional balance between accuracy and efficiency.