Spiking Transformers

Follow this topic Ask about this topic

Momentum

2 papers in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 23

All topics

Contrastive Attention Mitigates Spectral Bias in Spiking Transformers
Oct 1, 2026Xiaoli Liu, Malu Zhang, Yang YangSpiking Transformers Self-Attention
School of Computer Science and Engineering, University of Electronic Science and Technology of China
Rethinking Pairwise Token Interaction in Spiking Transformers
Sep 22, 2026Sicheng Shen, Dongcheng Zhao, Zhiyuan Li +5Spiking Transformers Transformer Attention
Institute of Automatoin, CAS · Center for Excellence in Brain Science and Intelligence Technology, State Key Laboratory of Brain Cognition and Brain-inspired Intelligence Technology,Institute of Neuroscience, CAS · Zhongguancun Academy
Lapis: Laplacian Spiking Attention via First-Spike Timing and Membrane Leakage
Aug 12, 2026Kaiwen Tang, Jiaqi Zheng, Zixuan Zhu +3Spiking Transformers Efficient Attention
School of Computing, National University of Singapore · 2Sea AI Lab · 4Shanghai Advanced Research Institute, Chinese Academy of Sciences +2
Rethinking Attention Locality in Spiking Transformers
Aug 9, 2026Zeqi Zheng, Zizheng Zhu, Yuping Yan +3Spiking Transformers Spiking Neural Networks
1Zhejiang University · 2Westlake University · 3Peking University
Spike-HTR: Spiking Neural Transformer for Handwritten Text Recognition
Aug 3, 2026Xiubo Liang, Jinxing Han, Yuke Li +3Efficient Neural Network Inference Spiking Transformers
School of Software Technology, Zhejiang University, Ningbo, China
SMM Transformer: Leveraging Spiking Neural Networks for Multimodal Tasks
Aug 3, 2026Xiubo Liang, Jinxing Han, Yuke Li +3Spiking Transformers Spiking Neural Networks
School of Software Technology, Zhejiang University, Ningbo, China · NetEase Yidun AI Lab, Hangzhou, China
The Sparsity Ceiling: Where Spiking Networks Can and Cannot Trade Activity for Energy
Jul 29, 2026Zeyu WangEfficient Neural Network Inference Spiking Transformers
Georgia Institute of Technology
Bio-SFT: Asymmetric Cortical Guidance and Retinal Adaptation for Robust HDR Reconstruction
Jul 20, 2026Tingyu Cheng, Ting Zhang, Chongyi Li +2Spiking Transformers Image Enhancement
Fujian Key Laboratory of Intelligent Processing and Wireless Transmission of Media Information, College of Physics and Information Engineering, Fuzhou University, Fujian 350108, China · School of Computer Science, Nankai University, Tianjin 300071, China · School of Electrical and Information Engineering, Tianjin University, Tianjin 300072, China
SpikeDS: Dual Sparsity Spikformer for Perineural Invasion Prediction in 3D MRI
Jul 13, 2026Induk Um, Youngung Han, Kyeonghun Kim +14Spiking Transformers 3D Medical Imaging
Chung-Ang University, Seoul, Republic of Korea · Seoul National University, Seoul, Republic of Korea · OUTTA, Seoul, Republic of Korea +5
SpikeLogBERT: Energy-Efficient Log Parsing Using Spiking Transformer Networks
Jun 30, 2026Thuan Bui, Duong Do, Tung Vu +2Efficient Neural Network Inference Spiking Transformers
Swinburne Vietnam, FPT University Hanoi, Vietnam · Posts and Telecommunications Institute of Technology Hanoi, Vietnam · Academy of Cryptography Techniques Hanoi, Vietnam +1
End-to-End Radar and Communication Modulation Recognition with Neuromorphic Computing
Jun 23, 2026Xiaohu Li, Chongxiao Qu, Caiyong Lin +2Efficient Neural Network Inference Spiking Transformers
China Nanhu Academy of Electronics and Information Technology, Jiaxing, China · University of Electronic Science and Technology of China, Chengdu, China
VL2Spike: Spike-driven Distillation from VLMs for Low-Power Visual Perception in Embodied AI
Jun 14, 2026Zinan Liu, Eric Zheng, Soumyaratna Debnath +3Cross-Modal Knowledge Distillation Spiking Transformers
School of EEE, Nanyang Technological University (NTU), Singapore · Department of Computer Science, University of Toronto and Advanced Micro Devices, Inc., Canada · State Key Laboratory of Extreme Photonics and Instrumentation, Zhejiang University, Hangzhou 310027, China +1
AQ4SViT: An Automated Quantization Framework with Search Gating Policy for Compressing Spiking Vision Transformers
Jun 14, 2026Rachmad Vidya Wicaksana Putra, Saad Iftikhar, Muhammad ShafiqueSpiking Transformers Neural Network Quantization
eBRAIN Lab, Division of Engineering, New York University (NYU) Abu Dhabi, United Arab Emirates (UAE)
Otters++: A Time-to-first-spike Based Energy Efficient Optical Spiking Transformer
Jun 11, 2026Zhanglu Yan, Jiayi Mao, Kaiwen Tang +6Efficient Transformer Inference Spiking Transformers
Department of Computer Science, National University of Singapore · School of Engineering, Westlake University · College of Computer Science and Technology, Zhejiang University +2
SpikeDecoder: Realizing the GPT Architecture with Spiking Neural Networks
Jun 10, 2026Claas Beger, Florian Walter, Alois KnollEfficient Neural Network Inference Spiking Transformers
Chair of Robotics, Artificial Intelligence and Real-time Systems Technical University of Munich, Germany
SpikingMoE: SDPrompt-Guided Dynamic Expert Fusion in Spiking Neural Networks
May 22, 2026Yukai Yang, Chenxi Qin, Jungang Li +3Spiking Transformers Dynamic Neural Networks
Tianjin University, Medical School, Tianjin 300072, China · Southern University of Science and Technology, Department of Computer Science and Engineering, Shenzhen, Guangdong 518055, China · Chinese Academy of Medical Sciences, Institute of Biomedical Engineering, Tianjin 300192, China
Plug-and-Play Spiking Operators: Breaking the Nonlinearity Bottleneck in Spiking Transformers
May 19, 2026Xinzhe Yuan, Xiang Peng, Bin Gu +1Spiking Transformers Spiking Neural Networks
IASM, Harbin Institute of Technology · IASM, Harbin Institute of Technology, China · School of Artificial Intelligence, Jilin University +1
Breaking Global Self-Attention Bottlenecks in Transformer-based Spiking Neural Networks with Local Structure-Aware Self-Attention
May 12, 2026Lingdong Li, Hangming Zhang, Qiang YuSpiking Transformers Efficient ViTs
School of Future Technology, Tianjin Key Laboratory of Cognitive Computing and Application, Tianjin University, Tianjin 300354, China · School of Artificial Intelligence, Tianjin Key Laboratory of Cognitive Computing and Application, College of Intelligence and Computing, Tianjin University, Tianjin 300354, China
Uncertainty-Aware Token Importance Estimation in Spiking Transformers
May 10, 2026Wenxuan Liu, Zecheng Hao, Tong Bu +2Spiking Transformers Uncertainty Quantification
School of Computer Science, Peking University · School of Computer Science, Peking University. · Institute for Artificial Intelligence, Peking University +2
SAFformer:Improving Spiking Transformer via Active Predictive Filtering
May 8, 2026Zequan Xie, Weiming Zeng, Yunhua Chen +3Spiking Transformers Efficient ViTs
School of Computer Science and Technology, Guangdong University of Technology, Guangzhou, China · Faculty of Science, Hong Kong Baptist University, China · School of Electronic Information, Wuhan University, China
Elastic Spiking Transformers for Efficient Gesture Understanding
May 4, 2026Alberto Ancilotto, Gianluca Amprimo, Stefano Di Carlo +1Efficient Neural Network Inference Spiking Transformers
Fondazione Bruno Kessler · Politecnico di Torino
BSViT: A Burst Spiking Vision Transformer for Expressive and Efficient Visual Representation Learning
Apr 25, 2026Hongxiang Peng, Dewei Bai, Hong QuSpiking Transformers Efficient ViTs
School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu 610054, China
Closing the Theory-Practice Gap in Spiking Transformers via Effective Dimension
Apr 17, 2026Dongxin Guo, Jikun Wu, Siu Ming YiuSpiking Transformers Transformer Attention
*The University of Hong Kong, Hong Kong, China · †Brain Investing Limited, Hong Kong, China

Oct 1, 2026, cs.AIJ/K move · Enter open · S save

Contrastive Attention Mitigates Spectral Bias in Spiking Transformers

Xiaoli Liu, Malu Zhang, Yang Yang

School of Computer Science and Engineering, University of Electronic Science and Technology of China

Spiking Transformers Self-Attention Spectral Bias Spiking Neural Networks

Open paper AskPDF arXiv

Spiking Transformers merge the energy-efficiency of spiking neural networks (SNNs) with the representational power of self-attention, creating a promising architecture for high-performance, energy-efficient computation. However, a performance gap persists versus its counterparts in artificial neural networks (ANNs). Unlike prior works attributing this to binary activations, we reveal that both spiking neurons and spiking self-attention (SSA) act as low-pass filters through multiscale spectral analysis. This characteristic leads to the dissipation of high-frequency components. To address this issue, we propose the Spiking Contrastive Attention (SCA) paradigm, which draw inspiration from the edge-detection and differential sensing properties of biological visual system. By extracting contrast prototypes via global contrastive aggregation and applying local differential refinement, SCA effectively enhances high-frequency information. Extensive experiments show that SCA is a general module that consistently boosts Spiking Transformers across image classification, semantic segmentation, and event-based tracking. Furthermore, it achieves lower complexity, offering superior efficiency over original SSA. These results establish its potential as a fundamental building block for energy-efficient Spiking Transformers.

Momentum

Related topics

Latest papers 23