Spiking Transformers

Ask about this topic

Overview

Spiking Transformers adapt attention-based models to spiking neural networks, where neurons communicate through discrete events over time, with the goal of reducing computation and energy use. Common approaches use leaky integrate-and-fire neurons trained with surrogate gradients, time-to-first-spike or binary temporal encodings, and spike-friendly attention; local attention, token pruning, and distillation further reduce cost or improve representations. Work spans image and event-based vision, language modeling, and robotics, with evaluations on benchmarks such as ImageNet, GLUE, and neuromorphic datasets, as well as emerging tests on specialized hardware. Many methods report competitive accuracy and lower estimated energy, but actual savings depend on task and hardware, and language-modeling quality and energy estimates remain important points of tension.

Latest papers

CardsList

42 papers6 papers in the last four weeks, up 100% on the four weeks before. 0.0% of all new papers.

  1. Softmax-Free Attention in Spiking Transformers as the Zero-Temperature Limit of Two-Level Systems

    Oct 11, 2026Roberto C. Sotero, Pouya Taghipour, Jose M. Sanchez-BornotSpiking TransformersAttention Mechanisms

    Department of Radiology, University of Calgary, Calgary, AB, Canada · Department of Biomedical Engineering, University of Calgary, Calgary, AB, Canada · Hotchkiss Brain Institute, University of Calgary, Calgary, AB, Canada +1

  2. SDPAD: A Fully Spike-Driven Pipeline for End-to-End Autonomous Driving

    Oct 8, 2026Chengjun Zhang, Yuhao Zhang, Jie Yang +1Autonomous Driving PlanningEnd-to-End Autonomous Driving

    Zhejiang Key Laboratory of 3D Micro/Nano Fabrication and Characterization, Westlake Institute for Optoelectronics, Fuyang, Hangzhou, China · Integrated-On-Chips Brain-Computer Interfaces Zhejiang Engineering Research Center, Hangzhou, Zhejiang, China · CenBRAIN Neurotech, School of Engineering, Westlake University, Hangzhou, Zhejiang, China

  3. Rethinking the Tradeoff Between Temporal Encoding and Nonlinear Computation in Spiking Language Models

    Oct 7, 2026Hanfei Liu, Shuchang Feng, Yanxia Chen +2Spiking Language ModelsLanguage Modeling

    Northeastern University, 110819, Shenyang, China · Hebei Key Laboratory of Marine Perception Network and Data Processing, 066004, Qinhuangdao, China

  4. Parallel Time-Aligned Spiking Self-Attention for Consistent Integer-Valued Training and Spike-Driven Inference

    Oct 2, 2026Peng Xue, Wei Fang, Kaiwei Che +4Spiking Neural NetworksSpiking Transformers

    Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, China · Pengcheng Laboratory, China · University of Chinese Academy of Sciences, China +2

  5. Contrastive Attention Mitigates Spectral Bias in Spiking Transformers

    Oct 1, 2026Xiaoli Liu, Malu Zhang, Yang YangSpiking TransformersSelf-Attention

    School of Computer Science and Engineering, University of Electronic Science and Technology of China

  6. Spike-driven Vision-Language-Action Model

    Sep 30, 2026Shuai Wang, Malu Zhang, Mingquan Liu +6Efficient VLA ModelsVision-Language-Action Models

    University of Electronic Science and Technology of China.

  7. MorphAtt: A Neuromorphic Accelerator for Efficient Multi-Head Attention Processing in Spiking Vision Transformers

    Sep 27, 2026Rachmad Vidya Wicaksana Putra, Amirhesam Jafari Rad, Muhammad ShafiqueAI Accelerator InferenceSpiking Transformers

    eBRAIN Lab, New York University (NYU) Abu Dhabi, Abu Dhabi, United Arab Emirates · University of Tehran, Tehran, Iran

  8. Rethinking Pairwise Token Interaction in Spiking Transformers

    Sep 22, 2026Sicheng Shen, Dongcheng Zhao, Zhiyuan Li +5Spiking TransformersTransformer Attention

    Institute of Automatoin, CAS · Center for Excellence in Brain Science and Intelligence Technology, State Key Laboratory of Brain Cognition and Brain-inspired Intelligence Technology,Institute of Neuroscience, CAS · Zhongguancun Academy

  9. Large Language Models with At Most One Spike per Neuron

    Sep 4, 2026Zhuoya Zhao, Parsa Omidi, Aref Jafari +1Spiking Language ModelsSpiking Neural Networks

  10. Parameter Efficient Continual Learning for Sparse Event-Based Transformers

    Aug 27, 2026Vaishnavi Nagabhushana, Kartikay Agrawal, Ayon BorthakurOnline Continual LearningClass-Incremental Learning

    SustainAI Lab, MFSDS&AI IIT Guwahati

  11. Event-triggered Implicit Perturbation for Zeroth-Order Fine-Tuning of Spiking Transformers

    Aug 21, 2026Tengteng Lei, Prabodh Katti, Rashi Dutt +5Zeroth-Order OptimizationSpiking Neural Networks

    Institute for Intelligent Networked Systems (INSI), Northeastern University London, London E1 8PH, U.K. · Centre for Intelligent Information Processing Systems (CIIPS), Department of Engineering, King’s College London, London WC2R 2LS, U.K.

  12. SAGE: Surrogate-gradient Adaptation via Attention-Guided Entropy for Spiking Transformers

    Aug 13, 2026Kiran Nair, Rodrigue Rizk, KC SantoshSpiking Neural NetworksNeural Network Optimization

    USD Artificial Intelligence Research Lab Department of Computer Science University of South Dakota Vermillion, SD 57069, USA

  13. Lapis: Laplacian Spiking Attention via First-Spike Timing and Membrane Leakage

    Aug 12, 2026Kaiwen Tang, Jiaqi Zheng, Zixuan Zhu +3Spiking TransformersEfficient Attention

    School of Computing, National University of Singapore · 2Sea AI Lab · 4Shanghai Advanced Research Institute, Chinese Academy of Sciences +2

  14. Spiking Local Interaction and Adaptive Complementary Fusion for Spiking Transformer

    Aug 11, 2026Dongcheng Zhao, Sicheng Shen, Zhenyu Yang +6Spiking TransformersImage Classification

    Center for Excellence in Brain Science and Intelligence Technology, State Key Laboratory of Brain Cognition and Brain-inspired Intelligence Technology, Institute of Neuroscience, Chinese Academy of Sciences · China Electric Power Research Institute Co., Ltd. (CEPRI)

  15. Rethinking Attention Locality in Spiking Transformers

    Aug 9, 2026Zeqi Zheng, Zizheng Zhu, Yuping Yan +3Spiking TransformersSpiking Neural Networks

    1Zhejiang University · 2Westlake University · 3Peking University

  16. Spike-HTR: Spiking Neural Transformer for Handwritten Text Recognition

    Aug 3, 2026Xiubo Liang, Jinxing Han, Yuke Li +3Efficient Neural Network InferenceSpiking Transformers

    School of Software Technology, Zhejiang University, Ningbo, China

  17. SMM Transformer: Leveraging Spiking Neural Networks for Multimodal Tasks

    Aug 3, 2026Xiubo Liang, Jinxing Han, Yuke Li +3Spiking TransformersSpiking Neural Networks

    School of Software Technology, Zhejiang University, Ningbo, China · NetEase Yidun AI Lab, Hangzhou, China

  18. The Sparsity Ceiling: Where Spiking Networks Can and Cannot Trade Activity for Energy

    Jul 29, 2026Zeyu WangEfficient Neural Network InferenceSpiking Transformers

    Georgia Institute of Technology

  19. Bio-SFT: Asymmetric Cortical Guidance and Retinal Adaptation for Robust HDR Reconstruction

    Jul 20, 2026Tingyu Cheng, Ting Zhang, Chongyi Li +2Spiking TransformersImage Enhancement

    Fujian Key Laboratory of Intelligent Processing and Wireless Transmission of Media Information, College of Physics and Information Engineering, Fuzhou University, Fujian 350108, China · School of Computer Science, Nankai University, Tianjin 300071, China · School of Electrical and Information Engineering, Tianjin University, Tianjin 300072, China

  20. SpikeDS: Dual Sparsity Spikformer for Perineural Invasion Prediction in 3D MRI

    Jul 13, 2026Induk Um, Youngung Han, Kyeonghun Kim +14Spiking Transformers3D Medical Imaging

    Chung-Ang University, Seoul, Republic of Korea · Seoul National University, Seoul, Republic of Korea · OUTTA, Seoul, Republic of Korea +5

  21. SpikeLogBERT: Energy-Efficient Log Parsing Using Spiking Transformer Networks

    Jun 30, 2026Thuan Bui, Duong Do, Tung Vu +2Efficient Neural Network InferenceSpiking Transformers

    Swinburne Vietnam, FPT University Hanoi, Vietnam · Posts and Telecommunications Institute of Technology Hanoi, Vietnam · Academy of Cryptography Techniques Hanoi, Vietnam +1

  22. End-to-End Radar and Communication Modulation Recognition with Neuromorphic Computing

    Jun 23, 2026Xiaohu Li, Chongxiao Qu, Caiyong Lin +2Efficient Neural Network InferenceSpiking Transformers

    China Nanhu Academy of Electronics and Information Technology, Jiaxing, China · University of Electronic Science and Technology of China, Chengdu, China

  23. VL2Spike: Spike-driven Distillation from VLMs for Low-Power Visual Perception in Embodied AI

    Jun 14, 2026Zinan Liu, Eric Zheng, Soumyaratna Debnath +3Cross-Modal Knowledge DistillationSpiking Transformers

    School of EEE, Nanyang Technological University (NTU), Singapore · Department of Computer Science, University of Toronto and Advanced Micro Devices, Inc., Canada · State Key Laboratory of Extreme Photonics and Instrumentation, Zhejiang University, Hangzhou 310027, China +1

  24. AQ4SViT: An Automated Quantization Framework with Search Gating Policy for Compressing Spiking Vision Transformers

    Jun 14, 2026Rachmad Vidya Wicaksana Putra, Saad Iftikhar, Muhammad ShafiqueSpiking TransformersNeural Network Quantization

    eBRAIN Lab, Division of Engineering, New York University (NYU) Abu Dhabi, United Arab Emirates (UAE)

  25. Otters++: A Time-to-first-spike Based Energy Efficient Optical Spiking Transformer

    Jun 11, 2026Zhanglu Yan, Jiayi Mao, Kaiwen Tang +6Efficient Transformer InferenceSpiking Transformers

    Department of Computer Science, National University of Singapore · School of Engineering, Westlake University · College of Computer Science and Technology, Zhejiang University +2

  26. SpikeDecoder: Realizing the GPT Architecture with Spiking Neural Networks

    Jun 10, 2026Claas Beger, Florian Walter, Alois KnollEfficient Neural Network InferenceSpiking Transformers

    Chair of Robotics, Artificial Intelligence and Real-time Systems Technical University of Munich, Germany

  27. SpikingMoE: SDPrompt-Guided Dynamic Expert Fusion in Spiking Neural Networks

    May 22, 2026Yukai Yang, Chenxi Qin, Jungang Li +3Spiking TransformersDynamic Neural Networks

    Tianjin University, Medical School, Tianjin 300072, China · Southern University of Science and Technology, Department of Computer Science and Engineering, Shenzhen, Guangdong 518055, China · Chinese Academy of Medical Sciences, Institute of Biomedical Engineering, Tianjin 300192, China

  28. Plug-and-Play Spiking Operators: Breaking the Nonlinearity Bottleneck in Spiking Transformers

    May 19, 2026Xinzhe Yuan, Xiang Peng, Bin Gu +1Spiking TransformersSpiking Neural Networks

    IASM, Harbin Institute of Technology · IASM, Harbin Institute of Technology, China · School of Artificial Intelligence, Jilin University +1

  29. Breaking Global Self-Attention Bottlenecks in Transformer-based Spiking Neural Networks with Local Structure-Aware Self-Attention

    May 12, 2026Lingdong Li, Hangming Zhang, Qiang YuSpiking TransformersEfficient ViTs

    School of Future Technology, Tianjin Key Laboratory of Cognitive Computing and Application, Tianjin University, Tianjin 300354, China · School of Artificial Intelligence, Tianjin Key Laboratory of Cognitive Computing and Application, College of Intelligence and Computing, Tianjin University, Tianjin 300354, China

  30. Uncertainty-Aware Token Importance Estimation in Spiking Transformers

    May 10, 2026Wenxuan Liu, Zecheng Hao, Tong Bu +2Spiking TransformersUncertainty Quantification

    School of Computer Science, Peking University · School of Computer Science, Peking University. · Institute for Artificial Intelligence, Peking University +2