Spiking Transformer

Spiking Transformers integrate the energy efficiency of spiking neural networks (SNNs) with the powerful attention mechanisms of Transformers, aiming to create high-performance, low-power deep learning models. Current research focuses on developing novel spiking self-attention mechanisms, improving training methods (including direct training and ANN-to-SNN conversion), and exploring various architectures like Spikformer and its variants for diverse applications such as image classification, video action recognition, and audio-visual processing. This research area is significant because it offers a potential pathway towards more energy-efficient and biologically plausible artificial intelligence, particularly relevant for resource-constrained devices and applications requiring real-time processing.

Papers

November 15, 2023

August 29, 2023

SpikeBERT: A Language Spikformer Learned from BERT with Knowledge Distillation
Changze Lv, Tianlong Li, Jianhan Xu, Chenxi Gu, Zixuan Ling, Cenyuan Zhang, Xiaoqing Zheng, Xuanjing Huang
Knowledge Distillation Ticket BERT Transformer Based Network Deep Architecture Spiking Transformer Two Stage Knowledge Distillation Spikformer Architecture

July 4, 2023

Spike-driven Transformer
Man Yao, Jiakui Hu, Zhaokun Zhou, Li Yuan, Yonghong Tian, Bo Xu, Guoqi Li
Self Attention Spiking Neural Network Spiking Transformer Spike Driven

July 1, 2023

AutoST: Training-free Neural Architecture Search for Spiking Transformers
Ziqing Wang, Qidong Zhao, Jinku Cui, Xu Liu, Dongkuan Xu
Training Free Neuromorphic Datasets Spiking Transformer Free Neural Architecture Search

May 10, 2023

Enhancing the Performance of Transformer-based Spiking Neural Networks by SNN-optimized Downsampling with Precise Gradient Backpropagation
Chenlin Zhou, Han Zhang, Zhaokun Zhou, Liutao Yu, Zhengyu Ma, Huihui Zhou, Xiaopeng Fan, Yonghong Tian
System Performance Spiking Transformer Gradient Backpropagation Downsampling Layer

April 24, 2023

Spikingformer: Spike-driven Residual Learning for Transformer-based Spiking Neural Network
Chenlin Zhou, Liutao Yu, Zhaokun Zhou, Zhengyu Ma, Han Zhang, Huihui Zhou, Yonghong Tian
Neuromorphic Hardware Spike Based Spiking Transformer

March 16, 2023

Event-based Human Pose Tracking by Spiking Spatiotemporal Transformer
Shihao Zou, Yuxuan Mu, Xinxin Zuo, Sen Wang, Li Cheng
Event Camera 3D Human Motion Spiking Transformer Spatiotemporal Transformer

February 2, 2023

Complex Dynamic Neurons Improved Spiking Transformer Network for Efficient Automatic Speech Recognition
Minglun Han, Qingyu Wang, Tielin Zhang, Yi Wang, Duzhen Zhang, Bo Xu
Automatic Speech Recognition Spiking Neural Network Neural Dynamic Spiking Transformer Neuronal Dynamic Leaky Integrate and Fire Neuron

October 3, 2022

Masked Spiking Transformer
Ziqing Wang, Yuetong Fang, Jiahang Cao, Qiang Zhang, Zhongrui Wang, Renjing Xu
State of the Art Random Masking Spiking Transformer SNN Conversion

September 29, 2022

Spikformer: When Spiking Neural Network Meets Transformer
Zhaokun Zhou, Yuesheng Zhu, Chao He, Yaowei Wang, Shuicheng Yan, Yonghong Tian, Li Yuan
Neural Network Transformer Based Self Attention Spiking Neural Network SNN Model Spiking Transformer Spikformer Architecture

Spiking Transformer

Papers

DISTA: Denoising Spiking Transformer with intrinsic plasticity and spatiotemporal attention

SparseSpikformer: A Co-Design Framework for Token and Weight Pruning in Spiking Transformer

SpikeBERT: A Language Spikformer Learned from BERT with Knowledge Distillation

Spike-driven Transformer

AutoST: Training-free Neural Architecture Search for Spiking Transformers

Enhancing the Performance of Transformer-based Spiking Neural Networks by SNN-optimized Downsampling with Precise Gradient Backpropagation

Spikingformer: Spike-driven Residual Learning for Transformer-based Spiking Neural Network

Event-based Human Pose Tracking by Spiking Spatiotemporal Transformer

Complex Dynamic Neurons Improved Spiking Transformer Network for Efficient Automatic Speech Recognition

Masked Spiking Transformer

Spikformer: When Spiking Neural Network Meets Transformer