Organizations: School of Computer Science, Peking University · State Key Laboratory for Multimedia Information Processing, Peking University · Beijing Key Laboratory of Brain-inspired Spiking Large Models, Peking University · Beijing Academy of Artificial Intelligence (BAAI)
Spiking Neural Networks (SNNs) promise event-driven and energy-efficient vision, yet existing spiking vision transformers (S-ViTs) largely confine spiking tokens to fixed and uniform firing schemes along the temporal, spatial and channel dimensions, whereas suppressing the computational redundancy thus incurred calls for adaptive firing mechanisms that operate jointly across all three axes at test time. To this end, we propose the Adaptive Efficient Integrate-and-Fire (AdaE-IF) model, which consults the attention score of each spiking token and routes salient regions to fine-grained firing patterns while assigning energy-frugal coarse patterns to the remaining regions. Building on AdaE-IF, we further introduce Spiking Gated Self-Attention (S-GSA) and Spiking Local-Gated Unit (S-LGU), which enforce selective spike emission at token and channel granularity. We integrate the aforementioned mechanisms and modules into AdaS-ViT, a test-time adaptive-routing architecture trained with a progressive spatio-temporal curriculum that stabilizes the co-adaptation of firing precision and routing masks. Extensive experiments across benchmarks spanning diverse data scales and modalities demonstrate that AdaS-ViT consistently achieves highly competitive performance. Notably, on ImageNet-1k it consumes only 1.02 to 2.45 millijoules per image and attains an accuracy-per-energy ratio of 32× to 72×, operating in an economical inference regime and thereby pushing the energy-efficiency boundary further.
Figures & tables
Figure 1: AdaS-ViT establishes a compelling accuracy-energy frontier for spiking vision transformers. (a) Qualitative comparison of training paradigms over six axes covering training cost and inference behavior: prior paradigms trade these objectives against one another, whereas our test-time adaptive routing maintains a uniformly strong profile across all of them. (b) Quantitative comparison with prior SNN-based works on ImageNet-1k. The horizontal axis denotes top-1 accuracy and the vertical axis reports Ratio AE, i.e., the accuracy delivered per millijoule of inference energy. AdaS-ViT occupies the top-right region alone with the smallest bubble area, indicating markedly higher energy efficiency at a fraction of the parameters of prior art.
Figure 2: AdaE-IF: from uniform to adaptive and efficient spike firing. Top-left: the firing-rate curve of a vanilla IF/LIF neuron is ideally a uniform staircase, yet diverse input current patterns {Itl} yield divergent firing rates F~1:Tl for identical average input current I~1:Tl , which accumulate across layers into conversion error and spatio-temporal gradient inconsistency. Bottom: AdaE-IF maps each spike firing pattern to its bit-granular equivalent, aligning training gradients and enabling precise regulation of Synaptic Operations (SOPs); guided by the attention score of the spiking feature map (Middle), it routes each token to a coarse-grained pattern that preserves energy efficiency, or a fine-grained pattern that enhances representation at only marginal extra cost; jointly scheduled under an ultra-low energy budget (Top-right), the two modes realize test-time adaptive firing precision.
Figure 3: Architecture of AdaS-ViT. Top: the macro pipeline stacks dual-mode AdaS-Blocks (S-GSA followed by S-LGU or S-FFN) over progressively downsampled token maps. Bottom: data flow of S-GSA, S-LGU and the parameterized S-FFN variants (I)(II) with convolution kernel size k and expansion ratio r .
Method
Type
Architecture
Param.(M)
T
Energy(mJ)
Acc.(%)
Ratio AE
Spiking ResNet [ 18 ]
ANN-SNN Conversion
ResNet-34
21.79
350
59.30
71.61
1.21
ResNet-50
25.56
350
70.93
72.75
1.03
STBP-tdBN [ 51 ]
STBP Training
Spiking ResNet-34
21.79
6
6.39
63.72
9.97
SEW ResNet [ 11 ]
STBP Training
SEW ResNet-34
21.79
4
4.04
67.04
16.59
SEW ResNet-50
25.56
4
4.89
67.78
13.86
SEW ResNet-101
44.55
4
8.91
68.76
7.72
Table 1: Comparison of representative SNN approaches across learning paradigms and network architectures on ImageNet-1k. Here T denotes the inference time-steps, Energy is the estimated inference energy per image, computed following the convention of [ 54 ] . Ratio AE quantifies the accuracy delivered per unit of energy cost, where a higher value indicates better energy efficiency.
Method
Type
CIFAR-10
CIFAR-100
CIFAR10-DVS
Time-step
Acc.(%)
Time-step
Acc.(%)
Time-step
Acc.(%)
STBP-tdBN [ 51 ]
STBP Training
4
92.92
-
-
10
67.8
TET [ 7 ]
STBP Training
4
94.44
4
74.47
10
83.17
QCFS [ 1 ]
ANN-SNN Conversion
8
94.82
8
78.48
-
-
SLTT [ 31 ]
Online Training
6
94.44
6
74.38
10
82.20
GAC-SNN [ 35 ]
STBP Training
4
96.24
4
79.83
-
-
Table 2: Generalization across data scales and modalities: comparison with prior SNN learning methods on static (CIFAR-10, CIFAR-100) and neuromorphic (CIFAR10-DVS) benchmarks.
Figure 4: Fine-grained energy breakdown of AdaS-ViT Large on ImageNet-1k. Module-level (a)–(c) and Operator-level (d)–(f) statistics of SOPs and average spike firing rates. For each block, Bar hues identify the module family, the total height of a bar denotes the theoretical SOPs Upper-bound while its dark lower segment denotes the actually incurred SOPs, the curve on the right axis reports the average firing rate.
Spiking Transformers combine attention-based architectures with event-driven spiking neurons to process information using sparse, temporally coded computation.
Spiking Vision Transformers (S-ViTs) offer a promising framework for energy-efficient visual learning. However, existing designs remain limited by two fundamental issues: the restricted information capacity of binary spike coding and the dense token interactions introduced by global self-attention. To address these challenges, this work proposes BSViT, a burst spiking-driven Vision Transformer featuring a Dual-Channel Burst Spiking Self-Attention (DBSSA) mechanism. DBSSA encodes queries with binary spikes and keys with burst spikes to enhance representational capacity. The value pathway adopts dual excitatory and inhibitory binary channels, enabling signed modulation and richer spike interactions. Importantly, the entire attention operation preserves addition-only computation, ensuring compatibility with energy-efficient neuromorphic hardware. To further reduce spike activity and incorporate spatial priors, a patch adjacency masking strategy is introduced to restrict attention to local neighborhoods, resulting in structure-aware sparsity and reduced computational overhead. In addition, burst spike coding is systematically integrated across the network to increase spike-level representational capacity beyond conventional binary spiking. Extensive experiments on both static and event-based vision benchmarks demonstrate that BSViT consistently outperforms existing spiking Transformers in accuracy while maintaining competitive energy efficiency.
Hongxiang Peng, Dewei Bai, Hong Qu
School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu 610054, China
Spiking Neural Networks (SNNs) offer superior energy efficiency over Artificial Neural Networks (ANNs). However, they encounter significant deficiencies in training and inference metrics when applied to Spiking Vision Transformers (S-ViTs). Existing paradigms including ANN-SNN Conversion and Spatial-Temporal Backpropagation (STBP) suffer from inherent limitations, precluding concurrent optimization of memory, accuracy and energy consumption. To address these issues, we propose Ge2mS-T, a novel architecture implementing grouped computation across temporal, spatial and network structure dimensions. Specifically, we introduce the Grouped-Exponential-Coding-based IF (ExpG-IF) model, enabling lossless conversion with constant training overhead and precise regulation for spike patterns. Additionally, we develop Group-wise Spiking Self-Attention (GW-SSA) to reduce computational complexity via multi-scale token grouping and multiplication-free operations within a hybrid attention-convolution framework. Experiments confirm that our method can achieve superior performance with ultra-high energy efficiency on challenging benchmarks. To our best knowledge, this is the first work to systematically establish multi-dimensional grouped computation for resolving the triad of memory overhead, learning capability and energy budget in S-ViTs. Code is available at https://github.com/hzc1208/Ge2mST.
Zecheng Hao, Shenghao Xie, Kang Chen +3
Peking University, School of Computer Science, State Key Laboratory for Multimedia Information Processing, Beijing, China · Peking University, School of Computer Science, Beijing Key Laboratory of Brain-inspired Spiking Large Models, Beijing, China
Integrating Spiking Neural Networks (SNNs) with Transformer architectures offers a promising pathway to balance energy efficiency and performance, particularly for edge vision applications. However, existing Spiking Transformers face two critical challenges: (i) a substantial performance gap compared to their Artificial Neural Networks (ANNs) counterparts and (ii) high memory overhead during inference. Through theoretical analysis, we attribute both limitations to the Spiking Self-Attention (SSA) mechanism: the lack of locality bias and the need to store large attention matrices. Inspired by the localized receptive fields (LRF) and membrane-potential dynamics of biological visual neurons, we propose LRF-Dyn, which uses spiking neurons with localized receptive fields to compute attention while reducing memory requirements. Specifically, we introduce a LRF method into SSA to assign higher weights to neighboring regions, strengthening local modeling and improving performance. Building on this, we approximate the resulting attention computation via charge-fire-reset dynamics, eliminating explicit attention-matrix storage and reducing inference-time memory. Extensive experiments on visual tasks confirm that our method reduces memory overhead while delivering significant performance improvements. These results establish it as a key unit for achieving energy-efficient Spiking Transformers.
Dehao Zhang, Fukai Guo, Shuai Wang +6
University of Electronic Science and Technology of China · Shenzhen Loop Area Institute · The Chinese University of Hong Kong (Shenzhen)