Event-triggered Implicit Perturbation for Zeroth-Order Fine-Tuning of Spiking Transformers
Authors: Tengteng Lei, Prabodh Katti, Rashi Dutt, Houssem Sifaou, Tan Peng, Osvaldo Simeone, Kai Xu, Bipin Rajendran
Organizations: Institute for Intelligent Networked Systems (INSI), Northeastern University London, London E1 8PH, U.K. · Centre for Intelligent Information Processing Systems (CIIPS), Department of Engineering, King’s College London, London WC2R 2LS, U.K.
Zeroth-order (ZO) optimization estimates gradients using only forward-pass evaluations, making it suitable for fine-tuning non-differentiable, event-driven spiking neural networks (SNNs). However, its deployment on in-memory computing (IMC) accelerators is constrained by the repeated read-modify-write (RMW) operations arising from explicit weight perturbation and the prohibitive hardware footprint of random number generators (RNGs) for statistically independent per-weight perturbations. To address these challenges, we propose an implicit-perturbation ZO (IPZO) architecture in which perturbation sums computed by an event-triggered perturbation generation unit (PGU) are combined with the weighted sums produced by the IMC array, eliminating perturbation-induced RMW operations while preserving weight-stationary execution of IMC. By exploiting spike sparsity, the PGU generates and accumulates perturbation contributions only for spike-activated weight rows, reducing the required row dimension of the RNG array. An address-driven XOR recombination scheme (PGU-XOR) is further introduced to mitigate the spatial correlations caused by direct RNG reuse (PGU-Reuse). The results show that (1) PGU-XOR matches software RNGs in accuracy on Spikingformer/CIFAR-10 (76.41% vs. 76.53%) and perplexity (PPL) on SpikeGPT/WikiText-2 (54.20 vs. 53.23), whereas PGU-Reuse degrades accuracy by 9.56 percentage points and increases PPL by 11.8; (2) implemented in a TSMC 16-nm CMOS technology, PGU-XOR incurs 40.3%-46.0% area and 15.2%-48.9% energy overhead per matrix-vector multiplication relative to PGU-Reuse, yet its faster convergence reduces the total perturbation energy to 0.51x that of PGU-Reuse at iso-accuracy; (3) IPZO reduces the perturbation energy to 0.46x-0.83x that of conventional explicit weight perturbation for a batch size of B=64 and T=4 time steps, with the advantage growing as BT decreases.
Figures & tables
Fig. 1: Comparison of (a) EPZO and (b) IPZO architectures for computing perturbed MVM outputs. The symbols x , θ , zi , and ϵ denote the input spike vector, weight matrix, i -th random perturbation vector, and perturbation scaling factor, respectively.
Fig. 2: Block diagram of the sparse IPZO architecture with a row-dimension-reduced PGU array. m and n denote the numbers of rows and columns of the IMC array, and k represents the number of PGU rows ( k≪m ).
Fig. 3: Perturbation generation schemes for the row-dimension-reduced PGU: (a) PGU-Reuse with group-based perturbation sharing across rows; (b) PGU-XOR with address decomposition and XOR-based combination of quotient- and remainder-associated LFSRs.
Fig. 4: (a) Circuit diagram of the perturbation accumulation architecture with a β×φ CLU array for PGU-Reuse and PGU-XOR, requiring c=k/β accumulation cycles. (b) Overlapping bit-slicing scheme using stride s to partition the w -bit CLU output rn into v pseudo-random segments.
Fig. 5: Data flow of the PGU-XOR across forward pass and parameter update phases.
Fig. 6: Mean firing rate ρ of Q, K, and V LIF neurons across transformer blocks at each time step. Shaded regions indicate its range across transformer blocks.
Fig. 7: Evolution of bit probability distribution in 128 random sequences rn generated by (a) PGU-Reuse and (b) PGU-XOR.
Fig. 8: Statistical correlation properties of random sequences rn : autocorrelation coefficient of a single sequence generated by (a) PGU-Reuse and (b) PGU-XOR; cross-correlation matrix among 128 sequences generated by (c) PGU-Reuse and (d) PGU-XOR.
Fig. 9: Cross-correlation characteristics of 8-bit ( u ) random sequences arn extracted from a 36-bit ( w ) PGU-XOR sequence with different stride configurations: (a) s=1 with 29 sequences; (b) s=2 with 15 sequences; (c) s=4 with 8 sequences; (d) s=7 with 5 sequences.
Fig. 10: Accuracy of a three-layer SNN-MLP (1024-128-10) trained from scratch on MNIST with MeZO under four perturbation schemes including PGU-Reuse, PGU-XOR, Randint, and Randn: (a) accuracy versus training epoch and (b) final accuracy and standard deviation over 10 independent runs.
Fig. 11: (a) Effective rank and (b) Frobenius norm ∥ΔW∥F of the weight change ΔW=Wfinal−Winit for the first layer of SNN-MLP trained from scratch using MeZO under four perturbation schemes including PGU-Reuse, PGU-XOR, Randint, and Randn.
Spikingformer [ 34 ]
SpikeGPT [ 47 ]
CIFAR-10
WikiText-2
WikiText-103
Acc. (%)
Epochs †
PPL
Steps †
PPL
Steps †
PGU-Reuse
66.85
247
66.01
3944
94.73
4000
PGU-XOR
76.41
0 84
54.20
0 687
83.98
0 777
Randint
76.53
0 80
53.23
0 633
82.67
0 641
Randn
76.56
0 83
53.07
0 653
82.32
0 742
TABLE I: Fine-tuning Performance Using MeZO Under Different Random Numbers.
Fig. 12: Layout comparison of the PGU-Reuse and PGU-XOR implementations in 16-nm CMOS technology under accumulation cycles of c=8 .
Fig. 13: (a) Normalized post-layout area and (b) throughput of PGU-Reuse and PGU-XOR under different accumulation cycles c in 16-nm CMOS technology. All designs are placed at ∼66% cell utilization. One operation (OP) denotes one multiplication or one addition.
Fig. 14: Post-layout (a) power and (b) energy per MVM of PGU-Reuse and PGU-XOR under different accumulation cycles c in 16-nm CMOS technology, evaluated for 8-bit signed (BIT8, open bars) and 1-bit Rademacher (BIT1, hatched bars) perturbations. One MVM denotes one spike–perturbation product x⋅zi , where x is the spike vector and zi is the perturbation matrix. (c) Energy cost of PGU-Reuse and PGU-XOR at iso-accuracy on CIFAR-10, normalized to PGU-Reuse.
Fig. 15: (a) Operation energies of a 128 × 64 SRAM in TSMC 16-nm technology: 64-bit read er , 64-bit write ew , and 8-bit addition ea . (b) Energy comparison between IPZO (PGU-XOR, per MVM) obtained from post-layout simulation, and EPZO (RMW, per batch) estimated from (a). (c) Energy ratio R=EIPZO/EEPZO as a function of batch size and number of time steps.
Zeroth-order optimization uses function values or comparisons, rather than gradients, to optimize objectives when differentiation is unavailable or costly.
Spiking neural networks (SNNs) are promising for energy-efficient inference, and time-to-first-spike (TTFS) coding is especially attractive because each neuron fires at most once. In practice, however, this benefit is often reduced by the cost of computing a temporal decay term and multiplying it by the synaptic weight. We address this issue by turning a physical hardware "bug," the natural signal decay in optoelectronic devices, into the main computation of TTFS, named Otters++. Specifically, we use the measured decay of a custom In2O3 optoelectronic synapse to directly realize the TTFS temporal term, removing the need for explicit digital decay computation. To scale this idea to Transformer models, we establish a layer-wise functional equivalence between the Otters++ and a quantized neural network (QNN), and develop a hybrid training method that uses device-faithful SNN computation in the forward pass and QNN straight-through gradients through the equivalent QNN path in the backward pass, together with model distillation. This avoids differentiation through discrete first-spike events and reduces the over-sparsity problem in direct TTFS-SNN training. We further make training aware of measured device noise by sampling run-to-run variation, and refine the system-level energy model by accounting for device sharing and multi-hop communication. On GLUE dataset, Otters++ improves the average score to 84.17% while maintaining a clear energy advantage over prior spiking Transformer baselines. These results show that physically grounded TTFS computing can be efficient, trainable, and robust under realistic hardware effects.
Zhanglu Yan, Jiayi Mao, Kaiwen Tang +6
Department of Computer Science, National University of Singapore · School of Engineering, Westlake University · College of Computer Science and Technology, Zhejiang University +2
ANN-to-SNN conversion offers a practical, training-free route to spiking large language models. However, current pipelines primarily focus on spike-driven realizations for Transformer linear-algebra operations, while providing limited support for key nonlinear operators. This gap limits compatibility with neuromorphic-style execution constraints, where such nonlinearities typically require division, exponentiation, or norm computations that are not naturally supported by standard leaky integrate-and-fire dynamics. To solve this problem, we propose a plug-and-play framework that implements spike-friendly approximations for Transformer nonlinearities and integrates into existing ANN-to-SNN pipelines. Our method decomposes these nonlinear computations into three recurring primitives -- division, exponentiation, and ℓ2 norms -- and realizes them via population computation using LIF neuron groups, combined with lightweight bit-shift scaling to avoid floating-point arithmetic. By composing these primitives as modular operator blocks, our framework supports common Transformer nonlinearities (e.g., Softmax, SiLU, and normalization) without any fine-tuning. Experiments on a range of LLMs Transformers show that selectively replacing the targeted nonlinear operators incurs less than a 1% accuracy drop across all evaluated tasks.
Xinzhe Yuan, Xiang Peng, Bin Gu +1
IASM, Harbin Institute of Technology · IASM, Harbin Institute of Technology, China · School of Artificial Intelligence, Jilin University +1
On neuromorphic chips, spiking neural networks (SNNs) consume far less energy than artificial neural networks (ANNs), but the limited memory of these chips requires low-bit synaptic weights and membrane potentials. Existing post-training quantization (PTQ) methods provide them without retraining and, like those for ANNs, minimize numerical errors such as the mean squared error (MSE). In an SNN, however, a quantization error reaches later layers only through the spikes it flips. We find that errors of equal MSE can cause very different accuracy losses, whereas errors that flip the same fraction of spikes cause the same loss. We therefore propose SpikeQ, a spike-guided PTQ framework. For weights, we select the level of each weight by simulating the downstream neurons and minimizing their spike errors. For membrane potentials, most existing methods quantize the state after the reset, whereas neuromorphic chips compare the threshold with the membrane potential stored at a fixed bit width. We target this pre-fire quantization and build a reset-aware grid that places the threshold on a rounding boundary, so rounding never flips the spike at the current time step. We evaluate SpikeQ on seven benchmarks of image classification, semantic segmentation, and object detection. With 2-bit pre-fire membrane potentials, every existing grid collapses, whereas SpikeQ retains most of the full-precision accuracy. With both at 4 bits, the models stay within 2.59 points of full precision, shrink their memory footprint 7.8--7.9 times, and use 6.4--6.9 times less energy.
Kai Sun, Peibo Duan, Zhanglu Yan +3
Monash University · National University of Singapore · Northeastern University