Event-triggered Implicit Perturbation for Zeroth-Order Fine-Tuning of Spiking Transformers

Tengteng Lei; Prabodh Katti; Rashi Dutt; Houssem Sifaou; Tan Peng; Osvaldo Simeone; Kai Xu; Bipin Rajendran

cs.ARAug 21, 2026

Event-triggered Implicit Perturbation for Zeroth-Order Fine-Tuning of Spiking Transformers

Authors: Tengteng Lei, Prabodh Katti, Rashi Dutt, Houssem Sifaou, Tan Peng, Osvaldo Simeone, Kai Xu, Bipin Rajendran

Organizations: Institute for Intelligent Networked Systems (INSI), Northeastern University London, London E1 8PH, U.K. · Centre for Intelligent Information Processing Systems (CIIPS), Department of Engineering, King’s College London, London WC2R 2LS, U.K.

Abstract

Zeroth-order (ZO) optimization estimates gradients using only forward-pass evaluations, making it suitable for fine-tuning non-differentiable, event-driven spiking neural networks (SNNs). However, its deployment on in-memory computing (IMC) accelerators is constrained by the repeated read-modify-write (RMW) operations arising from explicit weight perturbation and the prohibitive hardware footprint of random number generators (RNGs) for statistically independent per-weight perturbations. To address these challenges, we propose an implicit-perturbation ZO (IPZO) architecture in which perturbation sums computed by an event-triggered perturbation generation unit (PGU) are combined with the weighted sums produced by the IMC array, eliminating perturbation-induced RMW operations while preserving weight-stationary execution of IMC. By exploiting spike sparsity, the PGU generates and accumulates perturbation contributions only for spike-activated weight rows, reducing the required row dimension of the RNG array. An address-driven XOR recombination scheme (PGU-XOR) is further introduced to mitigate the spatial correlations caused by direct RNG reuse (PGU-Reuse). The results show that (1) PGU-XOR matches software RNGs in accuracy on Spikingformer/CIFAR-10 (76.41% vs. 76.53%) and perplexity (PPL) on SpikeGPT/WikiText-2 (54.20 vs. 53.23), whereas PGU-Reuse degrades accuracy by 9.56 percentage points and increases PPL by 11.8; (2) implemented in a TSMC 16-nm CMOS technology, PGU-XOR incurs 40.3%-46.0% area and 15.2%-48.9% energy overhead per matrix-vector multiplication relative to PGU-Reuse, yet its faster convergence reduces the total perturbation energy to 0.51x that of PGU-Reuse at iso-accuracy; (3) IPZO reduces the perturbation energy to 0.46x-0.83x that of conventional explicit weight perturbation for a batch size of B=64 and T=4 time steps, with the advantage growing as BT decreases.

Figures & tables

Fig. 1: Comparison of (a) EPZO and (b) IPZO architectures for computing perturbed MVM outputs. The symbols $x$ , $\theta$ , $z_{i}$ , and $\epsilon$ denote the input spike vector, weight matrix, $i$ -th random perturbation vector, and perturbation scaling factor, respectively.

Fig. 2: Block diagram of the sparse IPZO architecture with a row-dimension-reduced PGU array. $m$ and $n$ denote the numbers of rows and columns of the IMC array, and $k$ represents the number of PGU rows ( $k\ll m$ ).

Fig. 3: Perturbation generation schemes for the row-dimension-reduced PGU: (a) PGU-Reuse with group-based perturbation sharing across rows; (b) PGU-XOR with address decomposition and XOR-based combination of quotient- and remainder-associated LFSRs.

Fig. 4: (a) Circuit diagram of the perturbation accumulation architecture with a $\beta\times\varphi$ CLU array for PGU-Reuse and PGU-XOR, requiring $c=k/\beta$ accumulation cycles. (b) Overlapping bit-slicing scheme using stride $s$ to partition the $w$ -bit CLU output $rn$ into $v$ pseudo-random segments.

Fig. 5: Data flow of the PGU-XOR across forward pass and parameter update phases.

Fig. 6: Mean firing rate $\rho$ of Q, K, and V LIF neurons across transformer blocks at each time step. Shaded regions indicate its range across transformer blocks.

Fig. 7: Evolution of bit probability distribution in 128 random sequences $rn$ generated by (a) PGU-Reuse and (b) PGU-XOR.

Fig. 8: Statistical correlation properties of random sequences $rn$ : autocorrelation coefficient of a single sequence generated by (a) PGU-Reuse and (b) PGU-XOR; cross-correlation matrix among 128 sequences generated by (c) PGU-Reuse and (d) PGU-XOR.

Fig. 9: Cross-correlation characteristics of 8-bit ( $u$ ) random sequences $a_{rn}$ extracted from a 36-bit ( $w$ ) PGU-XOR sequence with different stride configurations: (a) $s=1$ with 29 sequences; (b) $s=2$ with 15 sequences; (c) $s=4$ with 8 sequences; (d) $s=7$ with 5 sequences.

Fig. 10: Accuracy of a three-layer SNN-MLP (1024-128-10) trained from scratch on MNIST with MeZO under four perturbation schemes including PGU-Reuse, PGU-XOR, Randint, and Randn: (a) accuracy versus training epoch and (b) final accuracy and standard deviation over 10 independent runs.

Fig. 11: (a) Effective rank and (b) Frobenius norm $\|\Delta W\|_{F}$ of the weight change $\Delta W=W_{\mathrm{final}}-W_{\mathrm{init}}$ for the first layer of SNN-MLP trained from scratch using MeZO under four perturbation schemes including PGU-Reuse, PGU-XOR, Randint, and Randn.

	Spikingformer [ 34 ]		SpikeGPT [ 47 ]
	CIFAR-10		WikiText-2		WikiText-103
	Acc. (%)	Epochs †	PPL	Steps †	PPL	Steps †
PGU-Reuse	66.85	247	66.01	3944	94.73	4000
PGU-XOR	76.41	0 84	54.20	0 687	83.98	0 777
Randint	76.53	0 80	53.23	0 633	82.67	0 641
Randn	76.56	0 83	53.07	0 653	82.32	0 742

TABLE I: Fine-tuning Performance Using MeZO Under Different Random Numbers.

Fig. 12: Layout comparison of the PGU-Reuse and PGU-XOR implementations in 16-nm CMOS technology under accumulation cycles of $c=8$ .

Fig. 13: (a) Normalized post-layout area and (b) throughput of PGU-Reuse and PGU-XOR under different accumulation cycles $c$ in 16-nm CMOS technology. All designs are placed at $\sim 66\%$ cell utilization. One operation (OP) denotes one multiplication or one addition.

Fig. 14: Post-layout (a) power and (b) energy per MVM of PGU-Reuse and PGU-XOR under different accumulation cycles $c$ in 16-nm CMOS technology, evaluated for 8-bit signed (BIT8, open bars) and 1-bit Rademacher (BIT1, hatched bars) perturbations. One MVM denotes one spike–perturbation product $x\cdot z_{i}$ , where $x$ is the spike vector and $z_{i}$ is the perturbation matrix. (c) Energy cost of PGU-Reuse and PGU-XOR at iso-accuracy on CIFAR-10, normalized to PGU-Reuse.

Fig. 15: (a) Operation energies of a 128 $\times$ 64 SRAM in TSMC 16-nm technology: 64-bit read $e_{r}$ , 64-bit write $e_{w}$ , and 8-bit addition $e_{a}$ . (b) Energy comparison between IPZO (PGU-XOR, per MVM) obtained from post-layout simulation, and EPZO (RMW, per batch) estimated from (a). (c) Energy ratio $R=E_{\mathrm{IPZO}}/E_{\mathrm{EPZO}}$ as a function of batch size and number of time steps.

Zeroth-Order Optimization

Zeroth-order optimization uses function values or comparisons, rather than gradients, to optimize objectives when differentiation is unavailable or costly.

See all 86 papers

Otters++: A Time-to-first-spike Based Energy Efficient Optical Spiking Transformer
Jun 11, 2026Zhanglu Yan, Jiayi Mao, Kaiwen Tang +6Efficient Transformer Inference Spiking Transformers
Department of Computer Science, National University of Singapore · School of Engineering, Westlake University · College of Computer Science and Technology, Zhejiang University +2
Plug-and-Play Spiking Operators: Breaking the Nonlinearity Bottleneck in Spiking Transformers
May 19, 2026Xinzhe Yuan, Xiang Peng, Bin Gu +1Spiking Transformers Spiking Neural Networks
IASM, Harbin Institute of Technology · IASM, Harbin Institute of Technology, China · School of Artificial Intelligence, Jilin University +1
SpikeQ: Spike-Guided Post-Training Quantization for Spiking Neural Networks
Oct 16, 2026Kai Sun, Peibo Duan, Zhanglu Yan +3Spiking Neural Networks Post-Training Quantization
Monash University · National University of Singapore · Northeastern University

Jun 11, 2026J/K move · Enter open · S save

Otters++: A Time-to-first-spike Based Energy Efficient Optical Spiking Transformer

Zhanglu Yan, Jiayi Mao, Kaiwen Tang, Fanfan Li+5

Department of Computer Science, National University of Singapore · School of Engineering, Westlake University · College of Computer Science and Technology, Zhejiang University+2

Efficient Transformer Inference Spiking Transformers Spiking Neural Networks Neuromorphic Computing Photonic Computing Spike-Based Signal Encoding

Open paper AskPDF arXiv

Spiking neural networks (SNNs) are promising for energy-efficient inference, and time-to-first-spike (TTFS) coding is especially attractive because each neuron fires at most once. In practice, however, this benefit is often reduced by the cost of computing a temporal decay term and multiplying it by the synaptic weight. We address this issue by turning a physical hardware "bug," the natural signal decay in optoelectronic devices, into the main computation of TTFS, named Otters++. Specifically, we use the measured decay of a custom In

_2

_3

optoelectronic synapse to directly realize the TTFS temporal term, removing the need for explicit digital decay computation. To scale this idea to Transformer models, we establish a layer-wise functional equivalence between the Otters++ and a quantized neural network (QNN), and develop a hybrid training method that uses device-faithful SNN computation in the forward pass and QNN straight-through gradients through the equivalent QNN path in the backward pass, together with model distillation. This avoids differentiation through discrete first-spike events and reduces the over-sparsity problem in direct TTFS-SNN training. We further make training aware of measured device noise by sampling run-to-run variation, and refine the system-level energy model by accounting for device sharing and multi-hop communication. On GLUE dataset, Otters++ improves the average score to 84.17% while maintaining a clear energy advantage over prior spiking Transformer baselines. These results show that physically grounded TTFS computing can be efficient, trainable, and robust under realistic hardware effects.

Abstract

Zeroth-Order Optimization

Explore similar work

Otters++: A Time-to-first-spike Based Energy Efficient Optical Spiking Transformer

Plug-and-Play Spiking Operators: Breaking the Nonlinearity Bottleneck in Spiking Transformers

SpikeQ: Spike-Guided Post-Training Quantization for Spiking Neural Networks