Organizations: Northeastern University, 110819, Shenyang, China · Hebei Key Laboratory of Marine Perception Network and Data Processing, 066004, Qinhuangdao, China
Spiking language models face a tradeoff between representing continuous semantic features over short temporal windows and retaining costly nonlinear attention operations. We introduce Spora, which jointly designs spike encodings and attention operators. Binary temporal weights let T spikes represent compositional values with up to T bits of capacity, compared with O(log2T) bits for spike-count readout. Unipolar Binary Spiking (UBS) uses thresholds and spike-triggered residual decay to produce non-negative integer codes; Bipolar Binary Spiking (BBS) separates sign and magnitude and learns a scale for signed activations. These representations support accumulation-and-shift dot products and integer-exponent mappings in attention. With four time steps, Spora achieves 76.6 average GLUE score and 44.1 CoLA MCC, improving over SpikeLM by 1.2 and 6.2 points, respectively. Extending BBS to six steps raises these scores to 78.2 and 47.4. Conditional-decay analysis, matched-budget activation-quantization comparisons, event-workload statistics, and fixed-point evaluation further characterize the connection between encoding fidelity and computational cost.
Figures & tables
Figure 1: Spike encodings: (a) rate, (b) TTFS, (c) rank-order, (d) Few-Spike, (e) UBS, and (f) BBS. Event labels denote readout weights; BBS traces show separate positive and negative inputs.
Figure 2: Spora architecture. (A) UBS temporal weights; (B) sign-separated BBS; (C) approximate attention, with ShiftApprox denoting shift-based Softmax approximation; (D) feed-forward block, where UBS implements the GELU approximation during inference.
Figure 3: Attention-output NMSE for four-step integer encoders: (a) raw-score and (b) shared-preprocessing references. Annotations report aggregate reductions relative to recalibrated no-decay UBS.
Method
Acc. (%)
MCC ×100
Static-QAT
76.61
40.97
Spora T=6
79.00
47.40
Table 2: CoLA comparison under matched representation budgets: 15 signed states and four-bit UBS.
Figure 4: Activation-derived workload on CoLA at padded length 128. Left: event density by component. Right: estimated event-driven accumulations relative to dense MACs in spike-to-linear paths.
Figure 5: Attention score visualization of BERT and Spora with time steps 4 and 6 on the CoLA sample “Janet broke the cup.” across 12 layers. Q denotes the query token and K denotes the key token.