Softmax-Free Attention in Spiking Transformers as the Zero-Temperature Limit of Two-Level Systems
Authors: Roberto C. Sotero, Pouya Taghipour, Jose M. Sanchez-Bornot
Organizations: Department of Radiology, University of Calgary, Calgary, AB, Canada · Department of Biomedical Engineering, University of Calgary, Calgary, AB, Canada · Hotchkiss Brain Institute, University of Calgary, Calgary, AB, Canada · Intelligent Systems Research Center, Ulster University, Derry Londonderry, Northern Ireland, United Kingdom
Spiking Transformers achieve energy efficiency by discarding softmax normalization in favor of thresholded, binary attention. However, this removal breaks the thermodynamic framework used to analyze attention networks, as the normalized measure and partition function cease to exist. We restore this theoretical foundation by proving that softmax-free spiking attention operates exactly as the zero-temperature limit of a product of independent two-level systems. This correspondence places spiking attention strictly in the frozen ordered phase of the thermodynamic phase diagram for all inputs. Consequently, the entropic cost of organizing attention collapses, replaced by a Landauer-type physical floor governed by action-potential generation. Furthermore, this ordered-phase pinning mathematically explains the optimization difficulty caused by vanishing surrogate gradients. We validate this theory with forward-relaxation experiments, placing both standard and neuromorphic attention mechanisms on a unified thermodynamic axis.
Figures & tables
Figure 1: The spiking-Boltzmann correspondence. Each column is read from top to bottom as a derivation, from the underlying ensemble to its Boltzmann measure. A : softmax attention. The N keys attended to by a query form a single coupled N -state categorical system. The scores sij=qi⊤kj/dk define the interaction energy Eij=−sij , a single global temperature T fixes the inverse temperature β=1/T , the shared normaliser Zi=∑je−βEij couples the keys, and the attention weights are the Boltzmann measure Aij(T)=e−βEij/Zi of Equation ( 1 ), with operating point β=1 . B : softmax-free spiking attention. Deleting the shared normaliser decouples the keys into a product of N independent two-level systems, one per query-key pair, each with energy gap Δij=ϑ−γsij and partition function zij=1+e−βsΔij . The single temperature splits into a forward value Ts=1/βs and a backward value Teff=1/α set by the surrogate slope. The two-level occupation pij=e−βsΔij/zij=σ(βs(γsij−ϑ)) is the counterpart of Aij , and the deployed hard spike aij=Θ(γsij−ϑ)∈{0,1} is its zero-temperature limit Ts→0 (Equation ( 8 )). Softmax attention therefore operates at a finite, entropy-bearing temperature, whereas spiking attention is the zero-temperature limit of the corresponding two-level Boltzmann measure.
Quantity
Symbol
Value
Dataset
CIFAR-10
Image size
32×32×3
Patch size
4
Embedding width
dmodel
64
Attention heads
4
Transformer blocks
2
Table 1: Experimental configuration. The surrogate slope α is fixed for the relaxation experiment (Figure 2 ) and swept for the backward-temperature experiment (Figure 3 ).
Figure 2: Forward relaxation of the matched models on CIFAR-10. The trained spiking operator (red) is shown against the two-level inverse temperature βs and the softmax baseline (blue) against a swept temperature T , on a shared axis. A : order parameter. The softmax model sits at the disordering crossover at its native point T=1 (dotted line), while the spiking model is pinned near ϕ(2)=1 at its operating point βs→∞ (star), validating the ordered-phase pinning of Equation ( 10 ); the comparison concerns operating-point placement, not curve identity. B : specific heat. The spiking heat capacity shows an interior Schottky anomaly peaking at Tc (dashed line), and the operating point at T→0 lies well cold of Tc . The softmax peak is larger because it sums the energy variance over all N keys, whereas Cv(2) is the heat capacity of one binary unit per pair.
Figure 3: Surrogate slope as the backward temperature on CIFAR-10. A : test accuracy against α=1/Teff , with error bars over seeds. Accuracy peaks across a broad intermediate plateau and collapses as α grows: large α is the cold backward temperature that starves the gradient, and the smallest α is the biased high-temperature side. B : active-band fraction, the fraction of pairs with ∣γsij−ϑ∣<1/α , against α on logarithmic axes; the power-law fit (slope in the legend) confirms the O(1/α) surrogate support of Equation ( 9 ). The fraction is averaged over the attention array, so its across-seed spread is below the marker size. Panels A and B give mechanism and consequence: as the band narrows as 1/α , trainability is lost.
Figure 4: Entropy dynamics of the spiking model during training on CIFAR-10, shown as the mean over K seeds with error bars. A : the per-input two-level entropy Sˉ(2) (blue), evaluated at the backward temperature βs=α , and the across-ensemble spike-pattern entropy Sspk of Equation ( 12 ) (red), against training epoch; the dotted line is the maximum binary entropy ln2 . The per-input entropy stays near zero for the whole run, so the attention pattern is frozen in the ordered phase at every epoch and the pinning of Equation ( 10 ) holds along the trajectory, whereas the across-ensemble entropy starts substantial and falls steadily, representing the informative reorganisation that learning performs. The across-seed spread of Sˉ(2) is below the marker, as for the averaged observables of Figure 2 . B : the evaluation loss over the same window. The loss falls together with Sspk , placing the cost of learning in the across-ensemble spike pattern rather than in the per-input distribution.
Spiking transformers achieve competitive accuracy with conventional transformers while offering 38-57× energy efficiency on neuromorphic hardware, yet no theoretical framework guides their design. This paper establishes the first comprehensive expressivity theory for spiking self-attention. We prove that spiking attention with Leaky Integrate-and-Fire neurons is a universal approximator of continuous permutation-equivariant functions, providing explicit spike circuit constructions including a novel lateral inhibition network for softmax normalization with proven O(1/T) convergence. We derive tight spike-count lower bounds via rate-distortion theory: ε-approximation requires Ω(Lf2nd/ε2) spikes, with rigorous information-theoretic derivation. Our key insight is input-dependent bounds using measured effective dimensions (deff=47--89 for CIFAR/ImageNet), explaining why T=4 timesteps suffice despite worst-case T≥10,000 predictions. We provide concrete design rules with calibrated constants (C=2.3, 95% CI: [1.9,2.7]). Experiments on Spikformer, QKFormer, and SpikingResformer across vision and language benchmarks validate predictions with R2=0.97 (p<0.001). Our framework provides the first principled foundation for neuromorphic transformer design.
Dongxin Guo, Jikun Wu, Siu Ming Yiu
*The University of Hong Kong, Hong Kong, China · †Brain Investing Limited, Hong Kong, China
We revisit the statistical mechanics of Transformer attention via modern Hopfield models. We present a free-energy formalism to characterize parameter regimes and critical thresholds for robust attention inference. The resulting free-energy landscape incorporates randomness in query states and attention weights. Attention then emerges as the equation of motion of the corresponding statistical-mechanical system (i.e., modern Hopfield model) through the standard free-energy minimization principle. This allows us to identify first-order phase transitions in attention between glassy and robust-inference phases. We further extend to attentions with stochastic queries and keys. Together, our results provide a rigorous statistical-mechanics framework for characterizing phase structure and inference reliability across a family of Transformer attention mechanisms.
Jerry Yao-Chieh Hu, Yu-An Chen, Maojiang Su +1
Center for Foundation Models and Generative AI, Northwestern University Department of Computer Science, Northwestern University · Department of Computer Science, College of Connected Computing, Vanderbilt University · Department of Physics, National Taiwan University +1
Integrating Spiking Neural Networks (SNNs) with Transformer architectures offers a promising pathway to balance energy efficiency and performance, particularly for edge vision applications. However, existing Spiking Transformers face two critical challenges: (i) a substantial performance gap compared to their Artificial Neural Networks (ANNs) counterparts and (ii) high memory overhead during inference. Through theoretical analysis, we attribute both limitations to the Spiking Self-Attention (SSA) mechanism: the lack of locality bias and the need to store large attention matrices. Inspired by the localized receptive fields (LRF) and membrane-potential dynamics of biological visual neurons, we propose LRF-Dyn, which uses spiking neurons with localized receptive fields to compute attention while reducing memory requirements. Specifically, we introduce a LRF method into SSA to assign higher weights to neighboring regions, strengthening local modeling and improving performance. Building on this, we approximate the resulting attention computation via charge-fire-reset dynamics, eliminating explicit attention-matrix storage and reducing inference-time memory. Extensive experiments on visual tasks confirm that our method reduces memory overhead while delivering significant performance improvements. These results establish it as a key unit for achieving energy-efficient Spiking Transformers.
Dehao Zhang, Fukai Guo, Shuai Wang +6
University of Electronic Science and Technology of China · Shenzhen Loop Area Institute · The Chinese University of Hong Kong (Shenzhen)