cs.LGOct 11, 2026

Softmax-Free Attention in Spiking Transformers as the Zero-Temperature Limit of Two-Level Systems

Authors: Roberto C. Sotero, Pouya Taghipour, Jose M. Sanchez-Bornot

Organizations: Department of Radiology, University of Calgary, Calgary, AB, Canada · Department of Biomedical Engineering, University of Calgary, Calgary, AB, Canada · Hotchkiss Brain Institute, University of Calgary, Calgary, AB, Canada · Intelligent Systems Research Center, Ulster University, Derry Londonderry, Northern Ireland, United Kingdom

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Spiking Transformers achieve energy efficiency by discarding softmax normalization in favor of thresholded, binary attention. However, this removal breaks the thermodynamic framework used to analyze attention networks, as the normalized measure and partition function cease to exist. We restore this theoretical foundation by proving that softmax-free spiking attention operates exactly as the zero-temperature limit of a product of independent two-level systems. This correspondence places spiking attention strictly in the frozen ordered phase of the thermodynamic phase diagram for all inputs. Consequently, the entropic cost of organizing attention collapses, replaced by a Landauer-type physical floor governed by action-potential generation. Furthermore, this ordered-phase pinning mathematically explains the optimization difficulty caused by vanishing surrogate gradients. We validate this theory with forward-relaxation experiments, placing both standard and neuromorphic attention mechanisms on a unified thermodynamic axis.

Figures & tables

Figure 1: The spiking-Boltzmann correspondence. Each column is read from top to bottom as a derivation, from the underlying ensemble to its Boltzmann measure. A : softmax attention. The NN keys attended to by a query form a single coupled NN -state categorical system. The scores sij=qi⊤kj/dks_{ij}=\mathbf{q}_{i}^{\top}\mathbf{k}_{j}/\sqrt{d_{k}} define the interaction energy Eij=−sijE_{ij}=-s_{ij} , a single global temperature TT fixes the inverse temperature β=1/T\beta=1/T , the shared normaliser Zi=∑je−βEijZ_{i}=\sum_{j}e^{-\beta E_{ij}} couples the keys, and the attention weights are the Boltzmann measure Aij(T)=e−βEij/ZiA_{ij}(T)=e^{-\beta E_{ij}}/Z_{i} of Equation ( 1 ), with operating point β=1\beta=1 . B : softmax-free spiking attention. Deleting the shared normaliser decouples the keys into a product of NN independent two-level systems, one per query-key pair, each with energy gap Δij=ϑ−γsij\Delta_{ij}=\vartheta-\gamma s_{ij} and partition function zij=1+e−βsΔijz_{ij}=1+e^{-\beta_{s}\Delta_{ij}} . The single temperature splits into a forward value Ts=1/βsT_{s}=1/\beta_{s} and a backward value Teff=1/αT_{eff}=1/\alpha set by the surrogate slope. The two-level occupation pij=e−βsΔij/zij=σ(βs(γsij−ϑ))p_{ij}=e^{-\beta_{s}\Delta_{ij}}/z_{ij}=\sigma(\beta_{s}(\gamma s_{ij}-\vartheta)) is the counterpart of AijA_{ij} , and the deployed hard spike aij=Θ(γsij−ϑ)∈{0,1}a_{ij}=\Theta(\gamma s_{ij}-\vartheta)\in\{0,1\} is its zero-temperature limit Ts→0T_{s}\to 0 (Equation ( 8 )). Softmax attention therefore operates at a finite, entropy-bearing temperature, whereas spiking attention is the zero-temperature limit of the corresponding two-level Boltzmann measure.
QuantitySymbolValue
Dataset CIFAR-10
Image size 32×32×332\times 32\times 3
Patch size 44
Embedding widthdmodeld_{\text{model}}6464
Attention heads 44
Transformer blocks 22
Table 1: Experimental configuration. The surrogate slope α\alpha is fixed for the relaxation experiment (Figure 2 ) and swept for the backward-temperature experiment (Figure 3 ).
Figure 2: Forward relaxation of the matched models on CIFAR-10. The trained spiking operator (red) is shown against the two-level inverse temperature βs\beta_{s} and the softmax baseline (blue) against a swept temperature TT , on a shared axis. A : order parameter. The softmax model sits at the disordering crossover at its native point T=1T=1 (dotted line), while the spiking model is pinned near ϕ(2)=1\phi^{(2)}=1 at its operating point βs→∞\beta_{s}\to\infty (star), validating the ordered-phase pinning of Equation ( 10 ); the comparison concerns operating-point placement, not curve identity. B : specific heat. The spiking heat capacity shows an interior Schottky anomaly peaking at TcT_{c} (dashed line), and the operating point at T→0T\to 0 lies well cold of TcT_{c} . The softmax peak is larger because it sums the energy variance over all NN keys, whereas Cv(2)C_{v}^{(2)} is the heat capacity of one binary unit per pair.
Figure 3: Surrogate slope as the backward temperature on CIFAR-10. A : test accuracy against α=1/Teff\alpha=1/T_{\text{eff}} , with error bars over seeds. Accuracy peaks across a broad intermediate plateau and collapses as α\alpha grows: large α\alpha is the cold backward temperature that starves the gradient, and the smallest α\alpha is the biased high-temperature side. B : active-band fraction, the fraction of pairs with ∣γsij−ϑ∣<1/α|\gamma s_{ij}-\vartheta|<1/\alpha , against α\alpha on logarithmic axes; the power-law fit (slope in the legend) confirms the O(1/α)O(1/\alpha) surrogate support of Equation ( 9 ). The fraction is averaged over the attention array, so its across-seed spread is below the marker size. Panels A and B give mechanism and consequence: as the band narrows as 1/α1/\alpha , trainability is lost.
Figure 4: Entropy dynamics of the spiking model during training on CIFAR-10, shown as the mean over KK seeds with error bars. A : the per-input two-level entropy Sˉ(2)\bar{S}^{(2)} (blue), evaluated at the backward temperature βs=α\beta_{s}=\alpha , and the across-ensemble spike-pattern entropy SspkS_{spk} of Equation ( 12 ) (red), against training epoch; the dotted line is the maximum binary entropy ln⁡2\ln 2 . The per-input entropy stays near zero for the whole run, so the attention pattern is frozen in the ordered phase at every epoch and the pinning of Equation ( 10 ) holds along the trajectory, whereas the across-ensemble entropy starts substantial and falls steadily, representing the informative reorganisation that learning performs. The across-seed spread of Sˉ(2)\bar{S}^{(2)} is below the marker, as for the averaged observables of Figure 2 . B : the evaluation loss over the same window. The loss falls together with SspkS_{spk} , placing the cost of learning in the across-ensemble spike pattern rather than in the per-input distribution.
More in this topic

Spiking Transformers

Spiking Transformers combine attention-based architectures with event-driven spiking neurons to process information using sparse, temporally coded computation.

See all 42 papers

Explore similar work

CardsList

  1. Closing the Theory-Practice Gap in Spiking Transformers via Effective Dimension

    Apr 17, 2026Dongxin Guo, Jikun Wu, Siu Ming YiuSpiking TransformersTransformer Attention

    *The University of Hong Kong, Hong Kong, China · †Brain Investing Limited, Hong Kong, China

  2. On the Statistical Mechanics of Attention

    Oct 12, 2026Jerry Yao-Chieh Hu, Yu-An Chen, Maojiang Su +1Transformer AttentionAttention Mechanisms

    Center for Foundation Models and Generative AI, Northwestern University Department of Computer Science, Northwestern University · Department of Computer Science, College of Connected Computing, Vanderbilt University · Department of Physics, National Taiwan University +1

  3. Neural Dynamics Self-Attention for Spiking Transformers

    Mar 9, 2026Dehao Zhang, Fukai Guo, Shuai Wang +6Spiking TransformersSpiking Neural Networks

    University of Electronic Science and Technology of China · Shenzhen Loop Area Institute · The Chinese University of Hong Kong (Shenzhen)