On the Statistical Mechanics of Attention

Jerry Yao-Chieh Hu; Yu-An Chen; Maojiang Su; Han Liu

cs.LGOct 12, 2026

On the Statistical Mechanics of Attention

Authors: Jerry Yao-Chieh Hu, Yu-An Chen, Maojiang Su, Han Liu

Organizations: Center for Foundation Models and Generative AI, Northwestern University Department of Computer Science, Northwestern University · Department of Computer Science, College of Connected Computing, Vanderbilt University · Department of Physics, National Taiwan University · Department of Statistics and Data Science, Northwestern University

Abstract

We revisit the statistical mechanics of Transformer attention via modern Hopfield models. We present a free-energy formalism to characterize parameter regimes and critical thresholds for robust attention inference. The resulting free-energy landscape incorporates randomness in query states and attention weights. Attention then emerges as the equation of motion of the corresponding statistical-mechanical system (i.e., modern Hopfield model) through the standard free-energy minimization principle. This allows us to identify first-order phase transitions in attention between glassy and robust-inference phases. We further extend to attentions with stochastic queries and keys. Together, our results provide a rigorous statistical-mechanics framework for characterizing phase structure and inference reliability across a family of Transformer attention mechanisms.

Figures & tables

Concept	Meaning for Attention	Dictionary
Fenchel Conjugate	It transforms a microscopic quantity into a macroscopic one, and constructs the free-energy objective.	Definitions C.6 , 2.1 and 2.1
Tsallis Entropy $\Psi_{\alpha}$	Attention Family: softmax $(\alpha=1)$ , entmax $(1<\alpha<2)$ , and sparsemax $(\alpha=2)$ .	Sections C.3 , 2.1 and 3.3
Query-Key Similarity (Overlap $z=\frac{1}{\sqrt{d}}\xi^{\top}x$ )	It measures the alignment between queries and memories. A larger overpal indicates a stronger retrieval preference for that memory.	Definition 3.2
Free-energy $\Gamma_{\alpha}$	It defines free-energy landscape and enables assigning energy value for each local minimum. It encodes equation of motion as attention rule.	Definition 2.1 , Propositions 3.2 and 3.3
Coexistence (Energy Crossing)	It occurs when two distinct stable retrieval states have the same free energy $\Gamma_{\alpha}$ simultaneously.	Definitions 4.2 and 4.1
Spinodal	It occurs when the energy barrier of a local mimimum disappers. This drives a retrieval state to the other.	Definitions 4.3 and 4.2

Table 1: Roadmap from the technical terms to their role in attention.

Panel of Figure 1 — Figure 1: Illustration of the procedure from the microscopic to macroscopic energy landscapes. Left. “(Microscopic) Fluctuations” means $x$ is random, has nonzero variance, and follows Gibbs distribution $p_{J}(x)$ . Built from microscopic $x$ , the local stable point $x_{J}$ exhibits randomness. Middle. $W[J]=\log(\int_\mathcal{X}\exp\{-\beta\mathcal{H}[x]+Jx\})$ encodes macroscopic responses, i.e., $\varphi=\mathbb{E}_{x\sim p_{J}}[x]=\nabla_{J}W[J]$ . Hence, the slope of $W[J]$ gives a deterministic average. Right. $\Gamma[\varphi]=\sup_{J}\{J\varphi-W[J]\}$ is the Fenchel conjugate of $W[J]$ . Its response is $J=\nabla_{\varphi}\Gamma[\varphi]$ , so local stable macroscopic points occurs at $J=0$ . $\Gamma[\varphi]$ describes the energy landscape built from macroscopic $\varphi$ after integrating out microscopic fluctuations.

Figure 2: Temperature analysis of the retrieval structure of MHM in the sparse regime (

\alpha=2

(M=100,d=64)

. Results are averaged over 10 random pattern realizations, and shaded regions indicate one standard deviation across different seeds. (a) Retrieval branches. The maximal overlap

z_{\text{max}}^{\star}

under up- and down-sweeps of

\beta

exhibits hysteresis. (b) Support size. We compare the exact support size (active components of the probability distribution

p

), the inverse participation ratio (IPR), and the fixed score-threshold count ( Section F.2 ). (c) Top branches. We track the top five stable branches ranked by

z_{\text{max}}^{\star}

. (d) Branch energies. We show the corresponding free energies of those top five branches.

Figure 3: Temperature analysis of the retrieval structure of MHM in the dense regime (

\alpha=1

(M=100,d=64)

. Results are averaged over 10 random pattern realizations, and shaded regions indicate one standard deviation across different seeds. (a) Retrieval branches. The maximal overlap

z_{\text{max}}^{\star}

under up- and down-sweeps of

\beta

shows no hysteresis. (b) Support size. We compare the exact support size (active components of the probability distribution

p

) and the inverse participation ratio (IPR) ( Section F.2 ). (c) Top branches. We observe one single stable branch across the entire range of

\beta

. (d) Branch energies. We show the corresponding free energy of the branch.

Figure 4: Memory-load analysis of the retrieval structure of MHM in the sparse regime (

\alpha=2

). (

d=48,\beta=180

). Results are averaged over 10 random pattern realizations, and shaded regions indicate one standard deviation across different seeds. (a) Retrieval branches. The maximal overlap

z_{\text{max}}^{\star}

under up- and down-sweeps of

\kappa

shows hysteresis. (b) Support size. We compare the exact support size

|\text{supp}(p^{\star})|

(active components of the probability distribution

p

) and the inverse participation ratio (IPR) ( Section F.2 ). (c) Top branches. We track the top five stable branches ranked by

z^{\star}_{\text{max}}

. (d) Branch energies. We plot the corresponding free energy of the five branches.

Figure 5: Memory-load analysis of the retrieval structure of MHM in the dense regime (

\alpha=1

). (

M=\exp(\kappa d),d=48,\beta=180

). Results are averaged over random pattern realizations, and shaded regions indicate one standard deviation across different seeds. (a) Retrieval branches. The maximal overlap

z_{\text{max}}^{\star}

under up- and down-sweeps of

\kappa

shows no hysteresis. (b) Support size. We compare the exact support size (active components of the probability distribution

p

) and the inverse participation ratio (IPR) ( Section F.2 ). (c) Top branches. We track and observe only a single stable branch across the entire range of

\kappa

. (d) Branch energies. We plot the corresponding free energy of the branch.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6: Retrieval and frustration in the classical Hopfield picture. (a) From a noisy query set, we retrieve a stored pattern by minimizing $\mathcal{H}$ . Equivalently, the overlap increases during this associative retrieval. (b) Two ferromagnetic bonds and one antiferromagnetic bond on a triangle is impossible to satisfy pairwise energy minimization simultaneously. This is the simplest illustration of frustration.

Panel of Figure 7 — Figure 7: One-pattern free-energy landscape in the large- $d$ limit. The curves show the one-pattern Hopfield model free energy and its local minima. Cooling the system turns the minimum at $q^{\star}=0$ into two retrieval minima.

Panel of Figure 8 — Figure 8: Phase-transition picture. (a) A metastable minimum is protected by a barrier. (b) We plot two smooth branches. In one smooth branch, the minimal Hessian eigenvalue vanishes at a spinodal. This leads to a jump onto another branch.

Panel of Figure 9 — Figure 9: Selection of a stationary state by an external source. This figure illustrates how a source $J$ selects one state from multiple degenerate stationary states of the free energy $\Gamma[\varphi]$ . Left. When $J=0$ , $\Gamma[\varphi]$ is symmetric and has two degenerate local minima $\pm\varphi^{\star}$ . Middle. Turning on a small external source $J>0$ tilts the landscape to $\Gamma[\varphi]-J\varphi$ . This breaks the symmetry so that $+\varphi^{\star}$ becomes preferred. Then ,the system flows toward the selected basin. Right. After the selection, we take the limit $J\to 0^{+}$ . Although the explicit source is removed, the system remains in the chosen local minimum $+\varphi^{\star}$ satisfying $\nabla_{\varphi}\Gamma[+\varphi^{\star}]=0$ . This source-selection procedure resolves the degeneracy between degenerate stationary states and specifies the retrieval state.

Figure 10: Dynamical retrieval stability of MHM gradient descent in the sparse regime (

\alpha=2

). (

M=100,d=64

). We study the gradient descent update

x(t+1)=x(t)-\eta\nabla\Gamma_{\alpha}^{\text{(mod)}}(x(t))

. We denote the critical learning rate as

\eta_{\text{crit}}=2/\lambda_{\text{max}}(\nabla^{2}\Gamma_{\alpha}^{(\text{mod})}(x^{\star}))

with

\lambda_{\text{max}}

the largest eigenvalue at a fixed stable point

x^{\star}

. (a) Distance to the fixed point. For

\eta<\eta_{\text{crit}}

, the training converges to the fixed point. Otherwise, the dynamics is unstable or even diverges. Maximal overlap. For

\eta<\eta_{\text{crit}}

, the training converges to the fixed value of overlap

z_{\text{max}}\approx 0.0156

. It is the saturation value of one-pattern perfect retrieval in Figure 2 (a). Otherwise, the dynamics is unstable or even diverges.

Figure 11: Dynamical retrieval stability of MHM gradient descent in the dense regime (

\alpha=1

). (

M=100,d=64

). We study the gradient descent update

x(t+1)=x(t)-\eta\nabla\Gamma_{\alpha}^{\text{(mod)}}(x(t))

. We denote the critical learning rate as

\eta_{\text{crit}}=2/\lambda_{\text{max}}(\nabla^{2}\Gamma_{\alpha}^{(\text{mod})}(x^{\star}))

with

\lambda_{\text{max}}

the largest eigenvalue at a fixed stable point

x^{\star}

. (a) Distance to the fixed point. For

\eta<\eta_{\text{crit}}

, the training converges to the fixed point. For

\eta>\eta_{\text{crit}}

, a divergence follows after a decay. This reflects that the iterate close to

x^{\star}

initially. After this, the unstable eigen-directions start to dominate. We provide the precise explanation in this section. (b) Maximal overlap. For

\eta<\eta_{\text{crit}}

, the training converges to the fixed value of overlap

z_{\text{max}}

. For

\eta>\eta_{\text{crit}}

, a growing oscillations follow after an initial convergence. This is similar for the behavior in the distance plot.

Figure 12: Capacity-reliability trade-off for selective retrieval. We evaluate the trained Transformer at fixed

\beta_{\text{train}}=1

while increasing

M

. (a). We measure the value top-1 accuracy to characterize the retrieval reliability. Sparsemax retain the highest accuracy. Furthermore, their accuracy decreases as

M

grows, indicating the capacity-reliability trade-off. (b). Softmax support remains full, while sparsemax use the least supports and is the most support-efficient.

Figure 13: Capacity-reliability trade-off for uniform-aggregated retrieval. We evaluate the trained Transformer at fixed

\beta_{\text{train}}=1

while increasing

M

. (a). We measure the ratio of the output error and the zero-output baseline to characterize the retrieval reliability. Sparsemax exhibits the worst reliability. Furthermore, their retrieval reliabilities become worse as

M

grows, indicating the capacity-reliability trade-off. (b). Softmax support remains full, while sparsemax use the least supports and is the most support-efficient.

Figure 14: Temperature effect on frozen-parameter Transformer. We train the Transformer parameters at

\beta_{\text{train}}=1

, then we fix those parameters, set

M=128

and vary

\beta

during inference. (a). The three attention families both exhibit high accuracy for a selective retrieval task. (b). For selective retrieval, the sparsemax uses the least effective support size (IPR), which shows the support-efficiency and reliability of high-

\beta

sparsemax. (c). For a uniform-aggregation task, the three attention families have worse error relative to the zero-output baseline as

\beta

increases. (d). The IPR relative to target-set size decline as

\beta

grows, which indicates that the retrieval becomes selective. This leads to a mismatch for multi-memory aggregation.

Figure 15: Temperature effect on matched-parameter Transformer. We train the Transformer parameters at each

\beta

, then we evaluate with

M=128

during inference. (a). All three attention families maintain high accuracy in a selective retrieval task. (b). Sparsemax and entmax-1.5 are support-efficient in a selective retrieval task. (c). Softmax has the highest reliability among the three in a uniform-aggregation task. (d). Softmax uses numerous memories, which indicates a poor support-efficiency.

Figure 16: Two-key coexistence and spinodal validation. We consider two unit keys with correlation

\epsilon

. (a). The solid blue curve presents

\beta_{\text{coexist}}=2/(1-\epsilon)

. The numerical points are the root of the free energy difference

\Delta\Gamma(\widehat{\beta}_{\text{coexist}})=0

. The numerical energy crossings coincide with the theoretical curve over the full correlation range. (b). We plot the relative errors of energy crossing (coexistence) and curvature zero (spinodal). The low values validate the theoretical fromulas in Proposition 4.1 and Proposition 4.2 .

Figure 17: General

\bm{k}

-support spinodal validation. We test the general

k

-support spinodal formula

\beta_{\text{jump}}=2/\lambda_{\text{max}}(PGP)

. We consider 50 systems with unit keys. The numerical points agree with the theoretical values, which validates Proposition 4.2 .

Explore similar work

Softmax-Free Attention in Spiking Transformers as the Zero-Temperature Limit of Two-Level Systems
Oct 11, 2026Roberto C. Sotero, Pouya Taghipour, Jose M. Sanchez-BornotSpiking Transformers Attention Mechanisms
Department of Radiology, University of Calgary, Calgary, AB, Canada · Department of Biomedical Engineering, University of Calgary, Calgary, AB, Canada · Hotchkiss Brain Institute, University of Calgary, Calgary, AB, Canada +1
Phase Transitions in Attention: A Bayesian Theory of Copy Head Emergence
Jun 10, 2026Itay Lavie, Kirsten Fischer, Andrey Lekov +3Softmax Attention Phase Transitions
Racah Institute of Physics, Hebrew University of Jerusalem, Jerusalem 91904, Israel · John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, MA 02138, USA · Institute for Advanced Simulation (IAS-6), Computational and Systems Neuroscience, Jülich Research Center, Jülich 52428, Germany +3
Attention as In-Context Empirical Bayes: A Two-Stage View via Particle Dynamics
May 28, 2026Matthew Smart, Soumya Ganguly, Nilava Metya +2Self-Attention Empirical Bayes
Lewis-Sigler Institute for Integrative Genomics, Princeton University, Princeton, NJ, USA · Department of Mathematics, Rutgers University, Piscataway, NJ, USA · Department of Physics and Astronomy, Rutgers University, Piscataway, NJ, USA +1

Oct 11, 2026J/K move · Enter open · S save

Softmax-Free Attention in Spiking Transformers as the Zero-Temperature Limit of Two-Level Systems

Roberto C. Sotero, Pouya Taghipour, Jose M. Sanchez-Bornot

Department of Radiology, University of Calgary, Calgary, AB, Canada · Department of Biomedical Engineering, University of Calgary, Calgary, AB, Canada · Hotchkiss Brain Institute, University of Calgary, Calgary, AB, Canada+1

Spiking Transformers Attention Mechanisms Neural Network Training Dynamics

Open paper AskPDF arXiv

Spiking Transformers achieve energy efficiency by discarding softmax normalization in favor of thresholded, binary attention. However, this removal breaks the thermodynamic framework used to analyze attention networks, as the normalized measure and partition function cease to exist. We restore this theoretical foundation by proving that softmax-free spiking attention operates exactly as the zero-temperature limit of a product of independent two-level systems. This correspondence places spiking attention strictly in the frozen ordered phase of the thermodynamic phase diagram for all inputs. Consequently, the entropic cost of organizing attention collapses, replaced by a Landauer-type physical floor governed by action-potential generation. Furthermore, this ordered-phase pinning mathematically explains the optimization difficulty caused by vanishing surrogate gradients. We validate this theory with forward-relaxation experiments, placing both standard and neuromorphic attention mechanisms on a unified thermodynamic axis.

Abstract

Explore similar work

Softmax-Free Attention in Spiking Transformers as the Zero-Temperature Limit of Two-Level Systems

Phase Transitions in Attention: A Bayesian Theory of Copy Head Emergence

Attention as In-Context Empirical Bayes: A Two-Stage View via Particle Dynamics