Authors: Jerry Yao-Chieh Hu, Yu-An Chen, Maojiang Su, Han Liu
Organizations: Center for Foundation Models and Generative AI, Northwestern University Department of Computer Science, Northwestern University · Department of Computer Science, College of Connected Computing, Vanderbilt University · Department of Physics, National Taiwan University · Department of Statistics and Data Science, Northwestern University
We revisit the statistical mechanics of Transformer attention via modern Hopfield models. We present a free-energy formalism to characterize parameter regimes and critical thresholds for robust attention inference. The resulting free-energy landscape incorporates randomness in query states and attention weights. Attention then emerges as the equation of motion of the corresponding statistical-mechanical system (i.e., modern Hopfield model) through the standard free-energy minimization principle. This allows us to identify first-order phase transitions in attention between glassy and robust-inference phases. We further extend to attentions with stochastic queries and keys. Together, our results provide a rigorous statistical-mechanics framework for characterizing phase structure and inference reliability across a family of Transformer attention mechanisms.
Figures & tables
Concept
Meaning for Attention
Dictionary
Fenchel Conjugate
It transforms a microscopic quantity into a macroscopic one, and constructs the free-energy objective.
It measures the alignment between queries and memories. A larger overpal indicates a stronger retrieval preference for that memory.
Definition 3.2
Free-energy Γα
It defines free-energy landscape and enables assigning energy value for each local minimum. It encodes equation of motion as attention rule.
Definition 2.1 , Propositions 3.2 and 3.3
Coexistence (Energy Crossing)
It occurs when two distinct stable retrieval states have the same free energy Γα simultaneously.
Definitions 4.2 and 4.1
Spinodal
It occurs when the energy barrier of a local mimimum disappers. This drives a retrieval state to the other.
Definitions 4.3 and 4.2
Table 1: Roadmap from the technical terms to their role in attention.
Figure 1: Illustration of the procedure from the microscopic to macroscopic energy landscapes. Left. “(Microscopic) Fluctuations” means x is random, has nonzero variance, and follows Gibbs distribution pJ(x) . Built from microscopic x , the local stable point xJ exhibits randomness. Middle. W[J]=log(∫Xexp{−βH[x]+Jx}) encodes macroscopic responses, i.e., φ=Ex∼pJ[x]=∇JW[J] . Hence, the slope of W[J] gives a deterministic average. Right. Γ[φ]=supJ{Jφ−W[J]} is the Fenchel conjugate of W[J] . Its response is J=∇φΓ[φ] , so local stable macroscopic points occurs at J=0 . Γ[φ] describes the energy landscape built from macroscopic φ after integrating out microscopic fluctuations.
Figure 2: Temperature analysis of the retrieval structure of MHM in the sparse regime ( α=2 ). (M=100,d=64) . Results are averaged over 10 random pattern realizations, and shaded regions indicate one standard deviation across different seeds. (a) Retrieval branches. The maximal overlap zmax⋆ under up- and down-sweeps of β exhibits hysteresis. (b) Support size. We compare the exact support size (active components of the probability distribution p ), the inverse participation ratio (IPR), and the fixed score-threshold count ( Section F.2 ). (c) Top branches. We track the top five stable branches ranked by zmax⋆ . (d) Branch energies. We show the corresponding free energies of those top five branches.
Figure 3: Temperature analysis of the retrieval structure of MHM in the dense regime ( α=1 ). (M=100,d=64) . Results are averaged over 10 random pattern realizations, and shaded regions indicate one standard deviation across different seeds. (a) Retrieval branches. The maximal overlap zmax⋆ under up- and down-sweeps of β shows no hysteresis. (b) Support size. We compare the exact support size (active components of the probability distribution p ) and the inverse participation ratio (IPR) ( Section F.2 ). (c) Top branches. We observe one single stable branch across the entire range of β . (d) Branch energies. We show the corresponding free energy of the branch.
Figure 4: Memory-load analysis of the retrieval structure of MHM in the sparse regime ( α=2 ). ( d=48,β=180 ). Results are averaged over 10 random pattern realizations, and shaded regions indicate one standard deviation across different seeds. (a) Retrieval branches. The maximal overlap zmax⋆ under up- and down-sweeps of κ shows hysteresis. (b) Support size. We compare the exact support size ∣supp(p⋆)∣ (active components of the probability distribution p ) and the inverse participation ratio (IPR) ( Section F.2 ). (c) Top branches. We track the top five stable branches ranked by zmax⋆ . (d) Branch energies. We plot the corresponding free energy of the five branches.
Figure 5: Memory-load analysis of the retrieval structure of MHM in the dense regime ( α=1 ). ( M=exp(κd),d=48,β=180 ). Results are averaged over random pattern realizations, and shaded regions indicate one standard deviation across different seeds. (a) Retrieval branches. The maximal overlap zmax⋆ under up- and down-sweeps of κ shows no hysteresis. (b) Support size. We compare the exact support size (active components of the probability distribution p ) and the inverse participation ratio (IPR) ( Section F.2 ). (c) Top branches. We track and observe only a single stable branch across the entire range of κ . (d) Branch energies. We plot the corresponding free energy of the branch.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Retrieval and frustration in the classical Hopfield picture. (a) From a noisy query set, we retrieve a stored pattern by minimizing H . Equivalently, the overlap increases during this associative retrieval. (b) Two ferromagnetic bonds and one antiferromagnetic bond on a triangle is impossible to satisfy pairwise energy minimization simultaneously. This is the simplest illustration of frustration.
Figure 7: One-pattern free-energy landscape in the large- d limit. The curves show the one-pattern Hopfield model free energy and its local minima. Cooling the system turns the minimum at q⋆=0 into two retrieval minima.
Figure 8: Phase-transition picture. (a) A metastable minimum is protected by a barrier. (b) We plot two smooth branches. In one smooth branch, the minimal Hessian eigenvalue vanishes at a spinodal. This leads to a jump onto another branch.
Figure 9: Selection of a stationary state by an external source. This figure illustrates how a source J selects one state from multiple degenerate stationary states of the free energy Γ[φ] . Left. When J=0 , Γ[φ] is symmetric and has two degenerate local minima ±φ⋆ . Middle. Turning on a small external source J>0 tilts the landscape to Γ[φ]−Jφ . This breaks the symmetry so that +φ⋆ becomes preferred. Then ,the system flows toward the selected basin. Right. After the selection, we take the limit J→0+ . Although the explicit source is removed, the system remains in the chosen local minimum +φ⋆ satisfying ∇φΓ[+φ⋆]=0 . This source-selection procedure resolves the degeneracy between degenerate stationary states and specifies the retrieval state.
Figure 10: Dynamical retrieval stability of MHM gradient descent in the sparse regime ( α=2 ). ( M=100,d=64 ). We study the gradient descent update x(t+1)=x(t)−η∇Γα(mod)(x(t)) . We denote the critical learning rate as ηcrit=2/λmax(∇2Γα(mod)(x⋆)) with λmax the largest eigenvalue at a fixed stable point x⋆ . (a) Distance to the fixed point. For η<ηcrit , the training converges to the fixed point. Otherwise, the dynamics is unstable or even diverges. Maximal overlap. For η<ηcrit , the training converges to the fixed value of overlap zmax≈0.0156 . It is the saturation value of one-pattern perfect retrieval in Figure 2 (a). Otherwise, the dynamics is unstable or even diverges.
Figure 11: Dynamical retrieval stability of MHM gradient descent in the dense regime ( α=1 ). ( M=100,d=64 ). We study the gradient descent update x(t+1)=x(t)−η∇Γα(mod)(x(t)) . We denote the critical learning rate as ηcrit=2/λmax(∇2Γα(mod)(x⋆)) with λmax the largest eigenvalue at a fixed stable point x⋆ . (a) Distance to the fixed point. For η<ηcrit , the training converges to the fixed point. For η>ηcrit , a divergence follows after a decay. This reflects that the iterate close to x⋆ initially. After this, the unstable eigen-directions start to dominate. We provide the precise explanation in this section. (b) Maximal overlap. For η<ηcrit , the training converges to the fixed value of overlap zmax . For η>ηcrit , a growing oscillations follow after an initial convergence. This is similar for the behavior in the distance plot.
Figure 12: Capacity-reliability trade-off for selective retrieval. We evaluate the trained Transformer at fixed βtrain=1 while increasing M . (a). We measure the value top-1 accuracy to characterize the retrieval reliability. Sparsemax retain the highest accuracy. Furthermore, their accuracy decreases as M grows, indicating the capacity-reliability trade-off. (b). Softmax support remains full, while sparsemax use the least supports and is the most support-efficient.
Figure 13: Capacity-reliability trade-off for uniform-aggregated retrieval. We evaluate the trained Transformer at fixed βtrain=1 while increasing M . (a). We measure the ratio of the output error and the zero-output baseline to characterize the retrieval reliability. Sparsemax exhibits the worst reliability. Furthermore, their retrieval reliabilities become worse as M grows, indicating the capacity-reliability trade-off. (b). Softmax support remains full, while sparsemax use the least supports and is the most support-efficient.
Figure 14: Temperature effect on frozen-parameter Transformer. We train the Transformer parameters at βtrain=1 , then we fix those parameters, set M=128 and vary β during inference. (a). The three attention families both exhibit high accuracy for a selective retrieval task. (b). For selective retrieval, the sparsemax uses the least effective support size (IPR), which shows the support-efficiency and reliability of high- β sparsemax. (c). For a uniform-aggregation task, the three attention families have worse error relative to the zero-output baseline as β increases. (d). The IPR relative to target-set size decline as β grows, which indicates that the retrieval becomes selective. This leads to a mismatch for multi-memory aggregation.
Figure 15: Temperature effect on matched-parameter Transformer. We train the Transformer parameters at each β , then we evaluate with M=128 during inference. (a). All three attention families maintain high accuracy in a selective retrieval task. (b). Sparsemax and entmax-1.5 are support-efficient in a selective retrieval task. (c). Softmax has the highest reliability among the three in a uniform-aggregation task. (d). Softmax uses numerous memories, which indicates a poor support-efficiency.
Figure 16: Two-key coexistence and spinodal validation. We consider two unit keys with correlation ϵ . (a). The solid blue curve presents βcoexist=2/(1−ϵ) . The numerical points are the root of the free energy difference ΔΓ(βcoexist)=0 . The numerical energy crossings coincide with the theoretical curve over the full correlation range. (b). We plot the relative errors of energy crossing (coexistence) and curvature zero (spinodal). The low values validate the theoretical fromulas in Proposition 4.1 and Proposition 4.2 .
Figure 17: General k -support spinodal validation. We test the general k -support spinodal formula βjump=2/λmax(PGP) . We consider 50 systems with unit keys. The numerical points agree with the theoretical values, which validates Proposition 4.2 .
Spiking Transformers achieve energy efficiency by discarding softmax normalization in favor of thresholded, binary attention. However, this removal breaks the thermodynamic framework used to analyze attention networks, as the normalized measure and partition function cease to exist. We restore this theoretical foundation by proving that softmax-free spiking attention operates exactly as the zero-temperature limit of a product of independent two-level systems. This correspondence places spiking attention strictly in the frozen ordered phase of the thermodynamic phase diagram for all inputs. Consequently, the entropic cost of organizing attention collapses, replaced by a Landauer-type physical floor governed by action-potential generation. Furthermore, this ordered-phase pinning mathematically explains the optimization difficulty caused by vanishing surrogate gradients. We validate this theory with forward-relaxation experiments, placing both standard and neuromorphic attention mechanisms on a unified thermodynamic axis.
Roberto C. Sotero, Pouya Taghipour, Jose M. Sanchez-Bornot
Department of Radiology, University of Calgary, Calgary, AB, Canada · Department of Biomedical Engineering, University of Calgary, Calgary, AB, Canada · Hotchkiss Brain Institute, University of Calgary, Calgary, AB, Canada +1
Attention is the key mechanism underlying in-context learning in transformers, and attention patterns have been observed empirically to emerge abruptly during training. We present a Bayesian theory of feature learning in attention; we then focus on how the copy subcircuit in the first layer of an induction head is learned by analyzing a single-layer softmax attention network trained on a copy task. We derive a closed-form posterior over the attention matrix and reduce it to a low-dimensional order parameter space. This reduction reveals a phase transition in the amount of training data, which we verify using both Bayesian sampling and standard training with Adam. We contrast our results with linear attention and find that softmax attention exhibits a first-order phase transition while in linear attention an initial second-order phase transition is followed by a smooth, continuous evolution toward the structured attention pattern (crossover). Our work provides a first-principles theoretical account of the abrupt emergence of the copy subcircuit, reminiscent of the one observed in training large language models.
Itay Lavie, Kirsten Fischer, Andrey Lekov +3
Racah Institute of Physics, Hebrew University of Jerusalem, Jerusalem 91904, Israel · John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, MA 02138, USA · Institute for Advanced Simulation (IAS-6), Computational and Systems Neuroscience, Jülich Research Center, Jülich 52428, Germany +3
We study minimal attention-only transformers under all-token corruption and show they admit a two-stage empirical Bayes interpretation. A single attention step computes a kernel-weighted posterior mean with respect to the empirical distribution defined by the context. Depth refines this distribution through particle dynamics (Stage 1), while a long-range skip-connection carries the noisy input as a query for posterior inference (Stage 2), revealing distinct statistical roles for depth and attention residuals. The framework isolates a minimal setting in which the context itself induces a depth-dependent energy landscape governing in-context inference. We show that effective denoising can emerge without an explicit noise schedule: a fixed kernel bandwidth and finite integration horizon suffice, yielding a principled depth-noise relationship. We further establish a posterior-mean recovery guarantee for a class of well-behaved priors, where the empirical estimator converges to the Bayes-optimal predictor under asymptotic conditions. Connecting these dynamics to reverse-diffusion limits, our results provide a statistical interpretation of attention as in-context inference via sample-based posterior estimation, without explicit density modeling.
Matthew Smart, Soumya Ganguly, Nilava Metya +2
Lewis-Sigler Institute for Integrative Genomics, Princeton University, Princeton, NJ, USA · Department of Mathematics, Rutgers University, Piscataway, NJ, USA · Department of Physics and Astronomy, Rutgers University, Piscataway, NJ, USA +1