Attention over noisy examples: generalization and memorization in the exponential Hopfield network

Francesco Alemanno

cond-mat.dis-nnOct 15, 2026

Attention over noisy examples: generalization and memorization in the exponential Hopfield network

Authors: Francesco Alemanno

Abstract

In Hebbian learning from examples, a Hopfield network retrieves archetypes seen only through noisy examples. We study it in the exponential Hopfield model: one softmax attention step with the unlabelled examples as keys. A signal-to-noise analysis gives the overlap with an unseen archetype in closed form: a majority-rule law in which an attention entropy replaces the dataset entropy. At high temperature the archetype is lost in the crosstalk; at low temperature the softmax freezes on the closest training example, as in a random energy model, and the layer memorizes. In between, in a window widening with the square root of the network size, the layer pools the examples of the correct class and retrieves the unseen archetype as well as their majority vote allows, at any fixed load. On structured datasets, such as MNIST and fashion-MNIST, the best accuracy lies beyond the freezing.

Figures & tables

Figure 1: The three regimes seen from one query. Attention weights ( 5 ) of the same query (a fresh example of archetype

\bm{\xi}^{1}

;

N=1000

K=20

M=16

\rho=0.2

) over the same training set at three inverse temperatures,

\beta=\beta_{\rm lo}/3

\sqrt{\beta_{\rm lo}\beta_{\rm hi}}

and

3\beta_{\rm hi}

, where

\beta_{\rm lo}

and

\beta_{\rm hi}

are the two edges of the generalization window derived in Sec. 3 , Eq. ( 18 ). In each panel the left part shows the

M

weights

w_{b}

of the keys of the correct class, the right part the total weight

V_{\nu}=\sum_{b\in\nu}w_{b}

of each of the other

K-1

classes (logarithmic scale, shared; bars below

10^{-5}

are not drawn).

W=\sum_{a}w_{1a}

, Eq. ( 8 ), is the sum of the blue bars and

M_{\rm eff}

, Eq. ( 9 ), their participation ratio;

m^{(1)}

and

n_{\max}

are the overlaps of the updated state with the archetype and with the closest example. (a) Crosstalk: the correct class weighs as much as any other. (b) Prototype: the correct class carries all the weight, spread over its examples. (c) Memorization: one example carries it, and is returned.

Figure 2: The three regimes in the $(\beta,\rho)$ plane, from the closed-form theory alone ( $N=1000$ , $K=100$ , $M=16$ ). (a) Crosstalk (second term of ( 17 ) above $\rho$ ), prototype and memorization ( $x>\sqrt{2\ln M}$ ) regions; solid and dashed lines are the edges ( 18 ); the dotted line is the position $\beta^{*}$ of the maximum of $m^{(1)}$ , obtained by maximizing ( 15 ) numerically and drawn where it lies below the condensation edge; above $\rho\simeq 1.3$ the maximum crosses the condensation edge, where the Gaussian closure is no longer reliable. The crosstalk edge moves with $\rho$ as $1/r^{2}$ , the condensation edge as $1/\sqrt{1-r^{2}}$ , and the drawn edges close at $\rho\simeq 1.8$ , Eq. ( 20 ) (with the self-coupling the closure moves to $\rho\simeq 1.4$ , since $K>K_{\times}$ there); at large $\rho$ the pooling loss is already appreciable at $\beta_{\rm lo}$ ( $x_{\rm lo}>1$ ), and the crosstalk edge marks the point where the crosstalk drops below $\rho$ , not a fixed value of $m^{(1)}$ . (b) One-step magnetization ( 15 ), with the contours $m^{(1)}=0.5$ , $0.7$ , $0.9$ : the prototype regime is where the layer pools the correct class, not where it necessarily retrieves the archetype, since $\rho_{\rm eff}\to\rho$ there; in the memorization region ( 15 ) overestimates $m^{(1)}$ , whose actual value tends to $r$ (Remark 3 ).

Figure 3: One attention step over the training set as a function of the inverse temperature

\beta

, for

N=1000

\alpha=0.1

and three datasets of increasing entropy (columns:

(\rho,M)=(0.05,32)

(0.2,16)

(0.5,8)

, i.e.

r=0.62

0.49

0.45

). Top: magnetization of the unseen archetype, theory ( 15 ) (line) versus Monte Carlo (full circles), and overlap

n_{\max}

with the closest training example (open squares); the dotted line is the quality

r

of a single example. Bottom: weight of the correct class

W

and pooled fraction

M_{\rm eff}/M

, theory ( 13 ), ( 14 ) (lines) versus Monte Carlo (symbols). The grey band is the analytic window ( 18 ). The three regimes of Corollary 1 are visible from left to right: crosstalk (

W\ll 1

), prototype (crosstalk below

\rho

M_{\rm eff}\simeq M

), memorization (

M_{\rm eff}\to 1

n_{\max}\to 1

m^{(1)}\to r

Figure 4: The load disappears. Retrieval of an unseen archetype from a fresh test example (

q=r

N=4000

M=16

\beta=0.012

; lines are theory, symbols Monte Carlo (3 samples

\times

6 queries, standard errors). (a) One-step magnetization against the load

\alpha

at three dataset entropies: the pairwise network, Eq. ( 10 ) (dashed lines, open squares), degrades as

\alpha

grows; the attention layer, Eqs. ( 15 )–( 16 ) (solid lines, full circles), does not. (b) Magnetization against

\rho

\alpha=0.2

: pairwise (open squares), attention after one step (full circles) and at the fixed point of the iterated map (triangles); the dotted line is the quality

r

of a single example. The iterated map retrieves the archetype only for

\rho\lesssim 0.06

and otherwise converges to a training example.

Figure 5: Size dependence of the generalization window at

\rho=0.2

M=16

\alpha=0.1

(lines: theory; symbols: Monte Carlo, 3 samples

\times

8 queries). (a) Against

\beta

the window widens with

N

. (b) Against

\beta N

the rising (crosstalk) edges collapse, up to the slow drift

\ln K=\ln(\alpha N)

of Eq. ( 18 ). (c) Against

\beta\sqrt{N}

the falling (condensation) edges collapse.

Figure 6: Condensation of the within-class softmax. The pooled fraction

M_{\rm eff}/M

measured in simulations with different

N

M

and

r

(symbols;

r=0.5

unless stated) collapses, as a function of the single variable

x=\beta\sqrt{N(1-r^{2})}

, on the participation ratio

\Phi_{M}(x)

of a random energy model with

M

levels (lines, Proposition 3 ); dotted verticals mark the REM freezing point

x_{c}=\sqrt{2\ln M}

Figure 7: All Monte Carlo points of Figs. 3 and 5 and of the $(\alpha,\rho)$ grid at $N=1000$ described in Appendix A, for the attention layer (every $\alpha$ , $\rho$ , $\beta$ , $N$ ; colour: pooled fraction $M_{\rm eff}/M$ ), plotted against the attention entropy $\rho_{\rm eff}$ evaluated from the measured weights, Eq. ( 16 ) with the self-coupling to first order, $\tau\to\beta$ ; the line is $\mathrm{erf}(1/\sqrt{2\rho_{\rm eff}})$ . Crosses: the pairwise network on the same grid, against the bare dataset entropy $\rho$ , which does not account for the crosstalk.

Figure 8: One attention step with keys

=

binarized MNIST and fashion-MNIST (F-MNIST) examples,

M=256

per class,

K=10

N=784

, queries from the test sets (3 samples

\times

400 queries). (a) Pooled fraction

M_{\rm eff}/M

against

x=\beta N\sigma_{m}

\sigma_{m}

being the measured standard deviation of the within-class overlaps; line:

\Phi_{256}(x)

of Proposition 3 ; dotted vertical:

\sqrt{2\ln M}

. (b) Distribution of the standardized within-class overlaps against the Gaussian of Lemma 1 . (c) Test accuracy of the one-step readout (class of largest total weight) against

\beta

; dashed: the

\beta\to\infty

limit, which returns the closest training example; dotted: the nearest majority-rule prototype,

\arg\max_{\mu}\bm{\sigma}\cdot\mathrm{sign}\bar{\bm{\eta}}^{\mu}

; shaded: the window ( 19 ) with the measured

\bar{m}

and

\sigma_{m}

(and

\rho=N\sigma_{m}^{2}/(M\bar{m})

); stars: the best accuracy over the

\beta

grid, measured on the test queries.

Figure 9: Attention head with keys

=

values and one slot per training example (

H=KM

), trained by gradient descent, without labels, on the denoising objective ( 22 ) (

r^{\prime}=0.5

), on the datasets of Definition 1 (

N=400

K=20

). In (a) and (b) full symbols refer to the trained head at its learned inverse temperature, crosses to the trained head read out at a hand-set

\beta_{2}

, large open symbols to the Hebbian head with keys

=

examples at its best

\beta

; colours and shapes distinguish

M

. (a)

m^{(1)}

on fresh examples; dotted: majority-rule value

\mathrm{erf}(1/\sqrt{2\rho})

. (b) Inverse temperatures on the phase map of Eq. ( 18 ) for each

M

(solid: crosstalk edge; dashed: condensation edge; shaded: window); the learned one is in the units of the Hebbian head. (c) Largest cosine of the learned slots with the archetypes (full symbols) and with the training examples (open symbols), averaged over the slots with weights given by their mean attention on the test queries.

Hopfield Networks

Hopfield networks are energy-based recurrent models that store patterns as attractors and retrieve them from noisy or partial cues.

See all 40 papers

Sep 10, 2026

Phases in a class of associative memories via hidden neurons

Associative memory in the Hopfield network is attractor dynamics in a disordered many-body system, and higher-order and exponential extensions turn its retrieval update into softmax attention. The polynomial and exponential regimes have been analyzed by different methods, with no common architecture in which to ask what fixes the storage scale. In this paper we study the bipartite architecture of Krotov and Hopfield, which we call the class

H

, whose model is fixed by a Lagrangian for each layer, taking the hidden neurons as the order parameter of retrieval. At polynomial load the replica method yields the replica-symmetric phase diagrams and closed-form capacities, and the crosstalk moment is common to Ising and spherical visible neurons, so their differences come from the visible entropy. With a softmax hidden layer the load is exponential, and a copy representation maps the thermodynamics onto random-energy-model counting, with paramagnetic, condensed, and frozen phases. Heating destabilizes retrieval by quantized reassignments of attention, and typical Gaussian patterns remain metastable at every load. The regimes differ in their crosstalk statistics, central-limit at polynomial load and large-deviation at exponential load, and the class

H

splits retrieval into two roles, the visible Lagrangian fixing stability and the hidden one the storage scale, two axes that may also guide the design of new Lagrangians.

Toshihiro Ota, Masato Taki

CyberAgent AI Lab, Shibuya, Tokyo 150–0002, Japan · RIKEN iTHEMS, Wako, Saitama 351–0198, Japan · Graduate School of Artificial Intelligence and Science, Rikkyo University, Toshima, Tokyo 171–8501, Japan

Phase Transitions Hopfield Networks Associative Memory Phase Transitions in Neural Networks

Ask PDF

Sep 10, 2026

Hierarchical Prototype Emergence in Modern Hopfield Models

Hierarchical correlations are a universal feature of any realistic model of data, and the question of how associative memory models may learn these correlations and generalize beyond them to construct new sensible images is an important step towards understanding more complex modern architectures such as diffusion models. We consider a hierarchical model for memories which are sampled and stored in a dense Hopfield network with polynomial activation. We analytically derive conditions for each level of this hierarchy to be locally stable - that is they are local energy minima. We use prototype reconstruction as a minimal model of generalization and we find that it takes only a quasi-polynomial amount of information to generalize beyond particular memories and even particular groups in the hierarchy. We observe a qualitatively analogous phase diagram in the number of memories, sharpness of the activation function (polynomial degree) for data from Fashion-MNIST.

Aditya Cowsik, Adithya Sriram

Department of Physics, Stanford University, Stanford, CA 94305, USA

Neural Network Generalization Hopfield Networks Hierarchical Memory Associative Memory

Ask PDF

Dec 16, 2025

Implicit Bias and Invariance: How Hopfield Networks Efficiently Learn Graph Orbits

Many learning problems are organized by group symmetries. While invariance is often imposed through architectures or group averaging, we ask when it can emerge from training on a finite random subset of an orbit. We study this question in classical Hopfield networks, where strict memorization can be expressed as a linear margin problem. Reparameterizing minimization of energy flow (MEF) as an exponential loss connects gradient descent to the corresponding minimum-norm hard-margin memorizer. Our main result shows that, for independent uniform samples from any finite permutation orbit, the exact sample hard-margin support vector machine (HSVM) concentrates exponentially around the invariant full-orbit HSVM. Consequently, an orbit-size-independent polynomial number of samples suffices both for approximate parameter invariance and for simultaneous memorization of every orbit element; directional convergence transfers this conclusion asymptotically to MEF gradient descent. For graph-isomorphism orbits, we characterize the invariant parameters as a three-dimensional subspace and show that every such orbit is memorizable. For cliques of fixed linear density, additional symmetry sharpens the uniform memorization bound to

O(v^4\log(1/δ))

, exponentially smaller than the orbit size. Together with experiments across several learning rules, these results give a finite-sample account of how optimization bias can recover symmetry from partial group-structured data.

Michael Murray, Tenzin Chan, Kedar Karhadker +1

Mathematical Sciences, University of Bath · Department of Mathematics, UCLA · Algebraic

Sample Complexity Neural Network Memorization Hopfield Networks Implicit Bias Permutation-Invariant Learning Invariant Representation Learning

Ask PDF

Abstract

Hopfield Networks

Explore similar work

Phases in a class of associative memories via hidden neurons

Hierarchical Prototype Emergence in Modern Hopfield Models

Implicit Bias and Invariance: How Hopfield Networks Efficiently Learn Graph Orbits