Basin-Preserving Discretizations of Modern Hopfield Retrieval Dynamics: Energy Cells, Dissipation, and the Attention Limit

Francisco R. Villatoro

math.NAAug 21, 2026

Basin-Preserving Discretizations of Modern Hopfield Retrieval Dynamics: Energy Cells, Dissipation, and the Attention Limit

Authors: Francisco R. Villatoro

Organizations: Dept. de Lenguajes y Ciencias de la Computación, Universidad de Málaga, Spain

Abstract

The retrieval dynamics of a modern Hopfield network is the gradient flow of a log-sum-exp energy, while the attention update is its exact difference-of-convex minimization step. We study which time discretizations preserve not only energy decay and equilibria but also basins of attraction. We introduce energy cells, connected components of sublevel sets containing one attractor and no other critical point. Our main theorem shows that every finite energy cell below the escape energy is contained simultaneously in the basin of the continuous flow, every relaxed attention map $Ψ_θ=(1-θ)\,\mathrm{id}+θ\,\mathrm{attention}$ for $0<θ<2$ , and implicit Euler throughout its uniqueness regime. A parameter-uniform unit-curvature majorant yields unconditional dissipation and a monotone interpolation of each discrete step. We also derive explicit local contraction bounds near well-separated patterns, with a certified optimal slight overrelaxation; characterize proximal tunneling and overshoot beyond the preservation regimes; compare first-order error constants; establish an order barrier for scalar reparametrizations of the relaxed family; construct a second-order scalar-auxiliary-variable scheme; and extend cell preservation to damped difference-of-convex iterations in Bregman geometry, including a certified overrelaxed window under bounded asymmetry. Nine numerical campaigns test the bounds and failure mechanisms. In two-dimensional basin experiments, all observed disagreements between continuous and discrete retrieval occur above the attractor-specific numerically inferred escape level.

Figures & tables

scheme	certified	observed	scheme	certified	observed
$\Psi_{0.25}$	$0.7608$	$0.7500$	IE, $\Delta t=0.25$	$0.8069$	$0.8000$
$\Psi_{0.5}$	$0.5215$	$0.5002$	IE, $\Delta t=1$	$0.5110$	$0.5000$
$\Psi_{1}$	$0.0430$	$0.0026$	IE, $\Delta t=4$	$0.2071$	$0.2001$
			IE, $\Delta t=16$	$0.0613$	$0.0588$

Table 1: Certified contraction factors

q_{\theta}=1-\theta(1-\ell_{\mu}(r))

and

\kappa_{\Delta t}=(1+\Delta t(1-\ell_{\mu}(r)))^{-1}

against worst observed per-step factors near

x^{\ast}_{1}

(

d=64

N=4

\beta=0.15

r=0.8

	$\Delta t=0.4$	$0.2$	$0.1$	$0.05$	$0.025$	EOC
EE	$1.10\times 10^{-2}$	$6.11\times 10^{-3}$	$3.18\times 10^{-3}$	$1.62\times 10^{-3}$	$8.17\times 10^{-4}$	$0.85\to 0.99$
CS	$1.46\times 10^{-2}$	$6.99\times 10^{-3}$	$3.40\times 10^{-3}$	$1.68\times 10^{-3}$	$8.31\times 10^{-4}$	$1.07\to 1.01$
ETD	$5.06\times 10^{-6}$	$2.36\times 10^{-6}$	$1.14\times 10^{-6}$	$5.62\times 10^{-7}$	$2.79\times 10^{-7}$	$1.10\to 1.01$
SAV	$2.23\times 10^{-3}$	$5.34\times 10^{-4}$	$1.33\times 10^{-4}$	$3.32\times 10^{-5}$	$8.32\times 10^{-6}$	$2.06\to 2.00$

Table 2: Global trajectory errors at

T=4

against an RK4 reference, and empirical orders of convergence across the step-size range.

Figure 1: Campaign E. Basins of attraction of the three attractors for the flow, computed by adaptive Runge–Kutta integration at tolerance $10^{-11}$ (left), and for $\Psi_{1.9}$ (right), on a $301\times 301$ grid; triangles mark the patterns, circles the minima, crosses the index-one saddles, plus the maximum. Solid curves are the stable manifolds of the saddles, the numerically resolved continuous separatrices; dashed curves are the numerically inferred level sets $E=\widehat{c}^{\,\ast}$ of each attractor. Every basin discrepancy between the two panels lies outside the dashed level set of the attractor the flow assigns to it, as Theorem 4.5 requires.

$\beta$	$\ell$	$\lambda_{\max}$	$\bar{\lambda}$	$q_{\theta_{\star}}$ (cert = obs)	$\theta_{\star}$	it( $1$ )	it( $\theta_{\star}$ )	it( $\theta_{\mathrm{o}}$ )
$0.09$	$0.460$	$7.69\times 10^{-2}$	$8.54\times 10^{-2}$	$0.2984$	$1.298$	$7.6$	$18.7$	$7.0$
$0.10$	$0.316$	$4.40\times 10^{-2}$	$5.45\times 10^{-2}$	$0.1877$	$1.188$	$6.8$	$14.0$	$6.0$
$0.12$	$0.145$	$1.44\times 10^{-2}$	$2.04\times 10^{-2}$	$0.0783$	$1.078$	$5.1$	$9.0$	$5.0$
$0.15$	$4.30\times 10^{-2}$	$2.61\times 10^{-3}$	$3.85\times 10^{-3}$	$0.0220$	$1.022$	$3.9$	$6.0$	$4.0$
$0.20$	$5.20\times 10^{-3}$	$1.41\times 10^{-4}$	$2.18\times 10^{-4}$	$2.61\times 10^{-3}$	$1.003$	$3.0$	$4.0$	$3.0$
$0.25$	$5.90\times 10^{-4}$	$7.20\times 10^{-6}$	$1.47\times 10^{-5}$	$2.95\times 10^{-4}$	$1.000$	$2.0$	$3.0$	$2.0$

Table 3: Campaign G. Certified versus observed overrelaxation across the

\beta

sweep. Iteration counts are means over ten initializations at radius

0.8\rho

to tolerance

10^{-10}

; at every

\beta

the observed factor at

\theta_{\star}

reproduces the certified

\ell/(2-\ell)

to the digits shown.

			certified fraction			empirical fraction
$(d,N)$	$\rho$	$\beta$ triple
$(64,4)$	$0$	$0.15/0.3/0.6$	$1.00$	$1.00$	$1.00$	$1.00$	$1.00$	$1.00$
$(64,4)$	$0.3$	$0.15/0.3/0.6$	$0.00$	$1.00$	$1.00$	$1.00$	$1.00$	$1.00$
$(64,4)$	$0.6$	$0.15/0.3/0.6$	$0.00$	$0.00$	$1.00$	$0.00$	$1.00$	$1.00$
$(64,32)$	$0$	$0.15/0.3/0.6$	$0.00$	$1.00$	$1.00$	$1.00$	$1.00$	$1.00$
$(64,32)$	$0.3$	$0.15/0.3/0.6$	$0.00$	$0.06$	$1.00$	$0.16$	$1.00$	$1.00$

Table 4: Campaign H. Worst case over three seeds of the fraction of patterns satisfying the certified retrieval condition ( 14 ) at

r=0.05\sqrt{d}

, and of the fraction retrieved empirically, across correlation

\rho

, aspect ratio

N/d

and inverse temperature

\beta

. Certified retrieval was accompanied by empirical retrieval in every configuration; the converse fails increasingly with correlation and overcompleteness.

Explore similar work

May 27, 2026

Continual Learning in Modern Hopfield Networks with an Application to Diffusion Models

Generative models, including diffusion models, are increasingly used as foundation models and adapted through sequential fine-tuning, making continual learning an essential problem setting. However, continual learning in such generative models remains poorly understood: after a task change, what aspects of the learned distribution are most easily lost, and what replay samples should be prioritized? We address these questions through the modern Hopfield energy. Recent links between modern Hopfield networks (MHNs) and diffusion models allow analyses in MHNs to be transferred to diffusion models. We introduce intrinsic forgetting as an increase in Hopfield energy after the task change. In tractable settings in an MHN, we prove that high-energy, outlier-like samples undergo a larger energy increase than cluster-like samples, implying that samples located in sharp, isolated basins are more forgettable. We further analyze memory replay and show that replay is particularly effective for high-energy samples, enabling an energy-based selection of replay samples. We validate these predictions in experiments on MHNs and two diffusion models under continual-learning settings: Stable Diffusion and a pixel-space DDPM. In these diffusion models, Hopfield energy tracks reconstruction-based forgetting, and replay experiments reveal energy-dependent mitigation of forgetting that is consistent with the MHN analysis.

Ken Takeda, Masafumi Oizumi, Ryo Karakida

Graduate School of Arts and Science, The University of Tokyo, Tokyo, Japan · Artificial Intelligence Research Center, AIST, Tokyo, Japan

Hopfield Networks Replay-Based Continual Learning Diffusion Models Catastrophic Forgetting

Ask PDF

May 1, 2026

Geometric and dynamical analysis of attractor boundaries and storage limits in kernel Hopfield networks

High-capacity associative memories based on Kernel Logistic Regression (KLR) exhibit strong storage capabilities, but the dynamical and geometric mechanisms underlying their stability remain poorly understood. This paper investigates the global geometry of attractor basins and the mechanisms governing the storage limit in KLR-trained Hopfield networks. We combine empirical evaluations using random sequences and real-world image embeddings (CIFAR-10) with morphing experiments and statistical Signal-to-Noise Ratio (SNR) analysis. Our experiments show that the network achieves a storage capacity for random sequences up to

P/N \approx 16

, while maintaining stable retrieval for structured data at effective loads near

P/N \approx 20

. Morphing analysis indicates that attractors on the "Ridge of Optimization" are separated by sharp, phase-transition-like boundaries, characterized by steep effective potential barriers and critical slowing down. Furthermore, by comparing an SNR analysis with a geometric reference point inspired by Cover's theorem, we show that the practical storage limit is governed primarily not by a lack of geometric separability in the feature space, but by the loss of dynamical stability against crosstalk noise. These findings suggest that KLR networks function as highly localized exemplar-based memories that operate near the onset of dynamical collapse, providing a useful perspective on the design of robust, large-scale retrieval systems.

Akira Tamamori

Faculty of Information Science, Aichi Institute of Technology 1247 Yachigusa, Yakusa-cho, Toyota-shi, Aichi 470-0392, Japan

Dynamical Systems Neural Representation Geometry Hopfield Networks Associative Memory

Ask PDF

Sep 10, 2026

Phases in a class of associative memories via hidden neurons

Associative memory in the Hopfield network is attractor dynamics in a disordered many-body system, and higher-order and exponential extensions turn its retrieval update into softmax attention. The polynomial and exponential regimes have been analyzed by different methods, with no common architecture in which to ask what fixes the storage scale. In this paper we study the bipartite architecture of Krotov and Hopfield, which we call the class

H

, whose model is fixed by a Lagrangian for each layer, taking the hidden neurons as the order parameter of retrieval. At polynomial load the replica method yields the replica-symmetric phase diagrams and closed-form capacities, and the crosstalk moment is common to Ising and spherical visible neurons, so their differences come from the visible entropy. With a softmax hidden layer the load is exponential, and a copy representation maps the thermodynamics onto random-energy-model counting, with paramagnetic, condensed, and frozen phases. Heating destabilizes retrieval by quantized reassignments of attention, and typical Gaussian patterns remain metastable at every load. The regimes differ in their crosstalk statistics, central-limit at polynomial load and large-deviation at exponential load, and the class

H

splits retrieval into two roles, the visible Lagrangian fixing stability and the hidden one the storage scale, two axes that may also guide the design of new Lagrangians.

Toshihiro Ota, Masato Taki

CyberAgent AI Lab, Shibuya, Tokyo 150–0002, Japan · RIKEN iTHEMS, Wako, Saitama 351–0198, Japan · Graduate School of Artificial Intelligence and Science, Rikkyo University, Toshima, Tokyo 171–8501, Japan

Phase Transitions Hopfield Networks Associative Memory

Ask PDF