Phase Kernel Lifts Capacity of Dense Associative Memory

Yifei Zhao; Ying Tang

cs.LGOct 15, 2026

Phase Kernel Lifts Capacity of Dense Associative Memory

Authors: Yifei Zhao, Ying Tang

Organizations: Institute of Fundamental and Frontier Sciences, University of Electronic Science and Technology of China, Chengdu 611731, China · School of Physics, University of Electronic Science and Technology of China, Chengdu 611731, China · Key Laboratory of Quantum Physics and Photonic Quantum Information, Ministry of Education, University of Electronic Science and Technology of China, Chengdu 611731, China · Non-classical Information Science Basic Discipline Research Center of Sichuan Province, University of Electronic Science and Technology of China, Chengdu 611731, China

Abstract

Dense associative memory underlies modern attention mechanisms; however, Modern Hopfield Networks (MHNs) with linear kernels yield overlapping attractor basins and spurious states under high memory load, limiting their theoretically exponential storage capacity in practice. We construct a class of nonlinear phase kernels that modify the energy landscape and retrieval dynamics, and prove that these kernels preserve exponential storage capacity and local fixed-point convergence under given conditions on kernel smoothness, memory separation, and memory distributions. The framework is instantiated by the Sigmoidal Phase Hopfield Network (S-Hop), which guarantees monotone energy descent and mitigates practical capacity loss. S-Hop has $8.3\times$ and $4\times$ gains in one-step noisy retrieval load over Uniform Hopfield Networks (U-Hop) on CIFAR-10 and TinyImageNet respectively, and reaches a $50\times$ gain in first-failure load for ImageNet. The results demonstrate that phase kernels provide a principled route to robust dense associative memory.

Figures & tables

Figure 1: Framework, energy landscape, and capacity gains of S-Hop. (a) Schematic of the energy-shaping motivation for stored patterns. (b) Illustrative 2D energy contours around $m_{1},m_{2}$ , independently scaled to $[0,1]$ for each kernel. (c) Retrieval-load summaries on CIFAR-10 ( $d=3072$ ) and ImageNet ( $224\times 224$ , $d=150528$ ) for S-Hop, U-Hop [ 7 ] , MHN [ 1 ] , and Cosine Hopfield Network (C-Hop). S-Hop achieves passing-load ratios of over $50\times$ than the previous methods.

Figure 2: Illustrative image retrieval from a shared memory bank of 500 ImageNet images. Each column shows a target image, its noisy query, and retrieval results after 20 updates ( $\beta=1$ ) without PCA or normalization. Queries are generated as $q=\operatorname{clip}(|x+\epsilon|,0,1)$ with $\epsilon\sim\mathcal{N}(0,I)$ . S-Hop successfully retrieves target images, while MHN, U-Hop and C-Hop converge to other stored images.

	MNIST	CIFAR-10	ImageNet		Time
Method	$28\times 28$	$32\times 32$	$64\times 64$	$224\times 224$	(ms/q)
MHN [ 1 ]	${}<10$	${}<10$	${}<10$	${}<10$	${\approx}0.1$
U-Hop ( max ) [ 7 ]	$40$	${}<10$	${}<10$	${}<10$	$10$ – $40$
U-Hop (public avg )	$\geq 500$	$60$	${}<10$	${}<10$	$1.24^{\dagger}$
C-Hop	${}<10$	${}<10$	${}<10$	${}<10$	${\approx}0.4$
S-Hop (ours)	$\mathbf{\geq 500}$	$\mathbf{\geq 500}$	$\mathbf{\geq 500}$	$\mathbf{\geq 500}$	${\approx}1.5$

Table 1: Retrieval loads at

90\%

MSE

{}<0.1

success: one update,

\sigma=0.3

. Public avg and ImageNet entries use three-seed means. U-Hop uses a block projection at

224\times 224

. Times amortize MNIST training over

256

queries at

N=300

(

\dagger

: RTX 5090). TinyImageNet: Appendix D.4 .

Figure 3: ImageNet retrieval at

224\times 224

, averaged over three seeds with U-Hop’s public avg objective. (a) MSE

{}<0.1

success; outlined points mark first failures below

90\%

or passing endpoints. (b) Mean retrieval error. Shared memories and queries; one update,

\beta=1

\sigma=0.3

, and no PCA.

Figure 4: ImageNet retrieval-load summaries at (a)

64\times 64

and (b)

224\times 224

, using three-seed mean success and U-Hop’s public avg objective. Bars mark passing endpoints for S-Hop (

N_{c}\geq 500

) and first failures for the baselines (

N_{c}<10

), under the protocol of Figure 3 .

Variant	Load	Key observation
S-Hop (full)	$\mathbf{\geq 500}$	Passing sweep endpoint
w/o PCA	—	Retrieval error increases under noise
P-Hop (2nd admissible kernel)	50	Below $90\%$ success at this load
$\downarrow$ linear (w/o nonlinear kernel)	10	Below threshold at the smallest load
C-Hop (cosine kernel)	10	Violates strong convexity
Cross-framework & metric robustness

Table 2: Ablation study and protocol checks on CIFAR-10 (

d=3072

). Loads are passing endpoints or first failures, identified in the final column. “

\downarrow

linear” denotes the linear-kernel ablation. U-Hop uses max unless indicated.

Figure 5: Efficiency trade-off between mean retrieval error and amortized wall-clock time per query, exemplified on CIFAR-10 under the original max protocol. U-Hop training is performed once per memory set and amortized over

256

queries at N=300; S-Hop uses a training-free retrieval map.

Appendix figures & tables35 assets

Supplementary material from the paper’s appendix.

Appendix

Notation	Description
Dimensions and Constants
$d$	Phase-space dimension.
$\alpha$	Capacity ratio, $N_{c}/d$ .
$D$	Feature-space dimension; $D=2d$ for S-Hop.
$N$	Number of stored memories.
$N_{c}$	Largest tested load with $\geq 90\%$ success at MSE ${}<0.1$ .

Appendix

Table 3: Key mathematical notation.

Notation	Description
Indices, Operators, and Kernel Geometry
$i,j$ ; $k,a,b$ ; $\ell$	Memory, coordinate, and scalar-feature indices ( $\ell=1,2$ ).
$t$ ; $\varphi_{t},\xi_{t}$	Iteration index and iterates; $t$ also denotes a scalar argument.
$I,I_{d}$ ; $\mathrm{id}$	Identity matrix (size $d$ for $I_{d}$ ); identity map.
$\\|\cdot\\|_{2},\\|\cdot\\|_{\infty}$	Euclidean/spectral norm; maximum absolute value or supremum.
$\langle\cdot,\cdot\rangle$ , $\preceq$	Euclidean inner product; positive-semidefinite matrix order.

Appendix

Table 4: Additional notation for geometry, proofs, and experiments.

Figure 6: MNIST retrieval-load summary with U-Hop ( max ;

d=784

). S-Hop attains

N_{c}\geq 500

, a passing sweep endpoint matching that reached on CIFAR-10.

Figure 7: MNIST retrieval curves with U-Hop ( max ;

d=784

). (a) Perfect retrieval rate under the

\mathrm{MSE}<0.1

criterion. (b) Mean retrieval error versus the number of stored patterns. S-Hop sustains above-threshold retrieval and low error across the full range, while baselines saturate early.

Figure 8: Second admissible kernel on CIFAR-10: P-Hop and S-Hop retrieval loads. P-Hop is plotted at

N=50

, the plotted threshold-crossing load, while S-Hop reaches the

N=500

endpoint. The bars mark a passing endpoint for S-Hop and a first failure for P-Hop.

Figure 9: Second admissible kernel on CIFAR-10: P-Hop and S-Hop retrieval curves. (a) Perfect retrieval rate versus the number of stored patterns. (b) Mean retrieval error versus the number of stored patterns. P-Hop improves on the baselines, while S-Hop achieves higher retrieval success and lower error over the tested loads.

Figure 10: Aligned SSD criterion on CIFAR-10: retrieval-load summary. S-Hop passes through

N=500

under the aligned threshold

\mathrm{SSD}<0.1d

. Baseline bars mark threshold-crossing loads, following the original plotting convention.

Figure 11: Aligned SSD criterion on CIFAR-10: retrieval-rate and error curves. (a) SSD-based perfect retrieval rate versus the number of stored patterns under the aligned threshold

\mathrm{SSD}<0.1\cdot d\approx 307

. (b) Mean retrieval SSD versus the number of stored patterns. S-Hop remains below the threshold across the full range

N\in[10,500]

, while all baselines cross it by

N\approx 30

Figure 12: Legacy pairwise-average U-Hop protocol on CIFAR-10: retrieval-load summary. With MSE

{}<0.1

, S-Hop passes through

500

; U-Hop fails at the plotted load

60

(

81.7\%

success).

Figure 13: Legacy pairwise-average U-Hop protocol on CIFAR-10: retrieval curves. (a) Perfect retrieval rate versus the number of stored patterns. S-Hop maintains above-threshold retrieval up to

N\approx 500

, while U-Hop is below

90\%

N=60

. (b) Mean retrieval error versus the number of stored patterns. U-Hop attains lower error at small

N

due to its learned separation loss that uniformizes memory distribution under low load [ 7 ] . Its error rises steeply as

N

grows; S-Hop sustains consistently low error across the full capacity range, retaining its advantage at larger loads.

Figure 14: Centered dot-product control on CIFAR-10: H-Hop - and S-Hop retrieval loads. All methods use pattern-space centering

\tilde{\xi}_{j}=\xi_{j}-\bar{\xi}

\tilde{x}=x-\bar{\xi}

, where

\bar{\xi}=N^{-1}\sum_{j}\xi_{j}

. H-Hop - denotes the dot-product degenerate variant of [ 34 ] , obtained by removing the

\|\xi_{j}\|^{2}

penalty from Eq. ( 136 ) while retaining the pattern-space centering. S-Hop passes through

N=500

; the H-Hop - bar marks

N=40

under the original plotting convention.

Figure 15: Centered dot-product control on CIFAR-10: H-Hop - and S-Hop retrieval curves. (a) Perfect retrieval rate versus the number of stored patterns. (b) Mean retrieval error versus the number of stored patterns. With the norm-penalty removed, H-Hop - degrades toward the linear baseline, whereas S-Hop preserves low error throughout the full range, showing the retrieval benefit of S-Hop relative to this centered dot-product control.

Panel of Figure 16 — Figure 16: CIFAR-10 kernel and preprocessing ablations: retrieval error and gains over MHN. (a) Retrieval error under increasing memory load for ablated variants and stress-test conditions, including removal of the nonlinear kernel, PCA preprocessing, cosine-kernel substitution, random truncation, and high-noise evaluation ( $\sigma=0.5$ ). (b) Percentage improvement in retrieval accuracy over the MHN baseline as a function of the number of stored patterns, isolating the contributions of kernel choice and preprocessing.

Figure 17: Gating control on CIFAR-10: retrieval-load summary. S-Hop passes through

N=500

; baseline bars show the plotted threshold-crossing loads.

Figure 18: Gating control on CIFAR-10: retrieval-rate and error curves. (a) Perfect retrieval rate and (b) mean retrieval error versus the number of stored patterns.

Figure 19: Downstream generalization to multiple-instance learning: training and final performance. (a) Learning curves with mean

\pm

standard deviation. (b) Best test accuracy over repeated runs.

Figure 20: Downstream CIFAR-10 classification: hybrid S-Hop/MHN architectures. Hybrid architectures combining two early S-Hop layers with four later MHN layers reach approximately

90\%

peak test accuracy, about

0.2

percentage points above the best baseline peak (seed

784

Figure 21: Three-seed CIFAR-10 scale controls with the U-Hop max objective. Left: matched logit scale; right: normalized logits. Curves average seeds

42,43,44

, with shared memories and noisy queries within each seed. Success means MSE

{}<0.1

after one update at

\sigma=0.3

; dashed lines mark

90\%

. Outlined points mark first failures or passing sweep endpoints.

Method	Mean feature norm	Raw $\sigma_{z}$	Normalized $\sigma_{z}$	Raw $H$	Normalized $H$
MHN	$28.84$	$180.38$	$0.05683$	$0.06556$	$6.21306$
U-Hop	$1508.50$	$4.8213\times 10^{5}$	$0.06093$	$0$	$6.21283$
S-Hop	$95.63$	$506.83$	$0.00282$	$4.05\times 10^{-19}$	$6.21460$

Appendix

Table 5: Feature and logit diagnostics at

N=500

, averaged over three seeds.

H

is softmax entropy in nats at base

\beta=1

;

\sigma_{z}

is the mean within-query logit standard deviation.

Control	MHN	U-Hop	S-Hop
Normalized logits	$870/880$	$880/890$	$\geq 5000$
Matched logit scale	$<10$	$<10$	$790/800$
Input centering	$500/510$	$\geq 1000$	$\geq 1000$
Matched $\beta\Delta_{\min}$	$<10$	$<10$	$790/800$

Appendix

Table 6: CIFAR-10 scale controls: last passing load / first failing load under one-step retrieval with

\sigma=0.3

, base

\beta=1

, MSE

{}<0.1

, and

90\%

success. A bound

\geq

denotes a passing sweep endpoint;

<10

denotes failure at the smallest tested load. U-Hop uses the implemented max objective. S-Hop uses one seed and the baselines use three.

Figure 22: Extended CIFAR-10 scale sweeps with the U-Hop max objective. Left: normalized logits; right: matched logit scale. Both use one update,

\sigma=0.3

, and MSE

{}<0.1

. S-Hop uses seed

42

; baselines average seeds

42,43,44

. Dashed lines mark

90\%

success. With normalized logits, S-Hop passes through

N=5000

Figure 23: CIFAR-10 geometry controls: input centering and signed minimum-margin matching. Left:

F(\psi-\bar{\psi})

; right: matched

\beta\Delta_{\min}

. Success means MSE

{}<0.1

after one update at

\sigma=0.3

; dashed lines mark

90\%

. S-Hop uses seed

42

; baselines average seeds

42,43,44

Figure 24: CIFAR-10 criterion sensitivity: four MSE thresholds. One-step retrieval uses thresholds

0.01

0.03

0.05

, and

0.1

, with

\sigma=0.3

, no PCA, and seed

42

. Dashed lines mark

90\%

success. At threshold

0.1

, S-Hop last passes at

750

and first fails at

800

; stricter thresholds fail at the smallest tested load,

10

Figure 25: CIFAR-10 output identification: cosine top-1 and cosine-threshold criteria. Retrieval uses one update at

\sigma=0.3

. Left: top-1 accuracy selects the stored memory with highest cosine similarity to the retrieved output. Right: success requires output–target cosine similarity

\geq 0.95

; the dashed line marks

90\%

success.

Figure 26: Common-norm CIFAR-10 dynamics at low load (

N=10

). Initial query noise is

\sigma=0.3

. Panels show mean MSE, Euclidean nearest-memory accuracy, the nearest-competitor minus target squared-distance margin divided by

d

, and successive-update RMSE at

t\in\{0,1,2,5,10,20\}

Step	Mean MSE	Accuracy (%)	Mean margin	Update RMSE
0	0.07218	100	0.04342	—
1	0.05026	100	0.02206	0.11148
2	0.04497	75	0.00600	0.06249
5	0.04729	15	$-0.02372$	0.02959
10	0.05517	15	$-0.03981$	0.01019
20	0.05836	15	$-0.04347$	0.00250

Appendix

Table 7: S-Hop multiple-update diagnostics at common norm, CIFAR-10,

N=60

(60 queries). Nearest-memory accuracy uses Euclidean distance. The margin is the mean nearest-competitor squared distance minus target squared distance, divided by

d

Figure 27: Common-norm CIFAR-10 dynamics at higher load (

N=60

). Panels use the four diagnostics in Figure 26 , with initial noise

\sigma=0.3

. The dashed line marks zero margin; S-Hop values appear in Table 7 .

Figure 28: CIFAR-10 mechanism controls: state-update components and embedding dimension. Both use one update,

\beta=1

\sigma=0.3

, no PCA, and seed

42

. S-Hop-Sim uses phase-kernel scores with raw-memory averaging; S-Hop-Jac restores the Jacobian and inverse local field jointly. Right: U-Hop learns a

2d\times d

projection. Success means MSE

{}<0.1

; dashed lines mark

90\%

. Outlined points mark first failures or passing sweep endpoints.

Figure 29: CIFAR-10 preprocessing controls: PCA, random truncation, and higher query noise. “w/o PCA” removes S-Hop’s PCA preprocessing; “Rand. Trunc.” retains

1024

random input coordinates. “High Noise” uses

\sigma=0.5

; other curves use

\sigma=0.3

. Retrieval uses one update at

\beta=1

; the error axis is logarithmic.

Figure 30: Cross-dataset separation control: CIFAR-10 and

32\times 32

TinyImageNet. Curves compare S-Hop before and after input-distance matching. Left: one-step MSE

{}<0.1

success at

\sigma=0.3

, with the

90\%

threshold dashed. Right: median nearest-neighbor squared distance per input dimension.

Method	Retrieval load
MHN	$10$
U-Hop ( max )	$10$
U-Hop (public avg )	$10$
C-Hop	$10$
S-Hop	$40$

Appendix

Table 8: Original TinyImageNet subset:

64\times 64

images, one update,

\sigma=0.3

, and a

90\%

MSE

{}<0.1

threshold. Loads mark the first threshold crossing. Public avg uses three-seed mean success.

Figure 31: Original TinyImageNet retrieval-load comparison. S-Hop reaches

40

, versus

10

for the baselines, a

4\times

gain. U-Hop uses the original max objective.

Figure 32: Geometry-selected TinyImageNet memories (

N=500

64\times 64

). (a) Reconstruction error. (b) MSE

{}<0.1

success and Euclidean nearest-memory accuracy. Bars show means and sample standard deviations across three noise seeds: one update,

\beta=1

\sigma=0.3

, no PCA or normalization.

Figure 33: ImageNet resolution comparison with U-Hop ( max ): one-step retrieval at

64\times 64

and

224\times 224

. Left and right use the same first

500

training images, seed

42

, no PCA,

\beta=1

\sigma=0.3

, and MSE

{}<0.1

. Dashed lines mark

90\%

success; outlined points mark first failures or passing endpoints. S-Hop passes through

N=500

in both runs. U-Hop uses a dense projection at

64\times 64

and a block-diagonal projection (block size

64

) at

224\times 224

Figure 36: CIFAR-10 retrieval curves under the original one-step protocol with U-Hop ( max ). (a) Perfect retrieval rate under the

\mathrm{MSE}<0.1

criterion (dots mark the first drop below

90\%

). (b) Mean retrieval error as a function of the number of stored patterns.

Hopfield Networks

Hopfield networks are energy-based recurrent models that store patterns as attractors and retrieve them from noisy or partial cues.

See all 40 papers

Geometric Entropy and Retrieval Phase Transitions in Continuous Thermal Dense Associative Memory
Apr 8, 2026Tatiana Petrova, Evgeny Polyachenko, Radu StateAssociative Memory Hopfield Networks
Interdisciplinary Centre for Security, Reliability and Trust (SnT), University of Luxembourg, 29 Avenue J.F. Kennedy, L-1855 Luxembourg.
Phases in a class of associative memories via hidden neurons
Sep 10, 2026Toshihiro Ota, Masato TakiPhase Transitions Hopfield Networks
CyberAgent AI Lab, Shibuya, Tokyo 150–0002, Japan · RIKEN iTHEMS, Wako, Saitama 351–0198, Japan · Graduate School of Artificial Intelligence and Science, Rikkyo University, Toshima, Tokyo 171–8501, Japan
Quantization robustness from dense representations of sparse functions in high-capacity kernel associative memory
Apr 22, 2026Akira TamamoriNeural Representation Geometry Hopfield Networks
Faculty of Information Science, Aichi Institute of Technology 1247 Yachigusa, Yakusa-cho, Toyota-shi, Aichi 470-0392, Japan

Apr 8, 2026J/K move · Enter open · S save

Geometric Entropy and Retrieval Phase Transitions in Continuous Thermal Dense Associative Memory

Tatiana Petrova, Evgeny Polyachenko, Radu State

Interdisciplinary Centre for Security, Reliability and Trust (SnT), University of Luxembourg, 29 Avenue J.F. Kennedy, L-1855 Luxembourg.

Associative Memory Hopfield Networks Statistical Physics of Learning

Open paper AskPDF arXiv

We study the thermodynamic memory capacity of modern Hopfield networks (Dense Associative Memory models) with continuous states under geometric constraints, extending classical analyses of pairwise associative memory. We derive thermodynamic phase boundaries for Dense Associative Memory networks with exponential capacity

M = e^{αN}

, comparing Gaussian (LSE) and Epanechnikov (LSR) kernels. For continuous neurons on an

N

-sphere, the geometric entropy depends solely on the spherical geometry, not the kernel. In the sharp-kernel regime, the maximum theoretical capacity

α= 0.5

is achieved at zero temperature; below this threshold, a critical line separates retrieval from non-retrieval. The two kernels differ qualitatively in their phase boundary structure: for LSE, a critical line exists at all loads

α> 0

. For LSR, the finite support introduces a threshold

α_{\text{th}}

below which no spurious patterns contribute to the noise floor, and no critical line exists -- retrieval is perfect at any temperature. These results advance the theory of high-capacity associative memory and clarify fundamental limits of retrieval robustness in modern attention-like memory architectures.

Abstract

Hopfield Networks

Explore similar work

Geometric Entropy and Retrieval Phase Transitions in Continuous Thermal Dense Associative Memory

Phases in a class of associative memories via hidden neurons

Quantization robustness from dense representations of sparse functions in high-capacity kernel associative memory