Phase Kernel Lifts Capacity of Dense Associative Memory
Authors: Yifei Zhao, Ying Tang
Organizations: Institute of Fundamental and Frontier Sciences, University of Electronic Science and Technology of China, Chengdu 611731, China · School of Physics, University of Electronic Science and Technology of China, Chengdu 611731, China · Key Laboratory of Quantum Physics and Photonic Quantum Information, Ministry of Education, University of Electronic Science and Technology of China, Chengdu 611731, China · Non-classical Information Science Basic Discipline Research Center of Sichuan Province, University of Electronic Science and Technology of China, Chengdu 611731, China
Dense associative memory underlies modern attention mechanisms; however, Modern Hopfield Networks (MHNs) with linear kernels yield overlapping attractor basins and spurious states under high memory load, limiting their theoretically exponential storage capacity in practice. We construct a class of nonlinear phase kernels that modify the energy landscape and retrieval dynamics, and prove that these kernels preserve exponential storage capacity and local fixed-point convergence under given conditions on kernel smoothness, memory separation, and memory distributions. The framework is instantiated by the Sigmoidal Phase Hopfield Network (S-Hop), which guarantees monotone energy descent and mitigates practical capacity loss. S-Hop has 8.3× and 4× gains in one-step noisy retrieval load over Uniform Hopfield Networks (U-Hop) on CIFAR-10 and TinyImageNet respectively, and reaches a 50× gain in first-failure load for ImageNet. The results demonstrate that phase kernels provide a principled route to robust dense associative memory.
Figures & tables
Figure 1: Framework, energy landscape, and capacity gains of S-Hop. (a) Schematic of the energy-shaping motivation for stored patterns. (b) Illustrative 2D energy contours around m1,m2 , independently scaled to [0,1] for each kernel. (c) Retrieval-load summaries on CIFAR-10 ( d=3072 ) and ImageNet ( 224×224 , d=150528 ) for S-Hop, U-Hop [ 7 ] , MHN [ 1 ] , and Cosine Hopfield Network (C-Hop). S-Hop achieves passing-load ratios of over 50× than the previous methods.
Figure 2: Illustrative image retrieval from a shared memory bank of 500 ImageNet images. Each column shows a target image, its noisy query, and retrieval results after 20 updates ( β=1 ) without PCA or normalization. Queries are generated as q=clip(∣x+ϵ∣,0,1) with ϵ∼N(0,I) . S-Hop successfully retrieves target images, while MHN, U-Hop and C-Hop converge to other stored images.
MNIST
CIFAR-10
ImageNet
Time
Method
28×28
32×32
64×64
224×224
(ms/q)
MHN [ 1 ]
<10
<10
<10
<10
≈0.1
U-Hop ( max ) [ 7 ]
40
<10
<10
<10
10 – 40
U-Hop (public avg )
≥500
60
<10
<10
1.24†
C-Hop
<10
<10
<10
<10
≈0.4
S-Hop (ours)
≥500
≥500
≥500
≥500
≈1.5
Table 1: Retrieval loads at 90% MSE <0.1 success: one update, σ=0.3 . Public avg and ImageNet entries use three-seed means. U-Hop uses a block projection at 224×224 . Times amortize MNIST training over 256 queries at N=300 ( † : RTX 5090). TinyImageNet: Appendix D.4 .
Figure 3: ImageNet retrieval at 224×224 , averaged over three seeds with U-Hop’s public avg objective. (a) MSE <0.1 success; outlined points mark first failures below 90% or passing endpoints. (b) Mean retrieval error. Shared memories and queries; one update, β=1 , σ=0.3 , and no PCA.
Figure 4: ImageNet retrieval-load summaries at (a) 64×64 and (b) 224×224 , using three-seed mean success and U-Hop’s public avg objective. Bars mark passing endpoints for S-Hop ( Nc≥500 ) and first failures for the baselines ( Nc<10 ), under the protocol of Figure 3 .
Variant
Load
Key observation
S-Hop (full)
≥500
Passing sweep endpoint
w/o PCA
—
Retrieval error increases under noise
P-Hop (2nd admissible kernel)
50
Below 90% success at this load
↓ linear (w/o nonlinear kernel)
10
Below threshold at the smallest load
C-Hop (cosine kernel)
10
Violates strong convexity
Cross-framework & metric robustness
Table 2: Ablation study and protocol checks on CIFAR-10 ( d=3072 ). Loads are passing endpoints or first failures, identified in the final column. “ ↓ linear” denotes the linear-kernel ablation. U-Hop uses max unless indicated.
Figure 5: Efficiency trade-off between mean retrieval error and amortized wall-clock time per query, exemplified on CIFAR-10 under the original max protocol. U-Hop training is performed once per memory set and amortized over 256 queries at N=300; S-Hop uses a training-free retrieval map.
Appendix figures & tables35 assets
Supplementary material from the paper’s appendix.
Appendix
Notation
Description
Dimensions and Constants
d
Phase-space dimension.
α
Capacity ratio, Nc/d .
D
Feature-space dimension; D=2d for S-Hop.
N
Number of stored memories.
Nc
Largest tested load with ≥90% success at MSE <0.1 .
Appendix
Table 3: Key mathematical notation.
Notation
Description
Indices, Operators, and Kernel Geometry
i,j ; k,a,b ; ℓ
Memory, coordinate, and scalar-feature indices ( ℓ=1,2 ).
t ; φt,ξt
Iteration index and iterates; t also denotes a scalar argument.
I,Id ; id
Identity matrix (size d for Id ); identity map.
∥⋅∥2,∥⋅∥∞
Euclidean/spectral norm; maximum absolute value or supremum.
Table 4: Additional notation for geometry, proofs, and experiments.
Figure 6: MNIST retrieval-load summary with U-Hop ( max ; d=784 ). S-Hop attains Nc≥500 , a passing sweep endpoint matching that reached on CIFAR-10.
Figure 7: MNIST retrieval curves with U-Hop ( max ; d=784 ). (a) Perfect retrieval rate under the MSE<0.1 criterion. (b) Mean retrieval error versus the number of stored patterns. S-Hop sustains above-threshold retrieval and low error across the full range, while baselines saturate early.
Figure 8: Second admissible kernel on CIFAR-10: P-Hop and S-Hop retrieval loads. P-Hop is plotted at N=50 , the plotted threshold-crossing load, while S-Hop reaches the N=500 endpoint. The bars mark a passing endpoint for S-Hop and a first failure for P-Hop.
Figure 9: Second admissible kernel on CIFAR-10: P-Hop and S-Hop retrieval curves. (a) Perfect retrieval rate versus the number of stored patterns. (b) Mean retrieval error versus the number of stored patterns. P-Hop improves on the baselines, while S-Hop achieves higher retrieval success and lower error over the tested loads.
Figure 10: Aligned SSD criterion on CIFAR-10: retrieval-load summary. S-Hop passes through N=500 under the aligned threshold SSD<0.1d . Baseline bars mark threshold-crossing loads, following the original plotting convention.
Figure 11: Aligned SSD criterion on CIFAR-10: retrieval-rate and error curves. (a) SSD-based perfect retrieval rate versus the number of stored patterns under the aligned threshold SSD<0.1⋅d≈307 . (b) Mean retrieval SSD versus the number of stored patterns. S-Hop remains below the threshold across the full range N∈[10,500] , while all baselines cross it by N≈30 .
Figure 12: Legacy pairwise-average U-Hop protocol on CIFAR-10: retrieval-load summary. With MSE <0.1 , S-Hop passes through 500 ; U-Hop fails at the plotted load 60 ( 81.7% success).
Figure 13: Legacy pairwise-average U-Hop protocol on CIFAR-10: retrieval curves. (a) Perfect retrieval rate versus the number of stored patterns. S-Hop maintains above-threshold retrieval up to N≈500 , while U-Hop is below 90% at N=60 . (b) Mean retrieval error versus the number of stored patterns. U-Hop attains lower error at small N due to its learned separation loss that uniformizes memory distribution under low load [ 7 ] . Its error rises steeply as N grows; S-Hop sustains consistently low error across the full capacity range, retaining its advantage at larger loads.
Figure 14: Centered dot-product control on CIFAR-10: H-Hop - and S-Hop retrieval loads. All methods use pattern-space centering ξ~j=ξj−ξˉ , x~=x−ξˉ , where ξˉ=N−1∑jξj . H-Hop - denotes the dot-product degenerate variant of [ 34 ] , obtained by removing the ∥ξj∥2 penalty from Eq. ( 136 ) while retaining the pattern-space centering. S-Hop passes through N=500 ; the H-Hop - bar marks N=40 under the original plotting convention.
Figure 15: Centered dot-product control on CIFAR-10: H-Hop - and S-Hop retrieval curves. (a) Perfect retrieval rate versus the number of stored patterns. (b) Mean retrieval error versus the number of stored patterns. With the norm-penalty removed, H-Hop - degrades toward the linear baseline, whereas S-Hop preserves low error throughout the full range, showing the retrieval benefit of S-Hop relative to this centered dot-product control.
Figure 16: CIFAR-10 kernel and preprocessing ablations: retrieval error and gains over MHN. (a) Retrieval error under increasing memory load for ablated variants and stress-test conditions, including removal of the nonlinear kernel, PCA preprocessing, cosine-kernel substitution, random truncation, and high-noise evaluation ( σ=0.5 ). (b) Percentage improvement in retrieval accuracy over the MHN baseline as a function of the number of stored patterns, isolating the contributions of kernel choice and preprocessing.
Figure 17: Gating control on CIFAR-10: retrieval-load summary. S-Hop passes through N=500 ; baseline bars show the plotted threshold-crossing loads.
Figure 18: Gating control on CIFAR-10: retrieval-rate and error curves. (a) Perfect retrieval rate and (b) mean retrieval error versus the number of stored patterns.
Figure 19: Downstream generalization to multiple-instance learning: training and final performance. (a) Learning curves with mean ± standard deviation. (b) Best test accuracy over repeated runs.
Figure 20: Downstream CIFAR-10 classification: hybrid S-Hop/MHN architectures. Hybrid architectures combining two early S-Hop layers with four later MHN layers reach approximately 90% peak test accuracy, about 0.2 percentage points above the best baseline peak (seed 784 ).
Figure 21: Three-seed CIFAR-10 scale controls with the U-Hop max objective. Left: matched logit scale; right: normalized logits. Curves average seeds 42,43,44 , with shared memories and noisy queries within each seed. Success means MSE <0.1 after one update at σ=0.3 ; dashed lines mark 90% . Outlined points mark first failures or passing sweep endpoints.
Method
Mean feature norm
Raw σz
Normalized σz
Raw H
Normalized H
MHN
28.84
180.38
0.05683
0.06556
6.21306
U-Hop
1508.50
4.8213×105
0.06093
0
6.21283
S-Hop
95.63
506.83
0.00282
4.05×10−19
6.21460
Appendix
Table 5: Feature and logit diagnostics at N=500 , averaged over three seeds. H is softmax entropy in nats at base β=1 ; σz is the mean within-query logit standard deviation.
Control
MHN
U-Hop
S-Hop
Normalized logits
870/880
880/890
≥5000
Matched logit scale
<10
<10
790/800
Input centering
500/510
≥1000
≥1000
Matched βΔmin
<10
<10
790/800
Appendix
Table 6: CIFAR-10 scale controls: last passing load / first failing load under one-step retrieval with σ=0.3 , base β=1 , MSE <0.1 , and 90% success. A bound ≥ denotes a passing sweep endpoint; <10 denotes failure at the smallest tested load. U-Hop uses the implemented max objective. S-Hop uses one seed and the baselines use three.
Figure 22: Extended CIFAR-10 scale sweeps with the U-Hop max objective. Left: normalized logits; right: matched logit scale. Both use one update, σ=0.3 , and MSE <0.1 . S-Hop uses seed 42 ; baselines average seeds 42,43,44 . Dashed lines mark 90% success. With normalized logits, S-Hop passes through N=5000 .
Figure 23: CIFAR-10 geometry controls: input centering and signed minimum-margin matching. Left: F(ψ−ψˉ) ; right: matched βΔmin . Success means MSE <0.1 after one update at σ=0.3 ; dashed lines mark 90% . S-Hop uses seed 42 ; baselines average seeds 42,43,44 .
Figure 24: CIFAR-10 criterion sensitivity: four MSE thresholds. One-step retrieval uses thresholds 0.01 , 0.03 , 0.05 , and 0.1 , with σ=0.3 , no PCA, and seed 42 . Dashed lines mark 90% success. At threshold 0.1 , S-Hop last passes at 750 and first fails at 800 ; stricter thresholds fail at the smallest tested load, 10 .
Figure 25: CIFAR-10 output identification: cosine top-1 and cosine-threshold criteria. Retrieval uses one update at σ=0.3 . Left: top-1 accuracy selects the stored memory with highest cosine similarity to the retrieved output. Right: success requires output–target cosine similarity ≥0.95 ; the dashed line marks 90% success.
Figure 26: Common-norm CIFAR-10 dynamics at low load ( N=10 ). Initial query noise is σ=0.3 . Panels show mean MSE, Euclidean nearest-memory accuracy, the nearest-competitor minus target squared-distance margin divided by d , and successive-update RMSE at t∈{0,1,2,5,10,20} .
Step
Mean MSE
Accuracy (%)
Mean margin
Update RMSE
0
0.07218
100
0.04342
—
1
0.05026
100
0.02206
0.11148
2
0.04497
75
0.00600
0.06249
5
0.04729
15
−0.02372
0.02959
10
0.05517
15
−0.03981
0.01019
20
0.05836
15
−0.04347
0.00250
Appendix
Table 7: S-Hop multiple-update diagnostics at common norm, CIFAR-10, N=60 (60 queries). Nearest-memory accuracy uses Euclidean distance. The margin is the mean nearest-competitor squared distance minus target squared distance, divided by d .
Figure 27: Common-norm CIFAR-10 dynamics at higher load ( N=60 ). Panels use the four diagnostics in Figure 26 , with initial noise σ=0.3 . The dashed line marks zero margin; S-Hop values appear in Table 7 .
Figure 28: CIFAR-10 mechanism controls: state-update components and embedding dimension. Both use one update, β=1 , σ=0.3 , no PCA, and seed 42 . S-Hop-Sim uses phase-kernel scores with raw-memory averaging; S-Hop-Jac restores the Jacobian and inverse local field jointly. Right: U-Hop learns a 2d×d projection. Success means MSE <0.1 ; dashed lines mark 90% . Outlined points mark first failures or passing sweep endpoints.
Figure 29: CIFAR-10 preprocessing controls: PCA, random truncation, and higher query noise. “w/o PCA” removes S-Hop’s PCA preprocessing; “Rand. Trunc.” retains 1024 random input coordinates. “High Noise” uses σ=0.5 ; other curves use σ=0.3 . Retrieval uses one update at β=1 ; the error axis is logarithmic.
Figure 30: Cross-dataset separation control: CIFAR-10 and 32×32 TinyImageNet. Curves compare S-Hop before and after input-distance matching. Left: one-step MSE <0.1 success at σ=0.3 , with the 90% threshold dashed. Right: median nearest-neighbor squared distance per input dimension.
Method
Retrieval load
MHN
10
U-Hop ( max )
10
U-Hop (public avg )
10
C-Hop
10
S-Hop
40
Appendix
Table 8: Original TinyImageNet subset: 64×64 images, one update, σ=0.3 , and a 90% MSE <0.1 threshold. Loads mark the first threshold crossing. Public avg uses three-seed mean success.
Figure 31: Original TinyImageNet retrieval-load comparison. S-Hop reaches 40 , versus 10 for the baselines, a 4× gain. U-Hop uses the original max objective.
Figure 32: Geometry-selected TinyImageNet memories ( N=500 , 64×64 ). (a) Reconstruction error. (b) MSE <0.1 success and Euclidean nearest-memory accuracy. Bars show means and sample standard deviations across three noise seeds: one update, β=1 , σ=0.3 , no PCA or normalization.
Figure 33: ImageNet resolution comparison with U-Hop ( max ): one-step retrieval at 64×64 and 224×224 . Left and right use the same first 500 training images, seed 42 , no PCA, β=1 , σ=0.3 , and MSE <0.1 . Dashed lines mark 90% success; outlined points mark first failures or passing endpoints. S-Hop passes through N=500 in both runs. U-Hop uses a dense projection at 64×64 and a block-diagonal projection (block size 64 ) at 224×224 .
Figure 36: CIFAR-10 retrieval curves under the original one-step protocol with U-Hop ( max ). (a) Perfect retrieval rate under the MSE<0.1 criterion (dots mark the first drop below 90% ). (b) Mean retrieval error as a function of the number of stored patterns.
We study the thermodynamic memory capacity of modern Hopfield networks (Dense Associative Memory models) with continuous states under geometric constraints, extending classical analyses of pairwise associative memory. We derive thermodynamic phase boundaries for Dense Associative Memory networks with exponential capacity M=eαN, comparing Gaussian (LSE) and Epanechnikov (LSR) kernels. For continuous neurons on an N-sphere, the geometric entropy depends solely on the spherical geometry, not the kernel. In the sharp-kernel regime, the maximum theoretical capacity α=0.5 is achieved at zero temperature; below this threshold, a critical line separates retrieval from non-retrieval. The two kernels differ qualitatively in their phase boundary structure: for LSE, a critical line exists at all loads α>0. For LSR, the finite support introduces a threshold αth below which no spurious patterns contribute to the noise floor, and no critical line exists -- retrieval is perfect at any temperature. These results advance the theory of high-capacity associative memory and clarify fundamental limits of retrieval robustness in modern attention-like memory architectures.
Tatiana Petrova, Evgeny Polyachenko, Radu State
Interdisciplinary Centre for Security, Reliability and Trust (SnT), University of Luxembourg, 29 Avenue J.F. Kennedy, L-1855 Luxembourg.
Associative memory in the Hopfield network is attractor dynamics in a disordered many-body system, and higher-order and exponential extensions turn its retrieval update into softmax attention. The polynomial and exponential regimes have been analyzed by different methods, with no common architecture in which to ask what fixes the storage scale. In this paper we study the bipartite architecture of Krotov and Hopfield, which we call the class H, whose model is fixed by a Lagrangian for each layer, taking the hidden neurons as the order parameter of retrieval. At polynomial load the replica method yields the replica-symmetric phase diagrams and closed-form capacities, and the crosstalk moment is common to Ising and spherical visible neurons, so their differences come from the visible entropy. With a softmax hidden layer the load is exponential, and a copy representation maps the thermodynamics onto random-energy-model counting, with paramagnetic, condensed, and frozen phases. Heating destabilizes retrieval by quantized reassignments of attention, and typical Gaussian patterns remain metastable at every load. The regimes differ in their crosstalk statistics, central-limit at polynomial load and large-deviation at exponential load, and the class H splits retrieval into two roles, the visible Lagrangian fixing stability and the hidden one the storage scale, two axes that may also guide the design of new Lagrangians.
Toshihiro Ota, Masato Taki
CyberAgent AI Lab, Shibuya, Tokyo 150–0002, Japan · RIKEN iTHEMS, Wako, Saitama 351–0198, Japan · Graduate School of Artificial Intelligence and Science, Rikkyo University, Toshima, Tokyo 171–8501, Japan
High-capacity associative memories based on Kernel Logistic Regression (KLR) achieve strong retrieval performance but typically require substantial computational resources. This paper investigates the compressibility of KLR Hopfield networks to clarify the geometric principles underlying their robust representations. We present a geometric interpretation based on spontaneous symmetry breaking and Walsh analysis, and examine it through compression experiments involving quantization and pruning. The experiments reveal a clear asymmetry: the network remains robust under low-precision quantization while exhibiting strong sensitivity to pruning. We interpret this behavior through a "sparse function, dense representation" principle, in which a sparse input mapping is implemented through a dense bimodal parameterization. These findings suggest a practical route toward hardware-efficient kernel associative memories and provide insight into the geometric principles underlying robust representation in neural systems.
Akira Tamamori
Faculty of Information Science, Aichi Institute of Technology 1247 Yachigusa, Yakusa-cho, Toyota-shi, Aichi 470-0392, Japan