A Geometry-Based Capacity Theory for Finite-Feature Associative Memory

Jianhai Zhang; Donghao Zhang; Pattarawut Charatpangoon; Bijoy Menon; M. Ethan MacDonald; Wu Qiu; Aravind Ganesh

cs.LGOct 6, 2026

A Geometry-Based Capacity Theory for Finite-Feature Associative Memory

Authors: Jianhai Zhang, Donghao Zhang, Pattarawut Charatpangoon, Bijoy Menon, M. Ethan MacDonald, Wu Qiu, Aravind Ganesh

Organizations: Department of Biomedical Engineering, Schulich School of Engineering, University of Calgary, and the Ron and Rene Ward Centre for Healthy Brain Aging Research, Hotchkiss Brain Institute, University of Calgary

Abstract

We develop a geometry-based capacity theory for exact-key retrieval in compressed finite-feature Hebbian associative memory. For random or approximately isotropic values, retrieval interference separates into finite-feature noise, which decreases with feature dimension, and structural interference, which is determined by squared kernel overlap among stored keys and persists in the infinite-feature limit. This yields a fit-free prediction of retrieval quality, reveals a geometry-dependent capacity ceiling, and predicts the feature budget required for a target retrieval quality. When stored values are correlated, we show that retrieval depends jointly on the key kernel and value Gram matrix, and derive finite-feature approximations that account for this interaction. We validate the theory on synthetic, visual, and medical-image representations. Overall, the framework links representation geometry directly to memory capacity and distinguishes when performance can be improved by increasing the feature budget and when the representation itself must be changed. Across these settings, the predicted retrieval curves closely match empirical behavior and correctly identify changes in the preferred memory design.

Figures & tables

Fig. 1: Overview of the analysis. (a) In a finite random-feature Hebbian memory, retrieval error separates into a finite-feature term that can be reduced by increasing the memory budget and a structural-interference term imposed by representation geometry. (b) With correlated values, the relevant deterministic quantities are the kernel-weighted value mixture and the associated value Gram geometry. (c) The resulting predicted capacity curves can be compared across preprocessing and kernel choices to estimate the required memory size, rank designs, or predict a crossover.

Experiment	MAE	RMSE	$r$
Clustered synthetic keys	0.000626	0.000885	0.999992
Low-rank synthetic keys	0.000683	0.001126	0.999777
Anisotropic synthetic keys	0.000445	0.000612	0.999930
ResNet/ViT multi-condition validation	0.003879	0.004838	0.999969
DINOv2/CLIP extension	0.000878	0.001308	0.999992
Public SynthRAD2023 finite- $R$ validation	0.002132	0.002868	0.999274

TABLE I: Summary of theory–empirical agreement across the main validation stages. The boundary-location row is measured in units of

R/N

; the other prediction-error rows are cosine errors.

Fig. 2: Synthetic theory validation over 448 i.i.d., clustered, low-rank, and anisotropic conditions. (a) Predicted versus empirical retrieval cosine; the dashed line is the identity line. (b) Residual

C_{\rm empirical}-C_{\rm predicted}

on an expanded vertical scale. The overall MAE is

5.79\times 10^{-4}

; the residual range is

[-0.00460,\,0.00708]

Backbone / mode	Dim.	Eff. rank	RMS cos.	mean $I_{\rm struct,i}$	mean $C_{i,\infty}$
ResNet50 RAW	2048	6.52	0.392	1.658	0.652
ResNet50 CENTERED	2048	66.50	0.122	0.0498	0.979
ResNet50 WHITENED	256	252.68	0.061	0.00835	0.996
ViT-B/16 RAW	768	32.98	0.173	1.354	0.680
ViT-B/16 CENTERED	768	98.74	0.099	0.436	0.850
ViT-B/16 WHITENED	256	253.70	0.061	0.187	0.920

TABLE II: Representative real-embedding geometry statistics. DINOv2/CLIP rows use CIFAR-10 at the base kernel scale (

\gamma\times 1.0

). Structural interference and the infinite-

R

ceiling are reported as separate item-wise averages; because the ceiling is nonlinear in

I_{\mathrm{struct},i}

, mean

C_{i,\infty}

is not obtained by substituting mean

I_{\mathrm{struct},i}

into Eq. ( 3 ).

Fig. 3: Geometry intervention on real CIFAR-100 ResNet50 embeddings. (a) Mean structural interference decreases from 1.658 (RAW) to 0.0498 (CENTERED) and 0.00835 (WHITENED), while the separately averaged item-wise infinite-

R

ceiling increases; whitening retains 256 components. (b) Lines show theoretical predictions, and markers show empirical retrieval for the same images under the three preprocessing choices.

Fig. 4: Real-representation validation across supervised, self-supervised, and vision-language features. (a) Predicted versus empirical retrieval cosine for ResNet50, ViT-B/16, DINOv2, and CLIP conditions. (b) Signed residuals

C_{\rm empirical}-C_{\rm predicted}

for the same conditions, showing the error scale directly. The geometry-aware law achieves MAE 0.003879 on the original ResNet/ViT validation and 0.000878 on the DINOv2/CLIP extension; for comparison, the finite-feature-only baseline yields MAE 0.289832 and 0.319933 on the two groups.

Cohort	Modality	Encoder	$\Delta I_{\rm struct}$	$\Delta C_{\infty}$	$C_{\infty}\downarrow$
Brain	CT	CLIP	$+0.273\,[0.255,0.292]$	$-0.0755\,[-0.0798,-0.0713]$	100/100
Brain	CT	DINOv2	$+0.301\,[0.292,0.310]$	$-0.0997\,[-0.1022,-0.0972]$	100/100
Brain	MRI	CLIP	$+0.392\,[0.367,0.418]$	$-0.0865\,[-0.0910,-0.0821]$	100/100
Brain	MRI	DINOv2	$+0.305\,[0.292,0.317]$	$-0.0975\,[-0.1005,-0.0944]$	100/100
Pelvis	CT	CLIP	$+0.063\,[0.054,0.072]$	$-0.0225\,[-0.0253,-0.0197]$	96/100
Pelvis	CT	DINOv2	$+0.104\,[0.096,0.112]$	$-0.0382\,[-0.0408,-0.0356]$	100/100

TABLE III: SynthRAD2023 matched-

N

same-patient clustering over 100 random constructions. Entries are differences between clustered and dispersed means; brackets give 95% bootstrap intervals for the mean construction-level contrast conditional on the observed cohort. Both conditions contain

N=120

stored slices.

Dataset	Representation	Predicted $R/N$
CIFAR-10	ResNet50 RAW	10.52
CIFAR-10	ResNet50 CENTERED	10.54
CIFAR-10	ResNet50 WHITENED	9.30
CIFAR-10	ViT-B/16 RAW	13.01
CIFAR-10	ViT-B/16 CENTERED	12.49
CIFAR-10	ViT-B/16 WHITENED	9.31

TABLE IV: Representative theory-predicted

R/N

values for a target cosine

C=0.95

on CIFAR-10/STL-10 unfitted-condition evaluations.

Fig. 5: Fixed-grid 20-seed target-cosine boundary revalidation on CIFAR-10/STL-10. Theory-predicted versus empirical

R/N

boundaries for the 38 conditions whose target crossings are bracketed within the evaluated budget. Empirical brackets are located on a fixed grid using discovery seeds and the boundaries are estimated with 20 independent confirmation seeds. The boundary-location MAE is 0.099807 in units of

R/N

, the mean relative absolute error is 1.517%, and

r=0.999894

Setting	MAE	RMSE	$r$
Key-only theory, all	0.113460	0.152437	0.865286
Joint key–value, all	0.004209	0.007842	0.999500
Joint key–value, CIFAR-10	0.002878	–	0.999718
Joint key–value, STL-10	0.005540	–	0.999400

TABLE V: Key-only theory versus joint key–value theory for real correlated values.

Fig. 6: Joint key–value controls across representation families. (a, b) Original ResNet

\rightarrow

ViT controls. (c) DINOv2/CLIP prediction error for aligned, permuted, and random values. (d) Repeated-permutation estimates of

\Delta C_{\infty}=C_{\infty}^{\rm aligned}-C_{\infty}^{\rm permuted}

for representative RAW conditions with

\gamma=1.0

; error bars are central 95% permutation intervals over 200 permutations. Across all 24 DINOv2/CLIP alignment conditions, the mean

\Delta C_{\infty}

and the corresponding permutation interval are strictly positive.

Group	$n$	Empirical mean	Simple mean	Cov. mean	Simple MAE	Cov. MAE
CIFAR-10 R50 $\rightarrow$ ViT	5	2.7867	5.0577	2.1620	2.2710	0.6247
STL-10 R50 $\rightarrow$ ViT	5	2.2956	5.1350	2.0911	2.8394	0.2045
CIFAR-10 ViT $\rightarrow$ R50	5	3.2725	6.7342	2.4528	3.4617	0.8197
STL-10 ViT $\rightarrow$ R50	5	2.3525	5.9092	2.1175	3.5567	0.2349
ALL	20	2.6768	5.7090	2.2058	3.0322	0.4710

TABLE VI: Cross-directional RAW-to-WHITENED crossover summary. Lower MAE is better.

Fig. 7: Theory-guided design. (a) Mean regret of two theory-based selectors and a fixed RAW default with

\gamma=1.0

. (b) Absolute crossover errors for the simple and covariance-aware predictions over 20 subset-level forward and reverse conditions (10 dataset–subset blocks). Every point lies below the equal-error line; overall MAE decreases from 3.0322 to 0.4710.

Explore similar work

Intrinsic Associative Memory on Riemannian Manifolds: Curvature, Capacity, and Emergent Modes
Sep 28, 2026Krishnakumar Balasubramanian, Zhaoyang ShiDense Associative Memory Riemannian Manifolds
Department of Statistics, University of California, Davis. · Center for Applied Mathematics, Fudan University.
Sharp Capacity Thresholds in Linear Associative Memory: From Winner-Take-All to Listwise Retrieval
May 6, 2026Nicholas Barnfield, Juno Kim, Eshaan Nichani +2Dense Associative Memory Full-Rank Correlation Matrices
Harvard University · University of California, Berkeley · Princeton University
Factual recall in linear associative memories: sharp asymptotics and mechanistic insights
May 11, 2026Alessio Giorlandino, Sebastian Goldt, Antoine MaillardDense Associative Memory Factual Recall
1International School of Advanced Studies (SISSA), Trieste, Italy · 2INRIA Paris & DI ENS, PSL University, Paris, France

Sep 28, 2026, cs.LGJ/K move · Enter open · S save

Intrinsic Associative Memory on Riemannian Manifolds: Curvature, Capacity, and Emergent Modes

Krishnakumar Balasubramanian, Zhaoyang Shi

Department of Statistics, University of California, Davis. · Center for Applied Mathematics, Fudan University.

Dense Associative Memory Riemannian Manifolds Local Curvature Geodesic

Open paper AskPDF arXiv

Geometry does more than constrain an associative memory: curvature determines what it remembers and which states it creates. We develop intrinsic dense associative memories on Riemannian manifolds by casting memory as Epanechnikov kernel-density mode seeking. We compare geodesic and volume-corrected energies and show that curvature separates their behavior. We prove that geodesic memory always retains an isolated pattern, while corrected memory obeys a sharp Ricci-curvature threshold: positive curvature can erase memories in high dimensions, while negative curvature reinforces them. We derive geodesic capacity scalings of

q_β^{-1/2}

for retaining every pattern and

q_β^{-1}

for a typical one, where

q_β

is the pairwise kernel-overlap probability. We show how overlap \emph{creates} novel memories: designed

N

-pattern configurations realize all

2^N-1

subset modes, but random data at the storage threshold yield only a Poisson number. We establish exact one-step recall using Riemannian mean shift. In simulations, we recover the predicted curvature transition and every designed mode. On WordNet's full noun hierarchy, we demonstrate that volume correction improves low-capacity retrieval. Together, our work shows that curvature is a design variable for associative memory, not merely a property of the data.

Abstract

Explore similar work

Intrinsic Associative Memory on Riemannian Manifolds: Curvature, Capacity, and Emergent Modes

Sharp Capacity Thresholds in Linear Associative Memory: From Winner-Take-All to Listwise Retrieval

Factual recall in linear associative memories: sharp asymptotics and mechanistic insights