Organizations: Sentra, 235 2nd Street, San Francisco, CA 94105, USA · Department of Mechanical Engineering, Massachusetts Institute of Technology, 77 Massachusetts Avenue, Cambridge, MA 02139, USA
Why do we forget? Why do we remember things that never happened? The conventional answer points to biological hardware. We propose a different one: geometry. Here we show that high-dimensional embedding spaces, subjected to noise, interference, and temporal degradation, reproduce quantitative signatures of human memory with no phenomenon-specific engineering. Power-law forgetting (b=0.460±0.183, human b≈0.5) arises from interference among competing memories, not from decay. The identical decay function without competitors yields b≈0.009, fifty times smaller. Time alone does not produce forgetting in this system. Competition does. Production embedding models (nominally 384--1{,}024 dimensions) concentrate their variance in only ∼16 effective dimensions, placing them deep in the interference-vulnerable regime. False memories require no engineering at all: cosine similarity on unmodified pre-trained embeddings reproduces the Deese--Roediger--McDermott false alarm rate (0.583 versus human ∼0.55) with zero parameter tuning and no boundary conditions. We did not build a false memory system. We found one already present in the raw geometry of semantic space. These results suggest that core memory phenomena are not bugs of biological implementation but features of any system that organizes information by meaning and retrieves it by proximity.
Figures & tables
Figure 1: Interference from competing memories produces human-like forgetting in low-dimensional embedding spaces. a , Forgetting curves steepen progressively as competing memories are added (light to dark blue), converging toward the classical human forgetting curve (red dashed; Ebbinghaus, 1885). All curves at d=64 with identical decay function. b , The fitted forgetting exponent b increases monotonically with competitor count at d=64 (blue) but remains near zero at d=128 (grey). Human b≈0.5 (red dashed). Shaded regions: bootstrap 95% CI. c , Maximum forgetting exponent as a function of effective dimensionality. Interference is substantial only below d≈100 , placing biological neural codes (estimated d=100 – 500 22 , 23 ; shaded) near the transition zone. Orange diamond: MiniLM ( dnom=384 , deff≈16 ) shows strong interference consistent with its low effective dimensionality. n=5 seeds throughout.
Figure 2: False memories arise from semantic clustering in the embedding space. a , UMAP projection of the SLEEP word list shows the critical lure ( sleep , red star) falling within the cluster of studied words (blue dots), while unrelated words (grey) remain separated. b , At θ=0.82 , the critical-lure false alarm rate ( 0.583 ) closely matches the human value ( ∼0.55 , red diamonds). c , Threshold operating curves show the lure false alarm rate remains elevated above unrelated across a wide range of thresholds. d , Per-list lure cosine similarity is consistently high across all 24 DRM word lists. n=5 seeds for panels b – d .
Figure 3: Spaced repetition survives age-dependent degradation through recency of the most recent trace. a , Timeline showing repetition schedules: long-spaced items retain a recent trace at test (14 days old), while massed items have only old traces (30 days). Retention values shown at right. b , Retention increases monotonically with spacing interval (blue), matching the human ordering (red diamonds): long > medium > short > massed. c , The spacing gradient emerges as noise increases: at σ=0 , all conditions achieve ceiling; as noise grows, conditions separate, with massed dropping fastest. Selected σ=0.25 (dashed line). Error bars: bootstrap 95% CI, n=5 seeds.
Figure 4: Exploratory: topological structure of the memory manifold shows a connectivity transition with transient loop structure. a , Connected components ( H0 , blue) exhibit a sharp 1,000→1 transition while loops ( H1 , red) peak at 534±24 at ϵ=1.0 (shaded: phase transition zone). b , Persistence diagram showing long-lived topological features (far from diagonal) versus noise (near diagonal). c , UMAP visualizations at three edge lengths: disconnected clusters ( ϵ=0.7 ), critical connectivity with loops ( ϵ=1.0 ), and fully connected ( ϵ=1.5 ). Points colored by topic. Error bands: 95% CI, n=5 seeds.
Figure 5: Lightweight projections align independent encoders for cross-modal retrieval. a , Cross-modal similarity matrix shows strong diagonal structure: image queries retrieve their matched text descriptions with high cosine similarity (dark diagonal), while unrelated pairs show low similarity. b , Recall at k∈{1,5,10} for image-to-text and text-to-image retrieval, exceeding random baseline by two orders of magnitude. c , Cross-dataset transfer (COCO → Flickr30k) shows minimal degradation. Error bars: 95% CI, n=5 seeds.
Figure 6: Qualitative comparison to canonical human benchmarks. For each phenomenon, the observed value (blue dot, with 95% CI) is compared to the corresponding human reference (red diamond). Short connecting lines indicate close matches; longer lines indicate discrepancies. The forgetting exponent and DRM false alarm rate show the closest correspondence; other phenomena show qualitative but not tight quantitative agreement.
Figure 7: Extended Data Fig. 1: System architecture. Conceptual overview of the embedding-based memory framework. Input modalities are encoded by frozen pre-trained models, projected into a shared space, and retrieved via cosine similarity with temporal decay modulation.
Figure 8: Extended Data Fig. 2: Memory-dependent reasoning on bAbI. a , Accuracy by task for contextual retrieval versus baselines. b , Memory scaling: precision declines with store size. c , Retrieval precision heatmap. Mean ± 95% CI, n=5 seeds.
Figure 9: Extended Data Fig. 3: Ebbinghaus parameter sweep. a , Fitted forgetting exponent b across the σ×β parameter grid. Contour line at b=0.5 (human value). b , Goodness-of-fit R2 for the same configurations.
Figure 10: Extended Data Fig. 4: Interference experiment full results. a , All forgetting curves at d=64 (near condition). b , Curves at d=128 remain flat. c , Near versus far distractor comparison. d , Raw retention by age bin. n=5 seeds.
Figure 11: Extended Data Fig. 5: Consolidation as an informative negative result. a , Backward transfer worsens with consolidation despite b , substantial memory compression. c , The compression–accuracy trade-off: geometric merging achieves compression at the cost of retrieval accuracy.
Figure 12: Extended Data Fig. 6: Cross-modal retrieval detailed results. a , Recall curves from k=1 to k=10 for both retrieval directions. b , Transfer comparison. c , Comparison with random baseline. n=5 seeds.
Figure 13: Extended Data Fig. 7: DRM per-list analysis. a , Lure cosine similarity across all 24 DRM word lists. b , Similarity distributions for studied words, critical lures, and unrelated words. c , Threshold operating curves for hit rate and false alarm rates. n=5 seeds.
Figure 14: Extended Data Fig. 8: Spacing sweep full results. a , Retention by noise level at 25K distractors. b , Retention by distractor count at σ=0.25 . c , Effect size (Cohen’s d , long versus massed) across the full σ× distractor grid. n=5 seeds.
Figure 15: Extended Data Fig. 9: Tip-of-tongue analysis. TOT rate ( 3.66±0.13% ) compared to the human baseline ( ∼1.5% ). The qualitative phenomenon (correct items known but not immediately retrieved) emerges from retrieval competition, though the rate exceeds the human value by a factor of ∼2.4 , suggesting the geometric operationalisation (rank 2–20) is looser than the human phenomenological criterion. n=5 seeds.
Figure 16: Extended Data Fig. 10: Reproducibility across seeds. a , Key metrics across all five random seeds, showing consistency of results. b , Bootstrap distribution of the forgetting exponent b with 95% confidence interval.
Figure 17: Extended Data Fig. 11: Effective dimensionality resolves the dimensionality paradox. a , Eigenvalue spectra of three embedding models show rapid drop-off; vertical dashed lines mark the participation ratio ( deff ). b , Cumulative explained variance: only 17–18 components account for 95% of variance regardless of nominal dimensionality (384, 768, or 1,024). c , Maximum interference exponent b as a function of effective dimensionality. MiniLM at native d=384 ( deff≈16 , orange diamond) shows stronger interference than BGE-large at PCA d=64 , consistent with its low effective dimensionality. d , Direct comparison: MiniLM d=384 without PCA shows catastrophic retrieval failure with near distractors, exceeding BGE-large PCA d=64 interference. n=5 seeds; error bars: bootstrap 95% CI.
Phase
Parameter
Value
Notes
1
Embedding model
MiniLM-L6-v2
384-dim
Positional encoding dim
384
Sinusoidal
ContextProjector
768→384
Linear + LayerNorm
InfoNCE temperature
0.07
Fixed
Learning rate
1×10−3
AdamW
Weight decay
1×10−4
—
Table 1: Hyperparameters across all phases. All values stored in YAML configuration files.
Dataset
Phase
Split
Size
Licence
bAbI QA (en-10k)
1
Train / Test
10K / 1K per task
BSD
TempLAMA
2
Full
Variable
MIT
CIFAR-100
3
Train / Test
50K / 10K
BSD
COCO Captions 2017
4
Train / Val
118K / 5K
CC BY 4.0
Flickr30k
4
Test
1K
Research
DRM word lists
5
—
24 lists × 16 words
Public domain
Table 2: Datasets used across all phases. All publicly available with permissive licences.
Method
Task 1
Task 2
Task 3
Task 4
Task 5
Contextual retrieval
0.597±0.009
0.341±0.014
0.281±0.012
0.365±0.006
0.793±0.008
No memory
0.000
0.000
0.006
0.011
0.000
Full context
0.773
0.487
0.267
0.305
0.840
Random retrieval
0.531±0.008
0.317±0.011
0.229±0.013
0.360±0.006
0.620±0.018
Overall
Contextual: 0.475±0.007 ; No memory: 0.003 ; Random: 0.411±0.004
Table 3: bAbI accuracy (mean ± std, 5 seeds) across tasks and methods.
Dim
Cond.
Distractors
b
95% CI
Retention
64
near
0
0.009
[0.006, 0.012]
0.990
near
1,000
0.038
[0.023, 0.062]
0.961
near
4,000
0.073
[0.057, 0.095]
0.919
near
10,000
0.105
[0.087, 0.124]
0.878
near
40,000
0.161
[0.126, 0.200]
0.814
far
10,000
0.078
[0.054, 0.116]
0.917
Table 4: Fitted power-law exponent b and retention by distractor count, condition, and dimensionality (mean, 95% CI, 5 seeds). MiniLM rows show native 384-dim without PCA projection, confirming that effective dimensionality, not nominal dimensionality, governs interference.
Model
dnom
deff
d95
d99
deff/dnom
MiniLM-L6-v2
384
15.7±0.0
17.0±0.0
19.0±0.0
0.041
BGE-base-en-v1.5
768
16.6±0.1
18.0±0.0
19.0±0.0
0.022
BGE-large-en-v1.5
1,024
16.3±0.1
17.6±0.5
19.0±0.0
0.016
Table 5: Effective dimensionality analysis. Despite 25 × –65 × differences in nominal dimensionality, all models concentrate variance in ∼16 effective dimensions. N=10,000 Wikipedia sentences per seed, 5 seeds.
Every major AI memory system in production today organises information by meaning. That organisation enables generalisation, analogy, and conceptual retrieval -- but it comes at a price. We prove that the same geometric structure enabling semantic generalisation makes interference, forgetting, and false recall inescapable. We formalise this tradeoff for semantically continuous kernel-threshold memories: systems whose retrieval score is a monotone function of an inner product in a semantic feature space with finite local intrinsic dimension. Within this class we derive four results: (1) semantically useful representations have finite effective rank; (2) finite local dimension implies positive competitor mass in retrieval neighbourhoods; (3) under growing memory, retention decays to zero, yielding power-law forgetting curves under power-law arrival statistics; (4) for associative lures satisfying a δ-convexity condition, false recall cannot be eliminated by threshold tuning. We test these predictions across five architectures: vector retrieval, graph memory, attention-based context, BM25 filesystem retrieval, and parametric memory. Pure semantic systems express the vulnerability directly as forgetting and false recall. Reasoning-augmented systems partially override these symptoms but convert graceful degradation into catastrophic failure. Systems that escape interference entirely do so by sacrificing semantic generalisation. The price of meaning is interference, and no architecture we tested avoids paying it.
Sambartha Ray Barman, Andrey Starenky, Sofia Bodnar +2
Sentra, 235 2nd Street, San Francisco, CA 94105, USA · Department of Mechanical Engineering, Massachusetts Institute of Technology, 77 Massachusetts Avenue, Cambridge, MA 02139, USA
Knowledge that a language model appears to forget during finetuning often remains stored and can be recovered, a phenomenon called spurious forgetting. Finetuning on new facts can even produce forgetting that undoes itself: recall of the old facts collapses, recovers as training continues on new facts alone, and only then erodes for good. We seek to understand when such forgetting is not catastrophic. A minimal associative memory reproduces these dynamics with three ingredients: keys with shared structure, concentrated new values, and normalization in the network. Finetuning moves all old representations along a common direction, hiding the old facts while preserving their relative geometry; normalization withdraws this shift once the new facts are learned, whereas fact-specific changes accumulate and cause the erosion. Moreover, subtracting the common shift eliminates the collapse in a Transformer trained on synthetic data, and removing a single direction from each weight update restores old facts in a pretrained language model. Forgetting thus combines a shared, reversible loss of access with a slow erosion of individual facts, and only the second is catastrophic. Which one dominates depends on whether the new data move old memories together or apart.
Vedant Palit, Florent Draye, Nicolas Zucchet +2
MPI for Intelligent Systems, Tübingen · Jinesis Lab, University of Toronto & Vector Institute · EuroSafeAI +3
We study whether geometric relationships between conversation turns, memory texts, and queries can predict answer correctness before generation. We evaluate four memory implementations with dense or hybrid retrieval and one observation-summary comparator. We embed these texts in a shared space. A logistic classifier uses 25 features describing query proximity, memory-conversation correspondence, and conversation structure. On LongMemEval-small, pooled AUROC is 0.640 over 2,367 question-system pairs, below a question-difficulty baseline of 0.655. Performance varies by question type: geometry reaches 0.841 on single-session-assistant questions, but system identity alone reaches 0.826. After controlling for difficulty and identity, geometry adds no statistically established pooled improvement (delta = +0.002, 95% CI [-0.010, 0.014]). A fixed-store context-selection experiment finds higher accuracy with proximity-ranked memories than with random subsets, consistent with ordinary retrieval relevance. On LoCoMo, nearest-turn similarity has modest predictive value for single-hop questions and near-chance estimates for several other categories. Training uses correctness labels; prediction uses pre-answer features and no additional LLM generation call. The results characterize associations between memory representations and correctness, rather than establish a reliable deployment diagnostic. In particular, embedding proximity does not verify fact coverage or distinguish memory-construction, retrieval, and generation errors.