Associative Memory

Ask about this topic

Overview

Associative memory asks how a system can store information so that a cue brings back a related pattern, fact, or sequence, a problem central to neural memory and efficient AI. Classical Hopfield networks, dense and kernel associative memories, and Hebbian learning use attractor dynamics or similarity-based retrieval; newer recurrent and linear-attention models add sparse key-value stores, while agent systems use linked episodic memories. Theory examines capacity, retrieval basins, stability, and interference, while experiments test noisy pattern completion, long-range recall, language-model memory, and hardware implementations. A persistent tension is that more storage does not guarantee reliable, selective recall: finite state and overlapping representations can cause interference, forgetting, and false associations, even as explicit retrieval improves access at added cost.

Latest papers

CardsList

120 papers16 papers in the last four weeks, down 16% on the four weeks before. 0.1% of all new papers.

Apr 14, 2026

Algorithmic Analysis of Dense Associative Memory: Finite-Size Guarantees and Adversarial Robustness

Dense Associative Memory (DAM) generalizes Hopfield networks through higher-order interactions and achieves storage capacity that scales as O(Nn−1)O(N^{n-1}) under suitable pattern separation conditions. Existing dynamical analyses primarily study the thermodynamic limit N→∞N\to\infty with randomly sampled patterns and therefore do not provide finite-size guarantees or explicit convergence rates. We develop an algorithmic analysis of DAM retrieval dynamics that yields finite-NN guarantees under explicit, verifiable pattern conditions. Under a separation assumption and a bounded-interference condition at high loading, we prove geometric convergence of asynchronous retrieval dynamics, which implies O(log⁡N)O(\log N) convergence time once the trajectory enters the basin of attraction. We further establish adversarial robustness bounds expressed through an explicit margin condition that quantifies the number of corrupted bits tolerable per sweep, and derive capacity guarantees that scale as Θ(Nn−1)Θ(N^{n-1}) up to polylogarithmic factors in the worst case, while recovering the classical Θ(Nn−1)Θ(N^{n-1}) scaling for random pattern ensembles. Finally, we show that DAM retrieval dynamics admit a potential-game interpretation that ensures convergence to pure Nash equilibria under asynchronous updates. Complete proofs are provided in the appendices, together with preliminary experiments that illustrate the predicted convergence, robustness, and capacity scaling behavior.

Madhava Gaikwad

Independent Researcher

Associative MemoryNeural Network Robustness
AskPDF
Apr 13, 2026

Beyond LLMs, Sparse Distributed Memory, and Neuromorphics <A Hyper-Dimensional SRAM-CAM "VaCoAl" for Ultra-High Speed, Ultra-Low Power, and Low Cost>

VaCoAl (Python: PyVaCoAl) attacks the Binding Problem algebraically rather than statistically: an SRAM/DRAM-CAM organised end-to-end around one primitive, XOR-and-shift over GF(2) via primitive-polynomial LFSRs. Rooted in Kanerva's Sparse Distributed Memory, it retrieves in a million-dimensional binary space by Galois-field diffusion; Binding and Unbinding are exactly reversible at O(L), giving compositional generalisation with post-hoc auditability. Unexpectedly, a path-dependent semantic selection mechanism emerges undesigned, functionally equivalent to Spike-Timing-Dependent Plasticity (STDP) and predictable a priori from a closed form matching measurement on tens of millions of records. It is the Don't Care (collision-tolerance) rule: not a defect to engineer away, but the source of its ability to rank paths by quality. Capability: reversible binding plus tolerated collisions yields a path-integral confidence measure (CR2) ranking candidates by accumulated reliability, absent from hash-based search. Necessity: repair the collisions fully and the ranking vanishes; a memory that never fails has no record of which paths were difficult. Position: we do not surpass large language models; the substrate is orthogonal, supplying the auditable path-ranking layer they lack. A mentor-student ontology of ~470,000 WIKIDATA scholars, traversed backwards from all 64 Fields Medalists to depth 57 and >25.5M paths, stress-tests multi-hop reasoning over a large DAG. Ablation shows Don't Care imposing a depth-dependent exponential penalty that prunes circuitous routes and preserves direct ones: an Occam's razor. Per-generation CR1 stays within 0.995-0.999, yet cumulative CR2 decays monotonically to ~0.905 by generation 56, matching the closed form 0.997^56 = 0.846 within observed variation. All measurements are from PyVaCoAl, a software DRAM-CAM; speed and power figures are projections.

Hiroyuki Chuma, Kanji Otsuka, Yoichi Sato

Professor Emeritus, Hitotsubashi University · Professor Emeritus, Meisei University · Shuhari System

Associative MemoryHyperdimensional Computing
AskPDF
Apr 9, 2026

Breaking the KV Cache Bottleneck: Fan Duality Model Achieves O(1) Decode Memory with Superior Associative Recall

We present FDM (Fan Duality Model), a linear sequence architecture that resolves the fundamental tension between memory efficiency and associative recall in sequence modeling. FDM separates sequence processing into two components: a wave component (recurrent scan via phase-preserving Givens rotations) that compresses long-range patterns into a fixed-size complex hidden state, and a particle component (local-global cache) that retrieves specific tokens via learned associative addressing with W+K=272 slots independent of sequence length N. This yields strictly O(1) decode memory: 867 MB fixed across all prompt lengths 128-8,192 tokens, versus Transformer's 853-4,247 MB (4.9x reduction at N=8,192). Beyond the architecture, we discover that jointly training the wave and particle components leads to suboptimal convergence. We propose Freeze-Scan, a two-phase training strategy that freezes the recurrent scan and optimizes the cache jointly with embeddings, achieving PPL=64.9 on WikiText-103 in 44K steps -- a 7.5x improvement over full fine-tuning (PPL=487). On Multi-Query Associative Recall (MQAR), FDM achieves 0.966 accuracy, surpassing Transformer (0.606) by 59.5%, while pure scan without cache scores only 0.011, confirming the necessity of the particle component. Finally, we introduce Holographic Reference Beam Decoding, interpreting the complex hidden state h_t as a holographic plate encoding the entire temporal history. Using the current input x_t as a reference beam to modulate h_t reduces PPL by up to 2.13 points (PPL=62.79) with a 4-head orthogonal reference beam using only 1.3M additional parameters, providing empirical support for the holographic interpretation. Code and pretrained weights: https://github.com/YasongFan/FDM

Yasong Fan

Independent Researcher

LLM Inference AccelerationAssociative MemoryPersistent Memory for Language ModelsLinear RNNs
AskPDF
Apr 8, 2026

Geometric Entropy and Retrieval Phase Transitions in Continuous Thermal Dense Associative Memory

We study the thermodynamic memory capacity of modern Hopfield networks (Dense Associative Memory models) with continuous states under geometric constraints, extending classical analyses of pairwise associative memory. We derive thermodynamic phase boundaries for Dense Associative Memory networks with exponential capacity M=eαNM = e^{αN}, comparing Gaussian (LSE) and Epanechnikov (LSR) kernels. For continuous neurons on an NN-sphere, the geometric entropy depends solely on the spherical geometry, not the kernel. In the sharp-kernel regime, the maximum theoretical capacity α=0.5α= 0.5 is achieved at zero temperature; below this threshold, a critical line separates retrieval from non-retrieval. The two kernels differ qualitatively in their phase boundary structure: for LSE, a critical line exists at all loads α>0α> 0. For LSR, the finite support introduces a threshold αthα_{\text{th}} below which no spurious patterns contribute to the noise floor, and no critical line exists -- retrieval is perfect at any temperature. These results advance the theory of high-capacity associative memory and clarify fundamental limits of retrieval robustness in modern attention-like memory architectures.

Tatiana Petrova, Evgeny Polyachenko, Radu State

Interdisciplinary Centre for Security, Reliability and Trust (SnT), University of Luxembourg, 29 Avenue J.F. Kennedy, L-1855 Luxembourg.

Associative MemoryHopfield NetworksStatistical Physics of Learning
AskPDF
Apr 6, 2026

Energy-Based Dynamical Models for Neurocomputation, Learning, and Optimization

Recent advances at the intersection of control theory, neuroscience, and machine learning have revealed novel mechanisms by which dynamical systems perform computation. These advances encompass a wide range of conceptual, mathematical, and computational ideas, with applications for model learning and training, memory retrieval, data-driven control, and optimization. This tutorial focuses on neuro-inspired approaches to computation that aim to improve scalability, robustness, and energy efficiency across such tasks, bridging the gap between artificial and biological systems. Particular emphasis is placed on energy-based dynamical models that encode information through gradient flows and energy landscapes. We begin by reviewing classical formulations, such as continuous-time Hopfield networks and Boltzmann machines, and then extend the framework to modern developments. These include dense associative memory models for high-capacity storage, oscillator-based networks for large-scale optimization, and proximal-descent dynamics for composite and constrained reconstruction. The tutorial demonstrates how control-theoretic principles can guide the design of next-generation neurocomputing systems, steering the discussion beyond conventional feedforward and backpropagation-based approaches to artificial intelligence.

Arthur N. Montanari, Francesco Bullo, Dmitry Krotov +1

Center for Network Dynamics and the Department of Physics & Astronomy, Northwestern University, Evanston, IL 60208, USA · Center for Control, Dynamical Systems, and Computation and the Department of Mechanical Engineering, University of California at Santa Barbara, Santa Barbara, CA 93106, USA · Independent Researcher, Cambridge, MA 02141, USA +1

Energy-Based ModelsNeural Network OptimizationNeuromorphic ComputingAssociative MemoryHopfield Networks
AskPDF
Apr 6, 2026

Phase-Associative Memory: Sequence Modeling in Complex Hilbert Space

Experiments probing natural language processing by both humans and LLMs suggest that the meaning of a semantic expression is indeterminate prior to the act of interpretation rather than being specifiable simply as the sum of its parts (i.e. compositionality). This observer-dependent act dynamically actualizes meaning under genuine contextuality more consistent with quantum logical mechanisms than with classical Boolean approaches that assume separability, motivating an approach to language modeling that utilizes a Hilbert space formalism. In this work, we introduce Phase-Associative Memory (PAM) -- a complex-valued sequence model whose state S_t \in \mathbb{C}^{d \times d} accumulates outer products of complex token embeddings retrieved through the conjugate inner product Re⟨K∣Q⟩/d\mathrm{Re}\langle K \mid Q\rangle / \sqrt{d} -- and evaluate it against a structurally matched real-valued ablation. Both architectures train stably across a 5M--100M parameter sweep on WikiText-103 under identical conditions; PAM sits at higher absolute loss at every measured scale but improves more rapidly with parameter count, with power-law exponents of −0.15-0.15 vs.\ −0.12-0.12 in loss and −0.65-0.65 vs.\ −0.49-0.49 in perplexity that narrow the gap between the two architectures monotonically. Further investigation of complex-valued sequence modeling at larger scales could reveal that the loss plateau characteristic of real-valued state-of-the-art language models (e.g. transformers) is reachable with PAM-style architectures with an order of magnitude fewer parameters than the current frontier (∼\sim1T), implying that similar capabilities are achievable at sizes runnable on consumer-grade hardware.

Gowrav Vishwakarma, Christopher J. Agostino

Xavoc Technocrats Pvt. Ltd., https: // xavoc. com , India · NPC Worldwide, Bloomington, Indiana 47403, USA

Autoregressive Language ModelingComplex-Valued Neural NetworksAssociative MemoryQuantum-Inspired MLMemory-Augmented Language Models
AskPDF
Apr 1, 2026

Oscillator-Based Associative Memory with Exponential Capacity: Theory, Algorithms, and Hardware Implementation

Associative memory systems enable content-addressable storage and retrieval of patterns, a capability central to biological neural computation and artificial intelligence. Classical implementations such as Hopfield networks face fundamental limitations in memory capacity, scaling at most linearly with network size. We present an associative memory architecture based on Kuramoto oscillator networks with honeycomb topology in which memories are encoded as stable phase-locked configurations. The honeycomb network consists of multiple cycles that share nodes in a chain-like arrangement, creating a one-dimensional lattice of chained+loops. We prove that this architecture achieves exponential memory capacity: a network of NN oscillators can store (2⌈nc/4⌉−1)m(2\lceil n_c/4 \rceil - 1)^m distinct patterns, where mm honeycomb cycles each contain ncn_c oscillators. Moreover, we fully characterize all stable configurations and prove that each memory's basin of attraction maintains a guaranteed minimum size independent of network scale. Simulations using charge-density-wave (CDW) oscillators validate predicted phase-locking behavior, demonstrating practical realizability in neuromorphic hardware.

Arie Ogranovich, Taosha Guo, Arvind R. Venkatakrishnan +3

Department of Mechanical Engineering, University of California, Santa Barbara, Santa Barbara, 93106, USA · Department of Electrical Engineering and Computer Science, University of California, Irvine, Irvine, CA, 92697, USA

Kuramoto ModelAssociative MemoryOscillatory Neural NetworksDynamical SystemsNeuromorphic Computing
AskPDF
Apr 1, 2026

Associative-State Universal Transformers: Sparse Retrieval Meets Structured Recurrence

We study whether a structured recurrent state can serve as a compact associative backbone for language modeling while still supporting exact retrieval. We introduce UniMatrix, a Universal Transformer style family that reuses a shared recurrent block across depth and augments it with hybrid state updates, a ROSA-style residual path, and token-conditioned embedding modulation. We evaluate these models on byte-level WikiText-2, synthetic associative recall, throughput profiling on Apple MPS, and a corrected benchmark for triple-token interactions. At small scale, UniMatrix-Core and UniMatrix-ROSA slightly outperform a parameter-matched Transformer on WikiText-2 while using many fewer parameters, reaching 5.084 and 5.083 bits-per-byte versus 5.124. The main negative result is equally important: on associative recall, the original UniMatrix family remains near chance while the Transformer reaches 25.4 percent, showing that compressed recurrent state alone is not enough for exact lookup. A retrieval-oriented follow-up, UniMatrix-Assoc, helps only marginally. By contrast, UniMatrix-SparsePointer, which adds sparse slot routing and direct pointer-logit fusion, reaches 75.6 percent on the original pilot recipe and 99.2 percent on a no-dropout follow-up while using 53.8 percent fewer parameters than the Transformer baseline. Ablations show that the gain comes from sufficient slot capacity and exact pointer-level output routing. Overall, structured recurrent state is promising and parameter-efficient, but strong long-range behavior still requires explicit sparse retrieval and better kernels.

Liu Xiao

Liu Xiao

Recurrent TransformersAutoregressive Language ModelingAssociative MemoryLearned Sparse Retrieval
AskPDF
Mar 30, 2026

Human-Like Lifelong Memory: A Neuroscience-Grounded Architecture for Infinite Interaction

Large language models lack persistent, structured memory for long-term interaction and context-sensitive retrieval. Expanding context windows does not solve this: recent evidence shows that context length alone degrades reasoning by up to 85% - even with perfect retrieval. We propose a bio-inspired memory framework grounded in complementary learning systems theory, cognitive behavioral therapy's belief hierarchy, dual-process cognition, and fuzzy-trace theory, organized around three principles: (1) Memory has valence, not just content - pre-computed emotional-associative summaries (valence vectors) organized in an emergent belief hierarchy inspired by Beck's cognitive model enable instant orientation before deliberation; (2) Retrieval defaults to System 1 with System 2 escalation - automatic spreading activation and passive priming as default, with deliberate retrieval only when needed, and graded epistemic states that address hallucination structurally; and (3) Encoding is active, present, and feedback-dependent - a thalamic gateway tags and routes information between stores, while the executive forms gists through curiosity-driven investigation, not passive exposure. Seven functional properties specify what any implementation must satisfy. Over time, the system converges toward System 1 processing - the computational analog of clinical expertise - producing interactions that become cheaper, not more expensive, with experience.

Diego C. Lerma-Torres

Universidad de Guanajuato

Persistent Memory for Language ModelsLong-Term Conversational MemoryMemory-Augmented Language ModelsContinual LearningAssociative Memory
AskPDF
Mar 28, 2026

The Price of Meaning: Why Every Semantic Memory System Forgets

Every major AI memory system in production today organises information by meaning. That organisation enables generalisation, analogy, and conceptual retrieval -- but it comes at a price. We prove that the same geometric structure enabling semantic generalisation makes interference, forgetting, and false recall inescapable. We formalise this tradeoff for semantically continuous kernel-threshold memories: systems whose retrieval score is a monotone function of an inner product in a semantic feature space with finite local intrinsic dimension. Within this class we derive four results: (1) semantically useful representations have finite effective rank; (2) finite local dimension implies positive competitor mass in retrieval neighbourhoods; (3) under growing memory, retention decays to zero, yielding power-law forgetting curves under power-law arrival statistics; (4) for associative lures satisfying a δδ-convexity condition, false recall cannot be eliminated by threshold tuning. We test these predictions across five architectures: vector retrieval, graph memory, attention-based context, BM25 filesystem retrieval, and parametric memory. Pure semantic systems express the vulnerability directly as forgetting and false recall. Reasoning-augmented systems partially override these symptoms but convert graceful degradation into catastrophic failure. Systems that escape interference entirely do so by sacrificing semantic generalisation. The price of meaning is interference, and no architecture we tested avoids paying it.

Sambartha Ray Barman, Andrey Starenky, Sofia Bodnar +2

Sentra, 235 2nd Street, San Francisco, CA 94105, USA · Department of Mechanical Engineering, Massachusetts Institute of Technology, 77 Massachusetts Avenue, Cambridge, MA 02139, USA

Semantic SearchAssociative MemoryCatastrophic Forgetting
AskPDF
Mar 27, 2026

Sharp Capacity Scaling of Spectral Optimizers in Learning Associative Memory

Spectral optimizers such as Muon have recently shown strong empirical performance in large-scale language model training, but the source and extent of their advantage remain poorly understood. We study this question through the linear associative memory problem, a tractable model for factual recall in transformer-based models. In particular, we go beyond orthogonal embeddings and consider Gaussian inputs and outputs, which allows the number of stored associations to greatly exceed the embedding dimension. Our main result sharply characterizes the recovery rates of one step of Muon, SGD, and Newton's method on the logistic regression loss under a power law frequency distribution. We show that the storage capacity of Muon significantly exceeds that of SGD, and even matches Newton's method while only using first-order information. Moreover, Muon saturates at a larger critical batch size. We further analyze the multi-step dynamics under a thresholded gradient approximation and show that Muon achieves a substantially faster initial recovery rate than SGD, while both methods eventually converge to the information-theoretic limit at comparable speeds. Experiments on synthetic tasks validate the predicted scaling laws. Our analysis provides a quantitative understanding of the signal amplification of spectral preconditioners and lays the groundwork for establishing scaling laws across more practical language modeling tasks and optimizers.

Juno Kim, Eshaan Nichani, Denny Wu +2

UC Berkeley · Princeton University · New York University +1

Spectral OptimizationNeural Network OptimizationMuon OptimizerSpectral MethodsAssociative Memory
AskPDF
Mar 27, 2026

On associative neural networks for sparse patterns with huge capacities

Generalized Hopfield models with higher-order or exponential interaction terms are known to have substantially larger storage capacities than the classical quadratic model. On the other hand, associative memories for sparse patterns, such as the Willshaw and Amari models, already exhibit enhanced storage capacities in the sparse regime. In this paper we combine these two mechanisms. We introduce higher-order versions of sparse associative memory models and study their storage capacities in the sense of fixed-pattern stability. For the Amari and Willshaw models with fixed interaction order nn, we obtain storage scales of order Nn(log⁡N)n\frac{N^n}{(\log N)^n}. When the interaction order grows logarithmically with the number of neurons, the resulting storage scale becomes super-polynomial. We also study higher-order interactions in the block-structured Gripon--Berrou architecture, where the natural storage scale is of order cnc^n. Our results show that the capacity increase caused by higher-order interactions persists in the sparse setting, while the precise storage scale depends on the underlying architecture.

Matthias Löwe, Franck Vermet

Associative MemoryHopfield Networks
AskPDF
Mar 27, 2026

The Geometry of Forgetting

Why do we forget? Why do we remember things that never happened? The conventional answer points to biological hardware. We propose a different one: geometry. Here we show that high-dimensional embedding spaces, subjected to noise, interference, and temporal degradation, reproduce quantitative signatures of human memory with no phenomenon-specific engineering. Power-law forgetting (b=0.460±0.183b = 0.460 \pm 0.183, human b≈0.5b \approx 0.5) arises from interference among competing memories, not from decay. The identical decay function without competitors yields b≈0.009b \approx 0.009, fifty times smaller. Time alone does not produce forgetting in this system. Competition does. Production embedding models (nominally 384--1{,}024 dimensions) concentrate their variance in only ∼16{\sim}16 effective dimensions, placing them deep in the interference-vulnerable regime. False memories require no engineering at all: cosine similarity on unmodified pre-trained embeddings reproduces the Deese--Roediger--McDermott false alarm rate (0.5830.583 versus human ∼0.55{\sim}0.55) with zero parameter tuning and no boundary conditions. We did not build a false memory system. We found one already present in the raw geometry of semantic space. These results suggest that core memory phenomena are not bugs of biological implementation but features of any system that organizes information by meaning and retrieves it by proximity.

Sambartha Ray Barman, Andrey Starenky, Sophia Bodnar +2

Sentra, 235 2nd Street, San Francisco, CA 94105, USA · Department of Mechanical Engineering, Massachusetts Institute of Technology, 77 Massachusetts Avenue, Cambridge, MA 02139, USA

Associative MemoryCognitive ModelingNeural Representation Geometry
AskPDF
Mar 26, 2026

Associative Memory using Attribute-Specific Neuron Groups-2: Learning and Sequential Associative Recall between Cue Neurons for different Cue Balls

This paper introduces a neural network model that learns multiple attributes as images and performs associated, sequential recall of the learned memories. Briefly, the model presented here is an associative memory model that extends previous models [1] by increasing the number of attributes. In the real world, memory recall generates a chain of associations consisting of complex and diverse data with meaningful relations. However, because this experimental system is designed to implement and verify the processing operations behind such operations, we believe it is not a problem if the associative memory (i.e., the chain of data) is composed of attributes that do not necessarily have clear relation with each other. Accordingly, the attribute-processing systems prepared in this study consist of five types: the C.CB-RN system for processing color attributes, the S.CB-RN system for shape attributes, and the V.CB-RN system for size attributes, as adopted in our previous paper [1], as well as the SV.CB-RN system for processing the names of the world's most beautiful scenery (spectacular view names) and the CN.CB-RN system for processing constellation names. As before, the data presented to each CB-RN system are represented as image patterns using QR codes [2]. These five types of CB-RN systems will be combined and trained with QR code pattern images of the attribute elements of each system. After that, when a pattern image of an attribute element is presented to any of the CB-RN systems, a mechanism will be constructed in which a chain (associative) recall of pattern images of related attribute elements in the other trained systems will be generated.

Hiroshi Inazawa

Center for Education in Information Systems, Kobe Shoin Women’s University１, 1-2-1 Shinohara-Obanoyama, Nada Kobe 657-0015, Japan.

Associative MemoryModular Neural Networks
AskPDF
Mar 22, 2026

Mixture of Chapters: Scaling Learnt Memory in Transformers

Transformers lack an explicit architectural mechanism for storing and organizing knowledge acquired during training. We introduce learnable sparse memory banks: a set of latent tokens, randomly initialized and trained end-to-end, that transformer layers query via cross-attention to retrieve stored knowledge. To scale memory capacity without prohibitive attention costs, we propose chapter-based routing inspired by Mixture-of-Experts architectures, partitioning the memory bank into chapters and training a router to select relevant subsets per input. This enables scaling to 262K memory tokens while maintaining tractable computation. We evaluate our approach against standard transformers (in iso-FLOP settings) on pre-training and instruction fine-tuning across relevant benchmarks. Our models surpass iso-FLOP baselines suggesting scope for a new axis of scaling, demonstrating that explicit associative memory provides complementary capacity to what is captured implicitly in model parameters. Additionally, we observe improved knowledge retention under continued training, with robustness to forgetting when transitioning between training phases (e.g., pretraining to instruction fine-tuning).

Tasmay Pankaj Tibrewal, Pritish Saha, Ankit Meda +2

IIT Kharagpur · Fractal AI

Memory-Augmented Language ModelsPersistent Memory for Language ModelsAssociative Memory
AskPDF
Mar 21, 2026

Sinkhorn Based Associative Memory Retrieval Using Spherical Hellinger Kantorovich Dynamics

We propose a dense associative memory for empirical measures (weighted point clouds). Stored patterns and queries are finitely supported probability measures, and retrieval is defined by minimizing a Hopfield-style log-sum-exp energy built from the debiased Sinkhorn divergence. We derive retrieval dynamics as a spherical Hellinger Kantorovich (SHK) gradient flow, which updates both support locations and weights. Discretizing the flow yields a deterministic algorithm that uses Sinkhorn potentials to compute barycentric transport steps and a multiplicative simplex reweighting. Under local separation and PL-type conditions we prove basin invariance, geometric convergence to a local minimizer, and a bound showing the minimizer remains close to the corresponding stored pattern. Under a random pattern model, we further show that these Sinkhorn basins are disjoint with high probability, implying exponential capacity in the ambient dimension. Experiments on synthetic Gaussian point-cloud memories demonstrate robust recovery from perturbed queries versus a Euclidean Hopfield-type baseline.

Aratrika Mustafi, Soumya Mukherjee

Department of Statistics, Pennsylvania State University

Sinkhorn DivergenceAssociative MemoryOptimal Transport
AskPDF
Mar 20, 2026

Hybrid Associative Memories

Recurrent neural networks (RNNs) and self-attention are both widely used sequence-mixing layers that maintain an internal memory. However, this memory is constructed using two orthogonal mechanisms: RNNs compress the entire past into a fixed-size state, whereas self-attention's state stores every past time step growing its state (the KV cache) linearly with the sequence length. This results in orthogonal strengths and weaknesses. Self-attention layers excel at retrieving information in the context but have large memory and computational costs, while RNNs are more efficient but degrade over longer contexts and underperform for precise recall tasks. Prior work combining these mechanisms has focused primarily on naively interleaving them to reduce computational cost without regard to their complementary mechanisms. We propose the Hybrid Associative Memory (HAM) layer, which combines self-attention and RNNs while leveraging their individual strengths: the RNN compresses the entire sequence, while attention supplements it only with information that is difficult for the RNN to predict, which is hence the most valuable information to explicitly store. HAM layers enable data-dependent growth of the KV cache, which can be precisely controlled by the user with a single, continuous threshold. We find that this fine-grained control of the KV cache growth rate has a smooth trade-off with loss and performance. Empirically, we show that our hybrid architecture offers strong, competitive performance relative to RNNs and Transformers even at substantially lower KV-cache usage.

Leon Lufkin, Tomás Figliolia, Beren Millidge +1

1Zyphra

Associative MemoryRecurrent TransformersHybrid Attention
AskPDF
Mar 19, 2026

From Topic to Transition Structure: Unsupervised Concept Discovery at Corpus Scale via Predictive Associative Memory

Embedding models group text by semantic content, what text is about. We show that temporal co-occurrence within texts discovers a different kind of structure: recurrent transition-structure concepts or what text does. We train a 29.4M-parameter contrastive model on 373 million co-occurrence pairs from 9,766 Project Gutenberg texts (24.96 million passages), mapping pre-trained embeddings into an association space where passages with similar transition structure cluster together. Under capacity constraint (42.75% accuracy), the model must compress across recurring patterns rather than memorise individual co-occurrences. Clustering at six granularities (k=50 to k=2,000) produces a multi-resolution concept map; from broad modes like "direct confrontation" and "lyrical meditation" to precise registers and scene templates like "sailor dialect" and "courtroom cross-examination." At k=100, clusters average 4,508 books each (of 9,766), confirming corpus-wide patterns. Direct comparison with embedding-similarity clustering shows that raw embeddings group by topic while association-space clusters group by function, register, and literary tradition. Unseen novels are assigned to existing clusters without retraining; the association model concentrates each novel into a selective subset of coherent clusters, while raw embedding assignment saturates nearly all clusters. Validation controls address positional, length, and book-concentration confounds. The method extends Predictive Associative Memory (PAM, arXiv:2602.11322) from episodic recall to concept formation: where PAM recalls specific associations, multi-epoch contrastive training under compression extracts structural patterns that transfer to unseen texts, the same framework producing qualitatively different behaviour in a different regime.

Jason Dury

Independent Researcher

Unsupervised Visual Concept DiscoveryText ClusteringAssociative MemoryPredictive Representation LearningUnsupervised ClusteringText Embedding Models+1
AskPDF
Mar 17, 2026

Attractor-Keyed Memory

Physical selectors (lasers choosing a mode, Ising machines settling on a ground state, condensates occupying a spin state) produce high-dimensional signatures at the moment of decision: full field amplitudes, multimode interference patterns, or scattering responses. These signatures are richer than the winner's index, yet they are routinely discarded. We show that when the signatures are repeatable across trials (stereotyped) and linearly independent across routes, a single linear decoder compiled from calibration data maps them to arbitrary payloads, merging selection and memory access into one event and eliminating the fetch that dominates latency and energy in sparse routing architectures. The construction requires one SVD of measured device responses, which certifies capability and bounds worst-case error for any downstream payload before the task is chosen. Runtime error separates into two independently diagnosable channels, decoding fidelity (controlled by dictionary conditioning σmin⁡(Φ)σ_{\min}(Φ)) and routing reliability (controlled by the margin-to-noise ratio Δ/TeffΔ/T_{\mathrm{eff}}), each with a distinct physical origin and targeted remedy. We derive the full error decomposition, give Ising-machine selector constructions, and validate the predicted scalings on synthetic speckle-signature simulations across three measurement modalities. No hardware demonstration exists; we provide a falsifiable four-step experimental protocol specifying what a first experiment must measure. Whether real device signatures satisfy stereotypy is the central open question.

Natalia G. Berloff

Department of Applied Mathematics and Theoretical Physics, University of Cambridge, Cambridge CB3 0WA, United Kingdom

Associative Memory
AskPDF
Mar 7, 2026

Thermal Robustness of Retrieval in Dense Associative Memories: LSE vs LSR Kernels

Understanding whether retrieval in dense associative memories survives thermal noise is essential for bridging zero-temperature capacity proofs with the finite-temperature conditions of practical inference and biological computation. We use Monte Carlo simulations to map the retrieval phase boundary of two continuous dense associative memories (DAMs) on the NN-sphere with an exponential number of stored patterns M=eαNM = e^{αN}: a log-sum-exp (LSE) kernel and a log-sum-ReLU (LSR) kernel. Both kernels share the zero-temperature critical load αc(0)=0.5α_c(0)=0.5, but their finite-temperature behavior differs markedly. The LSE kernel sustains retrieval at arbitrarily high temperatures for sufficiently low load, whereas the LSR kernel exhibits a finite support threshold below which retrieval is perfect at any temperature; for typical sharpness values this threshold approaches αcα_c, making retrieval nearly perfect across the entire load range. We also compare the measured equilibrium alignment with analytical Boltzmann predictions within the retrieval basin.

Tatiana Petrova

Interdisciplinary Centre for Security, Reliability and Trust · Interdisciplinary Centre for Security, Reliability and Trust (SnT), University of Luxembourg

Associative MemoryRetrieval RobustnessStatistical Physics of Learning
AskPDF
Mar 7, 2026

A Miniature Brain Transformer: Thalamic Gating, Hippocampal Lateralization, Amygdaloid Salience, and Prefrontal Working Memory in Attention-Coupled Latent Memory

We present a miniature brain transformer architecture that extends the attention-coupled latent memory framework with four additional brain-region analogues: a thalamic relay, an amygdaloid salience module, a prefrontal working-memory (PFC) buffer, and a cerebellar fast-path, all coupled by inhibitory callosal cross-talk between lateralized hippocampal banks. We evaluate on a two-domain benchmark -- MQAR (Multi-Query Associative Recall; episodic domain) and modular arithmetic (+1 mod 10; rule-based domain) -- using a seven-variant additive ablation. The central empirical finding is a surprise: inhibitory callosal coupling alone never lateralizes the banks (variants 1-5 maintain D_sep ~ 0.25 and P_ct ~ 0.25 for all 30 epochs). Functional lateralization requires the synergy of PFC and inhibition: only when the PFC buffer is added (variant 6) does a sharp, discontinuous phase transition fire -- at epoch 11 for the PFC-only variant and epoch 10 for the full model -- collapsing P_ct from 0.25 to ~0.002 and more than doubling D_sep from 0.251 to 0.501 in a single gradient step. The PFC buffer acts as a symmetry-breaker: its slowly drifting domain context creates the initial asymmetry that the inhibitory feedback loop then amplifies irreversibly. The cerebellar fast-path accelerates the transition by one epoch (epoch 10 vs. epoch 11) with no asymptotic change, confirming its convergence-acceleration role. The result constitutes a novel, falsifiable prediction -- no lateralization without working memory context -- and a principled, neurobiologically motivated blueprint for hierarchical persistent memory in sequence models.

Hong Jeong

Department of Computer Information Engineering, Inha University in Tashkent, Uzbekistan

Memory-Augmented Neural NetworksCognitive ModelingAssociative MemoryTransformer Attention
AskPDF
Mar 3, 2026

A Dynamical Theory of Sequential Retrieval in Input-Driven Hopfield Networks

Reasoning is the ability to integrate internal states and external inputs in a meaningful and semantically consistent flow. Contemporary machine learning (ML) systems increasingly rely on such sequential reasoning, from language understanding to multi-modal generation, often operating over dictionaries of prototypical patterns reminiscent of associative memory models. Understanding retrieval and sequentiality in associative memory models provides a powerful bridge to gain insight into ML reasoning. While the static retrieval properties of associative memory models are well understood, the theoretical foundations of sequential retrieval and multi-memory integration remain limited, with existing studies largely relying on numerical evidence. This work develops a dynamical theory of sequential reasoning in Hopfield networks. We consider the recently proposed input-driven plasticity (IDP) Hopfield network and analyze a two-timescale architecture coupling fast associative retrieval with slow reasoning dynamics. We derive explicit conditions for self-sustained memory transitions, including gain thresholds, escape times, and collapse regimes. Together, these results provide a principled mathematical account of sequentiality in associative memory models, bridging classical Hopfield dynamics and modern reasoning architectures.

Simone Betteti, Giacomo Baggio, Sandro Zampieri

The Italian Institute of Artificial Intelligence for Industry · Department of Information Engineering, Università degli Studi di Padova

Hopfield NetworksAssociative MemoryDynamical Systems
AskPDF
Mar 3, 2026

Look Forward to Walk Backward: Efficient Terrain Memory for Backward Locomotion with Forward Vision

Legged robots with egocentric forward-facing depth cameras can couple exteroception and proprioception to achieve robust forward agility on complex terrain. When these robots walk backward, the forward-only field of view provides no preview. Purely proprioceptive controllers can remain stable on moderate ground when moving backward but cannot fully exploit the robot's capabilities on complex terrain and must collide with obstacles. We present Look Forward to Walk Backward (LF2WB), an efficient terrain-memory locomotion framework that uses forward egocentric depth and proprioception to write a compact associative memory during forward motion and to retrieve it for collision-free backward locomotion without rearward vision. The memory backbone employs a delta-rule selective update that softly removes then writes the memory state along the active subspace. Training uses hardware-efficient parallel computation, and deployment runs recurrent, constant-time per-step inference with a constant-size state, making the approach suitable for onboard processors on low-cost robots. Experiments in both simulations and real-world scenarios demonstrate the effectiveness of our method, improving backward agility across complex terrains under limited sensing.

Shixin Luo, Songbo Li, Yuan Hao +4

The authors are with Institute of Cyber-Systems and Control, Zhejiang University, 310027, China. · The author is with College of Artificial Intelligence, Nankai University, 300350, China. · The author is with Hangzhou Public Library, 310016, China.

Robot LocomotionLegged RoboticsTerrain-Aware Robot LocomotionVision-Based Robot ControlAssociative Memory
AskPDF
Feb 27, 2026

Inhibitory Cross-Talk Enables Functional Lateralization in Attention-Coupled Latent Memory

We present a memory-augmented transformer in which attention serves simultaneously as a retrieval, consolidation, and write-back operator. The core update, A⊤AVWA^\top A V W, re-grounds retrieved values into persistent memory slots via the Gram matrix A⊤AA^\top A, providing a principled tripartite projection: observation space →\to latent memory →\to supervised transformation. We partition the memory into lateralized left and right banks coupled through a sign-controlled cross-talk matrix WsW_s, and show that the sign of this coupling is decisive for specialization. Excitatory cross-talk (s=+1s=+1) causes bank-dominance collapse: one bank monopolises all inputs and Pct→0.5\mathcal{P}_{ct} \to 0.5, despite lowering task loss. Inhibitory cross-talk (s=−1s=-1), motivated by the net inhibitory effect of callosal projections in human cortex, actively suppresses contralateral bank activation and achieves saturated specialization (Dsep=±1.00\mathcal{D}_{sep} = \pm 1.00, Pct≈0\mathcal{P}_{ct} \approx 0). On a controlled symbolic benchmark combining an episodic bijection cipher (requiring associative recall) with a strict arithmetic progression (requiring rule extraction), the inhibitory model reduces cipher-domain loss by 124×124{\times} over the baseline while matching it on the arithmetic domain, confirming that persistent lateralized memory is necessary for episodic recall but not for rule-based prediction.

Hong Jeong

Department of Computer Information Engineering, Inha University in Tashkent, Uzbekistan

Memory-Augmented Neural NetworksEpisodic MemoryAssociative Memory
AskPDF
Feb 16, 2026

Drift-Diffusion Matching: Embedding dynamics in latent manifolds of asymmetric neural networks

Recurrent neural networks (RNNs) provide a theoretical framework for understanding computation in biological neural circuits, yet classical results, such as Hopfield's model of associative memory, rely on symmetric connectivity that restricts network dynamics to gradient-like flows. In contrast, biological networks support rich time-dependent behaviour facilitated by their asymmetry. Here we introduce a general framework, which we term drift-diffusion matching, for training continuous-time RNNs to represent arbitrary, nonlinear stochastic differential equations (SDEs), with given drift and diffusion coefficients, within a low-dimensional latent subspace. Allowing asymmetric connectivity, we show that RNNs can faithfully embed the drift and diffusion of a given SDE, including nonlinear and nonequilibrium dynamics such as chaotic attractors. As an application, we construct RNN realisations of stochastic systems that transiently explore various attractors through both input-driven switching and autonomous transitions driven by nonequilibrium currents, which we interpret as models of associative and sequential (episodic) memory. To elucidate how these dynamics are encoded in the network, we introduce decompositions of the RNN based on its asymmetric connectivity and its time-irreversibility. Our results extend attractor neural network theory beyond equilibrium, showing that asymmetric neural populations can implement a broad class of dynamical computations within low-dimensional manifolds, unifying ideas from associative memory, nonequilibrium statistical mechanics, and neural computation.

Ramón Nartallo-Kaluarachchi, Renaud Lambiotte, Alain Goriely

Mathematical Institute, University of Oxford, Woodstock Road, Oxford, OX2 6GG, United Kingdom · Centre for Eudaimonia and Human Flourishing, University of Oxford, 7 Stoke Pl, Oxford, OX3 9BX, United Kingdom · Complexity Science Hub, Metternichgasse 8, Vienna, 1030, Austria

Recurrent Neural NetworksDynamical SystemsLatent Dynamics ModelingStochastic Differential EquationsAssociative Memory
AskPDF
Feb 12, 2026

RAM-Net: Linear-Time Sequence Modeling with Sparsely Addressable State

Linear attention offers an efficient alternative to full attention with a fixed-size recurrent state. However, this state is shared by all tokens, so information from distinct tokens becomes superposed within it and produces inter-token interference that degrades long-range fine-grained recall. To address this issue, we propose RAM-Net, which replaces dense access to a shared state with sparse address-based access. RAM-Net organizes the recurrent state as a fixed-size array of independent slots and uses an Address Decoder that maps each key or query into a sparse address, selecting a small subset of slots to write to or read from at each step. This design directs tokens with non-overlapping addresses to disjoint slots, suppressing inter-token interference, while keeping per-step state access dependent only on the number of selected slots rather than the total state size. Empirically, RAM-Net outperforms strong recurrent baselines on fine-grained long-range retrieval and achieves the lowest perplexity with competitive commonsense reasoning. It does so while accessing fewer state elements per step than all baselines, e.g., 8×8\times fewer than Mamba2.

Kaicheng Xiao, Haotian Li, Liran Dong +1

The Chinese University of Hong Kong

Memory-Augmented Neural NetworksLong-Context Language ModelingLinear AttentionAssociative Memory
AskPDF
Nov 28, 2025

Distributed Dynamic Associative Memory via Online Convex Optimization

An associative memory (AM) enables cue-response recall, and it has recently been recognized as a key mechanism underlying modern neural architectures such as Transformers. In this work, we introduce the concept of distributed dynamic associative memory (DDAM), which extends classical AM to settings with multiple agents and time-varying data streams. In DDAM, each agent maintains a local AM that must not only store its own associations but also selectively memorize information from other agents based on a specified interest matrix. To address this problem, we propose a novel tree-based distributed online gradient descent algorithm, termed DDAM-TOGD, which enables each agent to update its memory on the fly via inter-agent communication over designated routing trees. We derive rigorous performance guarantees for DDAM-TOGD, proving sublinear static regret in stationary environments and a path-length dependent dynamic regret bound in non-stationary environments. These theoretical results provide insights into how communication delays and network structure impact performance. Building on the regret analysis, we further introduce a combinatorial tree design strategy that optimizes the routing trees to minimize communication delays, thereby improving regret bounds. Numerical experiments demonstrate that the proposed DDAM-TOGD framework achieves superior accuracy and robustness compared to representative online learning baselines such as consensus-based distributed optimization, confirming the benefits of the proposed approach in dynamic, distributed environments.

Bowen Wang, Matteo Zecchin, Osvaldo Simeone

Department of Engineering, King’s College London, London WC2R 2LS, U.K. · Communication Systems Department, EURECOM, 06904 Sophia Antipolis, France · Institute for Intelligent Networked Systems (INSI) at Northeastern University London, London E1 8PH, U.K.

Online Convex OptimizationAssociative MemoryDecentralized Optimization
AskPDF
Nov 4, 2025

Redundancy Maximization as a Principle of Associative Memory Learning in Hopfield Networks

Associative memory, traditionally modeled by Hopfield networks, enables the retrieval of previously stored patterns from partial or noisy cues. Yet, the local computational principles which are required to enable this function remain incompletely understood. To formally characterize the local information processing in such systems, we employ a recent extension of information theory -- Partial Information Decomposition (PID). PID decomposes the contribution of different inputs to an output into unique information from each input, redundant information across inputs, and synergistic information that emerges from combining different inputs. Applying this framework to individual neurons in classical Hopfield networks we find that below the memory capacity, the information in a neuron's activity is characterized by high redundancy between the external pattern input and the internal recurrent input, while synergy and unique information are close to zero until the memory capacity is surpassed and performance drops steeply. Inspired by this observation, we use redundancy maximization at each neuron as an information-theoretic learning goal. This dramatically increases the network's memory capacity to 1.59, a more than tenfold improvement over the 0.14 capacity of classical Hopfield networks, and also outperforming recent state-of-the-art implementations of Hopfield networks. Overall, this work establishes redundancy maximization as a new design principle for associative memories and opens pathways for new associative memory models based on information-theoretic goals.

Mark Blümel, Andreas C. Schneider, Valentin Neuhaus +5

Complex Systems Theory, Max Planck Institute for Dynamics and Self-Organization, Göttingen, Germany · Faculty of Physics, Institute for the Dynamics of Complex Systems, University of Göttingen · Göttingen Campus Institute for Dynamics of Biological Networks, University of Göttingen, Göttingen, Germany +1

Neural Network MemorizationHopfield NetworksAssociative Memory
AskPDF
Jul 14, 2025

Dynamical stability for dense patterns in attractor neural networks

Recurrent neural networks are canonical models of biological memory. In these models, memories are represented by distributed patterns of neural activity that are stored in the recurrent connections between neurons, such that they become attractors of the network's dynamics. During memory recall, network dynamics thus converge toward one of these memory patterns when started from a noisy or partial cue. Therefore, memory performance critically hinges on the dynamical stability of the stored patterns. However, previous theoretical approaches only studied dynamical stability under highly restrictive conditions that do not readily apply to biological neural circuits. Here, we develop a theory of the local stability of discrete fixed points in a broad class of networks with graded neural activities and in the presence of noise. Using methods from random matrix theory, we analyze the bulk and outliers of the eigenvalue spectra of the Jacobians that characterize network dynamics around fixed points. We show that either all fixed points are stable or all of them are unstable, depending on whether their number is below a ``critical load for stability'', which is distinct from the classical critical capacity that measures the maximal number of achievable fixed points regardless of their stability. We further analyze the dependence of this critical load for stability on experimentally measurable quantities characterizing the statistics of memory patterns and the activation functions of neurons. Our analysis highlights the computational benefits of sparse-like patterns and threshold-linear activation functions and offers testable predictions for neural circuits supporting memory.

Uri Cohen, Máté Lengyel

Computational and Biological Learning Lab, Department of Engineering, University of Cambridge, Cambridge, UK · Center for Cognitive Computation, Department of Cognitive Science, Central European University, Budapest, Hungary

Neural Network MemorizationAssociative MemoryNeural ProcessesDynamical SystemsHopfield NetworksRecurrent Neural Networks+1
AskPDF
Jun 7, 2021

One-shot learning of paired association navigation with biologically plausible schemas

After learning a multiple paired-association (MPA) navigation task, rodents can rapidly learn new cue-location associations after a single exposure, a phenomenon proposed to be facilitated by cognitive schemas. However, it remains unclear what schemas support this behavior and how they could be implemented in biological neural circuits. Here, we identify a set of schemas sufficient for gradual then one-shot paired-association navigation, and implement them in a fully neural agent using biologically plausible plasticity rules. The agent gradually acquires a low-dimensional spatial metric representation using a path-integration based temporal-difference error, enabling self-localization across different environments. It also contains a memory network that forms one-shot associations between sensory cues and goal locations, implemented using reward-modulated exploratory Hebbian (EH) plasticity. A third network performs vector subtraction between the agent's current and goal location to compute goal-directed actions. Supplementing these schemas with an actor-critic enables one-shot navigation past obstacles. Lastly, learning a working-memory gating mechanism using the temporal-difference error enables one-shot learning even with brief cue and distractor presentation. The model reproduces key behavioral features of rodent learning in the MPA task and generates testable predictions about the neural mechanisms underlying schema-based one-shot learning.

M Ganesh Kumar, Cheston Tan, Camilo Libedinsky +2

Integrative Sciences and Engineering Programme, NUS Graduate School, National University of Singapore · The N.1 Institute for Health, National University of Singapore · Engineering Design and Innovation Centre, College of Design and Engineering, National University of Singapore +8

Few-Shot LearningKnowledge RepresentationEmbodied NavigationAssociative MemoryBiologically Plausible Learning
AskPDF