Associative Memory

Ask about this topic

Overview

Associative memory asks how a system can store information so that a cue brings back a related pattern, fact, or sequence, a problem central to neural memory and efficient AI. Classical Hopfield networks, dense and kernel associative memories, and Hebbian learning use attractor dynamics or similarity-based retrieval; newer recurrent and linear-attention models add sparse key-value stores, while agent systems use linked episodic memories. Theory examines capacity, retrieval basins, stability, and interference, while experiments test noisy pattern completion, long-range recall, language-model memory, and hardware implementations. A persistent tension is that more storage does not guarantee reliable, selective recall: finite state and overlapping representations can cause interference, forgetting, and false associations, even as explicit retrieval improves access at added cost.

Latest papers

CardsList

120 papers16 papers in the last four weeks, down 16% on the four weeks before. 0.1% of all new papers.

May 31, 2026

Blurry Window Attention

The Softmax Attention operation in Transformer language models has a quadratic complexity in the sequence length and a growing state size in the form of KV cache, which becomes a bottleneck in long context scenarios. To overcome this limitation, alternative architectures with linear complexity and finite state size have been introduced, such as State-Space Models (SSMs), Linear Attention (LA), and Attention with Bounded-memory Control (ABC). Though linear models achieve similar language perplexity as Transformers, they are still behind in tasks which require retrieval or recall of specific information. In this work, we introduce Blurry Window Attention (BLA) a novel ABC method inspired by SSMs. BLA stores a frequency window from which a blurry KV history is reconstructed via interpolation using Dirichlet kernels. BLA can be understood as a generalization of Sliding Window Attention (SWA) depending on the Dirichlet kernels resolution or as a special case of the Gated Slot Attention (GSA), where the decay factor is implemented with Dirichlet kernels. We describe in details the theory and efficient implementation of BLA. On the Multi-Query Associate Recall (MQAR) synthetic task, we show that the state efficiency of BLA is 8×\times better than SWA and is competitive with popular linear attention models, and in the RegBench synthetic task, only BLA and SWA improve their performance as the state size grows among the linear models we tested.

Axel Laborieux, Christos Sourmpis, Juan Gabriel Kostelec +1

Huawei, Zurich, Switzerland · Huawei Advanced Computing and Storage Lab, Shenzhen, China

Long-Context Language ModelingLinear AttentionAttention MechanismsAssociative Memory
AskPDF
May 29, 2026

Memory by Design: Bayesian Layers for Sequence Modeling

We introduce the design-model framework: a way to derive efficient recurrent sequence maps from explicit assumptions about memory. A design model writes evidence into memory by exact Bayesian filtering; a query- dependent readout produces a predictive distribution whose mean is the layer output. In our linear-Gaussian instantiation, the Bayesian Layer propagates both a mean and a covariance: the covariance tracks uncertainty over stored associations, steering writes toward uncertain directions, attenuating gains as evidence accumulates, and preserving confident memories. The same framework unifies several sub-quadratic recurrences: linear attention, GLA, and Mamba-2/SSD are exact filters under a latent-input design model, whereas DeltaNet and related Delta-rule models are covariance-reset reductions of the Bayesian Layer's design model. Restoring covariance propagation yields closed-form predictions for retrieval dynamics, which we verify empirically, and improves robustness beyond the training regime in controlled collision studies, learned associative recall, and the Zoology MQAR benchmark. Training from scratch on WikiText-103 under matched state budgets lowers perplexity on associative-recall hits. Distilling Bayesian Layers into a pretrained 340M Gated DeltaNet improves RULER long-context retrieval over a matched-compute control, at a 2.5--2.7% held-out perplexity cost.

Matthew Dowling, Hyungju Jeon, Cristina Savin +1

Champalimaud Research, Champalimaud Foundation, Portugal · Center for Neural Science, New York University, USA · Center for Data Science, New York University, USA

Long-Context RetrievalBayesian FilteringRecurrent Neural NetworksAssociative MemorySequence Modeling
AskPDF
May 23, 2026

Beyond Fixed Points: Superpolynomial Capacity of Asymmetric Hopfield Networks

Classical Hopfield networks are limited to static patterns due to symmetric weights, whereas asymmetric networks can encode temporal sequences via limit-cycle attractors. Achieving high-capacity storage of long sequences in classical synchronous asymmetric networks, however, has remained a challenge. We present a simple and robust construction within the classical asymmetric Hopfield model with binary neurons and synchronous updates, that allows nn neurons to support exp⁡ ⁣(Ω(n/(log⁡n)2))\exp\!\big(Ω(n/(\log n)^2)\big) distinct limit-cycle attractors, each with period exp⁡ ⁣(Ω(n/log⁡n))\exp\!\big(Ω(\sqrt n/\log n)\big) and robust to random noise with flip probability up to 12−o(1)\frac12-o(1), yielding superpolynomial capacity in both the number and length of stored sequences. This is the first demonstration of such capacity for asymmetric Hopfield networks, which we obtain by combining results from combinatorics, number theory and the analysis of opinion dynamics. Our findings show that synchronous asymmetric Hopfield networks possess a sequence-memory capacity which is larger and more robust than previously recognized, demonstrating that, in both biological and artificial neural systems, robust sequence representation can be achieved through coarse architectural motifs rather than complex nonlinearities.

Aakash Kumar, Anatoly Khina, Frederik Mallmann-Trenn +1

COATI, CNRS, Inria, I3S, Université Côte d’Azur, France · School of Electrical and Computer Engineering, Tel Aviv University, Israel · Department of Informatics, King’s College London, UK +1

Dynamical SystemsHopfield NetworksAssociative Memory
AskPDF
May 15, 2026

Bridging Silicon and the Hippocampus: Algebro-Deterministic Memory "VaCoAl" as a Substrate for Vector-HaSH and TEM

Vector-HaSH and the Tolman-Eichenbaum Machine (TEM) propose the hippocampal-entorhinal circuit factorizes memory via a grid-cell scaffold for compositional replay. Concurrently, human iEEG shows sharp-wave ripples gate recall and multi-hop replay fidelity decays multiplicatively. Yet, these fields lack a shared algebraic foundation. We introduce VaCoAl, an algebro-deterministic hyperdimensional memory architecture built on Galois-field linear-feedback shift registers. Its deterministic Galois-field diffusion offers a substrate-level alternative to Vector-HaSH's random projections, matching quasi-orthogonality while ensuring bit-exact reproducibility. Furthermore, the path-integral Confidence Ratio CR2 provides an algebraically tractable model for the empirically observed multiplicative replay decay. Biologically, VaCoAl's two operating regimes align with the EC-CA3 direct and EC-DG-CA3 trisynaptic pathways, explaining their 520-Myr conservation. Independent cellular evidence supports that the DG-CA3 pathway implements a biophysical homologue of Galois-field arithmetic. We also link this framework to Judea Pearl's Ladder of Causation. Reversible GF(2) binding provides the surgical algebra for the do-operator (Rung 2), and VaCoAl's dual-orthogonalizer architecture supplies the parallel substrate required for counterfactual reasoning (Rung 3). Ultimately, we prove these formal correspondences and derive testable iEEG predictions, uniting computational neuroscience, electrophysiology, and hyperdimensional computing.

Hiroyuki Chuma, Kanji Otsuka, Yoichi Sato

Institute of Innovation Research, Hitotsubashi University, Kunitachi, Tokyo 186-8603, Japan · Meisei University, Hino, Tokyo, Japan · Shuhari System, Tokyo, Japan

Hyperdimensional ComputingAssociative Memory
AskPDF
May 13, 2026

OSDN: Improving Delta Rule with Provable Online Preconditioning in Linear Attention

Linear attention and state-space models offer constant-memory alternatives to softmax attention, but often struggle with in-context associative recall. The Delta Rule mitigates this by writing each token via one step of online gradient descent. However, its step size relies on a single scalar gate that ignores the feature-wise curvature of the inner objective. We propose Online Scaled DeltaNet (OSDN), which augments the scalar gate with a diagonal preconditioner updated online via hypergradient feedback. Crucially, this right-preconditioning is algebraically equivalent to a per-feature scaling of the write-side key. This equivalence allows OSDN to strictly preserve the hardware-friendly chunkwise parallel pipeline of DeltaNet without incurring high-dimensional state overhead. Theoretically, by exploiting the exact-quadratic structure of the inner regression loss, we establish super-geometric convergence against a right-Newton comparator and prove an algorithm-aligned token-local residual contraction bound. To handle non-stationary contexts, we further introduce Adaptive Preconditioner Forgetting (APF) to dynamically refresh stale calibration. Empirically, OSDN demonstrates strong performance across scales. At the 340M-parameter scale, OSDN improves JRT-style in-context recall by 32% over DeltaNet. Scaling to 1.3B parameters, it achieves a 39% reduction in the recall residual ratio while maintaining parity on general downstream tasks (e.g., perplexity and LongBench) -- demonstrating that our online-preconditioning mechanism effectively transfers and amplifies at the billion-parameter scale.

Chenyu Zhou, Hongpei Li, Yuerou Liu +3

Shanghai Jiao Tong University · Northwestern University · Huazhong University of Science and Technology +1

Transformer AttentionLinear AttentionDelta-Rule Linear AttentionAssociative Memory
AskPDF
May 13, 2026

PaMM: Periodic Motif Memory for Atomistic Models with an Explicit Local-Structure Interface

Periodic crystals repeatedly instantiate similar local coordination motifs across translated cells and chemically related structures, but current equivariant atomistic models usually encode these patterns only implicitly in dense edge features. We introduce PaMM, a periodic motif memory that augments the UMA eSCN-MD edge encoder with explicit pair and triplet lookup features. Pair motifs are keyed by (Zj,Zi,br)(Z_j, Z_i, b_r) and triplet motifs by (Zj,Zi,Zk,bθ)(Z_j, Z_i, Z_k, b_θ), hashed into fixed-size tables and fused with the baseline edge representation through lightweight gate-only and affine-equipped variants. We evaluate PaMM in a matched UMA-S OMAT setting and focus on a narrow question: whether explicit motif memory helps at a fixed intermediate training budget. At the 10k-step checkpoint, both PaMM variants improve over the plain baseline; gate-only gives the best energy MAE, while the affine-equipped variant gives the best force MAE. A matched 20k follow-up keeps the same operating-point picture. Aligned controls show that the gain weakens for pair-only, triplet-only, random-bucket, and parameter-matched MLP alternatives, suggesting that the benefit is tied to structured pair/triplet organization rather than generic added capacity. A within-OMAT24 source-family evaluation also shows small but consistent gains across held-out generation families. We therefore make a focused claim: in the studied UMA-S + OMAT regime, explicit pair/ triplet motif memory is a useful inductive bias for periodic atomistic modeling. We do not claim broad cross-dataset transfer, a uniquely preferred fusion variant, or strong scientific interpretability beyond a more inspectable local-structure interface.

Ryan Dong

Independent Research

Memory-Augmented Neural NetworksMachine Learning Interatomic PotentialsAssociative Memory
AskPDF
May 13, 2026

Does Engram Do Memory Retrieval in Autoregressive Image Generation?

The Engram module -- a hash-keyed, O(1) associative memory injected into Transformer layers -- was recently shown to improve large language model pretraining, with the appealing interpretation that it provides a content-addressed shortcut to recurring local token patterns. We ask whether this interpretation transfers to autoregressive (AR) image generation, or whether the observed gains, if any, come from a different mechanism. We adapt the Engram module to vision with 2D spatial nn-gram hashing, gated fusion, and KV-cache-compatible incremental inference, and inject it into a class-conditional AR generator trained on ImageNet 256x256. Across a sweep of backbone-to-memory budget ratios ρ∈[0.17,0.90]ρ{\in}[0.17, 0.90], every Engram-augmented variant trails the pure AR baseline in FID, indicating that the module saves backbone FLOPs but does not, by itself, improve sample quality. We then probe how the module is used. A gate-clamp sweep shows that disabling the Engram pathway entirely is catastrophic, yet a tiny constant gate (g=0.10) matches or beats the learned gate -- inconsistent with a heavily content-addressed recall mechanism. A donor-probe experiment shows that swapping the hash inputs for matched, adversarial, or random same-class exemplars produces statistically indistinguishable next-token distributions, while collapsing or randomising the table degrades them by two to three orders of magnitude. Finally, training a model from scratch with the entire memory table frozen to N(0,1)\mathcal{N}(0, 1) noise costs only ΔFID=0.10Δ\text{FID}{=}0.10 and actually raises Inception Score. Together, these findings indicate that the Engram in AR image generation behaves not as a content-addressed retriever but as a gated architectural side-pathway: a hash-keyed residual stream whose benefit is dominated by the pathway itself, with the learned table contributing only a small distributional refinement.

Jinghao Wang, Qiyuan He, Chunbin Gu +1

1The Chinese University of Hong Kong · 2National University of Singapore

Memory-Augmented Neural NetworksImage GenerationAutoregressive Image GenerationAssociative Memory
AskPDF
May 12, 2026

δδ-mem: Efficient Online Memory for Large Language Models

Large language models increasingly need to accumulate and reuse historical information in long-term assistants and agent systems. Simply expanding the context window is costly and often fails to ensure effective context utilization. We propose δδ-mem, a lightweight memory mechanism that augments a frozen full-attention backbone with a compact online state of associative memory. δδ-mem compresses past information into a fixed-size state matrix updated by delta-rule learning, and uses its readout to generate low-rank corrections to the backbone's attention computation during generation. With only an 8×88\times8 online memory state, δδ-mem improves the average score to 1.10×1.10\times that of the frozen backbone and 1.15×1.15\times that of the strongest non-δδ-mem memory baseline. It achieves larger gains on memory-heavy benchmarks, reaching 1.31×1.31\times on MemoryAgentBench and 1.20×1.20\times on LoCoMo, while largely preserving general capabilities. These results show that effective memory can be realized through a compact online state directly coupled with attention computation, without full fine-tuning, backbone replacement, or explicit context extension.

Jingdi Lei, Di Zhang, Junxian Li +7

Nanyang Technological University · Mind Lab · Fudan University +3

Memory-Augmented Language ModelsMemorization in Language ModelsPersistent Memory for Language ModelsLLM MemoryMemorization in Generative ModelsAssociative Memory+1
AskPDF
May 12, 2026

A Fast and Energy-Efficient Latch-Based Memristive Analog Content-Addressable Memory

Analog content-addressable memories (aCAMs) based on memristors provide a promising pathway toward energy-efficient large-scale associative computing for Edge AI and embedded intelligence applications. They have been successfully applied to decision-tree inference and extend the capabilities of compute-in-memory (CIM) architectures beyond conventional vector-matrix multiplication. However, conventional designs such as the 6T2M architecture suffer from static search power, limited voltage gain, and pronounced match-line crosstalk, constraining analog precision and scalability. We introduce a strong-arm latched memristor (SALM) aCAM cell that replaces static voltage division with a dynamic current-race comparator, enabling high regenerative gain, intrinsic result latching, and near-zero static search power. Compared to 6T2M, SALM reduces read energy by 33% at identical latency while eliminating the gain and crosstalk limitations that prevent 6T2M from scaling to large arrays. SALM further enables scalable sequential and parallel latch sharing, and a dataset-aware optimization framework exposes an explicit energy-latency tradeoff, achieving up to 50% energy reduction at 3x latency across representative workloads. To enable architectural exploration, we develop a circuit-accurate behavioral model derived from SPICE lookup tables in 22 nm FD-SOI technology, capturing match-line dynamics and crosstalk. Integrated into the X-TIME decision-tree compiler, this framework demonstrates that SALM maintains near-software accuracy for high-dimensional datasets, whereas baseline designs degrade due to limited gain and cumulative crosstalk.

Paul-Philipp Manea, Aishwarya Natarajan, Jim Ignowski +2

PGI-14, Forschungszentrum Jülich, Jülich, Germany · RWTH Aachen University, Aachen, Germany · Hewlett Packard Enterprise, Fort Collins, CO, USA

Compute-in-MemoryEnergy-Efficient MLAssociative Memory
AskPDF
May 11, 2026

Variational Linear Attention: Stable Associative Memory for Long-Context Transformers

Linear attention reduces the quadratic cost of softmax attention to O(T)\mathcal{O}(T), but its memory state grows as O(T)\mathcal{O}(T) in Frobenius norm, causing progressive interference between stored associations. We introduce Variational Linear Attention (VLA), which reframes the memory update as an online regularised least-squares problem with an adaptive penalty matrix maintained via the Sherman-Morrison rank-1 formula. We prove that normalising the write direction to unit length gives the recurrence Jacobian spectral norm exactly 11 for all sequence lengths and head dimensions (Proposition 2), and that the state norm is self-limiting under bounded inputs (Proposition 1). Empirically, VLA reduces ∥St∥F\|S_t\|_F by 109×109\times relative to standard linear attention at T=1,000T{=}1{,}000, achieves near-perfect exact-match accuracy on multi-query associative recall within the effective per-head memory regime (npairs<dhn_\text{pairs} < d_h), maintaining substantially higher retrieval performance than DeltaNet and standard linear attention under increasing memory load, and maintains 62% accuracy at the per-head capacity boundary. A Triton-fused kernel achieves 14×14\times speedup over sequential Python and O(T)\mathcal{O}(T) scaling, crossing below softmax attention latency at approximately 43,000 tokens.

Vishal Pandey, Gopal Singh

Independent Researcher London, UK · Metriqual Athens, GR

Efficient AttentionLinear AttentionAssociative MemoryMemory-Efficient AttentionAttention Approximation
AskPDF
May 11, 2026

Factual recall in linear associative memories: sharp asymptotics and mechanistic insights

Large language models demonstrate remarkable ability in factual recall, yet the fundamental limits of storing and retrieving input--output associations with neural networks remain unclear. We study these limits in a minimal setting: a linear associative memory that maps pp input embeddings in Rd\mathbb{R}^d to their corresponding~dd-dimensional targets via a single layer, requiring each mapped input to be well separated from all other targets. Unlike in supervised classification, this strict separation induces~pp constraints per association and produces strong correlations between constraints that make a direct characterisation of the storage capacity difficult. Here, we provide a precise characterisation of this capacity in the following way. We first introduce a decoupled model in which each input has its own independent set of competing outputs, and provide numerical and analytical evidence that this decoupled model is equivalent to the original model in terms of storage capacity, spectra of the learnt weights, and storage mechanism. Using tools from statistical physics, we show that the decoupled model can store up to pclog⁡pc/d2=1/2p_c \log p_c / d^2 = 1 / 2 associations, and generalise the computation of pcp_c to linear two-layer architectures. Our analysis also gives mechanistic insight into how the optimal solution improves over a naïve Hebbian learning rule: rather than boosting input-output alignments with broad fluctuations, the optimal solution raises the correct scores just above the extreme-value threshold set by the competing outputs. These findings give a sharp statistical-physics characterisation of factual storage in linear networks and provide a baseline for understanding the memory capacity of more realistic neural architectures.

Alessio Giorlandino, Sebastian Goldt, Antoine Maillard

1International School of Advanced Studies (SISSA), Trieste, Italy · 2INRIA Paris & DI ENS, PSL University, Paris, France

Statistical Learning TheoryAssociative Memory
AskPDF
May 11, 2026

Hyperparameter Transfer for Dense Associative Memories

Dense Associative Memory (DenseAM) is a promising family of AI architectures that is represented by a neural network performing temporal dynamics on an energy landscape. While hyperparameter transfer methods are well-studied for feed-forward networks, these methods have not been developed for settings in which weights are shared across layers and within the layer, which is common in DenseAMs. Additionally, DenseAMs utilize rapidly peaking activation functions that are rarely used in feed-forward architectures. The confluence of these aspects makes DenseAM a challenging framework for using existing methods for hyperparameter transfer. Our work initiates the development of hyperparameter transfer methods for this class of models. We derive explicit prescriptions for how the hyperparameters tuned on small models can be transferred to models trained at scale. We demonstrate excellent agreement between these theoretical findings and empirical results.

Roi Holtzman, Dmitry Krotov, Boris Hanin

Rudolf Peierls Centre for Theoretical Physics, University of Oxford, Oxford OX1 3PU, UK · Dynamical Mind, IBM Research · Princeton

Hyperparameter TransferAssociative MemoryHyperparameter Optimization
AskPDF
May 8, 2026

Context-Gated Associative Retrieval: From Theory to Transformers

Hopfield networks and their generalizations have established deep connections among biological associative memories, statistical physics, and transformers. Yet most models treat retrieval as a fixed query-to-memory mapping, ignoring the role of external context in recall. In this work, we propose a two-stage associative memory architecture, wherein a context-gate subcircuit reshapes the retrieval energy landscape before and during recall. We show theoretically that context gating increases inter-memory separation while inducing sparsity, translating into exponential improvements in retrieval. Crucially, we prove that the system admits a unique self-consistent fixed point, revealing that the resulting retrieval state is driven by both a direct contextual bias and a second-order retrieval-gate feedback loop. We then bridge this theory to transformers; specifically, we evaluate a first-order approximation on Llama-3, confirming that in-context learning acts as context-gated retrieval. Native dynamics mirror our theory: context localizes a memory subspace, enabling the zero-shot query to cleanly discriminate. Ultimately, this framework provides a mechanistic link between associative memory theory and LLM phenomenology.

Moulik Choraria, Argyrios Gerogiannis, Vidhata Jayaraman +2

UIUC · University of Minnesota, Twin Cities · Stony Brook University

Transformer InterpretabilityIn-Context LearningAssociative Memory
AskPDF
May 7, 2026

Echo: KV-Cache-Free Associative Recall with Spectral Koopman Operators

Long chain-of-thought reasoning and agentic tool-calling produce traces spanning tens of thousands of tokens, yet Transformer KV caches grow linearly with sequence length, creating a memory bottleneck on commodity hardware. State-space models offer constant-memory recurrence but suffer a memory cliff: retrieval accuracy collapses once the gap between a stored fact and its query exceeds the effective horizon of the recurrent state. We introduce Echo, a KV-cache-free associative recall architecture built around Spectral Koopman Attention (SKA); a drop-in replacement for attention layers that augments SSM blocks with a closed-form dynamical operator whose sufficient statistics are accumulated in constant memory with no KV cache. Echo fits a spectral linear system to the key and value history via kernel ridge regression and retrieves through a learned power-iterated filter, all from O(r2)O(r^{2}) streaming state where rr is a small projection rank. On the Multi-Query Associative Recall benchmark, a pure Mamba-2 SSM fails to exceed chance accuracy (∼3%{\sim}3\%) across all gap lengths and KV-pair counts, while at the 50M parameter scale SKA-augmented models achieve 100%100\% retrieval accuracy on every configuration tested, including distractor gaps of 4,0964{,}096 tokens with 3232 KV pairs. Across five additional transfer benchmarks including needle-in-a-haystack, tool-trace, and multi-hop retrieval, SKA consistently outperforms both pure SSM and SSM+Attention hybrids while maintaining constant inference memory. Ablations confirm that the spectral operator, not the prefix masking strategy, drives the retrieval gain.

Anupama Sridhar, Alexander Johansen

California, USA · Stanford University Computer Science California, USA

State Space Language ModelsKoopman Operator LearningMemory-Efficient InferenceLanguage ModelingAssociative Memory
AskPDF
May 7, 2026

Adaptive Memory Decay for Log-Linear Attention

Sequence models face a fundamental tradeoff between memory capacity and computational efficiency. Transformers achieve expressive context modeling at quadratic cost, while linear attention and state-space models run in linear time by compressing context into a fixed-size hidden state, inherently limiting recall. Log-linear attention navigates this tradeoff by organizing memory across a Fenwick tree hierarchy, growing its hidden state logarithmically with sequence length at log-linear compute cost. However, its memory decay parameter λ is fixed and independent of the input, assigning uniform weights across all hierarchy levels regardless of the content, which introduces unnecessary rigidity. We propose learning λ directly from the input via a lightweight two-layer MLP, producing per-token, per-level decay that adapts to content rather than position. A softplus activation lets each Fenwick tree level scale independently, avoiding the inter-level competition that softmax introduces. This modification preserves log-linear complexity exactly and adds negligible parameter overhead. We evaluate on associative recall, selective copying, and language modeling, finding that input-dependent decay consistently outperforms the baseline, with the largest gains in long-range memory settings where baseline λ degrades or collapses entirely.

Yaxita Amin, Helen Zichen Li, Mengfan Zhang +1

University of Maryland

Long-Context ModelingLinear AttentionAssociative Memory
AskPDF
May 7, 2026

Efficient event-driven retrieval in high-capacity kernel Hopfield networks

High-capacity associative memory models, such as Kernel Logistic Regression (KLR) Hopfield networks, have demonstrated strong storage capabilities but typically rely on computationally expensive synchronous updates. This reliance poses a bottleneck for deployment on energy-efficient, event-driven neuromorphic hardware. In this paper, we investigate the asynchronous retrieval dynamics of KLR Hopfield networks. We show empirically that, under appropriately tuned kernel parameters, asynchronous sequential updates exhibit trajectories that are statistically indistinguishable from those of synchronous dynamics, while maintaining high recall accuracy within the tested regime for random patterns. Furthermore, we find that the asynchronous network achieves empirical storage capacities approaching P/N≈30P/N \approx 30 in static random pattern regimes, exceeding classical limits. To evaluate computational efficiency, we analyze the total number of state transitions (bit flips) required for error correction. The results show that the network converges using a number of events close to the initial Hamming distance from the target pattern, without observable spurious oscillations. These findings suggest that the large-margin attractors induced by KLR learning create a smooth energy landscape suited for sparse, event-driven computation, providing a basis for scalable and low-power associative memory on neuromorphic architectures.

Akira Tamamori

Faculty of Information Science, Aichi Institute of Technology 1247 Yachigusa, Yakusa-cho, Toyota-shi, Aichi 470-0392, Japan

Hopfield NetworksNeuromorphic ComputingAssociative Memory
AskPDF
May 6, 2026

Sharp Capacity Thresholds in Linear Associative Memory: From Winner-Take-All to Listwise Retrieval

How many key-value associations can a d×dd\times d linear memory store? We show that the answer depends not only on the d2d^2 degrees of freedom in the memory matrix, but also on the retrieval criterion. In an isotropic Gaussian model for the stored pairs, we show that top-1 retrieval, where every signal must beat its largest distractor, requires the logarithmic model-size scale d2≍nlog⁡nd^2\asymp n\log n. We prove that the correlation matrix memory construction, which stores associations by superposing key-target outer products, achieves this scale through a sharp phase transition, and that the same scaling is necessary for any linear memory. Thus the logarithm is the intrinsic extreme-value price of winner-take-all decoding. We next consider listwise retrieval, where the correct target need not be the unique top-scoring item but should remain among the strongest candidates. To formalize this regime, we propose the Tail-Average Margin (TAM), a convex upper-tail criterion that certifies inclusion of the correct target in a controlled candidate list. Under this listwise retrieval criterion, the capacity follows the quadratic scale d2≍nd^2\asymp n. At load n/d2→αn/d^2\toα, we develop an exact asymptotic theory for the TAM empirical-risk minimizer through a two-parameter scalar variational principle. The theory has a rich phenomenology: in the ridgeless limit it yields a closed-form critical load separating satisfiable and unsatisfiable phases, and it predicts the limiting laws of true scores, competitor scores, margins, and percentile profiles. Finally, a small-tail extrapolation further leads to the conjectural sharp top-1 threshold d2∼2nlog⁡nd^2\sim 2n\log n.

Nicholas Barnfield, Juno Kim, Eshaan Nichani +2

Harvard University · University of California, Berkeley · Princeton University

Information RetrievalAssociative Memory
AskPDF
May 6, 2026

Continual Knowledge Updating in LLM Systems: Learning Through Multi-Timescale Memory Dynamics

LLMs are trained once, then deployed into a world that never stops changing. External memory compensates for this, but most systems manage it explicitly rather than letting it adapt on its own. Biological memory works differently: coupled multi-timescale dynamics make new associations immediately usable, strengthen what repetition confirms, and let the rest fade. We argue that external memory should follow a similar principle. In Memini, this view takes the form of an associative memory that organizes knowledge as a directed graph. Each edge carries two coupled internal variables, one fast and one slow, following the Benna-Fusi model of synaptic consolidation. From this coupling, episodic sensitivity, gradual consolidation, and selective forgetting are expected to emerge as facets of a single mechanism, reframing external memory as a learning substrate that reorganizes through its own dynamics. This workshop article describes an early-stage conceptual design without experimental evaluation.

Andreas Pattichis, Constantine Dovrolis

The Cyprus Institute, Nicosia, Cyprus · School of Computer Science, Georgia Institute of Technology, Atlanta, GA, USA.

Continual Learning for LLMsLLM MemoryAssociative MemoryMemory Consolidation
AskPDF
May 6, 2026

The Impossibility Triangle of Long-Context Modeling

We identify and prove a fundamental trade-off governing long-sequence models: no model can simultaneously achieve (i) per-step computation independent of sequence length (Efficiency), (ii) state size independent of sequence length (Compactness), and (iii) the ability to recall a number of historical facts proportional to sequence length (Recall). We formalize this trade-off within an Online Sequence Processor abstraction that unifies Transformers, state space models, linear recurrent networks, and their hybrids. Using the Data Processing Inequality and Fano's Inequality, we prove that any model satisfying Efficiency and Compactness can recall at most O(poly(d)/log V) key-value pairs from a sequence of arbitrary length, where d is the model dimension and V is the vocabulary size. We classify 52 architectures published before March 2026 into the triangle, showing that each achieves at most two of the three properties and that hybrid architectures trace continuous trajectories in the interior. Experiments on synthetic associative recall tasks with five representative architectures validate the theoretical bound: empirical recall capacity lies strictly below the information-theoretic limit, and no architecture escapes the triangle.

Yan Zhou

School of Mathematics and Statistics Changsha University of Science and Technology Changsha, Hunan 410114, China

Long-Context ModelingAssociative MemorySequence ModelingInformation-Theoretic Lower Bounds
AskPDF
May 1, 2026

Benchmarking local Hebbian learning rules for memory storage and prototype extraction

Associative memory or content-addressable memory is an important component function in computer science and information processing, and at the same time a key concept in cognitive and computational brain science. Many different neural network architectures and learning rules have been proposed to model the brain's associative memory while investigating key component functions like figure-ground segmentation, perceptual reconstruction and rivalry. A less investigated but equally important capability of associative memory is prototype extraction where the training set comprises distorted prototype instances and the task is to recall the correct generating prototype given a new distorted instance. In this paper we benchmark associative memory function of seven different Hebbian learning rules employed in non-modular and modular recurrent networks with winner-take-all dynamics operating on moderately sparse binary patterns. We measure pattern storage and weight information capacity, prototype extraction capabilities, and sensitivity to correlations in data. The original additive Hebb rule comes out with worst capacity, covariance learning proves to be robust but with moderate capacity, and the Bayesian-Hebbian learning rules show highest capacity in almost all different conditions tested.

Anders Lansner, Andreas Knoblauch, Naresh B Ravichandran +1

1KTH Royal Institute of Technology, EECS, Stockholm · 2Stockholm University, Dept. of Mathematics, Stockholm · 3KEIM-Institute, Albstadt-Sigmaringen University, Germany +1

Hebbian LearningRecurrent Neural NetworksAssociative Memory
AskPDF
May 1, 2026

Geometric and dynamical analysis of attractor boundaries and storage limits in kernel Hopfield networks

High-capacity associative memories based on Kernel Logistic Regression (KLR) exhibit strong storage capabilities, but the dynamical and geometric mechanisms underlying their stability remain poorly understood. This paper investigates the global geometry of attractor basins and the mechanisms governing the storage limit in KLR-trained Hopfield networks. We combine empirical evaluations using random sequences and real-world image embeddings (CIFAR-10) with morphing experiments and statistical Signal-to-Noise Ratio (SNR) analysis. Our experiments show that the network achieves a storage capacity for random sequences up to P/N≈16P/N \approx 16, while maintaining stable retrieval for structured data at effective loads near P/N≈20P/N \approx 20. Morphing analysis indicates that attractors on the "Ridge of Optimization" are separated by sharp, phase-transition-like boundaries, characterized by steep effective potential barriers and critical slowing down. Furthermore, by comparing an SNR analysis with a geometric reference point inspired by Cover's theorem, we show that the practical storage limit is governed primarily not by a lack of geometric separability in the feature space, but by the loss of dynamical stability against crosstalk noise. These findings suggest that KLR networks function as highly localized exemplar-based memories that operate near the onset of dynamical collapse, providing a useful perspective on the design of robust, large-scale retrieval systems.

Akira Tamamori

Faculty of Information Science, Aichi Institute of Technology 1247 Yachigusa, Yakusa-cho, Toyota-shi, Aichi 470-0392, Japan

Dynamical SystemsNeural Representation GeometryHopfield NetworksAssociative Memory
AskPDF
Apr 29, 2026

Language Diffusion Models are Associative Memories Capable of Retrieving Unseen Data

When do language diffusion models memorize their training data, and how to quantitatively assess their true generative regime? We address these questions by showing that Uniform-based Discrete Diffusion Models (UDDMs) fundamentally behave as Associative Memories (AMs) with emergent creative capabilities\textit{with emergent creative capabilities}. The core idea of an AM is to reliably recover stored data points as memories\textit{memories} by establishing distinct basins of attraction around them. Historically, models like Hopfield networks use an explicit energy function to guarantee these stable attractors. We broaden this perspective by leveraging the observation that energy is not strictly necessary, as basins of attraction can also be formed via conditional likelihood maximization. By evaluating token recovery of training\textit{training} and test\textit{test} examples, we identify in UDDMs a sharp memorization-to-generalization transition governed by the size of the training dataset: as it increases, basins around training examples shrink and basins around unseen test examples expand, until both later converge to the same level. Crucially, we can detect this transition using only the conditional entropy of predicted token sequences: memorization is characterized by vanishing conditional entropy, while in the generalization regime the conditional entropy of most tokens remains finite. Thus, conditional entropy offers a practical probe for the memorization-to-generalization transition in deployed models.

Bao Pham, Mohammed J. Zaki, Luca Ambrogioni +2

1Rensselaer Polytechnic Institute (RPI) · 2Donders Institute for Brain, Cognition, and Behaviour, Radboud University · 3Dynamical Mind +1

Memorization in Generative ModelsNeural Network MemorizationDiffusion ModelsMemorization in Language ModelsDiffusion Language ModelsAssociative Memory+1
AskPDF
Apr 28, 2026

Emergent Self-Attention from Astrocyte-Gated Associative Memory Dynamics

We introduce a Hopfield-type associative memory in which effective connectivity is multiplicatively modulated by astrocytic gains evolving under an entropy-regularized replicator equation. The coupled neuron-astrocyte dynamics admit a Lyapunov function, ensuring global convergence. At fixed points, astrocytic gains implement a softmax-normalized allocation over pattern similarity scores, yielding a mechanistic realization of self-attention as emergent routing on the gain simplex. In regimes of high memory load and interference, the model significantly improves retrieval accuracy relative to classical Hopfield dynamics and recent neuron-astrocyte baselines. These results establish a dynamical systems framework linking glial modulation, competitive resource allocation, and attention-like computation.

Arnau Vivet, Alex Arenas

1Departament d’Enginyeria Inform`atica i Matem`atiques, Universitat Rovira i Virgili, 43007 Tarragona, Spain · 2Complexity Science Hub Vienna, Metternichgasse 8, 1030 Vienna, Austria

Softmax AttentionDynamical SystemsSelf-AttentionReplicator DynamicsNeural ProcessesHopfield Networks+2
AskPDF
Apr 22, 2026

Learning Hippo: Multi-attractor Dynamics and Stability Effects in a Biologically Detailed CA3 Extension of Hopfield Networks

We present a biologically detailed extension of the classical Hopfield/Marr auto-associative memory model for CA3, implementing ten populations (two asymmetric pyramidal subtypes, eight GABAergic interneuron classes), forty-seven compartments, multi-rule plasticity (recurrent Hebb, BCM anti-saturation, mossy-fiber short-term, endocannabinoid iLTD, burst-gated Hebb), and a bimodal cholinergic encoding/consolidation cycle. Evaluated on pattern completion across auto-associative, associative, and temporal regimes, and on a controlled inhibitory-proportion manipulation at N=256N{=}256, the full architecture exhibits three qualitative signatures absent from a minimal Hopfield baseline: (i)~multi-attractor cross-seed behaviour at K=5K{=}5 with biologically realistic inhibitory proportions, where two of five seeds converge to positive attractors with margin +0.10−0.22{+}0.10{-}0.22 (Cohen's d=0.71d{=}0.71, one-sided p=0.08p{=}0.08); (ii)~target-selective associative recall in paired (A,B)(A, B) memory at K≥5K{\geq}5, where the full model retrieves BB from a partial cue of AA while the minimal model echoes AA (Pearson margin Δ=+0.163Δ{=}{+}0.163 at K=5K{=}5); (iii)~reduced cross-seed variance of the full model below the minimal baseline under clean upstream, with ratios 1.0−3.01.0{-}3.0. These three signatures are architecture-specific: they appear consistently across independent regimes and are absent from the minimal control.

Daniele Corradetti, Renato Corradetti

Grupo de Física Matemática Instituto Superior Técnico Av. Rovisco Pais, 1049-001 Lisboa, Portugal · Dipartimento di Neuroscienze, Psicologia, Area del Farmaco e Salute del Bambino (NEUROFARBA) Università di Firenze, Italy22

Dynamical SystemsNeural ProcessesNeural Network MemorizationHopfield NetworksAssociative Memory
AskPDF
Apr 22, 2026

Quantization robustness from dense representations of sparse functions in high-capacity kernel associative memory

High-capacity associative memories based on Kernel Logistic Regression (KLR) achieve strong retrieval performance but typically require substantial computational resources. This paper investigates the compressibility of KLR Hopfield networks to clarify the geometric principles underlying their robust representations. We present a geometric interpretation based on spontaneous symmetry breaking and Walsh analysis, and examine it through compression experiments involving quantization and pruning. The experiments reveal a clear asymmetry: the network remains robust under low-precision quantization while exhibiting strong sensitivity to pruning. We interpret this behavior through a "sparse function, dense representation" principle, in which a sparse input mapping is implemented through a dense bimodal parameterization. These findings suggest a practical route toward hardware-efficient kernel associative memories and provide insight into the geometric principles underlying robust representation in neural systems.

Akira Tamamori

Faculty of Information Science, Aichi Institute of Technology 1247 Yachigusa, Yakusa-cho, Toyota-shi, Aichi 470-0392, Japan

Neural Representation GeometryHopfield NetworksAssociative MemoryNeural Network Quantization
AskPDF
Apr 22, 2026

To Know is to Construct: Schema-Constrained Generation for Agent Memory

Constructivist epistemology argues that knowledge is actively constructed rather than passively copied. Despite the generative nature of Large Language Models (LLMs), most existing agent memory systems are still based on dense retrieval. However, dense retrieval heavily relies on semantic overlap or entity matching within sentences. Consequently, embeddings often fail to distinguish instances that are semantically similar but contextually distinct, introducing substantial noise by retrieving context-mismatched entries. Conversely, directly employing open-ended generation for memory access risks "Structural Hallucination" where the model generates memory keys that do not exist in the memory, leading to lookup failures. Inspired by this epistemology, we posit that memory is fundamentally organized by cognitive schemas, and valid recall must be a generative process performed within these schematic structures. To realize this, we propose SCG-MEM, a schema-constrained generative memory architecture. SCG-MEM reformulates memory access as Schema-Constrained Generation. By maintaining a dynamic Cognitive Schema, we strictly constrain LLM decoding to generate only valid memory entry keys, providing a formal guarantee against structural hallucinations. To support long-term adaptation, we model memory updates via assimilation (grounding inputs into existing schemas) and accommodation (expanding schemas with novel concepts). Furthermore, we construct an Associative Graph to enable multi-hop reasoning through activation propagation. Experiments on the LoCoMo benchmark show that SCG-MEM substantially improves performance across all categories over retrieval-based baselines.

Lei Zheng, Weinan Song, Daili Li +1

UnionPay

Schema-Constrained GenerationLong-Term Conversational MemoryAssociative MemoryAgent Memory ManagementText Generation
AskPDF
Apr 21, 2026

FG2^2-GDN: Enhancing Long-Context Gated Delta Networks with Doubly Fine-Grained Control

Linear attention mechanisms have emerged as promising alternatives to softmax attention, offering linear-time complexity during inference. Recent advances such as Gated DeltaNet (GDN) and Kimi Delta Attention (KDA) have demonstrated that the delta rule, an online gradient descent update, enables superior associative recall compared to simple additive updates. While KDA refined the coarse head-wise decay gate into channel-wise decay, the learning rate βtβ_t in the delta update remains a scalar, limiting the model's capacity for dimension-specific adaptation. We introduce FG2^2-GDN, which replaces the scalar βtβ_t with a channel-wise vector analogous to the transition from SGD to per-coordinate adaptive optimizers such as AdaGrad and Adam. We further propose FG2^2-GDN+, which decouples the scaling for keys and values, enabling independent control of erasure strength and write strength. Experiments on synthetic and real-world benchmarks show that FG2^2-GDN and its variant improve associative recall and long-context understanding over GDN and KDA, with comparable computational efficiency.

Pingwei Sun, Yuxuan Hu, Jianchao Tan +6

Meituan · Beijing, China

Long-Context ModelingLinear AttentionDelta-Rule Linear AttentionAttention MechanismsGated Delta NetworksAssociative Memory
AskPDF
Apr 18, 2026

HeLa-Mem: Hebbian Learning and Associative Memory for LLM Agents

Long-term memory is a critical challenge for Large Language Model agents, as fixed context windows cannot preserve coherence across extended interactions. Existing memory systems represent conversation history as unstructured embedding vectors, retrieving information through semantic similarity. This paradigm fails to capture the associative structure of human memory, wherein related experiences progressively strengthen interconnections through repeated co-activation. Inspired by cognitive neuroscience, we identify three mechanisms central to biological memory: association, consolidation, and spreading activation, which remain largely absent in current research. To bridge this gap, we propose HeLa-Mem, a bio-inspired memory architecture that models memory as a dynamic graph with Hebbian learning dynamics. HeLa-Mem employs a dual-level organization: (1) an episodic memory graph that evolves through co-activation patterns, and (2) a semantic memory store populated via Hebbian Distillation, wherein a Reflective Agent identifies densely connected memory hubs and distills them into structured, reusable semantic knowledge. This dual-path design leverages both semantic similarity and learned associations, mirroring the episodic-semantic distinction in human cognition. Experiments on LoCoMo demonstrate superior performance across four question categories while using significantly fewer context tokens. Code is available on GitHub: https://github.com/ReinerBRO/HeLa-Mem

Jinchang Zhu, Jindong Li, Cheng Zhang +2

The Hong Kong University of Science and Technology (Guangzhou) · Jilin University · The Chinese University of Hong Kong

Memorization in Language ModelsHebbian LearningLLM Agent MemoryPersistent Memory for Language ModelsGraph-Based Agent MemoryLong-Term Conversational Memory+1
AskPDF
Apr 17, 2026

Generic Triple-Latent Compression with Gated Associative Retrieval

We study generic triple-latent sequence models that maintain a running token state and compressed pair-memory pathway to capture higher-order token interactions without benchmark-specific parsing. The triple-latent family improves a small Transformer baseline on byte-level WikiText-2 and on a tokenizer-based MiniMind language-model benchmark, while a recall-focused gated key-value retrieval extension improves associative recall but remains seed-sensitive and much slower in the current reference implementation.

Liu Xiao

Memory-Augmented Language ModelsAutoregressive Language ModelingAssociative MemorySequence Modeling
AskPDF
Apr 16, 2026

SRMU: Relevance-Gated Updates for Streaming Hyperdimensional Memories

Sequential associative memories (SAMs) are difficult to build and maintain in real-world streaming environments, where observations arrive incrementally over time, have imbalanced sampling, and non-stationary temporal dynamics. Vector Symbolic Architectures (VSAs) provide a biologically-inspired framework for building SAMs. Entities and attributes are encoded as quasi-orthogonal hyperdimensional vectors and processed with well defined algebraic operations. Despite this rich framework, most VSA systems rely on simple additive updates, where repeated observations reinforce existing information even when no new information is introduced. In non-stationary environments, this leads to the persistence of stale information after the underlying system changes. In this work, we introduce the Sequential Relevance Memory Unit (SRMU), a domain- and cleanup-agnostic update rule for VSA-based SAMs. The SRMU combines temporal decay with a relevance gating mechanism. Unlike prior approaches that solely rely on cleanup, the SRMU regulates memory formation by filtering redundant, conflicting, and stale information before storage. We evaluate the SRMU on streaming state-tracking tasks that isolate non-uniform sampling and non-stationary temporal dynamics. Our results show that the SRMU increases memory similarity by 12.6%12.6\% and reduces cumulative memory magnitude by 53.5%53.5\%. This shows that the SRMU produces more stable memory growth and stronger alignment with the ground-truth state.

Shay Snyder, Andrew Capodieci, David Gorsich +1

George Mason University · George Mason University, USA · Neya Robotics +3

Hyperdimensional ComputingVector Symbolic ArchitecturesState TrackingAssociative Memory
AskPDF