Overview

Associative memory asks how a system can store information so that a cue brings back a related pattern, fact, or sequence, a problem central to neural memory and efficient AI. Classical Hopfield networks, dense and kernel associative memories, and Hebbian learning use attractor dynamics or similarity-based retrieval; newer recurrent and linear-attention models add sparse key-value stores, while agent systems use linked episodic memories. Theory examines capacity, retrieval basins, stability, and interference, while experiments test noisy pattern completion, long-range recall, language-model memory, and hardware implementations. A persistent tension is that more storage does not guarantee reliable, selective recall: finite state and overlapping representations can cause interference, forgetting, and false associations, even as explicit retrieval improves access at added cost.

Latest papers

126 papers12 papers in the last four weeks, down 37% on the four weeks before. 0.1% of all new papers.

Nov 9, 2026

TANSAR: Editable and Attributable Correlation Matrix Memories under Correlated Keys, with Rule-Constrained Multi-Hop Knowledge Editing

Correlation Matrix Memories (CMMs) store associations as sums of rank-one outer products, providing closed-form addressability, exact deletion, and linear attribution. While classical analyses assume random, orthogonal keys, semantic encoders produce correlated representations. We analyse CMMs under correlated keys using frame-theoretic coherence: (i) capacity bounds yielding an encoder-independent limit n≤d(1+ε2)n \le d(1+\varepsilon^2) for mean squared crosstalk ε2\varepsilon^2; (ii) an exact characterisation of the collateral effect of deleting an association, guaranteeing that deletion never invalidates certified associations; and (iii) a geometric transparency bound demonstrating why attribution requires candidate sets of centred keys. On this basis, we introduce Topology-Aware Neuro-Symbolic Associative Retrieval (TANSAR). TANSAR traverses a directed acyclic association graph, prunes invalid paths via sound ground Horn constraints, and injects a representation derived strictly from the verified path into the residual stream of a frozen language model, guarded by a Clopper--Pearson selective-risk gate. Controlled experiments confirm all theoretical bounds. In multi-hop knowledge editing, Horn pruning reduces answer contradictions from 25.1% to 0.5%, whereas Hopfield retrieval incurs 24% contradictions when stale associations remain stored (and 0% if stale associations are removed). An end-to-end edit transaction requires 13.6 ms on a standard CPU and 2.37 ms on GPU. The symbolic component matches gold labels across all 600 ProofWriter theories and achieves 94.0% agreement on FOLIO, demonstrating sound and efficient multi-hop associative retrieval with exact unlearning.
Nov 3, 2026

Memory flows: geometry and dynamics of sequential retrieval in input-driven Hopfield networks

Associative memory models classically describe retrieval as convergence to stored patterns, but the theory of sequential retrieval remains comparatively limited. We develop a two-timescale input-driven Hopfield network in which fast associative retrieval is modulated by a slow feedback variable evolving over a prescribed memory-transition graph. This yields autonomous, graph-constrained transitions while preserving analytical tractability. Using slow-fast and geometric singular perturbation theory, we characterize memory fixation via memory manifolds and identify the geometric regions that govern stability loss and escape from retrieved memories. We further derive reduced slow dynamics and explicit criteria for transition onset, self-sustained retrieval, and collapse, expressed through escape times, gain thresholds, and fixed points. Together, these results provide a tractable framework for analyzing feedback-driven sequential retrieval in Hopfield networks.
Nov 3, 2026

SEAM: Situated Executable Associative Memory for a Co-located Community

In groups, knowledge is distributed among individuals, and how it is brought together depends on social ties. Taking inspiration from how a single mind recalls and reacts to memories and current stimulus, we build a shared memory that does this for a community, prototyped as a social app, SEAM. Members post arbitrary moments; SEAM works out how fragments relate, distills knowledge about the group, and replies in a situated and associative way. Some of these memories act as instructions that can alter how the system behaves, making it programmable by its members. We deployed SEAM to 46 people in a co-located community for two weeks of open-ended use, and ran a follow-up workshop with 14. Members shared their days, interrogated the memory, and instructed it to track routines and run games; replies introduced them to people and subjects they had never encountered. SEAM points towards communities where knowledge finds people.
Oct 30, 2026

Memory and Recall of Color Images Using Neural Networks-1 - The Cue-Ball Model based on the Associative Memory Mechanism -

This paper presents a neural network model for storing and recalling color images of different sizes using the Cue BallRecall Net systems [1,2,3,4,5]. The Cue Ball system consists of a Recall Net composed of multiple memory neurons each responsible for memorizing a single, distinct memory and a large number of neurons that represent image patterns. The images used in this study [6] consist of ten images(see Appendix). The pixel values of these images were converted into digital values and used for subsequent processing. Accordingly, the Cue Ball contains ten Cue neurons, corresponding to the ten images stored in memory. The number of pixels varies among the images, and therefore, the number of Recall neurons comprising the Recall Net also varies accordingly. In this study, the number of Recall neurons was prepared between 106,134 and 360,000, depending on the number of pixels in each image. In this simulation, after the connection weights (wjip, vijp) had been learned, we have used the learning and recall algorithms to investigate whether the Cue neuron which had memorized a specific image could be identified by the Recall Net when that a memorized image has been presented. Therefore, the most important simulation result is which Cue neuron responds when a memorized image is presented to the Recall Net. If the Cue neuron that memorizes the presented image responds, its output can be used to obtain the corresponding image data. By adding a BMP header to the resulting image data, the original memorized image can be accurately recalled.
Oct 28, 2026

Capacity of the Hopfield-Hebbian dynamics with correlated patterns

Allowing for small fraction of errors in retrieved stored binary patterns, Hopfield-Hebbian neural network (NN) associative memory dynamics [14] achieves linearly scaled capacity. The famous proportionality constant αc≈0.14α_c\approx 0.14 is obtained in a large dimensional scenario as the ratio of the number of reliably stored patterns mm and the patterns' length nn under the assumption that uniformly randomly chosen binary patterns are independent among themselves. While such an idealized scenario is now well understood, more practical scenarios with correlated/noisy datasets remaining challenging even for classical statistical mechanics replica approaches. We consider precisely such practical settings where each archetype pattern is replaced with MM correlated examples with parameter rr governing the quality of correlation. We develop a powerful Random Duality Theory (RDT) based framework to analyze correlated data and already on the first level of RDT lifting obtain capacity characterizations that remarkably well match behavior exhibited through simulated dynamics. Moreover, characterizations explicitly feature correlation parameters MM and rr, which allows to precisely determine their effects on phase-transitioning behavior of the underlying dynamics. Our results are also a well-fitting complement to the existing literature. Namely, to circumvent challenging statistical mechanics analyses, an alternative combination of spectral and replica methods is considered in [4]. We confirm that [4]'s elegant spectral characterizations are excellent approximations in the low correlated regime. At the same time, we also close the gap in the regimes with higher correlation levels where spectral methods deviate from the practical dynamics performance.
Oct 26, 2026

Streaming Recurrent Memory

Natural intelligence learns from a continuous stream of sensorimotor experience: data can arrive at high bandwidth, and behavior adapts to feedback as soon as it arrives, even when it is delayed. In contrast, most modern deep learning systems rely on stored datasets, replay buffers, and retained contexts through which credit is later assigned. At high data rates, storing and revisiting past experience can be impractical under tight memory and computational constraints. This paper studies recurrent memory under a streaming learning constraint: the learner learns from experience as soon as it arrives without storing raw experience. We ask whether a streaming learner can maintain a state that remains useful for future prediction and control from delayed feedback. We introduce Streaming Recurrent Memory (SRM), a recurrent memory designed for the streaming setting which preserves information that may matter later while remaining trainable with one-step backpropagation through time (BPTT). SRM provides a streaming-compatible recurrence that is designed to recall past information by content or time, track elapsed steps, and compose transformations whose outcome depends on the order in which inputs are observed. Across controlled memory tasks and streaming applications, SRM learns effectively with one-step BPTT, outperforming recurrent baselines in associative recall and online language modeling while achieving near-perfect success in associative T-Maze with delayed rewards. These results suggest that fixing how inputs are stored in the recurrent state, while learning how that state is read out, enables effective streaming learning without long-range backpropagation.
Oct 19, 2026

Sample-efficient learning with Dense Associative Memories

We analytically study the evolution of the energy function of Dense Associative Memories (DenseAMs) as they train on randomly sampled codewords of random expander low-density parity-check codes (a.k.a. solutions of random XORSAT problems). We identify two distinct regimes and a transition between them, reminiscent of grokking in neural networks. A regime of small training set size is where the model memorizes a few training examples but cannot discover the underlying rules of the entire codeword dictionary generative process. The opposite regime, corresponding to a larger training set size, is where the model has uncovered the sparse constraints that generated the entire data set after seeing only polynomially many patterns. We also design a novel class of DenseAMs with a thresholding learning rule, and show that any amount of thresholding enables the model to perform simplified sparse Hamiltonian learning, which exponentially reduces the number of training examples needed to learn the true underlying data set. In this regime, the model learns how to discover exp⁡[O(n)]\exp[\mathrm{O}(n)] solutions from only O(log⁡n)\mathrm{O}(\log n) training samples, which is asymptotically optimal for any learning algorithm. When training on codewords of certain random codes, the model can even learn the underlying code when a small O(1) fraction of bits are latent variables. These latent variables are not shown to the model during learning, but will be discovered by sparsifying observed higher-order correlations. For certain random codes, we show that a simple and conservative algorithm learns a minimal latent representation for the data set. This provides a transparent, physics-motivated, and analytically tractable toy model for how deep latent representations emerge in an associative memory training.
Oct 19, 2026

Kronecker Delta Rule for Long Context Linear Attention

We propose KronLSTM, a linear attention architecture based on the Kronecker Delta Rule, a novel update rule for associative matrix memories. KronLSTM extends the xLSTM family with a memory-revision mechanism that corrects targeted associations while modifying the memory only along a single direction in matrix space. This minimally invasive memory revision reduces interference with previously stored information compared with delta-rule updates, making it particularly well suited for long-context language modeling. The resulting memory dynamics can be mathematically described as a Markov chain that converges to a steady-state distribution, while the expected norm of the memory matrix remains uniformly bounded throughout the entire input sequence. Hence, KronLSTM ensures stable memory dynamics without requiring additional normalization and stabilization mechanisms for the memory state. Experiments show that KronLSTM is competitive with, and often outperforms, existing linear attention architectures, including models of the (Gated) DeltaNet family, with particularly strong performance on long context benchmarks. These results establish the Kronecker Delta Rule as an effective and analytically tractable foundation for stable associative memory in recurrent linear attention architectures. Our code (https://github.com/NX-AI/KronLSTM) and models (https://huggingface.co/collections/NX-AI/kronlstm) are open-source.
Oct 15, 2026

Phase Kernel Lifts Capacity of Dense Associative Memory

Dense associative memory underlies modern attention mechanisms; however, Modern Hopfield Networks (MHNs) with linear kernels yield overlapping attractor basins and spurious states under high memory load, limiting their theoretically exponential storage capacity in practice. We construct a class of nonlinear phase kernels that modify the energy landscape and retrieval dynamics, and prove that these kernels preserve exponential storage capacity and local fixed-point convergence under given conditions on kernel smoothness, memory separation, and memory distributions. The framework is instantiated by the Sigmoidal Phase Hopfield Network (S-Hop), which guarantees monotone energy descent and mitigates practical capacity loss. S-Hop has 8.3×8.3\times and 4×4\times gains in one-step noisy retrieval load over Uniform Hopfield Networks (U-Hop) on CIFAR-10 and TinyImageNet respectively, and reaches a 50×50\times gain in first-failure load for ImageNet. The results demonstrate that phase kernels provide a principled route to robust dense associative memory.
Oct 14, 2026

MemCode: Discrete Semantic Representations for Long-Term Agent Memory

Long-term memory remains a fundamental bottleneck for large language model (LLM) agents across sessions. Existing systems typically adopt a write-then-retrieve paradigm, storing context-dependent dialogue fragments and retrieving them via nearest-neighbor search in a continuous embedding space. This design yields unstable access units and systematically misses logically related but lexically distant evidence. We propose MemCode, a long-term agent memory framework that couples retrieval-oriented writing with discrete associative retrieval. MemCode rewrites dialogue into self-contained atomic memory units (AMUs), maps AMUs into a learnable codebook with isomorphic semantic quantization (ISQ), and uses codebook co-occurrence topology as a retrieval path complementary to vector search. For long-term memory stores, MemCode separates online atomic writes from asynchronous offline consolidation. On LongMemEval-S and LoCoMo with GPT-4o-mini and Qwen3-30B-A3B-Instruct-2507 backbones, MemCode achieves the best overall accuracy while keeping per-dialogue memory construction within roughly 40130 seconds, making it 321 times faster than LightMem and over an order of magnitude faster than A-MEM and MemoryOS.
Oct 13, 2026

Depth Memory

Residual connections propagate information through a network by additively accumulating layer outputs. Recently proposed alternatives, such as Manifold-Constrained Hyper-Connections (mHC) and Attention Residuals (AttnRes), instead use more expressive depth aggregation mechanisms that improve language modeling performance at scale. In this work, we consider the mixing of representations over depth as a problem of storing and retrieving information across layers. Formalizing this perspective, we introduce Depth Memory (DM), a unified framework that describes the standard pre-layer-norm (PreLN) transformer, mHC, and AttnRes in terms of streams that carry representations through the network. One such stream, for example, could be an explicit memory stream that is queried to enrich the input to every layer, and whose output is subsequently written back into the memory for use by later layers. For AttnRes, this memory stream is a growing cache of explicit layer representations. For mHC, it is an expanded residual stream. Building on this view, we propose Linear Depth Memory (LDM), which formulates information retrieval across depth as a regression problem solved using recursive least squares. The resulting architecture propagates three streams across depth: the standard residual stream, a linear associative memory stream, and an inverse covariance stream. For high-throughput training and inference, we propose a hardware-aware implementation of LDM that reduces peak activation memory and memory traffic. LDM consistently achieves the lowest perplexities among PreLN, mHC and AttnRes baselines at the 400M and 1.5B parameter scales, alongside competitive zero-shot accuracy.
Oct 13, 2026

AMemOps: Evaluating Memory Updates in Long Audio Streams

Audio memory must recognise recurring sources across acoustic changes while keeping their attributes up to date. Overall accuracy does not reveal how these demands interact. We introduce Audio Memory Operations (AMemOps), a benchmark for latest-attribute retrieval, preservation of unchanged attributes and absent-source rejection in long audio streams. Paired controls vary recordings, training identities and earlier events to examine their effects on updating. Training on other recordings of the test instruments increases a reference delta-rule memory's update advantage over matched attention by 13.4 percentage points. At retrieval delays of 120,s, matched histories separate benefits of consistent repetition from costs of conflicting earlier values, while additive-memory controls identify benefits of erasure.
Oct 12, 2026

Where Polanyi Meets Galois: Tacit Knowing as an Algebro-Deterministic Operation, Toward Re-rooting AI-Generated Science

Tacit knowing is an operating regime of one memory substrate, not a separate kind of content. In VaCoAl (CASRAM, PyVaCoAl), an algebro-deterministic hyperdimensional memory over GF(2), one DAG of bound records is read in two ways. With every collision repaired (Rescue), the trace is exactly auditable but no path outranks another; with collisions tolerated (Don't Care), a path-integrated confidence CR2CR_2 ranks direct over circuitous routes without being stated. This audit-valuation complementarity formalizes Polanyi's claim that tacit and explicit knowledge are not sharply divided. (1) Polanyi's from-to structure reads as non-commutative binding. (2) Full explication destroys the valuation (Lemma 1), as with Polanyi's pianist; tacitness is contingent or regime-bound, placing Polanyi, Nonaka, Tsoukas and Collins on one substrate. (3) The same lemma answers the popular dual-process moral ("train slow System 2 to tame fast System 1"), which mirrors the two-store reading of knowledge Polanyi rejected; Mercier and Sperber's reply, that reasoning is intuitive inference about reasons, matches his. (4) To know a large DAG is to cull it. Simon answered Polanyi's Meno argument: a test and a generator suffice. The AlphaGo-to-AlphaProof lineage builds that answer, yet a checker audits an endpoint and values no path: the tacit is relocated, not removed, and results are verified yet rootless. (5) Collins's three tacit knowledges map to bound structure, substrate valuation and value binding. (6) Our mentor-student DAG charts tacit transmission. (7) AI-solved problems with few follow-ups (Tao) are rootless explicit knowledge, re-rooted when LLMs generate, VaCoAl keeps the trace and the community values. (8) On Mercier and Sperber's theory an LLM is a sophist; truth-seeking lies in collective tacit knowing, Polanyi's conviviality, which we conjecture extends beyond humans. Testable predictions follow.
Oct 10, 2026

DeltaNet-PC: Test-Time Memory Without Backpropagation Through Time

Test-time memorization of sequential information is a foundational capability for any intelligent system that needs to rapidly store and retrieve data, as demonstrated by the attention mechanism in large language models. However, Transformers and linear attention variants are trained via backpropagation through time (BPTT), which has drawbacks such as its enormous memory requirements. Predictive coding (PC), an algorithm that optimizes a prediction-error energy function with local updates, offers a compelling solution but still lacks test-time memory for sequential data. We develop a solution to this problem by treating the per-step memory update ΔStΔS_t as a latent variable, as opposed to a slow weight to be learned. By taking the log probability of a standard PC-based probabilistic model, we derive a PC-native solution for verbatim sequence memory, and relate our update rule to common choices in the linear attention literature (e.g., DeltaNet). We present an extensive set of empirical tests to probe our system's properties, including its scaling capacity, memory footprint, and ability to infer a context from evidence (e.g., bank (river) vs. bank (money)). We show how DeltaNet-PC can be used for storing single sequences of data, multiple sequences of data, and key-value associative memory pairs with perfect recall accuracy.
Oct 9, 2026

Double-Take: Improving Capacity, Binding, and Selectivity In Long-Context Within Bidirectional Memory

Processing a long sequence does not guarantee that distant information remains usable. As context grows, a fixed-size memory must retain many associations, connect information across positions, and preserve relevant structure without being overwhelmed by everything it observes. These are distinct requirements: a memory may store more information without representing the relationships needed for retrieval, and it may transport those relationships without preserving the ones that matter. Thus, the challenge is not simply extending context length, but constructing a memory that remains usable as context grows. We study this problem in large-chunk test-time training (TTT) and identify two basic failures: single-step writes accumulate interference, limiting capacity, while self-associative writes cannot form cross-position bindings. We introduce Double-Take (DT), which repairs these failures with multi-step unpinned updates and cross-position mixing, and adds bidirectional change-sensitive writes for selective retention. Capacity alone leaves end-to-end recall at the 0.100.10 chance floor; adding binding raises it to 0.920.92. Trained only on 32 kb32\,\mathrm{kb} genomic contexts, DT reaches 0.9270.927 AUROC at 131 kb131\,\mathrm{kb} and 0.8560.856 near 1 Mb1\,\mathrm{Mb}, while transferring zero-shot from human to mouse. Together, these results identify capacity, binding, and selectivity as three requirements for usable long-range memory.
Oct 8, 2026

BRACE: Differential Privacy for Dense Associative Memory with LSR Energy

Dense associative memory (DAM) provides an energy-based framework for memory retrieval with close connections to attention mechanisms in modern artificial intelligence. Despite growing interest in differential privacy for AI, the privacy of DAM retrieval dynamics remains relatively unexplored. In this paper, we develop a differential privacy framework for log-sum-ReLU (LSR) dense associative memory, whose finite-support retrieval dynamics pose distinctive challenges for privacy-preserving computation. We propose the Boundary-Responsive Adaptive Correction Evolution (BRACE) algorithm, a differentially private retrieval mechanism for LSR-DAM that adaptively corrects boundary-sensitive perturbations to control their cumulative effect over the retrieval trajectory. In theory, we prove that our method is minimax optimal by deriving dimension-independent terminal and full-trajectory retrieval error rates, with optimal dependence on the inverse temperature and, in the growing-horizon regime, the retrieval horizon. We further establish central limit theorems that enable uncertainty quantification for private retrieval by characterizing its asymptotic distribution and the additional variability introduced by privacy. Numerical experiments compare our proposed method with baseline differential privacy approaches and evaluate its retrieval accuracy. Together, our results provide a theoretical foundation for optimal privacy-preserving retrieval and uncertainty quantification in energy-based associative memory systems.
Oct 6, 2026

A Geometry-Based Capacity Theory for Finite-Feature Associative Memory

We develop a geometry-based capacity theory for exact-key retrieval in compressed finite-feature Hebbian associative memory. For random or approximately isotropic values, retrieval interference separates into finite-feature noise, which decreases with feature dimension, and structural interference, which is determined by squared kernel overlap among stored keys and persists in the infinite-feature limit. This yields a fit-free prediction of retrieval quality, reveals a geometry-dependent capacity ceiling, and predicts the feature budget required for a target retrieval quality. When stored values are correlated, we show that retrieval depends jointly on the key kernel and value Gram matrix, and derive finite-feature approximations that account for this interaction. We validate the theory on synthetic, visual, and medical-image representations. Overall, the framework links representation geometry directly to memory capacity and distinguishes when performance can be improved by increasing the feature budget and when the representation itself must be changed. Across these settings, the predicted retrieval curves closely match empirical behavior and correctly identify changes in the preferred memory design.
Oct 1, 2026

Madeleine: Learning Involuntary Recall for Conversational Memory from Simulated Lives

A long-term conversational assistant must recall the right memory at the right moment, yet the memory that matters most is often not similar to what the user says now. Current systems recover such associations by letting an LLM reason at write or read time, at a cost of hundreds to over a thousand LLM calls per memory bank and up to several thousand context tokens per query. We argue that association is a learnable relevance: the pointwise mutual information of memories under how human lives unfold. We introduce Madeleine, which learns amortized association: offline, an LLM life simulator writes simulated lives, whose cue-trigger pairs teach a query encoder a residual association on top of frozen similarity; online, it calls no LLM and plugs into any vector memory by replacing only the query encoder. On LoCoMo-Plus under the official protocol, Madeleine (I) reaches 66.6 when plugged into HyperMem, the highest among all systems evaluated under this protocol; (II) used alone, reaches the score of HyperMem as released (52.4 vs. 52.9) with zero LLM calls and about 1/21 of its answer context; and (III) lifts T-Mem by 26.2 points, significantly outperforms the same untrained backbone inside both systems, and leaves ordinary QA intact on the 4B backbone.
Sep 30, 2026

Finite-Horizon Fisher Memory in Two-Sided Power-Bounded Recurrent Systems

We analyse allocation, admission and post-write retention in finite-horizon linear-Gaussian noisy recurrent memories. At every horizon, the directional Fisher memory MnM_n satisfies tr⁡Mn=N\operatorname{tr}M_n=N: non-normality redistributes information but cannot raise its spherical average, while normal carriers satisfy Mn=IM_n=I. For bi-power-bounded carriers, we derive uniform 1/n1/n lag bounds, identify the limit of MnM_n with the inverse of the classical Cesàro asymptotic limit of W⊤W^\top, and give finite-horizon error bounds. A time-varying coupling defines an end-to-end store operator. The writer-optimal direction need not be store-optimal. After writing ends, an invertible hold preserves the full stored Fisher matrix. Additive contamination bounded by αα times the closure covariance retains at least 1/(1+α)1/(1+α) of that matrix; a covariance-aware decoder attains the corresponding accuracy. With recurrent carriers held fixed, training input masks and linear readouts approached the task-specific optimum in 160 runs, with median normalized Rayleigh efficiency above 0.9980.998. Binary accuracy matched the Gaussian prediction to mean absolute error below 0.0020.002 over more than four orders of magnitude in JJ. In a separate pre-specified study of 320 runs, trained masks followed the designated input-time objective in both carrier types, in 16 of 16 draws. These studies used development-seen carriers and are pre-specified validations, not blind holdouts. The same fixed design reproduced the objective-specific result in 16 of 16 draws on carriers unused before run commitment. Exact isolation preserved information, while a decoder fixed at its training horizon fell to chance; inverse-adjoint transport restored its sampled decisions to numerical precision.
Sep 29, 2026

Non-Commutative Lie-Algebraic State-Space Models: Finite-Division Variety Laws, Semidirect Cascades, and Gradient Saddle Escape

Structured state-space sequence models (SSMs) achieve linear-time inference and logarithmic-time parallel training via associative scans, yet diagonal transitions generate solvable semigroups unable to track non-solvable group words or 3D rigid-body manifolds. We present Non-Commutative Lie-Algebraic State-Space Models (Lie-SSMs), parameterizing recurrent updates on Lie algebra tangent spaces via the matrix exponential. First, using an o-minimal cylindrical flag collar, we establish an unconditional single-layer finite-division criterion for contracting SO(3)SO(3) and SU(2)SU(2) heads, proving odd-prime Sylow abelianity, odd-pp-core centrality, and exponent-30 commutator obstructions excluding S5S_5 in SO(3)m,SU(2)m,SO(3)^m, SU(2)^m, and SO(4)mSO(4)^m while identifying the sharp threshold d∗(S5)=d∗(A6)=5d^*(S_5)=d^*(A_6)=5 in SO(5)SO(5). Second, Depth-2 semidirect and Schreier 22-cocycle cascades resolve S5S_5, 2I≅SL2(F5)2I\cong SL_2(F_5), and Heis(Fp)\mathrm{Heis}(F_p) in O(log⁡T)O(\log T) scan depth, whereas unnormalized contracting coupling collapses exponentially without Directional RMSNorm. Third, we characterize the Θ(εc−1)Θ(\varepsilon^{c-1}) Chen--Magnus--Hall--Witt commutator saddle and 1/∣[G,G]∣1/|[G,G]| coset-centroid dilution, proving generic first-order symmetry breaking at ε=0\varepsilon=0 and teacher-forced local Polyak--Łojasiewicz convergence under dense prefix supervision and factored readouts (99.90%/99.40%99.90\%/99.40\% on S5S_5 at T=128T=128; 100.00%100.00\% on D16D_{16}). Across SE2(3)SE_2(3) Left-Jacobian odometry, Hillis--Steele/Blelloch GPU scans (48.24×48.24\times forward, 20.27×/10.28×20.27\times/10.28\times full-layer speedup at T=2048T=2048), and three 3D IMU benchmarks (UCI HAR, UCI HAPT-PT, and 17-subject REALDISP), Lie-SO(3)SO(3) models reach 95.69%±0.14%95.69\%\pm 0.14\% on HAR, 88.44%±0.83%88.44\%\pm 0.83\% episode logit-vote (83.97%±1.20%83.97\%\pm 1.20\% C3 window in 30-subject 5-fold CV; 87.16%±0.64%87.16\%\pm 0.64\% 9-subject split) on HAPT-PT, and 84.23%±1.00%84.23\%\pm 1.00\% on REALDISP.
Sep 28, 2026

CyFA: Linear Sequence Modeling with Relative-Time-Partitioned Memory

Linear RNNs offer linear-time sequence processing and constant-memory decoding, but their fixed-size recurrent states must accommodate all past key--value associations. Existing forgetting mechanisms and Delta Rule updates reduce interference by selectively clearing or correcting the state, yet earlier associations can still become difficult to retrieve. We introduce CyFA (Cyclic Flow Attention), a Linear RNN with relative-time-partitioned memory. At each step, a learned clock controls the cyclic transport applied jointly to the key and value states before the current key--value pair enters the age-zero slot, thereby organizing stored associations across relative-time slots. We further derive an exact change to absolute-clock coordinates that expresses CyFA as two scalar-decay linear attention recurrences and enables efficient chunk-wise training. Across 400M--1.4B pretraining experiments with matched recurrent-state sizes, CyFA improves recall-intensive performance while maintaining competitive language modeling and high computational efficiency. At 400M, CyFA outperforms KDA on FDA (42.60 vs. 26.07) while requiring only 46.7% and 48.3% of KDA's forward and backward core-operator execution times, respectively. Our code is publicly available at this https URL.
Sep 28, 2026

Intrinsic Associative Memory on Riemannian Manifolds: Curvature, Capacity, and Emergent Modes

Geometry does more than constrain an associative memory: curvature determines what it remembers and which states it creates. We develop intrinsic dense associative memories on Riemannian manifolds by casting memory as Epanechnikov kernel-density mode seeking. We compare geodesic and volume-corrected energies and show that curvature separates their behavior. We prove that geodesic memory always retains an isolated pattern, while corrected memory obeys a sharp Ricci-curvature threshold: positive curvature can erase memories in high dimensions, while negative curvature reinforces them. We derive geodesic capacity scalings of qβ−1/2q_β^{-1/2} for retaining every pattern and qβ−1q_β^{-1} for a typical one, where qβq_β is the pairwise kernel-overlap probability. We show how overlap creates novel memories: designed NN-pattern configurations realize all 2N−12^N-1 subset modes, but random data at the storage threshold yield only a Poisson number. We establish exact one-step recall using Riemannian mean shift. In simulations, we recover the predicted curvature transition and every designed mode. On WordNet's full noun hierarchy, we demonstrate that volume correction improves low-capacity retrieval. Together, our work shows that curvature is a design variable for associative memory, not merely a property of the data.
Sep 27, 2026

SchemaMem: Schema-Indexed Recurrent Memory for Delayed State Retrieval

Attention provides direct access to past representations, but retaining an ever-growing history is costly. Recurrent models bound persistent state, yet must preserve selected information while processing subsequent inputs. We introduce SchemaMem, an attention-based recurrent memory architecture combining chunk-local attention with a persistent, schema-indexed phase state. Learned schema embeddings provide a shared representational reference for reading and writing. Reads use the current state, whereas writes use the layer input and static schema embeddings, excluding direct feedback from that layer's own state. Chunk-boundary commits aggregate bounded phase increments through forward computation. The same parameters also support full-history attention training before and during recurrent training. We studied selective updates, preservation, and delayed retrieval in a controlled address--value task, comparing three-layer models with approximately matched parameter counts and persistent-state dimensions. Across nine address/value settings and three training seeds, SchemaMem has higher mean written-value retention at four times the maximum training delay than both baselines, which are trained toward a higher in-range accuracy target. Updated-value recovery favors SchemaMem in all nine settings against Mamba-3 and seven against Gated DeltaNet. Defaults consistently favor Gated DeltaNet over SchemaMem at that delay, and SchemaMem requires substantially more optimization steps. These results identify a promising retention--optimization trade-off in schema-indexed recurrence.
Sep 27, 2026

How Linear Attention Remembers

Linear attention replaces the growing key--value (KV) cache of standard attention with a fixed-size recurrent state, substantially reducing memory growth with context length. This efficiency, however, changes how past information is stored: many tokens must share and repeatedly update the same memory. We study how this recurrent state functions as a memory system. Using an analytical decomposition together with controlled causal interventions in pretrained GLA and GDN models, we trace how recalled information is written, retained, and later accessed. We find that fact-specific information enters recurrent memory through concentrated, content-dependent writes and is later accessed through concentrated query-time read pathways. Multiple facts can remain selectively accessible within the same state, yet their internal representations exhibit cross-fact causal coupling rather than independent KV-like storage. As memory load increases, both recall and targeted editability degrade, whereas elapsed context alone has a substantially smaller effect within the tested regime. Causal interventions on subsequent writes further show that interference is shaped by their overlap with existing memory. Finally, in hybrid architectures that combine recurrent layers with full attention, the runtime memory directly supporting recall shifts predominantly to the full-attention KV state. Together, these results reveal how fixed-size recurrent memory supports selective recall despite shared storage, while exposing the interference and capacity limits that distinguish it from token-addressable KV memory.
Sep 26, 2026

DRAM: Delta-rule Recurrent Associative Memory for Robot Manipulation Policies

Robotic manipulation is inherently history-dependent, yet most pretrained robotic policies condition on only the current observation or a short temporal window. Equipping such policies with long-term memory remains challenging: existing approaches either feed the backbone multi-frame observation windows, which substantially increase inference cost, or rely on pre-defined semantic features, which limit task generality and may also require the retraining of the backbone to adapt to the memory. We introduce DRAM (Delta-rule Recurrent Associative Memory), a plug-and-play memory module that can be attached to a wide range of pretrained robotic policies, endowing them with long-horizon memory without architectural modification or backbone retraining, requiring only task-specific post-training of the memory module and action expert. DRAM maintains a fixed-size associative memory using gated delta-rule linear attention, with a modified update that incorporates all tokens within each frame in parallel. An architecture-agnostic readout integrates historical context into action prediction across different policy architectures. Experiments show that DRAM consistently improves frozen pretrained policies over short-context baselines and alternative compact memory designs, validating its effectiveness as a fixed-size, post-hoc memory module trained with the backbone frozen.
Sep 26, 2026

Graph Memory: Spectral Associative Memory via Dirichlet Energy

Dense associative memories have traditionally focused on storing and retrieving vector-valued patterns. Many modern machine learning problems, however, are naturally graph-structured, requiring memory mechanisms for relational patterns, graph diffusion geometries, community structures, and graph-based inductive biases. We propose a spectral dense associative memory for storage and retrieval of graph data, extending the classical vector-valued memories. Retrieval is performed through a log-sum-exp energy induced by Dirichlet energy with spectral norm distances, producing a softmax-weighted average of the stored Laplacians that remains a valid graph Laplacian. We prove exponential storage capacity and exponentially decaying retrieval error. Beyond graph retrieval, we establish theoretical guarantees for spectral quantities central to graph learning, including eigenvalues, eigenspaces, and diffusion operators. Experiments on synthetic graph data, real-world airline network, protein conformation data and wearable sensor data demonstrate robust graph retrieval while preserving the graph geometry of the data. Our framework provides a new associative memory paradigm for graph-structured data and bridges dense associative memory with modern graph learning and generative AI.
Sep 24, 2026

Boolean threshold functions, neuron capacity, and memory retrieval

How much information can a single neuron remember? How many memories can neural networks retrieve without creating false memories? These questions are related to a basic question: how many Boolean threshold functions f(x)=sgn⁡(a0+⟨a,x⟩)f(x)=\operatorname{sgn}(a_0+\langle a,x\rangle), x∈{−1,1}nx\in\{-1,1\}^n, are there? In this paper, we show that the number TnT_n of distinct Boolean threshold functions is Tn=2(2n−1n)(1+O(n−99)).T_n=2\binom{2^n-1}{n}\bigl(1+O(n^{-99})\bigr). Equivalently, the capacity of a single threshold neuron is n2−log⁡2(n!)+1+O(n−99)n^2-\log_2(n!)+1+O(n^{-99}) bits, improving the O(n)O(n) error term in the result of Kahn--Komlós--Szemerédi to O(n−99)O(n^{-99}). To prove this, we show that, for 1≤r≤n−11\le r\le n-1, and v1,…,vrv_1,\ldots,v_r are chosen at random from {−1,1}n\{-1,1\}^n, P ⁣{⟨v1,…,vr⟩∩{−1,1}n={±v1,…,±vr}}=1−O(n−99).\mathbb P\!\left\{ \langle v_1,\ldots,v_r\rangle\cap\{-1,1\}^n =\{\pm v_1,\ldots,\pm v_r\} \right\} =1-O(n^{-99}). In the context of the Kanter--Sompolinsky Hamiltonian for memory retrieval, this identifies r=n−1r=n-1 as a sharp threshold, at which, for almost every collection of rr memories, the only ground states are these memories and their negatives, confirming a weaker form of the Kalai--Linial--Odlyzko conjecture. It also settles a recent open problem posed by M. Anthony on the specification number of Boolean threshold functions. In addition, we show that, for every 1≤r≤n−11\le r\le n-1, P{v1,…,vr are linearly dependent}=2(r2) 2−n+O ⁣(2−ne−cn),\mathbb P\{v_1,\ldots,v_r\text{ are linearly dependent}\} =2\binom r2\,2^{-n}+O\!\left(2^{-n}e^{-cn}\right), confirming a conjecture of Kahn--Komlós--Szemerédi.
Sep 23, 2026

Constant-Memory Recall: Learned Associations in a Fixed Matrix State

Fixed-size recurrent memory limits storage growth during inference, but successful recall depends on the task and training. We study a small DeltaNet variant with fixed token-specific key biases, trained to remember 32 new key-value pairings per sequence. With 32 KiB of recurrent matrix state, it achieves 99.95% mean accuracy across three training seeds when choosing among the sequence's values. Recall remains near perfect when filler extends the pre-query context to 1,798 tokens without adding pairings. Zeroing the first memory block removes this recall. An exploratory 48-pair test remains near chance after one quarter of the primary training budget and does not locate a capacity limit. Parameter-matched vector and Transformer baselines remain near chance, including the Transformer after additional training searches. This unresolved baseline failure prevents a memory-efficiency comparison.
Sep 22, 2026

Latest Exact Match Attention

We introduce latest exact match attention (LEMA), an attention variant for transformers where queries and keys are binarized and each query attends only to the latest exactly matching key. We prove that LEMA transformers with chain of thought can simulate word-RAMs, as was recently shown for the less restrictive rightmost hard attention. In contrast to prior hard attention variants, the restriction to exact matches enables an efficient converse direction: word-RAMs can simulate LEMA transformers at a cost per token independent of the context length. Together, these results yield a close correspondence between the two computational models in terms of both compute and memory. Beyond the theory, we propose a training method for LEMA transformers that handles their non-differentiable operations with a straight-through estimator for the binarization and a soft attention surrogate annealed towards LEMA. On a synthetic associative recall task, LEMA models trained this way use their growing state to store and recall a large number of associations, outperforming gated DeltaNet (GDN) with its fixed state size. As a first scaling test, we train LEMA language models with up to 834 million parameters. They match softmax transformers of around half their size in loss and, on repeated rare phrases and a needle-retrieval task, remain behind softmax transformers but recall across longer distances than GDN models of comparable size. Finally, we implement dictionary-based inference for LEMA transformers and show constant generation speed comparable to GDN despite their growing state, with the dictionaries residing in main memory rather than VRAM. Code is available at https://github.com/moritzbroe/latest_exact_match_attention.
Sep 15, 2026

Bias-Induced Crossover in Absolute Capacity of Dense Associative Memory

The absolute capacity of dense associative memory has mainly been analyzed for unbiased patterns. Here we examine the effect of bias in centered binary patterns under the Krotov-Hopfield single-site criterion Perror=1/NP_{\mathrm{error}}=1/N, where PerrorP_{\mathrm{error}} is the probability that a single-site flip lowers the energy of a stored pattern and NN is the number of neurons. Each pattern component takes 1−q1-q with probability qq and −q-q otherwise, where 0<q≤1/20<q\le1/2. For polynomial interactions of order nn, a signal-to-noise analysis gives an absolute capacity of order Nn−1/ln⁡NN^{n-1}/\ln N at q=1/2q=1/2. For fixed q<1/2q<1/2, however, the capacity is O(Nn/2)O(N^{n/2}) for even n≥4n\ge4 and O(N(n+1)/2)O(N^{(n+1)/2}) for odd n≥5n\ge5. For n=3n=3, both the unbiased and fixed-bias capacities remain O(N2/ln⁡N)O(N^2/\ln N). For n≥4n\ge4, these different asymptotic forms imply a nonuniform large-NN limit near q=1/2q=1/2. Asymptotic matching predicts a bias-induced crossover in the region 1−2q=O(ln⁡N/N⌊n/2⌋−1)1-2q=O(\ln N/N^{\lfloor n/2\rfloor-1}). The crossover originates from a bias-dependent crosstalk mean that reduces the stability of sites carrying the more frequent value −q-q. Computer simulations are compared with the finite-size conditioned-Gaussian predictions. An activity-dependent control potential that cancels the conditional crosstalk mean restores the Nn−1/ln⁡NN^{n-1}/\ln N capacity for fixed 0<q<1/20<q<1/2 within the conditioned-Gaussian approximation.