Constant-Memory Recall: Learned Associations in a Fixed Matrix State

Samuel Larson

cs.LGSep 23, 2026

Constant-Memory Recall: Learned Associations in a Fixed Matrix State

Authors: Samuel Larson

Organizations: Pebble ML

Abstract

Fixed-size recurrent memory limits storage growth during inference, but successful recall depends on the task and training. We study a small DeltaNet variant with fixed token-specific key biases, trained to remember 32 new key-value pairings per sequence. With 32 KiB of recurrent matrix state, it achieves 99.95% mean accuracy across three training seeds when choosing among the sequence's values. Recall remains near perfect when filler extends the pre-query context to 1,798 tokens without adding pairings. Zeroing the first memory block removes this recall. An exploratory 48-pair test remains near chance after one quarter of the primary training budget and does not locate a capacity limit. Parameter-matched vector and Transformer baselines remain near chance, including the Transformer after additional training searches. This unresolved baseline failure prevents a memory-efficiency comparison.

Figures & tables

$\Delta$ vs flat-vector: mean 0.9656, 95% paired CI (0.9582, 0.9729)
model	seed 0	seed 1	seed 2	mean	above threshold?
matrix fast-weight	0.9995	1.0000	0.9990	0.9995	every seed
flat-vector recurrence	0.0322	0.0327	0.0369	0.0339	never
transformer (uncapped)	0.0271	0.0293	0.0286	0.0283	never
$\Delta$ vs transformer: mean 0.9712, 95% paired CI (0.9685, 0.9738) (descriptive)

Table 1: Recall of 32 pairings after 20,000 training steps, by training seed. Accuracy is restricted to the episode’s values; chance is 0.031 and the recall threshold is 0.094. The matrix model learns the task in all three seeds; both parameter-matched baselines remain near chance.

Figure 1: Recall after extending the pre-query context with filler, keeping 32 pairings fixed. Panel (a) shows the full accuracy range; (b) enlarges the baseline scores. The matrix model retains accuracy

\geq 0.998

with 32 KiB of matrix state across three seeds. Full-cache (dashed) and capped-cache (crosses,

M\in\{2,4,8,16,32\}

) Transformers remain near chance. Reference lines mark the recall threshold and chance. The protocol excludes

M=1

from this plot; its scores are reported in Table 3 .

Figure 2: Zeroing a memory block before the query, using 4,096 fixed queries per seed. Removing

S_{0}

reduces matrix-model accuracy below the 0.094 recall threshold; removing

S_{1}

leaves it nearly unchanged. Chance is

1/32

. Seed markers are offset vertically for readability.

chance 0.0208; exploratory threshold 0.0625; no model above threshold
	matrix fast-weight	flat-vector	transformer
$K{=}48$ ( $K/d{=}0.75$ ) accuracy	0.0189	0.0195	0.0218

Table 2: Recall with 48 pairings: one 5,000-step run per model, versus 20,000 steps for the primary 32-pair comparison. All scores are near chance (0.021). The shorter training budget prevents attributing the failure to memory load alone.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

model / KV budget	$T{=}454$	$T{=}902$	$T{=}1798$
matrix, fixed 32,768 B	0.9989	0.9989	0.9989
transformer, uncapped	0.0293	0.0314	0.0339
transformer, $M{=}32$ cap	0.0300	0.0251	0.0256
transformer, $M{=}16$ cap	0.0280	0.0240	0.0251
transformer, $M{=}8$ cap	0.0257	0.0264	0.0250
transformer, $M{=}4$ cap	0.0251	0.0277	0.0251

Appendix

Table 3: Mean recall across three seeds at pre-query context lengths

T

. The matrix state is fixed at 32,768 bytes; Transformer cache caps are

M

times that size. Chance is 0.031 and the recall threshold is 0.094. The protocol excluded the

M=1

eight-token cap because it is below the registered floor of 13 tokens (query plus one binding clause). Its descriptive scores, 0.021–0.030, are also near chance.

model	linear readout	cos. mean	[email protected]	shuffled cos.	gap
matrix	$S_{1}q$ (shallow query)	0.165	0.000	0.105	+0.060
matrix	$S_{0}q$ (bind-path query)	0.118	0.000	0.112	+0.006
matrix	$S_{1}q$ (true query)	0.167	0.000	0.104	+0.063
matrix	pre-LM-head hidden	0.894	0.674	0.094	+0.800
flat-vector	$S_{1}q$ (shallow query)	0.117	0.000	0.112	+0.005
flat-vector	$S_{0}q$ (bind-path query)	0.113	0.000	0.111	+0.002

Appendix

Table 4: Ridge-probe recovery of target value vectors on separate 20,000-step seed-0 calibration checkpoints. Probes fit 24,576 examples and evaluate 4,096 queries; regularization was selected on that evaluation set. “[email protected]” is the fraction above cosine 0.9; “gap” subtracts a shuffled-fit control. Only the matrix model’s pre-LM-head hidden state supports substantial recovery. These are descriptive probe results.

Figure 3: Training loss for the primary task, seed 0. After 20,000 steps, loss is 1.38 for the matrix model, 7.69 for the vector recurrence, and 7.51 for the Transformer. The baselines’ decreasing loss does not produce recall. These runs use the shared settings of Section 2 .

model	s0	s1	s2	s3	s4	s5	s6	s7	s8
matrix	0.040	0.029	0.334	0.036	0.031	0.479	0.028	0.391	0.041
flat-vector	0.034	0.028	0.037	0.031	0.031	0.032	0.034	0.031	0.035

Appendix

Table 5: Recorded accuracy under ambiguous hop supervision, nine seeds per model. Bold values exceed the original 0.094 threshold. The numbers are retained for documentation; crossing that threshold is not evidence of following a requested hop count, which was absent from the input.

Associative Memory

Associative memory studies how systems store patterns or associations and retrieve them from partial, noisy, or context-dependent cues.

See all 120 papers

Anatomy of Associative Recall in Fixed-State Recurrences: A Matched-State Decomposition, an Interference Wall, and a Curriculum That Breaks It
Sep 14, 2026Julian Boesch, Andrew WeeRecurrent Neural Networks Curriculum Learning
1Purdue University · 2Obit Research
SchemaMem: Schema-Indexed Recurrent Memory for Delayed State Retrieval
Sep 27, 2026Sungwoo Goo, Hwi-yeol Yun, Sangkeun JungMemory-Augmented Neural Networks Persistent Memory for Language Models
College of Pharmacy, Chungnam National University Daejeon, Republic of Korea · Department of Computer Science & Engineering, Chungnam National University Daejeon, Republic of Korea
DeltaNet-PC: Test-Time Memory Without Backpropagation Through Time
Oct 10, 2026Todd Morrill, Richard ZemelNeural Network Memorization Predictive Coding
Computer Science, Columbia University New York, NY, USA

Sep 14, 2026J/K move · Enter open · S save

Anatomy of Associative Recall in Fixed-State Recurrences: A Matched-State Decomposition, an Interference Wall, and a Curriculum That Breaks It

Julian Boesch, Andrew Wee

1Purdue University · 2Obit Research

Recurrent Neural Networks Curriculum Learning Associative Memory State Space Models

Open paper AskPDF arXiv

Fixed-state recurrences--linear attention and state-space models--are reported to lag behind attention on associative recall, but whole-architecture comparisons cannot say which ingredient is responsible. We decompose masked multi-query recall at a fixed state budget along three single-knob axes: a short causal convolution, the transition structure (rank-1 delta rule vs. diagonal), and decay. The convolution dominates (~+0.5 recall in both families under matched training): comparisons that pit convolution-free cells against a convolution-equipped Mamba measure the missing convolution, not the recurrence. The rank-1 transition beats its diagonal ablation by +0.19/+0.32 at 16/32 pairs, but the margin shrinks to +0.03 once both cells carry the convolution, and a state-matched Mamba-2 ties the unarmed rank-1 cell: no class claim survives. Cells that solve 32-pair recall degrade gracefully with load yet fall to chance retrieving 4 pairs from a distractor haystack--flat across lengths and transitions. Interference under sparse supervision, not capacity: a distance curriculum takes the unchanged architecture from 0.021 to 1.000. Training is a lock-in lottery--a seed either locks in or does not--and the curriculum is the lever. Lock-in rises from 1/10 to 7/10 (p=0.02); dense supervision adds nothing; at L=256 a shaped ramp reopens a boundary the uniform curriculum cannot (4/5 vs. 0/9); and at L=512, where the ramp collapses (0/6), gating it on measured accuracy locks in 6/6 (p=0.001). Bidirectional denoiser cells, reading the query before the haystack, show no measurable advantage over causal training (ten seeds), and collision-key retrieval needs two layers. Arming for recall is free on an S_5 state-tracking guardrail--the armed cell is significantly better at every depth (p<=0.0044). These replace "recurrent models are bad at recall" with a measured decomposition and two cheap interventions.

Abstract

Associative Memory

Explore similar work

Anatomy of Associative Recall in Fixed-State Recurrences: A Matched-State Decomposition, an Interference Wall, and a Curriculum That Breaks It

SchemaMem: Schema-Indexed Recurrent Memory for Delayed State Retrieval

DeltaNet-PC: Test-Time Memory Without Backpropagation Through Time