DeltaNet-PC: Test-Time Memory Without Backpropagation Through Time

Todd Morrill; Richard Zemel

cs.LGOct 10, 2026

DeltaNet-PC: Test-Time Memory Without Backpropagation Through Time

Authors: Todd Morrill, Richard Zemel

Organizations: Computer Science, Columbia University New York, NY, USA

Abstract

Test-time memorization of sequential information is a foundational capability for any intelligent system that needs to rapidly store and retrieve data, as demonstrated by the attention mechanism in large language models. However, Transformers and linear attention variants are trained via backpropagation through time (BPTT), which has drawbacks such as its enormous memory requirements. Predictive coding (PC), an algorithm that optimizes a prediction-error energy function with local updates, offers a compelling solution but still lacks test-time memory for sequential data. We develop a solution to this problem by treating the per-step memory update $ΔS_t$ as a latent variable, as opposed to a slow weight to be learned. By taking the log probability of a standard PC-based probabilistic model, we derive a PC-native solution for verbatim sequence memory, and relate our update rule to common choices in the linear attention literature (e.g., DeltaNet). We present an extensive set of empirical tests to probe our system's properties, including its scaling capacity, memory footprint, and ability to infer a context from evidence (e.g., bank (river) vs. bank (money)). We show how DeltaNet-PC can be used for storing single sequences of data, multiple sequences of data, and key-value associative memory pairs with perfect recall accuracy.

Figures & tables

Figure 1: We evaluate our system on several sequential memory tasks. A: Within-episode structure tests the model’s ability to handle non-Markovian dynamics in a single sequence. B: Multi-episode settings test the system’s ability to memorize and reason about multiple sequences, particularly with highly overlapping subsequences. C: Key-value binding tests whether the system can retrieve a value when cued with a key.

		Training	Inference	Local	Local	Sequential	Trained	Test-time
Model	Write rule	memory	memory	in space	in time	data	keys, values	write
Transformer ( Vaswani et al., 2017 )	append (KV cache)	$LD$	$LD$	$\times$	$\times$	✓	✓	✓
Linear attention ( Katharopoulos et al., 2020 )	outer product	$LD$	$D^{2}$	$\times$	$\times$	✓	✓	✓
DeltaNet ( Schlag et al., 2021 ; Yang et al., 2024b )	delta rule (LMS/NLMS)	$LD$	$D^{2}$	$\times$	$\times$	✓	✓	✓
Mesa layer ( von Oswald et al., 2024 ; von Oswald et al., 2026 )	RLS	$LD$	$D^{2}$	$\times$	$\times$	✓	✓	✓
Lin. assoc. memory ( Kohonen, 1972 ; Kohonen, 1974 )	outer product / RLS	–	$D^{2}$	✓	–	$\times$	$\times$	✓

Table 1: Situating DeltaNet-PC. Memory is one layer’s footprint;

L

the sequence length and

D

the latent space dimensionality. Local: weight updates use only pre- and postsynaptic activity (space) and only the current step (time). Sequential: stores ordered sequences. Trained: the system learns the keys and values, rather than given as raw data. Test-time: an episode is stored in one pass with no gradient steps on the weights. See Appendix A for an extended related work section and Appendix B.2 , which relates our work to linear attention.

Figure 2: A: DeltaNet-PC architecture. B: The per-position energy

E_{t}

. See Appendix B.4 for the full energy function.

Figure 3: A: a sample from our single episode sequential memory task that illustrates non-Markovian dynamics. B: when

L\leq D

, the memory can recall sequences with perfect accuracy. C: scaling

D

increases the memory’s capacity allowing it to store more memories.

Write rule	Untrained	Trained
Outer product, ${\bm{S}}\leftarrow{\bm{S}}+{\bm{v}}_{t}{\bm{k}}_{t}^{\top}$ ( Kohonen, 1972 ; Katharopoulos et al., 2020 )	0.13 $\pm$ 0.00	0.66 $\pm$ 0.03
Delta rule (NLMS), ${\bm{S}}\leftarrow{\bm{S}}+({\bm{v}}_{t}-{\bm{S}}{\bm{k}}_{t}){\bm{k}}_{t}^{\top}/\lVert{\bm{k}}_{t}\rVert^{2}$ ( Widrow & Hoff, 1960 ; Schlag et al., 2021 )	0.13 $\pm$ 0.00	0.26 $\pm$ 0.01
Recursive least squares, Equation 7 ( Kohonen, 1974 ; von Oswald et al., 2024 )	0.29 $\pm$ 0.01	1.00 $\pm$ 0.00

Table 2: Test recall accuracy under various write rules of untrained and trained systems. Sequential recall at

L=D=256

, chance

0.10

, 3 seeds, mean

\pm 1\sigma

. Untrained systems cannot solve the task and even in trained systems, the write rule matters: the recursive least squares update is the only one that achieves perfect recall.

Figure 4: Contextual inference heatmap. Top: cosine similarity between the inferred context and the 8 possible contexts at each point in time. We observe the system resolve its uncertainty about the context as it sees more transitions. Bottom: given the inferred context, we measure recall accuracy for the episodes and note that it behaves nearly like the ideal observer upper bound.

DeltaNet	1.000	1,024	$\times$	$\times$	0.53 M c	65 k	0.7 GB	55 GB	OOM
	MQAR		Local in				Peak training memory at $L$
Model	Test acc.	Streams a	time	space	Params	State b	512	65k	131k
DeltaNet-PC (ours)	1.000	$32$	✓	✓	0.20 M	131 k	0.25 GB	0.25 GB	0.25 GB

Table 3: Comparison of DeltaNet ( Yang et al., 2024b ) and DeltaNet-PC at width

D=256

on MQAR (512-token streams, 8,192 vocabulary). a Distinct training streams needed to achieve perfect test recall (one seed per model). b Fast-weight floats: DeltaNet’s per-head states, DeltaNet-PC’s memory

{\bm{S}}

and RLS inverse

{\bm{P}}

. c Excludes token embedding parameters. Memory measurements are based on a batch of 32 on an 80 GB GPU. Test accuracy is over 3 seeds for DeltaNet-PC and 1 seed for DeltaNet.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Event stream	Test scoring
Sequential recall	Observations and queries alternate: after each observation $({\bm{x}}_{t-1},{\bm{x}}_{t},t)$ the learner poses its own query $({\bm{x}}_{t-1},{\bm{x}}_{t},t)$ , to retrieve what was just written at its own position, against the memory as it stands. $L$ observations and $L$ queries per stream; online learning.	Ground truth ${\bm{x}}_{t}$ compared to prediction $\hat{{\bm{x}}}_{t}$ decoded from ${\bm{S}}_{L}R({\bm{W}}_{k}\,\operatorname{enc}(\hat{{\bm{x}}}_{t-1}),t)$ for $t=1,\dots,L$ , the final memory frozen.
Official MQAR	The $128$ context observations (64 pairs), then the query section: at each labelled position $t$ the task’s query $({\bm{x}}_{t-1},{\bm{y}}_{t},t)$ , the query key and its value, precedes the observation that writes the filler ${\bm{x}}_{t}$ . Every position is written.	Ground truth ${\bm{y}}_{t}$ compared to prediction $\hat{{\bm{y}}}_{t}$ decoded from ${\bm{S}}_{t-1}R({\bm{W}}_{k}\,\operatorname{enc}({\bm{x}}_{t-1}),t)$ for each $t$ containing a query.
Omniglot binding	Stream of $L$ observations, then the task’s queries: all queries have the form $({\bm{x}}_{t-1},{\bm{x}}_{t},L)$ meaning they are queries at the final position $L$ .	Ground truth ${\bm{x}}_{t}$ compared to prediction $\hat{{\bm{x}}}_{t}$ decoded from ${\bm{S}}_{L}\,R({\bm{W}}_{k}\,\operatorname{enc}({\bm{x}}_{t-1}),L)$ .

Appendix

Table 4: The event streams for the three tasks.

Task	Observation	$d_{\text{obs}}$	$L$ , $D$	$V$	Train / Val. / Test	Scored units per test set
Single-episode sequences	token	256	$L=64$ to $512$ at $D=256$ , $L=512$ at $D=64$ to $512$ , $L=D=1{,}024$	10 tokens	$1{,}024$ / $128$ / $128$	$128L$ relay tokens
Multi-episode sequences	token	256	$256$ ( $K=8\times\ell=32$ ), $D=256$	10 tokens	$1{,}024$ / $128$ / $128$	$32{,}768$ relay tokens
MQAR ( Arora et al., 2024 )	token	256	$512$ , $D=256$	$8{,}192$ tokens ( $4{,}095$ keys, $4{,}096$ values)	$100{,}000$ (or $32$ ) / $128$ / $3{,}000$	$192{,}000$ labelled positions
Omniglot-MNIST binding	image $28\times 28$	784	$256$ ( $128$ pairs), $D=256$	10 digits	$1{,}024$ / $128$ / $128$	$16{,}384$ probes

Appendix

Table 5: Summary of datasets.

Figure 5: Multi-episode examples. A : A single training example. B : An evaluation example for the contextual inference task from Section 4.2 .

Figure 6: Omniglot-MNIST key-value memory example.

Figure 7: Learned rotary frequencies of the three rows that use them, three seeds each, against the base-

10^{4}

initialization (dashed). The dotted line marks one cycle per horizon:

2\pi/256

on the sequential and Omniglot rows and

2\pi/32

, one episode, on the multi-episode row. Sequential and multi-episode recall raise the slow bands, and Omniglot binding, which reads after every write, sends most of them to zero.

	Untrained		Trained (decoder)
	Clean-up	Decoder	$32$ streams	$100{,}000$ streams
RoPE off	$1.000$	$0.667$	$1.000$	$1.000$
RoPE on	$0.570$	$0.043$	$0.964$	$0.999$

Appendix

Table 6: MQAR test accuracy of DeltaNet-PC at

D=256

with RoPE off (the setting of Table 3 ) and on (as in every other task). Clean-up : each memory read is matched to the nearest encoded codebook row by cosine similarity, with no settle, as in Table 2 . Decoder : the protocol of Table 3 , with the

16

-step test-time settle. Untrained and full-data cells are means over

3

seeds, and the

32

-stream cells are one seed.

Symbol	Name	Value	Description
Task
$L$	Sequence length	256	Positions per sequence (swept in Figure 3 ).
$V$	Vocabulary size	10	Distinct tokens, each a fixed Gaussian code of dimension $d_{\text{obs}}$ . 8,192 for MQAR.
$d_{\text{obs}}$	Observation dimension	256	Codebook row dimension (images: pixel count).
$n_{\text{train}}$	Training sequences	1024	Unique training sequences.
$n_{\text{test}}$	Test sequences	128	Held-out sequences. The validation split is the same size.

Appendix

Table 7: DeltaNet-PC default hyperparameters.

	Sequential	Multi-episode	MQAR	Omniglot
Full recipe	$1$	$1$	$1$	$0.9999$
No activity clip	$1$	$1$	$1$	$0.9999$
No value orthogonality ( $\lambda_{v}=0$ )	$1$	$1$	$1$	$0.926$
NLMS write (no interference term)	$0.282$	$0.178$	$0.978$	$0.807$
Frozen RoPE	$0.359$	$0.689$	–	$0.940$

Appendix

Table 8: Ablations of several system components—each row removes one component from the full recipe. Test accuracy on sequential recall (

L=D=256

), multi-episode recall (

K=8\times 32

), MQAR (

D=256

) and Omniglot-MNIST binding with

128

keys. On Omniglot the full recipe makes

2

errors in

16{,}384

probes and the variant without the clip makes

1

. The official MQAR row runs with RoPE off, so freezing it does not apply there.

	DeltaNet-PC	DeltaNet
Training, operations	$O(LD^{2}+PTD^{2})$ , $3.2$ GFLOP	$O(L(D^{2}+DV))$ , $8.7$ GFLOP
Test, operations	$O(LD^{2}+PDV)$ , $0.75$ ( $4.0$ ) GFLOP	$O(L(D^{2}+DV))$ , $2.9$ GFLOP
Training, time	$18$ ms	$0.42$ ms
Test, time	$3$ ( $12$ – $20$ ) ms	$0.8$ ms
To exact recall	$1{,}024$ observed streams, $3.3$ TFLOP, $18$ s	$200{,}192$ observed streams, $1.7$ PFLOP, $88$ s

Appendix

Table 9: Compute per MQAR stream (

L=512

P=64

queries,

D=256

V=8{,}192

T=64

) and measured time on one H100. DeltaNet-PC’s test figures in parentheses include the

16

-step test-time settle, e.g., 0.75 for no test-time settle, and 4.0 with a 16-step test-time settle. Training times are per stream at batch

32

for DeltaNet-PC and

128

for DeltaNet. Streams to exact recall count every training stream the model observed, including repeats.

Figure 8: tPC’s intrinsic memorization ceiling. Test recall accuracy on the

N

training sequences as a function of

N

, with a unique

\mathrm{BOS}

token per training sequence.

L=8

D=256

Neural Network Memorization

Neural network memorization studies how models store and retrieve training examples, how this differs from generalization, and how memory can be measured or controlled.

See all 64 papers

Oct 6, 2026

DeltaTTT: Layerwise Optimization for Nonlinear Recurrent Memory

Sequential test-time training adapts a memory network through successive updates, each computing an inner-loop gradient based on the network's previous state. Intuitively, this state dependence should allow each update to account for what the memory has already learned and better incorporate new information. However, we find that this expected advantage does not consistently materialize in nonlinear memories: a fixed-base parallel TTT baseline outperforms its serial counterpart. Our exploratory experiments point to a key underlying difficulty: nonlinear memories can be harder to optimize than linear ones within a single pass over the sequence. To alleviate this optimization difficulty, we introduce DeltaTTT, which replaces joint inner-loop optimization of a two-layer memory network with layerwise learning. Each layer is assigned a local prediction target and updated through a state-dependent delta rule. This formulation retains a nonlinear readout while enabling chunkwise parallel computation. Experiments on DeltaNet and LaCT backbones show improvements in language modeling and retrieval over their recurrent baselines.

Yining Li, Dongchen Han, Jie Fu +1

Qiuzhen College, Tsinghua University · LeapLab, Tsinghua University · IQuest Research

Deep Learning Optimization Neural Network Optimization Test-Time Training Language Modeling

Ask PDF

Sep 23, 2026

Constant-Memory Recall: Learned Associations in a Fixed Matrix State

Fixed-size recurrent memory limits storage growth during inference, but successful recall depends on the task and training. We study a small DeltaNet variant with fixed token-specific key biases, trained to remember 32 new key-value pairings per sequence. With 32 KiB of recurrent matrix state, it achieves 99.95% mean accuracy across three training seeds when choosing among the sequence's values. Recall remains near perfect when filler extends the pre-query context to 1,798 tokens without adding pairings. Zeroing the first memory block removes this recall. An exploratory 48-pair test remains near chance after one quarter of the primary training budget and does not locate a capacity limit. Parameter-matched vector and Transformer baselines remain near chance, including the Transformer after additional training searches. This unresolved baseline failure prevents a memory-efficiency comparison.

Samuel Larson

Pebble ML

Associative Memory Memory-Efficient Inference Gated Delta Networks Delta-Rule Linear Attention

Ask PDF

Apr 22, 2026

Preconditioned DeltaNet: Curvature-aware Sequence Modeling for Linear Recurrences

To address the increasing long-context compute limitations of softmax attention, several subquadratic recurrent operators have been developed. This work includes models such as Mamba-2, DeltaNet, Gated DeltaNet (GDN), and Kimi Delta Attention (KDA). As the space of recurrences grows, a parallel line of work has arisen to taxonomize them. One compelling view is the test-time regression (TTR) framework, which interprets recurrences as performing online least squares updates that learn a linear map from the keys to values. Existing delta-rule recurrences can be seen as first-order approximations to this objective, but notably ignore the curvature of the least-squares loss during optimization. In this work, we address this by introducing preconditioning to these recurrences. Starting from the theory of online least squares, we derive equivalences between linear attention and the delta rule in the exactly preconditioned case. Next, we realize this theory in practice by proposing a diagonal approximation: this enables us to introduce preconditioned variants of DeltaNet, GDN, and KDA alongside efficient chunkwise parallel algorithms for computing them. Empirically, we find that our preconditioned delta-rule recurrences yield consistent performance improvements across synthetic recall benchmarks and language modeling at the 340M and 1B scale.

Neehal Tumma, Noel Loo, Daniela Rus

1MIT · 2Liquid AI

Long-Context Language Modeling Recurrent Neural Networks Recurrent Transformers Linear Attention Delta-Rule Linear Attention Linear RNNs

Ask PDF

Abstract

Neural Network Memorization

Explore similar work

DeltaTTT: Layerwise Optimization for Nonlinear Recurrent Memory

Constant-Memory Recall: Learned Associations in a Fixed Matrix State

Preconditioned DeltaNet: Curvature-aware Sequence Modeling for Linear Recurrences