DeltaNet-PC: Test-Time Memory Without Backpropagation Through Time

Todd Morrill; Richard Zemel

cs.LGOct 10, 2026

DeltaNet-PC: Test-Time Memory Without Backpropagation Through Time

Authors: Todd Morrill, Richard Zemel

Organizations: Computer Science, Columbia University New York, NY, USA

Abstract

Test-time memorization of sequential information is a foundational capability for any intelligent system that needs to rapidly store and retrieve data, as demonstrated by the attention mechanism in large language models. However, Transformers and linear attention variants are trained via backpropagation through time (BPTT), which has drawbacks such as its enormous memory requirements. Predictive coding (PC), an algorithm that optimizes a prediction-error energy function with local updates, offers a compelling solution but still lacks test-time memory for sequential data. We develop a solution to this problem by treating the per-step memory update $ΔS_t$ as a latent variable, as opposed to a slow weight to be learned. By taking the log probability of a standard PC-based probabilistic model, we derive a PC-native solution for verbatim sequence memory, and relate our update rule to common choices in the linear attention literature (e.g., DeltaNet). We present an extensive set of empirical tests to probe our system's properties, including its scaling capacity, memory footprint, and ability to infer a context from evidence (e.g., bank (river) vs. bank (money)). We show how DeltaNet-PC can be used for storing single sequences of data, multiple sequences of data, and key-value associative memory pairs with perfect recall accuracy.

Figures & tables

Figure 1: We evaluate our system on several sequential memory tasks. A: Within-episode structure tests the model’s ability to handle non-Markovian dynamics in a single sequence. B: Multi-episode settings test the system’s ability to memorize and reason about multiple sequences, particularly with highly overlapping subsequences. C: Key-value binding tests whether the system can retrieve a value when cued with a key.

		Training	Inference	Local	Local	Sequential	Trained	Test-time
Model	Write rule	memory	memory	in space	in time	data	keys, values	write
Transformer ( Vaswani et al., 2017 )	append (KV cache)	$LD$	$LD$	$\times$	$\times$	✓	✓	✓
Linear attention ( Katharopoulos et al., 2020 )	outer product	$LD$	$D^{2}$	$\times$	$\times$	✓	✓	✓
DeltaNet ( Schlag et al., 2021 ; Yang et al., 2024b )	delta rule (LMS/NLMS)	$LD$	$D^{2}$	$\times$	$\times$	✓	✓	✓
Mesa layer ( von Oswald et al., 2024 ; von Oswald et al., 2026 )	RLS	$LD$	$D^{2}$	$\times$	$\times$	✓	✓	✓
Lin. assoc. memory ( Kohonen, 1972 ; Kohonen, 1974 )	outer product / RLS	–	$D^{2}$	✓	–	$\times$	$\times$	✓

Table 1: Situating DeltaNet-PC. Memory is one layer’s footprint;

L

the sequence length and

D

the latent space dimensionality. Local: weight updates use only pre- and postsynaptic activity (space) and only the current step (time). Sequential: stores ordered sequences. Trained: the system learns the keys and values, rather than given as raw data. Test-time: an episode is stored in one pass with no gradient steps on the weights. See Appendix A for an extended related work section and Appendix B.2 , which relates our work to linear attention.

Figure 2: A: DeltaNet-PC architecture. B: The per-position energy

E_{t}

. See Appendix B.4 for the full energy function.

Figure 3: A: a sample from our single episode sequential memory task that illustrates non-Markovian dynamics. B: when

L\leq D

, the memory can recall sequences with perfect accuracy. C: scaling

D

increases the memory’s capacity allowing it to store more memories.

Write rule	Untrained	Trained
Outer product, ${\bm{S}}\leftarrow{\bm{S}}+{\bm{v}}_{t}{\bm{k}}_{t}^{\top}$ ( Kohonen, 1972 ; Katharopoulos et al., 2020 )	0.13 $\pm$ 0.00	0.66 $\pm$ 0.03
Delta rule (NLMS), ${\bm{S}}\leftarrow{\bm{S}}+({\bm{v}}_{t}-{\bm{S}}{\bm{k}}_{t}){\bm{k}}_{t}^{\top}/\lVert{\bm{k}}_{t}\rVert^{2}$ ( Widrow & Hoff, 1960 ; Schlag et al., 2021 )	0.13 $\pm$ 0.00	0.26 $\pm$ 0.01
Recursive least squares, Equation 7 ( Kohonen, 1974 ; von Oswald et al., 2024 )	0.29 $\pm$ 0.01	1.00 $\pm$ 0.00

Table 2: Test recall accuracy under various write rules of untrained and trained systems. Sequential recall at

L=D=256

, chance

0.10

, 3 seeds, mean

\pm 1\sigma

. Untrained systems cannot solve the task and even in trained systems, the write rule matters: the recursive least squares update is the only one that achieves perfect recall.

Figure 4: Contextual inference heatmap. Top: cosine similarity between the inferred context and the 8 possible contexts at each point in time. We observe the system resolve its uncertainty about the context as it sees more transitions. Bottom: given the inferred context, we measure recall accuracy for the episodes and note that it behaves nearly like the ideal observer upper bound.

DeltaNet	1.000	1,024	$\times$	$\times$	0.53 M c	65 k	0.7 GB	55 GB	OOM
	MQAR		Local in				Peak training memory at $L$
Model	Test acc.	Streams a	time	space	Params	State b	512	65k	131k
DeltaNet-PC (ours)	1.000	$32$	✓	✓	0.20 M	131 k	0.25 GB	0.25 GB	0.25 GB

Table 3: Comparison of DeltaNet ( Yang et al., 2024b ) and DeltaNet-PC at width

D=256

on MQAR (512-token streams, 8,192 vocabulary). a Distinct training streams needed to achieve perfect test recall (one seed per model). b Fast-weight floats: DeltaNet’s per-head states, DeltaNet-PC’s memory

{\bm{S}}

and RLS inverse

{\bm{P}}

. c Excludes token embedding parameters. Memory measurements are based on a batch of 32 on an 80 GB GPU. Test accuracy is over 3 seeds for DeltaNet-PC and 1 seed for DeltaNet.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Event stream	Test scoring
Sequential recall	Observations and queries alternate: after each observation $({\bm{x}}_{t-1},{\bm{x}}_{t},t)$ the learner poses its own query $({\bm{x}}_{t-1},{\bm{x}}_{t},t)$ , to retrieve what was just written at its own position, against the memory as it stands. $L$ observations and $L$ queries per stream; online learning.	Ground truth ${\bm{x}}_{t}$ compared to prediction $\hat{{\bm{x}}}_{t}$ decoded from ${\bm{S}}_{L}R({\bm{W}}_{k}\,\operatorname{enc}(\hat{{\bm{x}}}_{t-1}),t)$ for $t=1,\dots,L$ , the final memory frozen.
Official MQAR	The $128$ context observations (64 pairs), then the query section: at each labelled position $t$ the task’s query $({\bm{x}}_{t-1},{\bm{y}}_{t},t)$ , the query key and its value, precedes the observation that writes the filler ${\bm{x}}_{t}$ . Every position is written.	Ground truth ${\bm{y}}_{t}$ compared to prediction $\hat{{\bm{y}}}_{t}$ decoded from ${\bm{S}}_{t-1}R({\bm{W}}_{k}\,\operatorname{enc}({\bm{x}}_{t-1}),t)$ for each $t$ containing a query.
Omniglot binding	Stream of $L$ observations, then the task’s queries: all queries have the form $({\bm{x}}_{t-1},{\bm{x}}_{t},L)$ meaning they are queries at the final position $L$ .	Ground truth ${\bm{x}}_{t}$ compared to prediction $\hat{{\bm{x}}}_{t}$ decoded from ${\bm{S}}_{L}\,R({\bm{W}}_{k}\,\operatorname{enc}({\bm{x}}_{t-1}),L)$ .

Appendix

Table 4: The event streams for the three tasks.

Task	Observation	$d_{\text{obs}}$	$L$ , $D$	$V$	Train / Val. / Test	Scored units per test set
Single-episode sequences	token	256	$L=64$ to $512$ at $D=256$ , $L=512$ at $D=64$ to $512$ , $L=D=1{,}024$	10 tokens	$1{,}024$ / $128$ / $128$	$128L$ relay tokens
Multi-episode sequences	token	256	$256$ ( $K=8\times\ell=32$ ), $D=256$	10 tokens	$1{,}024$ / $128$ / $128$	$32{,}768$ relay tokens
MQAR ( Arora et al., 2024 )	token	256	$512$ , $D=256$	$8{,}192$ tokens ( $4{,}095$ keys, $4{,}096$ values)	$100{,}000$ (or $32$ ) / $128$ / $3{,}000$	$192{,}000$ labelled positions
Omniglot-MNIST binding	image $28\times 28$	784	$256$ ( $128$ pairs), $D=256$	10 digits	$1{,}024$ / $128$ / $128$	$16{,}384$ probes

Appendix

Table 5: Summary of datasets.

Figure 5: Multi-episode examples. A : A single training example. B : An evaluation example for the contextual inference task from Section 4.2 .

Figure 6: Omniglot-MNIST key-value memory example.

Figure 7: Learned rotary frequencies of the three rows that use them, three seeds each, against the base-

10^{4}

initialization (dashed). The dotted line marks one cycle per horizon:

2\pi/256

on the sequential and Omniglot rows and

2\pi/32

, one episode, on the multi-episode row. Sequential and multi-episode recall raise the slow bands, and Omniglot binding, which reads after every write, sends most of them to zero.

	Untrained		Trained (decoder)
	Clean-up	Decoder	$32$ streams	$100{,}000$ streams
RoPE off	$1.000$	$0.667$	$1.000$	$1.000$
RoPE on	$0.570$	$0.043$	$0.964$	$0.999$

Appendix

Table 6: MQAR test accuracy of DeltaNet-PC at

D=256

with RoPE off (the setting of Table 3 ) and on (as in every other task). Clean-up : each memory read is matched to the nearest encoded codebook row by cosine similarity, with no settle, as in Table 2 . Decoder : the protocol of Table 3 , with the

16

-step test-time settle. Untrained and full-data cells are means over

3

seeds, and the

32

-stream cells are one seed.

Symbol	Name	Value	Description
Task
$L$	Sequence length	256	Positions per sequence (swept in Figure 3 ).
$V$	Vocabulary size	10	Distinct tokens, each a fixed Gaussian code of dimension $d_{\text{obs}}$ . 8,192 for MQAR.
$d_{\text{obs}}$	Observation dimension	256	Codebook row dimension (images: pixel count).
$n_{\text{train}}$	Training sequences	1024	Unique training sequences.
$n_{\text{test}}$	Test sequences	128	Held-out sequences. The validation split is the same size.

Appendix

Table 7: DeltaNet-PC default hyperparameters.

	Sequential	Multi-episode	MQAR	Omniglot
Full recipe	$1$	$1$	$1$	$0.9999$
No activity clip	$1$	$1$	$1$	$0.9999$
No value orthogonality ( $\lambda_{v}=0$ )	$1$	$1$	$1$	$0.926$
NLMS write (no interference term)	$0.282$	$0.178$	$0.978$	$0.807$
Frozen RoPE	$0.359$	$0.689$	–	$0.940$

Appendix

Table 8: Ablations of several system components—each row removes one component from the full recipe. Test accuracy on sequential recall (

L=D=256

), multi-episode recall (

K=8\times 32

), MQAR (

D=256

) and Omniglot-MNIST binding with

128

keys. On Omniglot the full recipe makes

2

errors in

16{,}384

probes and the variant without the clip makes

1

. The official MQAR row runs with RoPE off, so freezing it does not apply there.

	DeltaNet-PC	DeltaNet
Training, operations	$O(LD^{2}+PTD^{2})$ , $3.2$ GFLOP	$O(L(D^{2}+DV))$ , $8.7$ GFLOP
Test, operations	$O(LD^{2}+PDV)$ , $0.75$ ( $4.0$ ) GFLOP	$O(L(D^{2}+DV))$ , $2.9$ GFLOP
Training, time	$18$ ms	$0.42$ ms
Test, time	$3$ ( $12$ – $20$ ) ms	$0.8$ ms
To exact recall	$1{,}024$ observed streams, $3.3$ TFLOP, $18$ s	$200{,}192$ observed streams, $1.7$ PFLOP, $88$ s

Appendix

Table 9: Compute per MQAR stream (

L=512

P=64

queries,

D=256

V=8{,}192

T=64

) and measured time on one H100. DeltaNet-PC’s test figures in parentheses include the

16

-step test-time settle, e.g., 0.75 for no test-time settle, and 4.0 with a 16-step test-time settle. Training times are per stream at batch

32

for DeltaNet-PC and

128

for DeltaNet. Streams to exact recall count every training stream the model observed, including repeats.

Figure 8: tPC’s intrinsic memorization ceiling. Test recall accuracy on the

N

training sequences as a function of

N

, with a unique

\mathrm{BOS}

token per training sequence.

L=8

D=256

Neural Network Memorization

Neural network memorization studies how models store and retrieve training examples, how this differs from generalization, and how memory can be measured or controlled.

See all 64 papers

DeltaTTT: Layerwise Optimization for Nonlinear Recurrent Memory
Oct 6, 2026Yining Li, Dongchen Han, Jie Fu +1Deep Learning Optimization Neural Network Optimization
Qiuzhen College, Tsinghua University · LeapLab, Tsinghua University · IQuest Research
Constant-Memory Recall: Learned Associations in a Fixed Matrix State
Sep 23, 2026Samuel LarsonAssociative Memory Memory-Efficient Inference
Pebble ML
Preconditioned DeltaNet: Curvature-aware Sequence Modeling for Linear Recurrences
Apr 22, 2026Neehal Tumma, Noel Loo, Daniela RusLong-Context Language Modeling Recurrent Neural Networks
1MIT · 2Liquid AI

Oct 6, 2026J/K move · Enter open · S save

DeltaTTT: Layerwise Optimization for Nonlinear Recurrent Memory

Yining Li, Dongchen Han, Jie Fu, Gao Huang

Qiuzhen College, Tsinghua University · LeapLab, Tsinghua University · IQuest Research

Deep Learning Optimization Neural Network Optimization Test-Time Training Language Modeling

Open paper AskPDF arXiv

Sequential test-time training adapts a memory network through successive updates, each computing an inner-loop gradient based on the network's previous state. Intuitively, this state dependence should allow each update to account for what the memory has already learned and better incorporate new information. However, we find that this expected advantage does not consistently materialize in nonlinear memories: a fixed-base parallel TTT baseline outperforms its serial counterpart. Our exploratory experiments point to a key underlying difficulty: nonlinear memories can be harder to optimize than linear ones within a single pass over the sequence. To alleviate this optimization difficulty, we introduce DeltaTTT, which replaces joint inner-loop optimization of a two-layer memory network with layerwise learning. Each layer is assigned a local prediction target and updated through a state-dependent delta rule. This formulation retains a nonlinear readout while enabling chunkwise parallel computation. Experiments on DeltaNet and LaCT backbones show improvements in language modeling and retrieval over their recurrent baselines.

Abstract

Neural Network Memorization

Explore similar work

DeltaTTT: Layerwise Optimization for Nonlinear Recurrent Memory

Constant-Memory Recall: Learned Associations in a Fixed Matrix State

Preconditioned DeltaNet: Curvature-aware Sequence Modeling for Linear Recurrences