Streaming Recurrent Memory

Mohamed Elsayed; Minh Pham; A. Rupam Mahmood

cs.LGOct 26, 2026

Streaming Recurrent Memory

Authors: Mohamed Elsayed, Minh Pham, A. Rupam Mahmood

Organizations: Department of Computing Science, University of Alberta · CIFAR Canada AI Chair, Amii

Abstract

Natural intelligence learns from a continuous stream of sensorimotor experience: data can arrive at high bandwidth, and behavior adapts to feedback as soon as it arrives, even when it is delayed. In contrast, most modern deep learning systems rely on stored datasets, replay buffers, and retained contexts through which credit is later assigned. At high data rates, storing and revisiting past experience can be impractical under tight memory and computational constraints. This paper studies recurrent memory under a streaming learning constraint: the learner learns from experience as soon as it arrives without storing raw experience. We ask whether a streaming learner can maintain a state that remains useful for future prediction and control from delayed feedback. We introduce Streaming Recurrent Memory (SRM), a recurrent memory designed for the streaming setting which preserves information that may matter later while remaining trainable with one-step backpropagation through time (BPTT). SRM provides a streaming-compatible recurrence that is designed to recall past information by content or time, track elapsed steps, and compose transformations whose outcome depends on the order in which inputs are observed. Across controlled memory tasks and streaming applications, SRM learns effectively with one-step BPTT, outperforming recurrent baselines in associative recall and online language modeling while achieving near-perfect success in associative T-Maze with delayed rewards. These results suggest that fixing how inputs are stored in the recurrent state, while learning how that state is read out, enables effective streaming learning without long-range backpropagation.

Figures & tables

Task	Target at query time
Presence	Whether a queried symbol appeared in the sequence.
Successor	The symbol that appeared right after the queried symbol.
Predecessor	The symbol that appeared right before the queried symbol.
Count	The number of occurrences of the queried symbol.
Order	Whether the first queried symbol appeared before the second in the presentation.
Position	The symbol presented at a queried position.

Table 1: The task suite. Each task tests how well a unit encodes and retains one kind of information. Each sequence consists of 5 to 8 task-dependent input symbols, followed by a sequence of waiting symbols. At the end of the sequence, the learner is prompted to give an answer, at which point feedback is provided.

Figure 2: Accuracy of ridge-probe over recurrent states unfolded to delay checkpoints, without training. Lines connect points from each of the 10 random initializations. Dotted lines indicate chance-level accuracy.

Figure 3: (Left) Alignment score ( $-1$ to $1$ ; higher is better) throughout readout training, for all of $10$ SRM initializations. (Right) Squared negative alignment ( $0$ to $1$ ; lower is better), averaged over all steps and runs.

Figure 4: Training accuracy on the task suite under one-step BPTT (solid lines) and RTRL (dashed lines), averaged over

10

runs. Shading shows standard deviation for one-step BPTT. Dotted lines indicate chance.

	MQAR accuracy (%) $\uparrow$						TinyStories loss $\downarrow$
	Sequence length 64			Sequence length 128			Bytes seen
Unit	Small	Medium	Large	Small	Medium	Large	$10$ M	$20$ M
GLA	11.1	17.6	23.3	1.3	8.3	11.3	$1.193\pm 0.029$	$1.192\pm 0.048$
KDA	5.6	20.1	92.4	2.7	5.2	62.3	$1.076\pm 0.006$	$0.994\pm 0.009$
RTU	0.8	0.8	14.0	0.9	0.7	2.3	$0.942\pm 0.002$	$0.892\pm 0.002$
eLSTM	7.3	13.8	22.6	2.3	1.8	9.6	$1.307\pm 0.017$	$1.130\pm 0.015$

Table 2: MQAR: accuracy over the final

10\%

of training sequences at three FLOPs-matched model dimensions (Table 5 ), averaged over

10

runs. TinyStories: online next-byte loss averaged over the

1

M bytes preceding the

10

M- and

20

M-byte marks, reported as mean

\pm

standard deviation over three independent runs; all language models have around

13

M parameters. Best value per column is shown in bold.

Figure 5: Associative T-Maze with

K

key–value pairs. Left four panels: online success rate during training averaged over

10

runs with shaded area representing one standard error. Dotted line is chance. Rightmost panel shows online success over the final

10\%

of episodes (dots are individual seeds).

Figure 6: (Left) Ablation of SRM’s components: tracking channel, clock, and previous-input projections. Results show the average and standard deviation of 10 runs. (Right) SRM’s recall accuracy degrades when the write weights (

{\bm{K}}

and

{\bm{V}}

) and their bias terms are made learnable, across various input alphabet sizes.

Figure 7: Merging the tracking channel and the memory, with SRM at initialization. (a) Fraction of a stored key left at its written position after number of inputs. (b) Key–value retrieval accuracy; the dotted line is chance. (c) Linear-probe accuracy for the ball position in a five-cup shell game.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Unit	Input width	State width	Read heads	tracking dimension
SRM	32	32	2	8
GLA	32	42	2	–
KDA	32	20	2	–
RTU	32	60	–	–
eLSTM	32	45	–	–

Appendix

Table 3: Size of the recurrent unit of SRM and baselines used throughout the diagnostics experiments. SRM uses 8 rotors for the clock and 8 tracking dimensions of the tracking channel.

Task	$\bm{\|\Sigma\|}$	Chance	Examples
			Sequence	Target
Presence	16	$0.594$	$\langle 15,9,4,0,7,\bullet,8\rangle$ $\langle 10,4,8,5,15,\bullet,4\rangle$	$0$ $1$
Successor	8	$0.190$	$\langle 7,1,2,6,0,\bullet,1\rangle$ $\langle 5,1,2,7,6,3,4,0,\bullet,2\rangle$	$2$ $7$
Predecessor	8	$0.190$	$\langle 7,1,2,6,0,\bullet,2\rangle$ $\langle 5,1,2,7,6,3,4,0,\bullet,7\rangle$	$1$ $2$
Count	4	$0.250$	$\langle 0,1,0,1,3,\bullet,3\rangle$ $\langle 3,0,2,1,0,1,0,\bullet,0\rangle$	$1$ $3$
Order	8	$0.500$	$\langle 7,1,2,6,0,\bullet,2,0\rangle$ $\langle 7,3,4,2,1,\bullet,3,1\rangle$	$1$ $1$

Appendix

Table 4: The synthetic task suite.

|\Sigma|

denotes the presentation alphabet size. Chance is the expected accuracy of a task-appropriate guesser, with variable presentation length accounted for.

Figure 8: (Top) Alignment score for parameters of SRM over the training horizon of the readout, across the task suite, with all of non-readout parameters fixed throughout this training. Results are smoothed with a 5000-point centered rolling average. (Bottom) Squared negative alignment, averaged over training steps and runs. This value ranges from 0 to 1, with higher values indicating that local updates oppose the gradient contributions from earlier steps.

	Model dimension
Model	Small	Medium	Large
SRM	32	64	128
GLA	40	76	156
KDA	36	72	144
RTU	42	90	184
eLSTM	37	77	157

Appendix

Table 5: FLOPs-matched model dimensions for SRM and baselines, for MQAR.

Figure 9: (a) MQAR: Average accuracy across three model dimensions, with dashed and solid lines denoting the shorter and longer task settings. (b) Online next-byte loss on TinyStories, over

100

M training bytes. Each line is one of three independent runs. All models use networks that have around

13

M parameters.

Setting	Value
Discount $\gamma$ , trace decay $\lambda_{\text{tr}}$	$0.99$ , $0.8$
Step sizes (critic head $\alpha$ , recurrent networks $\alpha_{\text{rec}}$ )	$(10^{-2},10^{-3})$ or $(10^{-3},10^{-2})$
Actor head step size	$\alpha/2$
Entropy coefficient	$0.1$
Training episodes	$20{,}000$ ( $30{,}000$ for $K=8$ )

Appendix

Table 6: Associative T-Maze settings. The step sizes and the entropy coefficient are selected per unit and per

K

Unit	Model width $M$	State width $S$
RTU	$1024$	$1024$
eLSTM	$1024$	$512$
GLA	$768$	$512$
KDA	$768$	$512$
SRM	$768$	$512$

Appendix

Table 7: Widths of the four-layer streaming language models. All models have approximately the same number of parameters around

13

Panel of Figure 10 — Figure 10: Architecture used throughout the paper. RN denotes RMSNorm. $M$ is the model dimension and $S$ is the state dimension. $\sigma$ denotes a sigmoid nonlinearity.

Figure 11: Online next-byte loss of RTU trained with one-step BPTT under RMSprop with different

\beta_{2}

and under StreamingOptimizer (equivalent to Adamax with

\beta_{1}=0

), over the first

30

M bytes.

Figure 12: Next-byte loss of SRM and RTU across the scaling sweep, against (a) the number of parameters and (b) the training FLOPs. Error bars show the standard error of the mean. The configurations are listed in Tables 8 and 9 .

Params	Layers	Width $M$	Rows $S$	Memory $S\times M/4$	State / layer	Total state
$2.8$ M	$3$	$384$	$256$	$256\times 96$	$25{,}040$	$75$ K
$6.8$ M	$4$	$512$	$384$	$384\times 128$	$49{,}744$	$199$ K
$13.6$ M	$4$	$768$	$512$	$512\times 192$	$99{,}152$	$397$ K
$36.0$ M	$6$	$1024$	$704$	$704\times 256$	$181{,}328$	$1.09$ M
$71.5$ M	$8$	$1280$	$832$	$832\times 320$	$267{,}600$	$2.14$ M

Appendix

Table 8: SRM configurations of the scaling sweep.

M

is the model width and

S

is the number of memory rows.

Params	Layers	Width $M$	State width $S$	State / layer	Total state
$2.6$ M	$3$	$512$	$512$	$512$	$1.5$ K
$6.9$ M	$4$	$736$	$736$	$736$	$2.9$ K
$13.1$ M	$4$	$1024$	$1024$	$1{,}024$	$4.1$ K
$36.4$ M	$6$	$1408$	$1408$	$1{,}408$	$8.4$ K
$72.6$ M	$8$	$1728$	$1728$	$1{,}728$	$13.8$ K

Appendix

Table 9: RTU configurations of the scaling sweep. The state of width

S

is held as

S/2

complex units.

Model	one-step BPTT	RTRL	Ratio
Task Suite Model	113,796	201,543	$1.771\times$
MQAR small	629,715	720,729	$1.145\times$
MQAR medium	1,965,923	2,656,873	$1.351\times$
MQAR large	6,767,235	12,150,921	$1.796\times$
Language	259,341,436	1,773,230,984	$6.837\times$

Appendix

Table 10: FLOPs per timestep for SRM training with one-step BPTT and RTRL. Costs include forward computation, backward computation, and one optimizer update.

Figure 13: RTRL gradients alignment against exact full BPTT gradients throughout training.

Recurrent Neural Networks

Recurrent neural networks process sequences by updating a persistent hidden state, enabling temporal modeling, memory, and streaming computation.

See all 307 papers

Streaming Reinforcement Learning under Partial Observability with Real-Time Recurrent Learning
May 23, 2026Noah Farr, Aryaman Reddi, Carlo D'Eramo +1Reinforcement Learning Partially Observable RL
Technical University of Darmstadt · Zuse School ELIZA · Center for Artificial Intelligence and Data Science, University of Würzburg +1
SRMU: Relevance-Gated Updates for Streaming Hyperdimensional Memories
Apr 16, 2026Shay Snyder, Andrew Capodieci, David Gorsich +1Hyperdimensional Computing Vector Symbolic Architectures
George Mason University · George Mason University, USA · Neya Robotics +3
Streaming Deep Reinforcement Learning Finally Works
Oct 18, 2024Mohamed Elsayed, Elena Sorina Lupu, Gautham Vasan +1Reinforcement Learning Actor-Critic Methods
Department of Computing Science, University of Alberta, Edmonton, Canada · Alberta Machine Intelligence Institute (Amii) · Canada CIFAR AI Chair

May 23, 2026J/K move · Enter open · S save

Streaming Reinforcement Learning under Partial Observability with Real-Time Recurrent Learning

Noah Farr, Aryaman Reddi, Carlo D'Eramo, Jan Peters

Technical University of Darmstadt · Zuse School ELIZA · Center for Artificial Intelligence and Data Science, University of Würzburg+1

Reinforcement Learning Partially Observable RL Recurrent Neural Networks Streaming Algorithms

Open paper AskPDF arXiv

Streaming reinforcement learning has emerged as an online learning paradigm that conforms to the restrictions of natural learning agents that process data incrementally, i.e. with a batch size of 1 and no replay buffer. While streaming RL has recently been shown to scale with deep function approximation with full observability, partially observable settings have remained out of reach. Truncated backpropagation through time collapses to a one-step gradient horizon under the streaming setting, and exact real-time recurrent learning is prohibitively expensive. We close this gap using recurrent trace units, a diagonal recurrent architecture that enables exact RTRL with linear time and memory complexity in the parameter count, and show that they integrate cleanly into existing streaming algorithms across both discrete and continuous control. On a MemoryChain diagnostic with chain lengths from 2 to 128, our method sustains performance where streaming TBPTT(1) baselines using feedforward, GRU, and RTU networks collapse. On five POPGym tasks and on partially observable MuJoCo continuous control, the streaming approach is competitive with batched PPO on POPGym and recovers a substantial fraction of batched performance on masked MuJoCo, despite using no replay buffer or batched updates.

Abstract

Recurrent Neural Networks

Explore similar work

Streaming Reinforcement Learning under Partial Observability with Real-Time Recurrent Learning

SRMU: Relevance-Gated Updates for Streaming Hyperdimensional Memories

Streaming Deep Reinforcement Learning Finally Works