Immediate Derivatives Suffice for Online Recurrent Adaptation

Aur Shalev Merin

cs.LGMar 30, 2026

Immediate Derivatives Suffice for Online Recurrent Adaptation

Authors: Aur Shalev Merin

Organizations: Independent Researcher

Abstract

For three decades online recurrent learning has been assumed to require propagating a Jacobian tensor through the network's dynamics at $O(n^4)$ per step. We show it doesn't. Dropping the propagation entirely ( $d=0$ , $O(n^2)$ memory) matches full RTRL within CI on held-out BCI cross-session drift (TOST equivalent within $\pm 3$ pp at $n=20$ , Adam, float64), and across vanilla-RNN synthetic cells (sine and Lorenz under Adam and SGD) and LSTM/sine under Adam. A decomposition $g_{RTRL} = g_{imm} + g_{past}$ explains why. On BCI, $g_{past}$ concentrates in a single direction (top-1 singular fraction 0.62-0.74 across four optimizers, vs 0.333 for $g_{imm}$ ), and the four-optimizer full-RTRL-vs- $d=0$ recovery gap tracks each optimizer's per-layer update-magnitude ratio $\|ΔW_{hh}\|/\|ΔW_{out}\|$ monotonically. A stationary (no-drift) control collapses both concentrations to ~0.6: the drift-specific signal is the differential, not $g_{past}$ 's absolute rank-1 structure. The signature and the behavioral gap both collapse on LSTM, consistent with a mechanism specific to additive linear recurrence. On synthetic sine, $g_{imm}$ is redundant with $g_{past}$ , which predicts the synthetic null. Full RTRL's one robust advantage is LARS (+17 to +27 pp), but $d=0$ +LARS also fails to adapt independently; the gap is an optimizer $\times$ method interaction, not a method-quality claim. We characterize the regime: $d=0$ +Adam+float64 is robust; SGD, Adafactor, and float32 have specific fragilities documented in the paper. On the evaluated cells, the $1000\times$ memory saving at $n=1024$ ( $O(n^2)$ vs $O(n^4)$ ) comes with no measured recovery cost.

Figures & tables

Arch.	Task	$\cos(g_{\text{imm}},g_{\text{past}})$	CI contains 0?
Vanilla RNN	Sine/Lorenz, $n\in\{64,128\}$	$\cos\in[-0.021,-0.003]$	marginal (4 cells)
LSTM	Sine ( $n{=}64$ )	$\mathbf{-0.104}\;[-0.190,-0.027]$	no
LSTM	Lorenz ( $n{=}64$ )	$-0.004\;[-0.046,+0.036]$	yes
RetNet	Sine ( $n{=}64$ )	$\mathbf{-0.102}\;[-0.147,-0.057]$	no
RetNet	Lorenz ( $n{=}64$ )	$+0.050\;[-0.015,+0.115]$	yes
BCI †	Cross-session	$\mathbf{-0.119}\;[-0.230,-0.014]$	no

Table 1:

\cos(g_{\text{imm}},g_{\text{past}})

during online adaptation (mean across 5 seeds with 95% bootstrap CI, averaged over the post-shift window; vanilla RNN uses exact full-RTRL sensitivity tensor, LSTM/RetNet/BCI use TBPTT

W{=}200

|\cos|\leq 0.25

across all 9 cells, sign-stable across hardware, magnitude hardware-dependent on BCI (0.12–0.24 across the two hardware configurations we measured). On vanilla RNN,

\cos

is near-zero at both scales. On LSTM/sine and RetNet/sine,

|\cos|\approx 0.10

with CIs excluding zero (bold): small but statistically non-zero. The TBPTT approximation is validated against exact RTRL on vanilla RNN sine (

-0.021

in both). † BCI uses TBPTT

W{=}200

on a FullRankRNN (

H{=}64

, input dim

=40

Condition	$\text{top1}(G_{\text{past}})$	$\text{top1}(G_{\text{imm}})$	gap	$W_{hh}$ share of $u_{1}$
SGD, drift	$0.743$	$0.333$	$+0.41$	$60.0\%$
Adam, drift	$0.708$	—	—	$58.2\%$
LARS, drift	$0.623$	—	—	$59.4\%$
Adafactor, drift	$0.657$	—	—	$53.0\%$
SGD, stationary (control)	$0.586$	$0.600$	$-0.01$	$74.2\%$

Table 2: Structure of

g_{\text{past}}

during BCI cross-session adaptation, f64, local AVX-2, per-optimizer f64 fr-best LR, TBPTT window

w{=}200

. Top four rows: cross-session drift (session B adapt,

n{=}5

seeds, 41 windows).

\text{top1}(G_{\text{imm}})

measured for SGD only at matched conditions. Bottom row: stationary control (SGD fr-logged on the tail of session A, pretrained weights, no distribution shift;

n{=}5

seeds, 25 windows). The drift-induced differential

\text{top1}(G_{\text{past}})-\text{top1}(G_{\text{imm}})

+0.41

on drift and

-0.01

on stationary;

g_{\text{past}}

’s absolute rank-1 concentration is partly a cumulative-trace SVD geometry (present at

\sim 0.6

stationary), but

g_{\text{imm}}

’s diffuseness emerges only under drift.

W_{hh}

share compared against the parameter-count null of

|W_{hh}|/d=4096/6850\approx 59.8\%

;

n{=}5

drift values are at null,

n{=}10

SGD hardening gives

74.1\%

above null.

Optimizer	$\\|\Delta W_{hh}\\|/\\|\Delta W_{out}\\|$	fr $-$ $d{=}0$ gap (pp)
SGD	$4.30$	$-10$
Adafactor	$4.82$	$-7$
Adam	$6.20$	$\approx 0$
LARS	$10.49$	$+20$

Table 3: Per-layer update-magnitude ratio

\|\Delta W_{hh}\|/\|\Delta W_{out}\|

and measured fr-vs-

d{=}0

held-out recovery gap on BCI f64 cross-session drift (80/20 split convention from our cross-hardware held-out sweeps; the 50/50 convention gives Adam within

0.3

pp of

0

(Section 8 reproduction) and preserves the LARS direction). Higher ratio

\rightarrow

larger fr advantage. Interpretation: SGD’s step per layer scales with the raw per-layer gradient (ratio

\sim 4

); LARS’s per-layer normalization divides by

\|W_{\text{layer}}\|

, and because pretrained

\|W_{hh}\|\gg\|W_{out}\|

the effective ratio inflates to

\sim 10

; Adam’s per-coordinate

v

-buffer produces an intermediate ratio; Adafactor’s factored preconditioner is close to SGD.

n{=}4

is too few for an out-of-sample prediction test; the ordering is consistent with the mechanism but we report it as a measured correspondence, not a prediction.

Figure 1: (a) Memory: at

n{=}1024

d{=}0

uses

12.6

MB vs full RTRL’s

12.9

GB (

1000\times

). (b) BCI held-out gap (full RTRL

-

d{=}0

, pp) at

80/20

, float32: three (hardware, kernel) configurations per optimizer at fixed Kaggle-tuned LR. Adam sign-reverses across configs; LARS spans

43

pp; method comparisons unreliable at f32. (c) Same at float64: range bars = 3 configs at Kaggle-tuned LR (transported across hardware); diamonds = per-hardware-best LR for both methods (open Kaggle, filled local). Adam equivalent within seed noise; LARS fr-wins

+16

+25

pp (

d{=}0{+}

LARS independently fails); Adafactor hardware-inconsistent (

\approx 8

pp sign-flip). At

50/50

, Adam is equivalent to within

0.3

pp (reproduced to

0.04

pp).

Figure 2: BCI held-out gap (full RTRL

-

d{=}0

, pp) vs adapt-horizon, four optimizers,

80/20

, local f64 AVX-2, 5 seeds. LARS retains a

+17

+27

pp fr advantage at every horizon (CIs exclude zero). SGD, Adafactor, Adam gaps are within

\pm 10

pp of zero with overlapping CIs. Seed 42’s fr+Adam is a consistent outlier (

19

–

31\%

recovery vs

40

–

50\%

), widening Adam fr CIs but not changing the zero-overlap verdict. Presented as adapt-horizon-invariance of the LARS advantage, not a prediction test.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Domain	$d{=}0$ best optim.	$d{=}0$ Adam (default)	$k{=}4$ RTRL / UORO §	Full RTRL
Sine ( $n{=}64$ )	SGD lr=0.1: 130%	102	125	100
Delayed ( $t$ +50)	RMSprop: 179	147	130	100
Lorenz (chaotic)	Adafactor: 125	49	–	100
BCI, Adam (f64, 50/50)	$d{=}0$ +Adam: $37.6$	$37.6$	–	$37.3$ (gap $-0.3$ pp)
BCI, Adam (f64, 80/20)	$d{=}0$ +Adam: $41.7$	$41.7$	UORO: $14.7$	$50.3$ (fr $+8.6$ pp, CI $\ni 0$ )
BCI, Adam (f64, last-layer) §	Adam: $22.0$ ; SGD: $22.1$	$22.0$	–	– (floor; cf. $41.7$ full-rec.)

Appendix

Table 4: Head-to-head comparison: immediate derivatives (

d{=}0

) vs. Jacobian-based methods across domains. All values are held-out or adapt-window recovery (%) unless noted; “–” indicates not tested. Synthetic rows report adapt-window recovery under per-optimizer LR tuning. BCI rows use the held-out protocol of Appendix C with per-cell best LR on disjoint validation; split convention in parentheses. BCI numbers at float32 are hardware-fragile (see Appendix D ); we report float64 numbers here as the numerically stable estimate .

\dagger

Language row reports cross-entropy (lower is better); no held-out validation was run.

\ddagger

Full RTRL cannot run at

n{=}1024

; the high recovery reflects an unreliable denominator.

\lx@sectionsign

UORO: rank-1 Jacobian approximation of Tallec and Ollivier (2017) .

\mathsection

Last-layer-only:

W_{ih}

W_{hh}

b_{h}

frozen; only

W_{out}

and

b_{out}

adapt, quantifying the readout-remap floor for comparison against

d{=}0

’s full-recurrent adaptation.

Optimizer	Sine ( $n{=}64$ )	BCI adapt window	Lorenz $\rho{=}28{\to}35$
SGD	130% (lr=0.1)	160 $\pm$ 40% (OVERFIT)	124 $\pm$ 26%
Adam	102% (lr= $10^{-3}$ )	61 $\pm$ 16% (generalizes)	49 $\pm$ 27%
signSGD	50%	61 $\pm$ 13%	95 $\pm$ 42%
signSGD + momentum	30%	43 $\pm$ 5%	54 $\pm$ 15%
LARS	50%	123 $\pm$ 23% (OVERFIT)	123 $\pm$ 24%
Gradient normalization	40%	32 $\pm$ 6%	50 $\pm$ 11%

Appendix

Table 5: Per-optimizer recovery (%) with

d{=}0

, per-optimizer LR tuning, 5 seeds, evaluated on the adaptation window (not held out). Note that adapt-window recovery on non-stationary tasks is inflated by overfitting to the adaptation stream; see the held-out validation in Appendix C . On stationary sine, all optimizers adapt. The BCI adapt-window column is measured at float32 on Kaggle CPU; given the float32 hardware fragility documented in Appendix D , treat absolute BCI adapt-window numbers here as environment-specific measurements rather than method-level properties. The OVERFIT markers reflect the held-out result (Appendix C ); the Adam generalization claim is robust across configurations (Appendix D ).

Method	Optimizer	Best LR	Held-out rec. (%, CI)	Verdict
$d{=}0$	Adam	$3{\times}10^{-4}$	$\mathbf{44.9}\;[41.6,48.5]$	generalizes
$d{=}0$	Adafactor	$10^{-3}$	$25.9\;[20.6,32.6]$	partial
$d{=}0$	SGD	$3{\times}10^{-3}$	$42.2\;[38.5,46.3]$	generalizes
$d{=}0$	LARS	$10^{-2}$	$11.3\;[3.7,21.2]$	weak
Full RTRL	Adam	$10^{-3}$	$\mathbf{52.1}\;[48.7,55.4]$	generalizes
Full RTRL	Adafactor	$10^{-3}$	$49.7\;[47.8,52.5]$	generalizes

Appendix

Table 6: BCI held-out validation with per-cell LR tuning (5 seeds, per-cell best LR from a grid sweep, selected to maximize mean held-out recovery). Top block:

d{=}0

(immediate derivatives). Bottom block: full RTRL (TBPTT

w{=}50

). Each method

\times

optimizer cell receives its own best LR from the same grid. The

d{=}0

vs full-RTRL comparison within each optimizer is therefore fair (same grid, best-LR-per-method).

Optimizer	default	default	AVX-2	Gap range (f32)
	Kaggle f32	Local f32	Local f32
Adam	$+7.14$	$-9.74$	$-14.35$	$-14$ to $+7$ (sign-flip)
Adafactor	$+23.79$	$+1.77$	$+10.45$	$+1$ to $+24$
LARS	$+44.20$	$+31.02$	$+0.78$	$+1$ to $+44$
	Kaggle f64	Kaggle f64	Local f64
	default	AVX-2	AVX-2	Gap range (f64)

Appendix

Table 7: BCI held-out recovery gap (full-RTRL

-

d{=}0

, pp) at Kaggle-tuned LR transported across configurations. Positive favours full RTRL. Kaggle f32 is the initial Kaggle-f32 protocol. Cross-hardware comparisons at this fixed LR conflate method differences with LR-portability effects; see Table 8 for gaps at per-hardware best LR.

Optimizer	Kaggle f32 gap	Local f32 gap	Kaggle f64 gap	Local f64 gap	Verdict at f64
Adam	$+7.1$	$-3.9$	$-6.05$ / $-2.47$	$-0.34$	equivalent
SGD	$-11.27$	$-7.25$	–	$-9.93$	$d{=}0$ wins $7$ – $11$ pp
Adafactor	$+23.8$ ∗	$+5.3$	$+1.37$	$-6.67$	hardware-inconsistent
LARS	$+44.2$	$+23.2$	$+15.85$ / $+22.87$	$+25.09$	full RTRL wins $+16$ to $+25$ pp

Appendix

Table 8: BCI held-out gap (full-RTRL

-

d{=}0

) at true per-hardware best LR for both methods, both hardwares . All f64 cells use LRs selected from tighter-than-initial grids on each hardware. The Kaggle Adafactor row resolves an initial-grid-coarseness issue (the grid missed the true

d{=}0

Adafactor optimum at

3\times 10^{-4}

on Kaggle). Compare with Table 7 , which uses fixed Kaggle-tuned LRs transported across hardware.

Optimizer	$\lambda{=}0.0$	$\lambda{=}0.1$	$\lambda{=}0.5$	$\lambda{=}0.95$
Adam	$100.0\%$	$99.3\%$	$100.2\%$	$90.2\%$
Adafactor	$100.9\%$	$100.6\%$	$98.3\%$	$100.9\%$
RMSprop	$100.8\%$	$100.5\%$	$98.2\%$	$98.3\%$
SGD	$104.1\%$	$103.9\%$	$102.2\%$	$103.4\%$

Appendix

Table 9: Recovery (%) at best per-cell LR. Sine

n{=}64

, 5 seeds, no clipping. Reference: Adam

d{=}0

at lr

=10^{-3}

(mse

=0.00068

). Every

(\lambda,\text{optimizer})

cell achieves

\geq 90\%

recovery: decay value is empirically compensable by LR tuning, consistent with the small-

|\cos|

finding of Section 3 .

Figure 3: Recovery vs. trace decay on sine frequency-shift (Adam at lr

=10^{-3}

, 5 seeds, mean

\pm

1 std). At the standard Adam learning rate, the conventional default

\lambda{=}0.95

produces 0% recovery; values below 0.5 form a safe plateau. Per-decay LR tuning rescues high-decay configurations (Table 9 ), but

\lambda{=}0

is the simpler choice: it removes a hyperparameter and works at every standard learning rate we tested.

Architecture	Recovery ( $d{=}0$ + Adam)	Notes
Vanilla RNN	102%	reference
LSTM	99.7%	gated cell state
CTRNN	89.8%	continuous-time
xLSTM (sLSTM)	91.5%	exponential gating
RWKV	adapts	unstable on chaotic dynamics with Adam (App. R )
RetNet	93%	multiplicative key-value

Appendix

Table 10: Cross-architecture

d{=}0

recovery on sine frequency-shift (5 seeds, BPTT

w{=}1

gradients, Adam at lr

=10^{-3}

). Six architectures have numerical recovery; four are reported as “adapts” (post-shift MSE below frozen; full-RTRL reference impractical at scale or the output-bypass structure makes the ratio uninformative).

Figure 4: Cross-architecture structural measurements (7 architectures, 5 seeds each; RWKV excluded at initialization). (a) Gradient norm ratio between output and recurrent parameters. (b) Condition number of the recurrent parameter Jacobian

\partial h/\partial\theta_{\text{rec}}

, measured after training. Five of seven architectures are near-isotropic (

\text{cond}<10

, blue band). RetNet is an exception (cond

\sim 10^{4}

): its key-value outer product concentrates sensitivity in few directions.

Depth	Task	Best trace (d, lr)	MSE	Best RTRL MSE	Wins?
1	Sine	$d{=}0.0$ , $10^{-3}$	0.0007	0.0006	$\approx$ †
2	Sine	$d{=}0.0$ , $3{\times}10^{-4}$	0.000129	0.000142	Yes
3	Sine	$d{=}0.1$ , $10^{-4}$	0.000178	0.000218	Yes
3	Lorenz	$d{=}0.0$ , $10^{-3}$	0.000352	0.000573	Yes

Appendix

Table 11: Multi-layer results: best trace configuration vs. best full RTRL at each depth (5 seeds, post-shift MSE). † Within seed variance.

cond $(W_{hh})$	$d{=}0$ MSE	full-RTRL MSE	within CI?
$5$	$\sim 0.0006$	$\sim 0.0006$	yes
$50$	$\sim 0.0006$	$\sim 0.0006$	yes
$500$	$\sim 0.0006$	$\sim 0.0006$	yes
$5000$	$\sim 0.0006$	$\sim 0.0006$	yes

Appendix

Table 12: Post-shift MSE (mean

\pm

seed std) on sine frequency-shift under imposed

\operatorname{cond}(W_{hh})

, Adam pretrained, per-cell best LR,

n{=}5

seeds. The

d{=}0

-vs-full-RTRL post-shift MSE is within CI at every imposed condition level, but because training returns

\operatorname{cond}(\partial h/\partial\theta)

[2,7]

regardless of imposed

\operatorname{cond}(W_{hh})

(paragraph above), this null result does not refute the isotropy hypothesis, since the intervention does not vary the quantity the hypothesis is about.

$\rho$	Vanilla RNN cos	RetNet cos
$0.00$	$-0.001$ $[-0.013,+0.012]$	$-0.000$ $[-0.003,+0.003]$
$0.30$	$-0.004$ $[-0.015,+0.005]$	$+0.002$ $[-0.009,+0.015]$
$0.60$	$-0.010$ $[-0.032,+0.005]$	$+0.019$ $[-0.009,+0.049]$
$0.90$	$+0.021$ $[+0.012,+0.030]$	$+0.002$ $[-0.075,+0.071]$
$0.99$	$+0.009$ $[-0.002,+0.024]$	$-0.125$ $[-0.243,-0.008]$

Appendix

Table 13:

\cos(g_{\text{imm}},g_{\text{past}})

across input autocorrelation

\rho

10

seeds, bootstrap

95\%

CI. RetNet cos remains small across

\rho\in[0,0.9]

; at

\rho{=}0.99

the sign reverses (wrong direction under the hypothesis). The decorrelation hypothesis is falsified.

Opt.	LR	Recovery (%, CI)	top1 $(G_{\text{past}})$	top1 $(G_{\text{imm}})$	gap
SGD	$3{\times}10^{-4}$	$+45.7\;[+41.2,+50.9]$	$0.361$	$0.348$	$+0.013$
Adam	$3{\times}10^{-3}$	$+63.8\;[+62.7,+64.9]$	$0.299$	$0.264$	$+0.035$
LARS	$1{\times}10^{-2}$	$+52.9\;[+48.8,+56.0]$	$0.403$	$0.345$	$+0.058$
Adafactor	$3{\times}10^{-3}$	$+66.4\;[+63.3,+69.0]$	$0.364$	$0.312$	$+0.052$

Appendix

Table 14: LSTM

H{=}64

BCI cross-session, f64 AVX-2, 5 seeds per optimizer at per-optimizer LR-swept best. All four cells fail the pre-registered primary kill (top-1

\geq 0.50

and top-1 gap vs.

G_{\text{imm}}\geq 0.20

). Compare RNN equivalents of

0.62

–

0.74

and

+0.41

. The rank-1 signature is specific to vanilla RNN’s additive linear recurrence.

config	h $\to$ h params	fr LR	fr rec	$d{=}0$ rec	gap (pp, 95% CI)	abs MSE benefit
RNN $H{=}64$	4096	$3{\times}10^{-3}$	$+35.59$	$+17.83$	$+17.76$ [ $+14.00,+21.47$ ]	$+0.119$
LSTM $H{=}32$	4096	$3{\times}10^{-3}$	$+57.45$	$+23.05$	$+34.40$ [ $+32.47,+36.37$ ]	$+0.255$
LSTM $H{=}64$	16384	$3{\times}10^{-3}$	$+65.31$	$+28.52$	$+36.79$ [ $+34.09,+39.15$ ]	$+0.286$

Appendix

Table 15: signSGD fr-vs-

d{=}0

gap at matched hidden-to-hidden recurrent capacity. LSTM

H{=}32

has

4\cdot 32^{2}=4096

\to

h parameters, identical to RNN

H{=}64

’s

W_{hh}

. The gap at LSTM

H{=}32

is within CI of the gap at LSTM

H{=}64

, and both are

\approx 2\times

RNN.

5

seeds,

95\%

bootstrap CIs, BCI f64, per-method fr/

d{=}0

best LR.

Cross-arch ratio (RNN)/(LSTM): $4.19$ (signSGD) vs $4.20$ (SGD) — matches to two decimal places
metric	RNN / signSGD	RNN / SGD	LSTM / signSGD	LSTM / SGD
$\|g_{\text{past}}\|/\|g_{\text{imm}}\|$ median	$20.6$ [ $11.5,31.5$ ]	$14.4$ [ $6.1,22.8$ ]	$4.93$ [ $4.59,5.28$ ]	$3.44$ [ $3.18,3.64$ ]
past-wins-magnitude fraction	$1.000$	$0.981$	$0.940$	$0.885$
$\sigma(\mathrm{sign}\,g_{\text{past}})$	$0.134$	$0.258$	$0.152$	$0.334$
$\mathrm{sign}(g_{\text{tot}})\neq\mathrm{sign}(g_{\text{imm}})$ rate	$0.376$	$0.351$	$0.326$	$0.290$

Appendix

Table 16: Per-coordinate structural measurements on recurrent parameters (RNN

W_{hh}

and LSTM

H{=}64

lstm.W ), BCI cross-session, f64,

41

TBPTT windows at

W{=}200

5

seeds,

95\%

bootstrap CIs. Columns under signSGD and under SGD use each optimizer’s fr-best LR. Cross-architecture ratio of

|g_{\text{past}}|/|g_{\text{imm}}|

preserved to two decimal places across optimizers (

4.19

4.20

Model	Params	Recovery ( $d{=}0$ + Adam)
Mamba-1.4B	1.4B	$\mathbf{133.5\%}$
Mamba-130M	130M	$51.6\%$
TinyLlama	1.1B	$31.6\%$
Qwen2	7B	$31.3\pm 7.3\%$
GPT-2	124M	$15.0\pm 0.9\%$

Appendix

Table 17: Online LoRA adaptation across pretrained LMs with

d{=}0

+ Adam at lr

=10^{-3}

(rank-4 adapters, code-to-Wikipedia shift, single-pass over 20K tokens with shift at 10K, no replay). Adapt-window recovery only; no held-out validation was run. Subject to the stream-fitting confound documented in Appendix C ; see caveat above. 5 seeds for Qwen2, 3 seeds for others (single-seed reported as point estimate). Baseline denominator is full RTRL+Adam on the same model with truncated-BPTT (

w{=}64

) as a practical proxy; Mamba-1.4B’s

133.5\%

therefore reflects exceeding this proxy, not an exact RTRL baseline (full closed-form RTRL on a 1.4B-parameter model is computationally infeasible and never run).

Architecture	Delay	$d{=}0$ + Adam	Full RTRL	Frozen	Ratio
Vanilla RNN	$t{+}50$	0.0044	0.0031	1.25	$\approx$
Vanilla RNN	$t{+}100$	0.31	0.32	0.71	$\approx$
Vanilla RNN	$t{+}200$	0.065	0.083	0.69	$0.8\times$
Vanilla RNN	$t{+}500$	0.0079	0.0077	0.31	$\approx$
LSTM	$t{+}50$	0.0052	0.0082	0.65	$0.6\times$
LSTM	$t{+}100$	0.19	NaN (diverged)	1.72	–

Appendix

Table 18: Long-horizon delayed sine prediction. Post-shift MSE (5 seeds).

d{=}0

+ Adam matches or beats RTRL at every delay, up to

t{+}500

Task	$w{=}1$	$w{=}4$	$w{=}16$	$w{=}64$
Sine	0.0009 (1.0 $\times$ )	0.0005 (0.5 $\times$ )	0.0013 (1.5 $\times$ )	0.0024 (2.7 $\times$ )
Lorenz	0.00009 (1.0 $\times$ )	0.00003 (0.3 $\times$ )	0.00006 (0.6 $\times$ )	0.001 (12 $\times$ )
BCI (Adam)	0.339 (1.0 $\times$ )	0.321 (0.9 $\times$ )	0.294 (0.9 $\times$ )	0.356 (1.0 $\times$ )
BCI (RMSprop)	0.254 (1.0 $\times$ )	0.242 (1.0 $\times$ )	0.275 (1.1 $\times$ )	0.309 (1.2 $\times$ )

Appendix

Table 19: Truncated BPTT window comparison (5 seeds, converged pre-shift model, window varied at shift only). Values are post-shift MSE; ratio to

w{=}1

in parentheses. Lower is better.

Explore similar work

Massive Redundancy in Gradient Transport Enables Sparse Online Learning
Mar 16, 2026Aur Shalev MerinOnline Learning Recurrent Neural Networks
Independent Researcher
Learning Length-Extrapolatable Recurrent Models
Sep 8, 2026Hanwen JiangBackpropagation Temporal Credit Assignment
Adobe Research
LoRA-RC: Reservoir Computing with Low-Rank Adaptation
Sep 11, 2026Wenbin WanDynamical Systems Time Series Forecasting
Department of Mechanical Engineering, University of New Mexico, Albuquerque, NM 87131, USA

Mar 16, 2026J/K move · Enter open · S save

Massive Redundancy in Gradient Transport Enables Sparse Online Learning

Aur Shalev Merin

Independent Researcher

Online Learning Recurrent Neural Networks

Open paper AskPDF arXiv

Real-time recurrent learning (RTRL) computes exact online gradients by propagating a Jacobian tensor forward through recurrent dynamics, but at O(n^4) cost per step. Prior work has sought structured approximations (rank-1 compression, graph-based sparsity, Kronecker factorization). We show that, in the continuous error signal regime, the recurrent Jacobian is massively redundant:propagating through a random 6% of paths (k=4 of n=64) recovers 84 +/- 6% of full RTRL's adaptation ability across five seeds, and the absolute count k=4 remains effective from n=64 to n=256 (6% to 1.6%, recovery 84 to 78%), meaning sparse RTRL becomes relatively cheaper as networks grow. In RNNs, the recovery is selection-invariant (even adversarial path selection works) and exhibits a step-function transition from zero to any nonzero propagation. Spectral analysis reveals the mechanism: the Jacobian is full-rank but near-isotropic (condition numbers 2.6-6.5), so any random subset provides a directionally representative gradient estimate. On chaotic dynamics (Lorenz attractor), sparse propagation is more numerically stable than full RTRL (CV 13% vs. 88%), as subsampling avoids amplifying pathological spectral modes. The redundancy extends to LSTMs (k=4 matches full RTRL) and to transformers via sparse gradient transport (50% head sparsity outperforms the dense reference; 33% is borderline), with higher thresholds reflecting head specialization rather than isotropy. On real primate neural data, sparse RTRL (k=4) adapts online to cross-session electrode drift (80 +/- 11% recovery, 5 seeds), where sparse propagation is again more stable than full RTRL. Without continuous error signal, Jacobian propagation accumulates numerical drift and degrades all RTRL variants, a scope condition for all forward-mode methods. Results hold with SGD (92 +/- 1% recovery), suggesting independence from optimizer choice.

Abstract

Explore similar work

Massive Redundancy in Gradient Transport Enables Sparse Online Learning

Learning Length-Extrapolatable Recurrent Models

LoRA-RC: Reservoir Computing with Low-Rank Adaptation