Immediate Derivatives Suffice for Online Recurrent Adaptation

Aur Shalev Merin

cs.LGMar 30, 2026

Immediate Derivatives Suffice for Online Recurrent Adaptation

Authors: Aur Shalev Merin

Organizations: Independent Researcher

Abstract

For three decades online recurrent learning has been assumed to require propagating a Jacobian tensor through the network's dynamics at $O(n^4)$ per step. We show it doesn't. Dropping the propagation entirely ( $d=0$ , $O(n^2)$ memory) matches full RTRL within CI on held-out BCI cross-session drift (TOST equivalent within $\pm 3$ pp at $n=20$ , Adam, float64), and across vanilla-RNN synthetic cells (sine and Lorenz under Adam and SGD) and LSTM/sine under Adam. A decomposition $g_{RTRL} = g_{imm} + g_{past}$ explains why. On BCI, $g_{past}$ concentrates in a single direction (top-1 singular fraction 0.62-0.74 across four optimizers, vs 0.333 for $g_{imm}$ ), and the four-optimizer full-RTRL-vs- $d=0$ recovery gap tracks each optimizer's per-layer update-magnitude ratio $\|ΔW_{hh}\|/\|ΔW_{out}\|$ monotonically. A stationary (no-drift) control collapses both concentrations to ~0.6: the drift-specific signal is the differential, not $g_{past}$ 's absolute rank-1 structure. The signature and the behavioral gap both collapse on LSTM, consistent with a mechanism specific to additive linear recurrence. On synthetic sine, $g_{imm}$ is redundant with $g_{past}$ , which predicts the synthetic null. Full RTRL's one robust advantage is LARS (+17 to +27 pp), but $d=0$ +LARS also fails to adapt independently; the gap is an optimizer $\times$ method interaction, not a method-quality claim. We characterize the regime: $d=0$ +Adam+float64 is robust; SGD, Adafactor, and float32 have specific fragilities documented in the paper. On the evaluated cells, the $1000\times$ memory saving at $n=1024$ ( $O(n^2)$ vs $O(n^4)$ ) comes with no measured recovery cost.

Figures & tables

Arch.	Task	$\cos(g_{\text{imm}},g_{\text{past}})$	CI contains 0?
Vanilla RNN	Sine/Lorenz, $n\in\{64,128\}$	$\cos\in[-0.021,-0.003]$	marginal (4 cells)
LSTM	Sine ( $n{=}64$ )	$\mathbf{-0.104}\;[-0.190,-0.027]$	no
LSTM	Lorenz ( $n{=}64$ )	$-0.004\;[-0.046,+0.036]$	yes
RetNet	Sine ( $n{=}64$ )	$\mathbf{-0.102}\;[-0.147,-0.057]$	no
RetNet	Lorenz ( $n{=}64$ )	$+0.050\;[-0.015,+0.115]$	yes
BCI †	Cross-session	$\mathbf{-0.119}\;[-0.230,-0.014]$	no

Table 1:

\cos(g_{\text{imm}},g_{\text{past}})

during online adaptation (mean across 5 seeds with 95% bootstrap CI, averaged over the post-shift window; vanilla RNN uses exact full-RTRL sensitivity tensor, LSTM/RetNet/BCI use TBPTT

W{=}200

|\cos|\leq 0.25

across all 9 cells, sign-stable across hardware, magnitude hardware-dependent on BCI (0.12–0.24 across the two hardware configurations we measured). On vanilla RNN,

\cos

is near-zero at both scales. On LSTM/sine and RetNet/sine,

|\cos|\approx 0.10

with CIs excluding zero (bold): small but statistically non-zero. The TBPTT approximation is validated against exact RTRL on vanilla RNN sine (

-0.021

in both). † BCI uses TBPTT

W{=}200

on a FullRankRNN (

H{=}64

, input dim

=40

Condition	$\text{top1}(G_{\text{past}})$	$\text{top1}(G_{\text{imm}})$	gap	$W_{hh}$ share of $u_{1}$
SGD, drift	$0.743$	$0.333$	$+0.41$	$60.0\%$
Adam, drift	$0.708$	—	—	$58.2\%$
LARS, drift	$0.623$	—	—	$59.4\%$
Adafactor, drift	$0.657$	—	—	$53.0\%$
SGD, stationary (control)	$0.586$	$0.600$	$-0.01$	$74.2\%$

Table 2: Structure of

g_{\text{past}}

during BCI cross-session adaptation, f64, local AVX-2, per-optimizer f64 fr-best LR, TBPTT window

w{=}200

. Top four rows: cross-session drift (session B adapt,

n{=}5

seeds, 41 windows).

\text{top1}(G_{\text{imm}})

measured for SGD only at matched conditions. Bottom row: stationary control (SGD fr-logged on the tail of session A, pretrained weights, no distribution shift;

n{=}5

seeds, 25 windows). The drift-induced differential

\text{top1}(G_{\text{past}})-\text{top1}(G_{\text{imm}})

+0.41

on drift and

-0.01

on stationary;

g_{\text{past}}

’s absolute rank-1 concentration is partly a cumulative-trace SVD geometry (present at

\sim 0.6

stationary), but

g_{\text{imm}}

’s diffuseness emerges only under drift.

W_{hh}

share compared against the parameter-count null of

|W_{hh}|/d=4096/6850\approx 59.8\%

;

n{=}5

drift values are at null,

n{=}10

SGD hardening gives

74.1\%

above null.

Optimizer	$\\|\Delta W_{hh}\\|/\\|\Delta W_{out}\\|$	fr $-$ $d{=}0$ gap (pp)
SGD	$4.30$	$-10$
Adafactor	$4.82$	$-7$
Adam	$6.20$	$\approx 0$
LARS	$10.49$	$+20$

Table 3: Per-layer update-magnitude ratio

\|\Delta W_{hh}\|/\|\Delta W_{out}\|

and measured fr-vs-

d{=}0

held-out recovery gap on BCI f64 cross-session drift (80/20 split convention from our cross-hardware held-out sweeps; the 50/50 convention gives Adam within

0.3

pp of

0

(Section 8 reproduction) and preserves the LARS direction). Higher ratio

\rightarrow

larger fr advantage. Interpretation: SGD’s step per layer scales with the raw per-layer gradient (ratio

\sim 4

); LARS’s per-layer normalization divides by

\|W_{\text{layer}}\|

, and because pretrained

\|W_{hh}\|\gg\|W_{out}\|

the effective ratio inflates to

\sim 10

; Adam’s per-coordinate

v

-buffer produces an intermediate ratio; Adafactor’s factored preconditioner is close to SGD.

n{=}4

is too few for an out-of-sample prediction test; the ordering is consistent with the mechanism but we report it as a measured correspondence, not a prediction.

Figure 1: (a) Memory: at

n{=}1024

d{=}0

uses

12.6

MB vs full RTRL’s

12.9

GB (

1000\times

). (b) BCI held-out gap (full RTRL

-

d{=}0

, pp) at

80/20

, float32: three (hardware, kernel) configurations per optimizer at fixed Kaggle-tuned LR. Adam sign-reverses across configs; LARS spans

43

pp; method comparisons unreliable at f32. (c) Same at float64: range bars = 3 configs at Kaggle-tuned LR (transported across hardware); diamonds = per-hardware-best LR for both methods (open Kaggle, filled local). Adam equivalent within seed noise; LARS fr-wins

+16

+25

pp (

d{=}0{+}

LARS independently fails); Adafactor hardware-inconsistent (

\approx 8

pp sign-flip). At

50/50

, Adam is equivalent to within

0.3

pp (reproduced to

0.04

pp).

Figure 2: BCI held-out gap (full RTRL

-

d{=}0

, pp) vs adapt-horizon, four optimizers,

80/20

, local f64 AVX-2, 5 seeds. LARS retains a

+17

+27

pp fr advantage at every horizon (CIs exclude zero). SGD, Adafactor, Adam gaps are within

\pm 10

pp of zero with overlapping CIs. Seed 42’s fr+Adam is a consistent outlier (

19

–

31\%

recovery vs

40

–

50\%

), widening Adam fr CIs but not changing the zero-overlap verdict. Presented as adapt-horizon-invariance of the LARS advantage, not a prediction test.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Domain	$d{=}0$ best optim.	$d{=}0$ Adam (default)	$k{=}4$ RTRL / UORO §	Full RTRL
Sine ( $n{=}64$ )	SGD lr=0.1: 130%	102	125	100
Delayed ( $t$ +50)	RMSprop: 179	147	130	100
Lorenz (chaotic)	Adafactor: 125	49	–	100
BCI, Adam (f64, 50/50)	$d{=}0$ +Adam: $37.6$	$37.6$	–	$37.3$ (gap $-0.3$ pp)
BCI, Adam (f64, 80/20)	$d{=}0$ +Adam: $41.7$	$41.7$	UORO: $14.7$	$50.3$ (fr $+8.6$ pp, CI $\ni 0$ )
BCI, Adam (f64, last-layer) §	Adam: $22.0$ ; SGD: $22.1$	$22.0$	–	– (floor; cf. $41.7$ full-rec.)

Appendix

Table 4: Head-to-head comparison: immediate derivatives (

d{=}0

) vs. Jacobian-based methods across domains. All values are held-out or adapt-window recovery (%) unless noted; “–” indicates not tested. Synthetic rows report adapt-window recovery under per-optimizer LR tuning. BCI rows use the held-out protocol of Appendix C with per-cell best LR on disjoint validation; split convention in parentheses. BCI numbers at float32 are hardware-fragile (see Appendix D ); we report float64 numbers here as the numerically stable estimate .

\dagger

Language row reports cross-entropy (lower is better); no held-out validation was run.

\ddagger

Full RTRL cannot run at

n{=}1024

; the high recovery reflects an unreliable denominator.

\lx@sectionsign

UORO: rank-1 Jacobian approximation of Tallec and Ollivier (2017) .

\mathsection

Last-layer-only:

W_{ih}

W_{hh}

b_{h}

frozen; only

W_{out}

and

b_{out}

adapt, quantifying the readout-remap floor for comparison against

d{=}0

’s full-recurrent adaptation.

Optimizer	Sine ( $n{=}64$ )	BCI adapt window	Lorenz $\rho{=}28{\to}35$
SGD	130% (lr=0.1)	160 $\pm$ 40% (OVERFIT)	124 $\pm$ 26%
Adam	102% (lr= $10^{-3}$ )	61 $\pm$ 16% (generalizes)	49 $\pm$ 27%
signSGD	50%	61 $\pm$ 13%	95 $\pm$ 42%
signSGD + momentum	30%	43 $\pm$ 5%	54 $\pm$ 15%
LARS	50%	123 $\pm$ 23% (OVERFIT)	123 $\pm$ 24%
Gradient normalization	40%	32 $\pm$ 6%	50 $\pm$ 11%

Appendix

Table 5: Per-optimizer recovery (%) with

d{=}0

, per-optimizer LR tuning, 5 seeds, evaluated on the adaptation window (not held out). Note that adapt-window recovery on non-stationary tasks is inflated by overfitting to the adaptation stream; see the held-out validation in Appendix C . On stationary sine, all optimizers adapt. The BCI adapt-window column is measured at float32 on Kaggle CPU; given the float32 hardware fragility documented in Appendix D , treat absolute BCI adapt-window numbers here as environment-specific measurements rather than method-level properties. The OVERFIT markers reflect the held-out result (Appendix C ); the Adam generalization claim is robust across configurations (Appendix D ).

Method	Optimizer	Best LR	Held-out rec. (%, CI)	Verdict
$d{=}0$	Adam	$3{\times}10^{-4}$	$\mathbf{44.9}\;[41.6,48.5]$	generalizes
$d{=}0$	Adafactor	$10^{-3}$	$25.9\;[20.6,32.6]$	partial
$d{=}0$	SGD	$3{\times}10^{-3}$	$42.2\;[38.5,46.3]$	generalizes
$d{=}0$	LARS	$10^{-2}$	$11.3\;[3.7,21.2]$	weak
Full RTRL	Adam	$10^{-3}$	$\mathbf{52.1}\;[48.7,55.4]$	generalizes
Full RTRL	Adafactor	$10^{-3}$	$49.7\;[47.8,52.5]$	generalizes

Appendix

Table 6: BCI held-out validation with per-cell LR tuning (5 seeds, per-cell best LR from a grid sweep, selected to maximize mean held-out recovery). Top block:

d{=}0

(immediate derivatives). Bottom block: full RTRL (TBPTT

w{=}50

). Each method

\times

optimizer cell receives its own best LR from the same grid. The

d{=}0

vs full-RTRL comparison within each optimizer is therefore fair (same grid, best-LR-per-method).

Optimizer	default	default	AVX-2	Gap range (f32)
	Kaggle f32	Local f32	Local f32
Adam	$+7.14$	$-9.74$	$-14.35$	$-14$ to $+7$ (sign-flip)
Adafactor	$+23.79$	$+1.77$	$+10.45$	$+1$ to $+24$
LARS	$+44.20$	$+31.02$	$+0.78$	$+1$ to $+44$
	Kaggle f64	Kaggle f64	Local f64
	default	AVX-2	AVX-2	Gap range (f64)

Appendix

Table 7: BCI held-out recovery gap (full-RTRL

-

d{=}0

, pp) at Kaggle-tuned LR transported across configurations. Positive favours full RTRL. Kaggle f32 is the initial Kaggle-f32 protocol. Cross-hardware comparisons at this fixed LR conflate method differences with LR-portability effects; see Table 8 for gaps at per-hardware best LR.

Optimizer	Kaggle f32 gap	Local f32 gap	Kaggle f64 gap	Local f64 gap	Verdict at f64
Adam	$+7.1$	$-3.9$	$-6.05$ / $-2.47$	$-0.34$	equivalent
SGD	$-11.27$	$-7.25$	–	$-9.93$	$d{=}0$ wins $7$ – $11$ pp
Adafactor	$+23.8$ ∗	$+5.3$	$+1.37$	$-6.67$	hardware-inconsistent
LARS	$+44.2$	$+23.2$	$+15.85$ / $+22.87$	$+25.09$	full RTRL wins $+16$ to $+25$ pp

Appendix

Table 8: BCI held-out gap (full-RTRL

-

d{=}0

) at true per-hardware best LR for both methods, both hardwares . All f64 cells use LRs selected from tighter-than-initial grids on each hardware. The Kaggle Adafactor row resolves an initial-grid-coarseness issue (the grid missed the true

d{=}0

Adafactor optimum at

3\times 10^{-4}

on Kaggle). Compare with Table 7 , which uses fixed Kaggle-tuned LRs transported across hardware.

Optimizer	$\lambda{=}0.0$	$\lambda{=}0.1$	$\lambda{=}0.5$	$\lambda{=}0.95$
Adam	$100.0\%$	$99.3\%$	$100.2\%$	$90.2\%$
Adafactor	$100.9\%$	$100.6\%$	$98.3\%$	$100.9\%$
RMSprop	$100.8\%$	$100.5\%$	$98.2\%$	$98.3\%$
SGD	$104.1\%$	$103.9\%$	$102.2\%$	$103.4\%$

Appendix

Table 9: Recovery (%) at best per-cell LR. Sine

n{=}64

, 5 seeds, no clipping. Reference: Adam

d{=}0

at lr

=10^{-3}

(mse

=0.00068

). Every

(\lambda,\text{optimizer})

cell achieves

\geq 90\%

recovery: decay value is empirically compensable by LR tuning, consistent with the small-

|\cos|

finding of Section 3 .

Figure 3: Recovery vs. trace decay on sine frequency-shift (Adam at lr

=10^{-3}

, 5 seeds, mean

\pm

1 std). At the standard Adam learning rate, the conventional default

\lambda{=}0.95

produces 0% recovery; values below 0.5 form a safe plateau. Per-decay LR tuning rescues high-decay configurations (Table 9 ), but

\lambda{=}0

is the simpler choice: it removes a hyperparameter and works at every standard learning rate we tested.

Architecture	Recovery ( $d{=}0$ + Adam)	Notes
Vanilla RNN	102%	reference
LSTM	99.7%	gated cell state
CTRNN	89.8%	continuous-time
xLSTM (sLSTM)	91.5%	exponential gating
RWKV	adapts	unstable on chaotic dynamics with Adam (App. R )
RetNet	93%	multiplicative key-value

Appendix

Table 10: Cross-architecture

d{=}0

recovery on sine frequency-shift (5 seeds, BPTT

w{=}1

gradients, Adam at lr

=10^{-3}

). Six architectures have numerical recovery; four are reported as “adapts” (post-shift MSE below frozen; full-RTRL reference impractical at scale or the output-bypass structure makes the ratio uninformative).

Figure 4: Cross-architecture structural measurements (7 architectures, 5 seeds each; RWKV excluded at initialization). (a) Gradient norm ratio between output and recurrent parameters. (b) Condition number of the recurrent parameter Jacobian

\partial h/\partial\theta_{\text{rec}}

, measured after training. Five of seven architectures are near-isotropic (

\text{cond}<10

, blue band). RetNet is an exception (cond

\sim 10^{4}

): its key-value outer product concentrates sensitivity in few directions.

Depth	Task	Best trace (d, lr)	MSE	Best RTRL MSE	Wins?
1	Sine	$d{=}0.0$ , $10^{-3}$	0.0007	0.0006	$\approx$ †
2	Sine	$d{=}0.0$ , $3{\times}10^{-4}$	0.000129	0.000142	Yes
3	Sine	$d{=}0.1$ , $10^{-4}$	0.000178	0.000218	Yes
3	Lorenz	$d{=}0.0$ , $10^{-3}$	0.000352	0.000573	Yes

Appendix

Table 11: Multi-layer results: best trace configuration vs. best full RTRL at each depth (5 seeds, post-shift MSE). † Within seed variance.

cond $(W_{hh})$	$d{=}0$ MSE	full-RTRL MSE	within CI?
$5$	$\sim 0.0006$	$\sim 0.0006$	yes
$50$	$\sim 0.0006$	$\sim 0.0006$	yes
$500$	$\sim 0.0006$	$\sim 0.0006$	yes
$5000$	$\sim 0.0006$	$\sim 0.0006$	yes

Appendix

Table 12: Post-shift MSE (mean

\pm

seed std) on sine frequency-shift under imposed

\operatorname{cond}(W_{hh})

, Adam pretrained, per-cell best LR,

n{=}5

seeds. The

d{=}0

-vs-full-RTRL post-shift MSE is within CI at every imposed condition level, but because training returns

\operatorname{cond}(\partial h/\partial\theta)

[2,7]

regardless of imposed

\operatorname{cond}(W_{hh})

(paragraph above), this null result does not refute the isotropy hypothesis, since the intervention does not vary the quantity the hypothesis is about.

$\rho$	Vanilla RNN cos	RetNet cos
$0.00$	$-0.001$ $[-0.013,+0.012]$	$-0.000$ $[-0.003,+0.003]$
$0.30$	$-0.004$ $[-0.015,+0.005]$	$+0.002$ $[-0.009,+0.015]$
$0.60$	$-0.010$ $[-0.032,+0.005]$	$+0.019$ $[-0.009,+0.049]$
$0.90$	$+0.021$ $[+0.012,+0.030]$	$+0.002$ $[-0.075,+0.071]$
$0.99$	$+0.009$ $[-0.002,+0.024]$	$-0.125$ $[-0.243,-0.008]$

Appendix

Table 13:

\cos(g_{\text{imm}},g_{\text{past}})

across input autocorrelation

\rho

10

seeds, bootstrap

95\%

CI. RetNet cos remains small across

\rho\in[0,0.9]

; at

\rho{=}0.99

the sign reverses (wrong direction under the hypothesis). The decorrelation hypothesis is falsified.

Opt.	LR	Recovery (%, CI)	top1 $(G_{\text{past}})$	top1 $(G_{\text{imm}})$	gap
SGD	$3{\times}10^{-4}$	$+45.7\;[+41.2,+50.9]$	$0.361$	$0.348$	$+0.013$
Adam	$3{\times}10^{-3}$	$+63.8\;[+62.7,+64.9]$	$0.299$	$0.264$	$+0.035$
LARS	$1{\times}10^{-2}$	$+52.9\;[+48.8,+56.0]$	$0.403$	$0.345$	$+0.058$
Adafactor	$3{\times}10^{-3}$	$+66.4\;[+63.3,+69.0]$	$0.364$	$0.312$	$+0.052$

Appendix

Table 14: LSTM

H{=}64

BCI cross-session, f64 AVX-2, 5 seeds per optimizer at per-optimizer LR-swept best. All four cells fail the pre-registered primary kill (top-1

\geq 0.50

and top-1 gap vs.

G_{\text{imm}}\geq 0.20

). Compare RNN equivalents of

0.62

–

0.74

and

+0.41

. The rank-1 signature is specific to vanilla RNN’s additive linear recurrence.

config	h $\to$ h params	fr LR	fr rec	$d{=}0$ rec	gap (pp, 95% CI)	abs MSE benefit
RNN $H{=}64$	4096	$3{\times}10^{-3}$	$+35.59$	$+17.83$	$+17.76$ [ $+14.00,+21.47$ ]	$+0.119$
LSTM $H{=}32$	4096	$3{\times}10^{-3}$	$+57.45$	$+23.05$	$+34.40$ [ $+32.47,+36.37$ ]	$+0.255$
LSTM $H{=}64$	16384	$3{\times}10^{-3}$	$+65.31$	$+28.52$	$+36.79$ [ $+34.09,+39.15$ ]	$+0.286$

Appendix

Table 15: signSGD fr-vs-

d{=}0

gap at matched hidden-to-hidden recurrent capacity. LSTM

H{=}32

has

4\cdot 32^{2}=4096

\to

h parameters, identical to RNN

H{=}64

’s

W_{hh}

. The gap at LSTM

H{=}32

is within CI of the gap at LSTM

H{=}64

, and both are

\approx 2\times

RNN.

5

seeds,

95\%

bootstrap CIs, BCI f64, per-method fr/

d{=}0

best LR.

Cross-arch ratio (RNN)/(LSTM): $4.19$ (signSGD) vs $4.20$ (SGD) — matches to two decimal places
metric	RNN / signSGD	RNN / SGD	LSTM / signSGD	LSTM / SGD
$\|g_{\text{past}}\|/\|g_{\text{imm}}\|$ median	$20.6$ [ $11.5,31.5$ ]	$14.4$ [ $6.1,22.8$ ]	$4.93$ [ $4.59,5.28$ ]	$3.44$ [ $3.18,3.64$ ]
past-wins-magnitude fraction	$1.000$	$0.981$	$0.940$	$0.885$
$\sigma(\mathrm{sign}\,g_{\text{past}})$	$0.134$	$0.258$	$0.152$	$0.334$
$\mathrm{sign}(g_{\text{tot}})\neq\mathrm{sign}(g_{\text{imm}})$ rate	$0.376$	$0.351$	$0.326$	$0.290$

Appendix

Table 16: Per-coordinate structural measurements on recurrent parameters (RNN

W_{hh}

and LSTM

H{=}64

lstm.W ), BCI cross-session, f64,

41

TBPTT windows at

W{=}200

5

seeds,

95\%

bootstrap CIs. Columns under signSGD and under SGD use each optimizer’s fr-best LR. Cross-architecture ratio of

|g_{\text{past}}|/|g_{\text{imm}}|

preserved to two decimal places across optimizers (

4.19

4.20

Model	Params	Recovery ( $d{=}0$ + Adam)
Mamba-1.4B	1.4B	$\mathbf{133.5\%}$
Mamba-130M	130M	$51.6\%$
TinyLlama	1.1B	$31.6\%$
Qwen2	7B	$31.3\pm 7.3\%$
GPT-2	124M	$15.0\pm 0.9\%$

Appendix

Table 17: Online LoRA adaptation across pretrained LMs with

d{=}0

+ Adam at lr

=10^{-3}

(rank-4 adapters, code-to-Wikipedia shift, single-pass over 20K tokens with shift at 10K, no replay). Adapt-window recovery only; no held-out validation was run. Subject to the stream-fitting confound documented in Appendix C ; see caveat above. 5 seeds for Qwen2, 3 seeds for others (single-seed reported as point estimate). Baseline denominator is full RTRL+Adam on the same model with truncated-BPTT (

w{=}64

) as a practical proxy; Mamba-1.4B’s

133.5\%

therefore reflects exceeding this proxy, not an exact RTRL baseline (full closed-form RTRL on a 1.4B-parameter model is computationally infeasible and never run).

Architecture	Delay	$d{=}0$ + Adam	Full RTRL	Frozen	Ratio
Vanilla RNN	$t{+}50$	0.0044	0.0031	1.25	$\approx$
Vanilla RNN	$t{+}100$	0.31	0.32	0.71	$\approx$
Vanilla RNN	$t{+}200$	0.065	0.083	0.69	$0.8\times$
Vanilla RNN	$t{+}500$	0.0079	0.0077	0.31	$\approx$
LSTM	$t{+}50$	0.0052	0.0082	0.65	$0.6\times$
LSTM	$t{+}100$	0.19	NaN (diverged)	1.72	–

Appendix

Table 18: Long-horizon delayed sine prediction. Post-shift MSE (5 seeds).

d{=}0

+ Adam matches or beats RTRL at every delay, up to

t{+}500

Task	$w{=}1$	$w{=}4$	$w{=}16$	$w{=}64$
Sine	0.0009 (1.0 $\times$ )	0.0005 (0.5 $\times$ )	0.0013 (1.5 $\times$ )	0.0024 (2.7 $\times$ )
Lorenz	0.00009 (1.0 $\times$ )	0.00003 (0.3 $\times$ )	0.00006 (0.6 $\times$ )	0.001 (12 $\times$ )
BCI (Adam)	0.339 (1.0 $\times$ )	0.321 (0.9 $\times$ )	0.294 (0.9 $\times$ )	0.356 (1.0 $\times$ )
BCI (RMSprop)	0.254 (1.0 $\times$ )	0.242 (1.0 $\times$ )	0.275 (1.1 $\times$ )	0.309 (1.2 $\times$ )

Appendix

Table 19: Truncated BPTT window comparison (5 seeds, converged pre-shift model, window varied at shift only). Values are post-shift MSE; ratio to

w{=}1

in parentheses. Lower is better.

Explore similar work

Mar 16, 2026

Massive Redundancy in Gradient Transport Enables Sparse Online Learning

Real-time recurrent learning (RTRL) computes exact online gradients by propagating a Jacobian tensor forward through recurrent dynamics, but at O(n^4) cost per step. Prior work has sought structured approximations (rank-1 compression, graph-based sparsity, Kronecker factorization). We show that, in the continuous error signal regime, the recurrent Jacobian is massively redundant:propagating through a random 6% of paths (k=4 of n=64) recovers 84 +/- 6% of full RTRL's adaptation ability across five seeds, and the absolute count k=4 remains effective from n=64 to n=256 (6% to 1.6%, recovery 84 to 78%), meaning sparse RTRL becomes relatively cheaper as networks grow. In RNNs, the recovery is selection-invariant (even adversarial path selection works) and exhibits a step-function transition from zero to any nonzero propagation. Spectral analysis reveals the mechanism: the Jacobian is full-rank but near-isotropic (condition numbers 2.6-6.5), so any random subset provides a directionally representative gradient estimate. On chaotic dynamics (Lorenz attractor), sparse propagation is more numerically stable than full RTRL (CV 13% vs. 88%), as subsampling avoids amplifying pathological spectral modes. The redundancy extends to LSTMs (k=4 matches full RTRL) and to transformers via sparse gradient transport (50% head sparsity outperforms the dense reference; 33% is borderline), with higher thresholds reflecting head specialization rather than isotropy. On real primate neural data, sparse RTRL (k=4) adapts online to cross-session electrode drift (80 +/- 11% recovery, 5 seeds), where sparse propagation is again more stable than full RTRL. Without continuous error signal, Jacobian propagation accumulates numerical drift and degrades all RTRL variants, a scope condition for all forward-mode methods. Results hold with SGD (92 +/- 1% recovery), suggesting independence from optimizer choice.

Aur Shalev Merin

Independent Researcher

Online Learning Recurrent Neural Networks

Ask PDF

Sep 8, 2026

Learning Length-Extrapolatable Recurrent Models

Recurrent models provide a natural path to long-context modeling, yet models trained with backpropagation through time (BPTT) often fail beyond their training horizon. Classical analyses emphasize gradients that vanish or explode along temporal paths. However, dense per-token losses can still train a shared recurrent rule despite severe decay, showing that decay alone does not determine whether learning fails. We instead study state credit: the signal through which future losses reach earlier recurrent states before contributing to parameter updates. Accordingly, we intervene directly on state credit and propose Credit Stabilization through Time (CST). During backward propagation, CST locally rescales the state-credit signal to stabilize its norm without rotating the component being corrected, while leaving the forward computation unchanged. Because controlled synthetic tasks and real data exhibit different credit dynamics, we specialize CST to each regime. In both settings, CST improves performance beyond the training horizon, with gains observed at up to 128x the training length.

Hanwen Jiang

Adobe Research

Backpropagation Temporal Credit Assignment Recurrent Neural Networks Length Generalization Long-Context Modeling

Ask PDF

Sep 11, 2026

LoRA-RC: Reservoir Computing with Low-Rank Adaptation

Reservoir computing (RC) trains only a linear readout over a fixed recurrent layer, making it fast and data-efficient for online prediction. However, a static reservoir degrades under system drift, readout-only adaptation is then insufficient, and unconstrained reservoir adaptation can destroy the echo-state and incremental stability properties that make RC reliable. This paper proposes LoRA-RC, which adapts the recurrent matrix through a low-rank correction driven by streaming prediction errors. The base reservoir and adaptation bases are fixed offline; a small core matrix is adapted online, projected onto a spectral-norm ball, and low-pass filtered at each step. The projection guarantees that every applied recurrent matrix remains within a certified contraction set, and an incremental input-to-state stability bound is established for the reservoir along each online adaptation path, with path-independent rate and gain. On a Lorenz system with an abrupt parameter drift, LoRA-RC cuts post-drift prediction error by 56% versus a fixed RC and 51% versus readout-only adaptation; ablations over 20 seeds show that removing the projection inflates this error by more than a factor of 40.

Wenbin Wan

Department of Mechanical Engineering, University of New Mexico, Albuquerque, NM 87131, USA

Dynamical Systems Time Series Forecasting Reservoir Computing Low-Rank Adaptation

Ask PDF