Massive Redundancy in Gradient Transport Enables Sparse Online Learning

Aur Shalev Merin

cs.LGMar 16, 2026

Massive Redundancy in Gradient Transport Enables Sparse Online Learning

Authors: Aur Shalev Merin

Organizations: Independent Researcher

Abstract

Real-time recurrent learning (RTRL) computes exact online gradients by propagating a Jacobian tensor forward through recurrent dynamics, but at O(n^4) cost per step. Prior work has sought structured approximations (rank-1 compression, graph-based sparsity, Kronecker factorization). We show that, in the continuous error signal regime, the recurrent Jacobian is massively redundant:propagating through a random 6% of paths (k=4 of n=64) recovers 84 +/- 6% of full RTRL's adaptation ability across five seeds, and the absolute count k=4 remains effective from n=64 to n=256 (6% to 1.6%, recovery 84 to 78%), meaning sparse RTRL becomes relatively cheaper as networks grow. In RNNs, the recovery is selection-invariant (even adversarial path selection works) and exhibits a step-function transition from zero to any nonzero propagation. Spectral analysis reveals the mechanism: the Jacobian is full-rank but near-isotropic (condition numbers 2.6-6.5), so any random subset provides a directionally representative gradient estimate. On chaotic dynamics (Lorenz attractor), sparse propagation is more numerically stable than full RTRL (CV 13% vs. 88%), as subsampling avoids amplifying pathological spectral modes. The redundancy extends to LSTMs (k=4 matches full RTRL) and to transformers via sparse gradient transport (50% head sparsity outperforms the dense reference; 33% is borderline), with higher thresholds reflecting head specialization rather than isotropy. On real primate neural data, sparse RTRL (k=4) adapts online to cross-session electrode drift (80 +/- 11% recovery, 5 seeds), where sparse propagation is again more stable than full RTRL. Without continuous error signal, Jacobian propagation accumulates numerical drift and degrades all RTRL variants, a scope condition for all forward-mode methods. Results hold with SGD (92 +/- 1% recovery), suggesting independence from optimizer choice.

Figures & tables

Method	Mechanism	Cost	Constraint
UORO ( Tallec and Ollivier, 2018 )	Rank-1 compression	$O(n^{2})$	High variance
SnAp ( Menick et al., 2021 )	Graph-structured sparsity	$O(n^{2}s)$	Weight-sparse RNN
OK ( Benzing et al., 2019 )	Kronecker-sum approx	$O(n^{3})$	None
Element-wise ( Irie et al., 2024 )	Kill inter-neuron recurrence	$O(n^{2})$	Architectural change
RFLO ( Murray, 2019 )	Drop Jacobian term	$O(n^{2})$	$=$ eligibility traces

Table 1: RTRL approximation methods. All propose structured approximations with specific design tradeoffs. None tests whether unstructured random sparsification suffices.

Method	Cost per step	Notes
Full RTRL	$O(n^{4})$	Exact forward-mode gradients
Sparse RTRL ( $k$ neighbors)	$O(k\cdot n^{3})$	$k\ll n$ ; dense weights
Eligibility traces	$O(n^{2})$	No Jacobian propagation
UORO	$O(n^{2})$	Rank-1, high variance
SnAp	$O(n^{2}\cdot s)$	$s$ = propagation steps; weight-sparse

Table 2: Per-step computational cost for RTRL variants.

$k$	Fraction	Post-shift MSE	Recovery (%)
0 (traces)	0%	$0.23\pm 0.15$	0
4	6%	$0.0015\pm 0.0005$	$84\pm 6$
8	12.5%	$0.0016\pm 0.0005$	$82\pm 5$
16	25%	$0.0017\pm 0.0005$	$82\pm 6$
32	50%	$0.0012\pm 0.0002$	$87\pm 3$
64 (full)	100%	$0.0006\pm 0.0000$	100

Table 3: Post-shift MSE and gap recovery on sine frequency shift (

n{=}64

, 5 seeds, mean

\pm

s.d.). The jump from

k{=}0

to any

k\geq 4

spans two orders of magnitude; further increasing

k

adds no systematic improvement.

Figure 1: The Jacobian is massively redundant. (a) Post-shift MSE drops by two orders of magnitude from

k{=}0

(eligibility traces) to

k{=}4

(6% of paths), then stays flat through

k{=}64

(full RTRL). (b) Log-scale gap recovery is a step function: any

k\geq 4

recovers

82

–

87\%

of full RTRL. Error bars:

\pm 1

s.d. across 5 seeds.

	$n{=}64$		$n{=}128$
Seed	$k{=}4$	Full RTRL	$k{=}4$	Full RTRL
42	0.015	0.015	0.016	0.067
123	0.016	0.011	0.015	0.021
7	0.018	0.046	0.015	0.876
2024	0.012	0.013	0.018	0.063
31337	0.014	0.093	0.018	0.012

Table 4: Post-shift MSE on Lorenz attractor (5 seeds).

k{=}4

is stable across all seeds; full RTRL diverges on chaotic dynamics. Bold values indicate failure (MSE

>

\times

the

k{=}4

mean).

Figure 2: Sparse RTRL is more stable than full RTRL on chaotic dynamics. Per-seed loss curves on the Lorenz attractor (

n{=}64

, 5 seeds). Full RTRL (red) degrades on 2 of 5 seeds;

k{=}4

(blue) is stable on all seeds. Gray: eligibility traces (

k{=}0

). Vertical dotted line: parameter shift at

t{=}2000

Seed	Traces	$k{=}4$	Full RTRL	Recovery (%)
42	0.100	0.0072	0.0022	69
123	0.178	0.0037	0.0019	85
7	0.202	0.0028	0.0020	93
2024	0.040	0.0057	0.0020	65
31337	0.100	0.0035	0.0023	89
Mean $\pm$ s.d.		$0.0046\pm 0.0016$	$0.0021\pm 0.0001$	$80\pm 12$

Table 5: Post-shift MSE and gap recovery on multi-regime sine at first shift point (

n{=}64

, 5 seeds). Unlike Lorenz, full RTRL is stable on non-chaotic dynamics.

Task	Ring	Random	Oracle	Anti-oracle	Dynamic
Sine	$\mathbf{.0015\pm.0005}$	$.0016\pm.0006$	$.0016\pm.0004$	$\mathbf{.0014\pm.0008}$	$.0019\pm.0004$
Lorenz	$\mathbf{.0149\pm.0019}$	$.0156\pm.0006$	$.0155\pm.0021$	$.0164\pm.0010$	$.0199\pm.0063$

Table 6: Post-shift MSE for five selection strategies, all at

k{=}4

(

n{=}64

, 5 seeds, mean

\pm

s.d.). No strategy is systematically better; the ranking changes across seeds.

Method	Copy T=5 (accuracy $\uparrow$ )	Adding T=50 (MSE $\downarrow$ )
Trivial baseline	0.125 (chance)	0.167 (predict mean)
BPTT	0.289	0.189
$k{=}0$ (traces)	0.239	0.284
$k{=}4$	0.239	0.281
$k{=}8$	0.292	0.278
$k{=}64$ (full RTRL)	0.130	0.289 (worst)

Table 7: Standard RTRL benchmarks with vanilla tanh RNN (

n{=}64

). No method succeeds; full RTRL is worst on both tasks. The architecture cannot solve these tasks regardless of training method.

Seed	Traces	$k{=}4$	Full RTRL	Notes
42	0.508	0.004	0.002	Clean
123	0.002	0.001	0.015	Inverted
7	0.548	0.003	0.015	$k{=}4$ beats full
2024	0.220	0.001	0.0002	Clean
31337	0.090	0.001	0.004	$k{=}4$ beats full
Mean $\pm$ s.d.		$0.002\pm 0.001$	$0.007\pm 0.006$

Table 8: LSTM gate Jacobian sparsification on sine frequency shift (

n{=}64

T{=}4000

, shift at

t{=}2000

, 5 seeds). The vanilla RNN pattern holds: traces fail,

k{=}4

recovers near-full RTRL performance. Seed 123 excluded from recovery (inverted gap; traces converge better than full RTRL on that seed).

Seed	$k{=}4$ post-shift MSE	$k{=}256$ (full)	Recovery (%)
42	0.0036	0.0003	66
123	0.0054	0.0003	62
7	0.0036	0.0010	80
2024	0.0034	0.0003	68
31337	0.0032	0.0065	116
Mean $\pm$ s.d.	$0.0038\pm 0.0008$		$78\pm 20$

Table 9: Scale test results at

n{=}256

(sine frequency shift, 5 seeds).

k{=}4

at 1.6% recovers

78\pm 20\%

of the log-space gap. Seed 31337:

k{=}4

outperforms full RTRL (recovery

>

100%), another stability win consistent with the Lorenz findings.

Method	$k/H$	Seed 42	Seed 43	Seed 44	Mean	Recovery
		Avg Accuracy
bptt_ref	12/12	0.407	0.397	0.374	0.393	—
dense_ref	12/12	0.403	0.389	0.401	0.398	100%
sparse $k{=}6$	6/12	0.406	0.382	0.415	0.401	107%
sparse $k{=}4$	4/12	0.387	0.376	0.384	0.383	77%
sparse $k{=}2$	2/12	0.353	0.325	0.336	0.338	8%

Table 10: Sparse gradient transport on Split CIFAR-10 (

H{=}12

heads, 3 seeds). All heads contribute to the forward pass; gradient flows through only

k

heads. At

k{=}6

(50%), sparse transport outperforms the dense reference (regularization effect). The threshold for

>

70% recovery is

{\sim}33\%

of heads, higher than RNNs (

{\sim}6\%

), reflecting greater head specialization.

Method	Seed 7	Seed 42	Seed 43	Seed 44	Seed 2024	Mean
	Recovery (%)
Fisher oracle (top-6)	96.9	97.6	93.7	89.2	99.1	95.3
Entropy oracle (top-6)	58.0	70.0	76.9	87.8	89.5	76.4
Random	84.0	84.2	66.3	80.6	72.5	77.5
Entropy anti (bottom-6)	82.3	61.1	82.8	66.9	85.3	75.7
Fisher anti (bottom-6)	70.1	63.3	69.7	63.7	88.3	71.0

Table 11: Head selection strategies on Split CIFAR-10 (

H{=}12

k{=}6

, 5 seeds). Fisher oracle selects heads with the highest squared-gradient importance; entropy oracle selects heads with the lowest attention entropy (most peaked attention, i.e., least dormant). The Fisher oracle–anti gap of

+

24.3pp contrasts sharply with RNNs (

-0.6

pp; see text), while entropy oracle

\approx

random shows dormancy-based selection adds no useful signal. Seed 2024 is an outlier (Fisher anti recovery 88.3%); the remaining 4 seeds show gaps of

+

25–34pp.

	Late session B MSE
Method	Seed 42	Seed 123	Seed 31337	Seed 7	Seed 2024
Frozen (no adapt.)	0.734	0.547	0.674	0.628	0.580
$k{=}0$ (traces)	0.479	0.360	0.432	0.464	0.423
$k{=}4$ (6%)	0.400	0.328	0.386	0.306	0.360
$k{=}64$ (full RTRL)	0.303	0.406	0.390	0.347	0.466
BPTT (ref)	0.351	0.329	0.330	0.317	0.333

Table 12: Cross-session neural decoding (7-month electrode drift,

n{=}64

, 5 seeds). Frozen decoder degrades massively (

+

0.29–0.47 MSE). Sparse RTRL (

k{=}4

) recovers

80\pm 11\%

of the drift gap, with lower variance than full RTRL. Full RTRL is unstable on 2 of 5 seeds (bold), reproducing the pattern from chaotic dynamics (Table 4 ).

Figure 3: Gradient transport redundancy across architectures and real neural data. (a) Recovery threshold comparison: RNNs require

{\sim}6\%

of neurons, transformers

{\sim}33

–

50\%

of heads. The difference reflects isotropy vs. specialization. (b) BCI cross-session adaptation:

k{=}4

sparse RTRL adapts to 7-month electrode drift while the frozen decoder degrades and full RTRL is unstable on 2/5 seeds.

† 3 of 5 seeds; 2 seeds diverged (condition number $>10^{4}$ ; see text).
Task	$n$	Seeds	$r_{95}$	Cond. ( $\sigma_{1}/\sigma_{n}$ )
Sine	64	5	$59.8\pm 0.4$	$3.97\pm 0.81$
Lorenz	64	3 †	$56.3\pm 0.5$	$2.95\pm 0.36$
Sine	128	5	$117.6\pm 0.5$	$6.51\pm 0.26$
Sine	256	3	$236.7\pm 0.4$	$2.97\pm 0.33$

Table 13: Spectral analysis of the Jacobian tensor along the neuron axis. The Jacobian is full-rank but near-isotropic: condition numbers of 2.6–6.5 indicate nearly uniform singular values. Values are mean

\pm

s.d. where multiple seeds are available.

Figure 4: The Jacobian is full-rank and near-isotropic. (a) Singular value spectrum of the Jacobian tensor (reshaped to

n\times n^{2}

) at the shift point (

n{=}64

, sine task), with nearly uniform singular values (condition number

{\sim}4

). Individual seeds shown. (b) Condition number remains low (2–5) throughout training across all 5 seeds. Isotropy is not an initialization artifact. Red dashed line: condition

=10

, above which random subsampling would degrade.

Figure 5: Sparse gradients are directionally aligned with full RTRL gradients. (a) Cosine similarity between

k

-sparse and full RTRL gradients over time (5-seed mean

\pm

s.d.). Pre-shift cosine is

{\sim}0.92

; post-shift dip to

{\sim}0.82

k{=}4

reflects the harder adaptation regime but never collapses. (b) Post-shift cosine increases monotonically with

k

, consistent with the flat recovery curve.

† $k{=}4$ is borderline (high seed variance); $k{=}6$ is robust (107% recovery, 3 seeds).
Architecture	Transport unit	Threshold	Recovery at threshold	Degenerate
Vanilla RNN	Neuron	$k{=}4$ /64 (6%)	$84\pm 6\%$	$k{=}0$ (fails)
LSTM	Gate Jacobian	$k{=}4$ /64 (6%)	$\geq$ 100%	$k{=}0$ (fails)
Transformer	Attention head	$k{=}4$ –6/12 (33–50%)	77–107% †	$k{=}1$ (0%)

Table 14: Gradient transport redundancy across architectures. The threshold for

>

70% recovery varies from

{\sim}6\%

(RNNs) to

{\sim}33\%

(transformers), reflecting different specialization structures. Transformer threshold measured with sparse gradient transport (Section 4.7 ).

Explore similar work

Mar 30, 2026

Immediate Derivatives Suffice for Online Recurrent Adaptation

For three decades online recurrent learning has been assumed to require propagating a Jacobian tensor through the network's dynamics at

O(n^4)

per step. We show it doesn't. Dropping the propagation entirely (

d=0

O(n^2)

memory) matches full RTRL within CI on held-out BCI cross-session drift (TOST equivalent within

\pm 3

pp at

n=20

, Adam, float64), and across vanilla-RNN synthetic cells (sine and Lorenz under Adam and SGD) and LSTM/sine under Adam. A decomposition

g_{RTRL} = g_{imm} + g_{past}

explains why. On BCI,

g_{past}

concentrates in a single direction (top-1 singular fraction 0.62-0.74 across four optimizers, vs 0.333 for

g_{imm}

), and the four-optimizer full-RTRL-vs-

d=0

recovery gap tracks each optimizer's per-layer update-magnitude ratio

\|ΔW_{hh}\|/\|ΔW_{out}\|

monotonically. A stationary (no-drift) control collapses both concentrations to ~0.6: the drift-specific signal is the differential, not

g_{past}

's absolute rank-1 structure. The signature and the behavioral gap both collapse on LSTM, consistent with a mechanism specific to additive linear recurrence. On synthetic sine,

g_{imm}

is redundant with

g_{past}

, which predicts the synthetic null. Full RTRL's one robust advantage is LARS (+17 to +27 pp), but

d=0

+LARS also fails to adapt independently; the gap is an optimizer

\times

method interaction, not a method-quality claim. We characterize the regime:

d=0

+Adam+float64 is robust; SGD, Adafactor, and float32 have specific fragilities documented in the paper. On the evaluated cells, the

1000\times

memory saving at

n=1024

(

O(n^2)

O(n^4)

) comes with no measured recovery cost.

Aur Shalev Merin

Independent Researcher

Recurrent Neural Networks Gradient Descent Dynamics

Ask PDF

May 29, 2026

Dynamics and Representation Structure of Local Approximations to Gradient-Based Learning in Linear Recurrent Neural Networks

Biological and neuromorphic recurrent neural networks (RNNs) are subject to spatial and temporal locality constraints on the information that can plausibly be used during learning. A common strategy to satisfy these constraints is to modify gradient descent by neglecting non-local terms to varying degrees, as in random feedback local online (RFLO) learning and truncated backpropagation through time (tBPTT). However, the learning dynamics of these algorithms, and how they compare with BPTT, remain poorly understood. We apply dynamical systems theory to data-aligned linear RNNs -- whose dynamics can be separated into orthogonal modes -- to compare stationary solutions, stability properties, and convergence rates, finding qualitatively distinct behaviour for RFLO versus BPTT and one-step tBPTT. We further observe that the solutions learned by RFLO are restricted to low-rank perturbations of initial parameters, a result which holds beyond the data-aligned setting. Our work provides analytical insight into how locality constraints shape learning dynamics, with implications for neuroscientific models of learning and alternative optimization approaches for RNNs.

Ezekiel Williams, Alexandre Payeur, Guillaume Lajoie

Department of Mathematics and Statistics, Universit´e de Montr´eal, Canada · Mila Qu´ebec, Canada · Imperial College London, United Kingdom

Recurrent Neural Networks Linear RNNs Neural Network Training Dynamics Local Learning

Ask PDF

Feb 20, 2026

Learning Long-Range Dependencies with Temporal Predictive Coding

Temporal Predictive Coding provides a layer-local, parallelisable mechanism for learning in recurrent systems, making it an attractive candidate for online local learning on neuromorphic and edge hardware. However, its recurrent parameter update captures only local temporal relationships, neglecting the historic influence of parameters along the latent-state trajectory, and therefore struggles to assign credit over longer temporal horizons. This work combines for the first time Temporal Predictive Coding with Real-Time Recurrent Learning (tPC-RTRL), incorporating an online influence matrix that tracks this historic effect whilst preserving the spatial and temporal locality properties valued by neuromorphic implementations. Under explicit assumptions, we prove that tPC-RTRL recovers the gradients of backpropagation-through-time exactly. Empirically, a near-equivalence holds across several tasks of varying scale and complexity, including byte-level language modelling on WikiText-103 (tPC-RTRL vs. BPTT: 1.865 vs. 1.864 validation BPC), English--French translation on a CCMatrix subset (20.23 vs. 20.29 BLEU), and a realistic nanodrone system-identification benchmark (0.506m vs. 0.505m mean position error). Finally, we show that the iterative inference mechanism used during training can be reused at deployment time to incorporate intermittent state observations, halving final-position error relative to open-loop rollout on the nanodrone task (0.402m vs. 0.805m) and suggesting a path towards unifying learning and filtering within the same computational framework.

Tom Potter, Oliver Rhodes

International Centre for Neuromorphic Systems, The University of Manchester, Manchester, United Kingdom

Predictive Coding Recurrent Neural Networks Local Learning

Ask PDF