SpatialPC: Deep Computation with Sublinear Local Credit Propagation

Benjamin Prada; Shion Matsumoto; Ankur Arjun Mali

cs.LGOct 26, 2026

SpatialPC: Deep Computation with Sublinear Local Credit Propagation

Authors: Benjamin Prada, Shion Matsumoto, Ankur Arjun Mali

Organizations: Trustworthy Knowledge-Driven AI Lab Bellini College of Artificial Intelligence, Cybersecurity and Computing University of South Florida Tampa, FL 33620

Abstract

On a chain of $N$ states, any local learning rule with bounded communication range needs $Θ(N)$ rounds before the loss can influence the earliest parameters, and this latency is usually attributed to depth. We show that it is a property of the communication geometry rather than of depth. For $N=m^d$ states placed at constant density in $d$ dimensions, a light-cone argument gives an $Ω(N^{1/d})$ lower bound on the number of bounded-range rounds that any local algorithm needs before its gradient estimates can be sign-correct at every parameter. We construct a unit-range, degree- $2d$ directed acyclic graph that contains a Hamiltonian computation of length $N-1$ and in which every state lies within directed distance $d(m-1)=Θ(N^{1/d})$ of the output, matching the lower bound. In the realizable-circuit model, depth and credit depth are therefore independent parameters: represented depth $Θ(N)$ and causal credit latency $Θ(N^{1/d})$ coexist, with a separation of order $N^{1-1/d}$ . SpatialPC realizes this separation within predictive coding. Credit is propagated by adjoint prediction dynamics (APD), a local update whose $K$ -round state on a settled acyclic graph is the truncated reverse-path sum $a^K=\sum_{k<K}(J^\top)^k b$ , with error exactly $(J^\top)^K a^\star$ . Graph geometry fixes when credit can first arrive at a state; the Jacobian-weighted path tail fixes how many further rounds accurate credit requires. A sparse two-dimensional Moore compiler retains the $N-1$ backbone with $\tfrac32N-2$ unit-range edges and has measured worst-node reachability $\sqrt N+2$ up to $N=65{,}536$ , where a 65,535-edge backbone is reached in 258 local hops. With $T=2\sqrt N$ rounds, SpatialPC keeps finite-round task gradients aligned with settled gradients (cosine $0.93$ on MNIST and $0.97$ on Fashion-MNIST at $N=256$ ) and reaches $97.6\%$ and $88.1\%$ test accuracy after 20 epochs.

Figures & tables

Panel of Figure 1 — Figure 1: Represented depth and local credit latency are different resources. (a) On a chain, represented depth and loss reachability are both $\Theta(N)$ . (b) The snake-ordered grid of Theorem 4.6 ( $d=2$ ) keeps an $N-1$ Hamiltonian computation (dark arrows) while every state lies within $2(\sqrt{N}-1)$ unit-range hops of the output; the blue route is a shortest credit path. (c) APD accumulates reverse paths by length $k$ . Geometry fixes the first round at which a state can receive credit; the Jacobian-weighted tail $(J^{\top})^{K}a^{\star}$ fixes how many further rounds accurate credit requires.

Panel of Figure 2 — Figure 2: Sparse Moore compiler for two grids. Solid arrows form the Hamiltonian backbone; dashed arrows are the cut edge and the enclosed-cell shortcuts. Every edge is unit length and respects the common topological order. The state(s) farthest from the output is state 1 at distance $6$ for the $4\times 4$ case and states 3, 5, 9 at distance $10$ for the $8\times 8$ case. Both cases satisfy $D_{\text{reach}}=m+2=\sqrt{N}+2$ .

Figure 3: Represented depth and reachability depth versus state count. A chain has

D_{\mathrm{reach}}=N-1

; the snake-ordered grid of Theorem 4.6 has

D_{\mathrm{reach}}=2(\sqrt{N}-1)

; the Moore compiler follows

D_{\mathrm{reach}}=\sqrt{N}+2

over the measured sizes, with

D_{\mathrm{seq}}=N-1

in all cases.

$\bm{m}$	$\bm{N=m^{2}}$	$\bm{\|E\|}$	$\bm{D_{\mathrm{seq}}}$	$\bm{D_{\mathrm{reach}}}$	separation
4	16	22	15	6	$2.5\times$
8	64	94	63	10	$6.3\times$
16	256	382	255	18	$14.2\times$
32	1,024	1,534	1,023	34	$30.1\times$
64	4,096	6,142	4,095	66	$62.0\times$
128	16,384	24,574	16,383	130	$126.0\times$

Table 1: Structural scaling of the Moore compiler.

|E|=\tfrac{3}{2}N-2

( Proposition 4.12 ),

D_{\mathrm{seq}}=N-1

is the backbone length,

D_{\mathrm{reach}}=\sqrt{N}+2

is the exact worst-node distance to the output, and the last column is

D_{\mathrm{seq}}/D_{\mathrm{reach}}

Dataset	Method	# of States	T	Test Acc. (%) $\uparrow$		Task-grad cos. to settled $\uparrow$
MNIST	SpatialPC-v1	16	8	91.71 $\pm$	0.14	0.8040 $\pm$	0.0122
		64	16	92.27 $\pm$	0.08	0.9271 $\pm$	0.0139
		256	32	92.75 $\pm$	0.23	0.9283 $\pm$	0.0086
	SpatialPC-v2	16	8	84.44 $\pm$	0.97	0.8379 $\pm$	0.0116
		64	16	89.90 $\pm$	0.23	0.9419 $\pm$	0.0082
		256	32	91.22 $\pm$	0.27	0.6467 $\pm$	0.0279

Table 2: Gradient fidelity under the geometric budget

T=2\sqrt{N}

as the number of states grows (

H=128

\lambda=100

, one epoch). Gradient cosine compares the finite-round APD task gradient with the settled task gradient of the same network.

Dataset	Method	$\bm{\lambda}$	Test Acc. (%) $\uparrow$		Task-grad cos. to settled $\uparrow$		Residual RMS $\downarrow$
MNIST	SpatialPC-v1	0	92.67 $\pm$	0.19	0.3525 $\pm$	0.1812	0.0245 $\pm$	0.0040
		1	92.68 $\pm$	0.20	0.5057 $\pm$	0.2094	0.0204 $\pm$	0.0025
		10	92.69 $\pm$	0.21	0.7688 $\pm$	0.0708	0.0144 $\pm$	0.0012
		30	92.73 $\pm$	0.21	0.8637 $\pm$	0.0253	0.0112 $\pm$	0.0007
		100	92.75 $\pm$	0.23	0.9283 $\pm$	0.0086	0.0079 $\pm$	0.0003
	SpatialPC-v2	0	91.27 $\pm$	0.22	0.0872 $\pm$	0.0431	0.0272 $\pm$	0.0025

Table 3: Effect of the feasibility weight

\lambda

N=256

T=32

H=128

(one epoch). Accuracy is nearly constant while residual RMS falls and gradient cosine rises.

Dataset	Method	$\bm{\lambda}$	Test Acc. (%) $\uparrow$		Task-grad cos. to settled $\uparrow$		Residual RMS $\downarrow$
MNIST	SpatialPC-v1	8	91.12 $\pm$	0.14	0.7009 $\pm$	0.0126	0.0249 $\pm$	0.0001
		16	92.16 $\pm$	0.24	0.9451 $\pm$	0.0036	0.0125 $\pm$	0.0001
		32	92.75 $\pm$	0.23	0.9283 $\pm$	0.0086	0.0079 $\pm$	0.0003
		64	93.15 $\pm$	0.21	0.9550 $\pm$	0.0131	0.0047 $\pm$	0.0005
	SpatialPC-v2	8	86.69 $\pm$	0.07	0.4624 $\pm$	0.0103	0.0169 $\pm$	0.0002
		16	89.95 $\pm$	0.17	0.7785 $\pm$	0.0186	0.0106 $\pm$	0.0000

Table 4: Effect of the round budget

T

N=256

H=128

\lambda=100

(one epoch). Every state is reachable once

T>D_{\mathrm{reach}}=18

, yet gradient fidelity keeps improving to

T=64

Panel of Figure 4 — Figure 4: Finite-round SpatialPC over 20 epochs at $N=256$ , $T=32$ . Gradient cosine compares the finite-round APD task gradient with the settled gradient of the same Moore network. The dashed accuracy curve is the depth-128 backpropagation reference.

			Accuracy (%)				Gradient Alignment
Dataset	Method	Epoch	Test $\uparrow$		Settled $\uparrow$		Task-grad cos. $\uparrow$		RMS $\bm{(\times 10^{-3})}$ $\downarrow$
MNIST	BP	1	93.40 $\pm$	0.31	–		–		–
		5	96.54 $\pm$	0.15	–		–		–
		10	97.23 $\pm$	0.32	–		–		–
		20	97.55 $\pm$	0.09	–		–		–
	SPCv1	1	92.74 $\pm$	0.23	92.58 $\pm$	0.19	0.934 $\pm$	0.005	7.9 $\pm$	0.3

Table 5: Twenty-epoch training at

N=256

T=32

H=128

(mean

\pm

std over three seeds). “Test” is finite-round accuracy from

h^{T}

and “Settled” uses the same parameters after an exact sweep; RMS is the residual RMS. The BP row is a depth-128 chain reference, for which the SpatialPC diagnostics do not apply.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Loss	Readout	Activation	Gradient cosine $\uparrow$	Relative error $\downarrow$
CE	$H^{-1/2}$	ReLU	1.0000000000	$2.83\times 10^{-16}$
CE	$H^{-1/2}$	tanh	1.0000000000	$1.54\times 10^{-16}$
CE	$H^{-1}$	ReLU	1.0000000000	$2.54\times 10^{-16}$
CE	$H^{-1}$	tanh	1.0000000000	$2.43\times 10^{-16}$
MSE	$H^{-1/2}$	ReLU	1.0000000000	$2.71\times 10^{-16}$
MSE	$H^{-1/2}$	tanh	1.0000000000	$1.48\times 10^{-16}$

Appendix

Table 6: Double-precision verification of the finite-round SpatialPC/APD gradient against automatic differentiation.

Dataset	Method	Graph	Test Acc. (%) $\uparrow$		Task-grad cos. to settled $\uparrow$
MNIST	SpatialPC-v1	Chain	9.80 $\pm$	0.00	0.0000 $\pm$	0.0000
		Moore	92.75 $\pm$	0.23	0.9283 $\pm$	0.0086
	SpatialPC-v2	Chain	9.80 $\pm$	0.00	0.0000 $\pm$	0.0000
		Moore	91.22 $\pm$	0.27	0.6467 $\pm$	0.0279
Fashion-MNIST	SpatialPC-v1	Chain	10.00 $\pm$	0.00	0.0000 $\pm$	0.0000
		Moore	82.85 $\pm$	0.18	0.9745 $\pm$	0.0008

Appendix

Table 7: Chain control at matched

N=256

T=32

H=128

. With

T<N

the chain output never receives input information, so accuracy equals the class-0 frequency and the task gradient is zero.

Dataset	Method	Activation	Test Acc. (%) $\uparrow$		Task-grad cos. to settled $\uparrow$
MNIST	SpatialPC-v1	ReLU	92.75 $\pm$	0.23	0.9283 $\pm$	0.0086
		Tanh	91.63 $\pm$	0.25	0.9628 $\pm$	0.0078
	SpatialPC-v2	ReLU	91.22 $\pm$	0.27	0.6467 $\pm$	0.0279
		Tanh	88.86 $\pm$	0.19	0.9539 $\pm$	0.0031
Fashion-MNIST	SpatialPC-v1	ReLU	82.85 $\pm$	0.18	0.9745 $\pm$	0.0008
		Tanh	82.14 $\pm$	0.13	0.9843 $\pm$	0.0037

Appendix

Table 8: Activation ablation at the reference configuration (

N=256

T=32

H=128

\lambda=100

, one epoch).

Dataset	Width	BP	$\bm{\mu}$ PC	PC	PC-ALM	SPC-v1	SPC-v2
MNIST	8	38.80	18.67	9.53	37.90	83.00	65.70
	16	75.85	35.65	9.51	71.47	87.60	77.94
	32	87.78	61.02	8.98	85.79	89.78	84.91
	64	91.51	74.98	9.75	89.78	91.42	89.18
	128	93.40	84.06	9.87	92.25	92.74	91.23
Fashion-MNIST	8	47.56	24.48	11.41	43.44	74.88	64.77

Appendix

Table 9: One-epoch test accuracy (%) at matched hidden width

H

. SpatialPC uses the

N=256

Moore graph; the chain baselines use depth 128. Batch sizes are 32 for SpatialPC and 64 for the baselines. Bold marks the best entry in each row.

Local Learning

Local learning develops neural training rules that use nearby activity or layerwise objectives instead of global backpropagated errors.

See all 52 papers

May 29, 2026

Augmented Lagrangian Predictive Coding

Predictive coding (PC) is a local-learning alternative to backpropagation (BP), training deep networks via local energy-minimization dynamics rather than a global backward pass. We introduce Augmented Lagrangian Predictive Coding (PC-ALM), which maintains PC's inference budget but aligns each weight update toward BP by accumulating per-layer constraint errors into a layer-local Lagrange multiplier. In linear PC networks, PC-ALM converges to an equilibrium with exact BP gradients distributed across the network via only layer-local updates. We analyze PC-ALM in nonlinear PC networks up to depth 128 and show that it matches BP performance across all width-depth regimes, notably in deep narrow networks where PC underperforms. PC-ALM introduces recurrent dynamics in each layer's activations. Compared to PC's heat flow on a scalar energy, PC-ALM dynamics are driven by dual ascent on the augmented Lagrangian. We observe "ballistic" credit propagation across very deep networks, with credit signals evenly distributed across layers, compared to PC's slow, diffusive credit propagation. Beyond the algorithm itself, the augmented Lagrangian framework offers a generalization of PC, and may yield insights into how distributed systems could compute and propagate BP-like credit signals through purely local dynamics.

Jeffrey Seely, Julian Gould

Sakana AI

Backpropagation Predictive Coding Augmented Lagrangian Method Neural Network Training Dynamics Local Learning

Ask PDF

Jul 15, 2026

Weight Feedback Computes the Jacobian Transpose Locally in Modern Deep Networks

Predictive Coding (PC) offers a biologically motivated alternative to backpropagation via local weight updates, yet routing error between layers still relies on an autograd Jacobian-transpose (

J^\top

) product - the last non-local operation in PC. We show that this dependency is largely avoidable. For any layer

f(x)=\mathrm{Act}(\mathrm{Norm}(L(x)))

with frozen normalization statistics, the exact

J^\top

factors into three locally available terms,

J^\top v = L^\top(s \odot σ'(z) \odot v)

, where

σ'

is the activation derivative,

z

is the pre-activation, and

s=γ/σ_{\mathrm{run}}

is the normalization gain. Prior weight-feedback methods omitted both corrections; restoring them closes the transport gap for this layer class. Locality here holds up to three assumptions, which we state upfront: weight symmetry (

L^\top

mirrors the forward operator, as assumed by all PC), a soft spectral-norm control that is not synapse-local, and a nearest-neighbour approximation for MaxPool. Substituting the identity into PC yields WF-Act-PC, which removes the autograd backward pass from error transport. On CIFAR-10/100 (50 epochs, 5 seeds), WF-Act-PC is the only PC method whose accuracy improves with depth, surpassing iPC - the strongest classical PC baseline - by 2.7-22.3 pp on CIFAR-10. With both methods tuned per architecture, it matches or exceeds a comparably-tuned backpropagation baseline on the deeper CIFAR-10 architectures (VGG-9: 93.57% vs. 92.43%; ResNet-18: 92.76% vs. 91.54%) and on the harder Tiny-ImageNet benchmark, while trailing tuned BP on the deeper CIFAR-100 VGG cells. Our WF-Act-PC implementation is publicly available at https://github.com/jlshen025/pcax

Junlong Shen, Xingyu Li

Department of Electrical and Computer Engineering, University of Alberta, Edmonton, Canada · Alberta Machine Intelligence Institute (Amii), Edmonton, Canada

Predictive Coding Local Learning

Ask PDF

Sep 11, 2026

Temporal Recurrence Favors Fewer Layers

In streaming tasks, recurrent models can carry latent computation across time, allowing each update to build on representations produced earlier. This raises a basic question: once temporal recurrence provides sequential computation across steps, how much depth is still needed within each step? Prior work has shown that recurrence can make shallow models competitive. We instead study this question as a compute-allocation problem, varying within-step depth, expert width, and the number of parallel experts per layer across several compute budgets. For each budget, we compare the best observed recurrent and non-recurrent allocations and the performance they achieve under approximately matched per-step computation. Across Sokoban and autoregressive FineWeb language modeling, we find that temporal recurrence shifts the best observed compute allocation toward substantially fewer layers, with comparable or better performance.

Ivan Anokhin, Johan Obando-Ceron, Irina Rish +1

Mila – Qu´ebec AI Institute · Universit´e de Montr´eal · Sakana AI

Recurrent Neural Networks Recurrent Language Models

Ask PDF

Abstract

Local Learning

Explore similar work

Augmented Lagrangian Predictive Coding

Weight Feedback Computes the Jacobian Transpose Locally in Modern Deep Networks

Temporal Recurrence Favors Fewer Layers