Accelerated Predictive Coding Networks via Direct Kolen-Pollack Feedback Alignment

Davide Casnici; Martin Lefebvre; Justin Dauwels; Charlotte Frenkel

cs.LGFeb 17, 2026

Accelerated Predictive Coding Networks via Direct Kolen-Pollack Feedback Alignment

Authors: Davide Casnici, Martin Lefebvre, Justin Dauwels, Charlotte Frenkel

Organizations: Department of Microelectronics, Delft University of Technology, Delft, Netherlands.

Abstract

Predictive coding (PC) is a biologically inspired algorithm for training neural networks that relies only on local updates, allowing parallel learning across layers. However, practical implementations face two key limitations: error signals must still propagate from the output to early layers through multiple inference-phase steps, and feedback decays exponentially during this process, leading to vanishing updates in early layers. We propose direct Kolen-Pollack predictive coding (DKP-PC), which simultaneously addresses both feedback delay and exponential decay, yielding a more efficient and scalable variant of PC while preserving update locality. Leveraging direct feedback alignment and direct Kolen-Pollack algorithms, DKP-PC introduces learnable feedback connections from the output layer to all hidden layers, establishing a direct pathway for error transmission. This yields an algorithm that reduces the theoretical error propagation time complexity from O(L), with L being the network depth, to O(1), removing depth-dependent delay in error signals. Moreover, empirical results demonstrate that DKP-PC achieves performance at least comparable to, and often exceeding, that of standard PC, while offering improved latency and computational performance, supporting its potential for custom hardware-efficient implementations.

Figures & tables

Figure 1: DKP-PC embeds DKP within the PC framework to address the error feedback delay and exponential decay issues of PC. Blue arrows represent forward connections, red arrows represent feedback connections. Neural activities are shown as gray circles, with clamped values in darker gray;

x_{0}

denotes the input and

y

the target.

\mathcal{L}

is the loss function,

\delta_{\ell}

are the BP errors,

\tilde{\delta}_{\ell}

their approximations, and

\epsilon_{\ell}

the PC error neurons, represented as triangles. (A) BP propagates the global error sequentially. (B) DFA and DKP propagate the error directly from the output to each layer. (C) PC minimizes local errors through an inference phase, followed by a learning phase that updates weights. (D) DKP-PC employs direct feedback to deliver instantaneous error signals to every layer, accelerating error propagation during the PC inference phase while preserving the locality of weight updates.

Figure 2: Error propagation in PC (A) and DKP-PC (B) during the inference phase of a VGG-9 network trained on a single CIFAR-10 batch, at different magnitudes of the neural activity learning rate $\gamma$ . In (A), PC exhibits both an error decay problem, where the error magnitude decreases exponentially with network depth, and an error delay problem, as the error signal flows through the network sequentially, undermining the theoretical parallelism. White colour represents values equal to zero or below the numerical precision. In (B), DKP-PC mitigates both issues, generating a more uniform error signal across all layers at the start of neural activity optimization.

% Accuracy	DKP	PC	iPC	CN-PC	DKP-PC	BP
MLP
MNIST	$98.03^{\pm{0.10}}$	$98.26^{\pm{0.04}}$	$\mathbf{98.45^{\pm{0.09}}}$	$98.23^{\pm 0.09}$	$98.02^{\pm{0.09}}$	$98.29^{\pm{0.08}}$
FashionMNIST	$88.86^{\pm{0.13}}$	$89.58^{\pm{0.13}}$	$\mathbf{89.90^{\pm{0.06}}}$	$89.56^{\pm 0.05}$	$89.42^{\pm{0.25}}$	$89.48^{\pm{0.07}}$
VGG-7
CIFAR-10	$77.98^{\pm{0.39}}$	$81.91^{\pm{0.30}}$	$80.15^{\pm{0.18}}$	$\mathbf{88.40^{\pm 0.12}}$	$82.36^{\pm{0.18}}$	$89.91^{\pm{0.10}}$
CIFAR-100 (Top-1)	$36.96^{\pm{0.62}}$	$37.52^{\pm{2.60}}$	$43.99^{\pm{0.30}}$	$\mathbf{64.76^{\pm 0.17}}$	$50.42^{\pm{0.38}}$	$65.36^{\pm{0.15}}$

Table 1: Test accuracy in % (mean

\pm

standard deviation) averaged over 5 random seeds. Results for PC, iPC, CN-PC and BP are taken from Pinchetti et al. (2024) . The best results among local algorithms are highlighted in bold.

Seconds	DKP	PC	iPC	DKP-PC	BP
MLP
MNIST	$4.70^{\pm{0.10}}$	$4.71^{\pm{0.06}}$	$4.79^{\pm{0.09}}$	$4.74^{\pm{0.09}}$	$4.70^{\pm{0.06}}$
FashionMNIST	$4.62^{\pm{0.06}}$	$4.77^{\pm{0.13}}$	$5.07^{\pm{0.13}}$	$4.70^{\pm{0.14}}$	$4.62^{\pm{0.07}}$
VGG-7
CIFAR-10	$7.21^{\pm{0.26}}$	$31.39^{\pm{0.20}}$	$54.48^{\pm{0.12}}$	$11.13^{\pm{0.09}}$	$7.27^{\pm{0.19}}$
CIFAR-100	$7.11^{\pm{0.06}}$	$31.48^{\pm{0.17}}$	$54.69^{\pm{0.22}}$	$11.67^{\pm{0.04}}$	$7.15^{\pm{0.04}}$

Table 2: Training time per epoch in seconds (mean

\pm

standard deviation), averaged over 5 trials. All results were obtained via sequential execution of the algorithms.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Forward weight gradients alignment across layers of a VGG9-like CNN trained for 50 epochs on CIFAR-100. Each curve shows the cosine similarity between the instantaneous forward-weight gradient produced by DKP and DKP-PC algorithms, compared to the one computed with BP. All gradients exclude weight decay and momentum and are smoothed using an exponential moving average with a window of 100 batches. DKP (brown) displays positive but slow alignment with BP, progressively deteriorating with increasing distance from the output layer. DKP-PC (yellow) gradient is computed as sum of the gradients resulting from DKP and PC stages. It achieves consistently faster, higher, and more stable alignment across all layers compared to standard DKP. The light blue curve shows that disabling the PC forward-weight update in DKP-PC causes alignment to collapse in all layers, confirming its role in injecting alignment information into the forward weights. The blue curve, obtained by disabling the feedback-weight update in DKP-PC, demonstrates that the alignment and regularization terms introduced by the PC stage also improve the update of the feedback matrices, resulting in worse alignment when disabled.

Figure 4: Energy evolution of four-layer MLP networks on a Fashion-MNIST batch for three different neural activity learning rate magnitudes. DKP-PC and its incremental variant are shown in blue and light blue, respectively, while standard PC and iPC are represented in brown and yellow. Both DKP-PC variants start from higher energy values due to the immediate error term at every layer, and converge to levels similar to those of the standard PC and iPC networks. Although DKP-PC exhibits slower convergence due to the additional terms in the neural activity dynamics, its incremental version equals the convergence speed of iPC across all evaluated learning rates, suggesting that updating forward parameters during inference effectively compensates for the additional complexity introduced by DKP.

Figure 5: Test accuracy distributions over 30 trials are shown as a function of the total number of neural activity optimization steps. Blue boxplots correspond to a four-layer MLP trained on Fashion-MNIST with DKP-PC, while light blue boxplots show the same architecture trained with iDKP-PC. In line with PC theory, both methods display a positive correlation between the number of optimization steps and the final test accuracy.

	MLP	VGG-7	VGG-9
Conv. channels	–	[128x2, 256x2, 512x2]	[128x2, 256x2, 512x2]
Kernel sizes	–	[3, 3, 3, 3, 3, 3]	[3, 3, 3, 3, 3, 3]
Strides	–	[1, 1, 1, 1, 1, 1]	[1, 1, 1, 1, 1, 1]
Paddings	–	[1, 1, 1, 0, 1, 0]	[1, 1, 1, 1, 1, 1]
Pool window	–	$2\times 2$	$2\times 2$
Pool stride	–	2	2

Appendix

Table 3: Architectural details. FC size refers to the number of units in the fully-connected layers after the convolutional ones (if any).

	MLP		VGG-7		VGG-9
	MNIST	FMNIST	CIFAR-10	CIFAR-100	CIFAR-10	CIFAR-100	Tiny ImageNet
activation	gelu	gelu	gelu	tanh	leaky	leaky	gelu
fw-lr	$4.616\mathrm{e}{-4}$	$5.254\mathrm{e}{-4}$	$1.458\mathrm{e}{-4}$	$2.482\mathrm{e}{-4}$	$1.609\mathrm{e}{-4}$	$1.602\mathrm{e}{-4}$	$7.373\mathrm{e}{-5}$
fw-decay	$3.737\mathrm{e}{-2}$	$2.744\mathrm{e}{-5}$	$3.626\mathrm{e}{-4}$	$9.664\mathrm{e}{-2}$	$5.271\mathrm{e}{-2}$	$1.040\mathrm{e}{-2}$	$2.893\mathrm{e}{-5}$
fw-opt	adamw	adamw	adam	adam	adam	adam	adamw
i-lr	$1.068\mathrm{e}{-3}$	$8.297\mathrm{e}{-1}$	$5.655\mathrm{e}{-2}$	$1.036\mathrm{e}{-2}$	$1.113\mathrm{e}{-3}$	$1.169\mathrm{e}{-2}$	$3.136\mathrm{e}{-3}$

Appendix

Table 4: Hyperparameters employed for the DKP-PC networks in our experiments.

Figure 6: The first row compares the training time on a logarithmic scale, measured as the sum of the forward pass and the complete parameter updates, with the contribution of the forward pass illustrated as a red dashed line. Results are averaged over 20 samples. The second row compares the minimum FLOPs requirements, estimated from the core MAC operations. In all plots, for both PC and iPC, the number of inference-phase steps is assumed to equal the network depth. In contrast, for DKP-PC, only a single step is considered, as it has been empirically demonstrated to be enough to achieve comparable performance.

Predictive Coding

Predictive coding explains perception and learning through predictions, prediction errors, and updates to internal models.

See all 59 papers

Jun 22, 2026

Error Highways: Scaling Predictive Coding to Very Deep Networks

Predictive coding networks (PCNs) offer a biologically-plausible, local-learning alternative to back-propagation of errors (backprop). Nevertheless, they have remained largely confined to shallow architectures and evaluated on simple machine intelligence benchmarks. A central obstacle to scaling PCNs is that the learning signal decays rapidly as it propagates away from the clamped boundaries, leaving interior layers effectively unchanged. To directly counter this problem, we propose highway error propagation (HEP), a scheme that augments the free energy function underlying predictive coding (PC) by altering its neural structure with feedback matrices

V_{L\to i}

that couple selected hidden states directly to the clamped output error. Since this coupling is linear in the hidden state, the highway pathway delivers a correction at every inference step whose magnitude is independent of depth, in contrast to vanilla PC where the output error reaches the

i

-th hidden layer with attenuation that decays exponentially in depth. This bypasses the Jacobian chain while preserving the local PC synaptic update rule. On MNIST and Fashion-MNIST, we show that HEP effectively trains MLPs of up to 128 layers with accuracy that is robust with respect to depth.

Amirhossein Mohammadi, Alexander G. Ororbia

SingularityNET Zug, Switzerland · Rochester Institute of Technology Rochester, NY, USA

Predictive Coding Local Learning

Ask PDF

May 29, 2026

Augmented Lagrangian Predictive Coding

Predictive coding (PC) is a local-learning alternative to backpropagation (BP), training deep networks via local energy-minimization dynamics rather than a global backward pass. We introduce Augmented Lagrangian Predictive Coding (PC-ALM), which maintains PC's inference budget but aligns each weight update toward BP by accumulating per-layer constraint errors into a layer-local Lagrange multiplier. In linear PC networks, PC-ALM converges to an equilibrium with exact BP gradients distributed across the network via only layer-local updates. We analyze PC-ALM in nonlinear PC networks up to depth 128 and show that it matches BP performance across all width-depth regimes, notably in deep narrow networks where PC underperforms. PC-ALM introduces recurrent dynamics in each layer's activations. Compared to PC's heat flow on a scalar energy, PC-ALM dynamics are driven by dual ascent on the augmented Lagrangian. We observe "ballistic" credit propagation across very deep networks, with credit signals evenly distributed across layers, compared to PC's slow, diffusive credit propagation. Beyond the algorithm itself, the augmented Lagrangian framework offers a generalization of PC, and may yield insights into how distributed systems could compute and propagate BP-like credit signals through purely local dynamics.

Jeffrey Seely, Julian Gould

Sakana AI

Backpropagation Predictive Coding Augmented Lagrangian Method Neural Network Training Dynamics Local Learning

Ask PDF

May 19, 2026

Closed-form predictive coding via hierarchical Gaussian filters

Predictive coding (PC) offers a local and biologically grounded alternative to backpropagation in the training of artificial neural networks, yet to date, it remains slower, and performance degrades sharply as network depth increases. We trace both problems to a single simplification: current PC networks fix the precision matrix to the identity, discarding precision-weighted prediction errors that the variational derivation requires to be fast, local, and Bayesian. We close this gap by expressing predictive coding networks as deep hierarchical Gaussian filters (HGFs) and restore precision-weighted message passing, yielding dynamic uncertainty estimates and Hebbian-compatible update rules at every layer. The resulting networks can simultaneously learn activations, weights, and precisions under a single free-energy objective, with no global error signal, and resolve inference without requiring iterations or automatic differentiation. On FashionMNIST, our solution approaches backpropagation in epoch-level wall-clock cost while converging in fewer epochs, and outperforms it on online, data efficiency, and concept-drift tasks. We thus establish that closed-form variational inference with online precision learning provides a tractable foundation for deep predictive coding networks, retaining biological and interpretative advantages, without requiring iterative relaxation or global error signals.

Aleksandrs Baskakovs, Sylvain Estebe, Kenneth Enevoldsen +3

Center for Humanities Computing Aarhus University Aarhus, Denmark · Interacting Minds Center Aarhus University Aarhus, Denmark

Predictive Coding Variational Inference Local Learning

Ask PDF

Abstract

Predictive Coding

Explore similar work

Error Highways: Scaling Predictive Coding to Very Deep Networks

Augmented Lagrangian Predictive Coding

Closed-form predictive coding via hierarchical Gaussian filters