Accelerated Predictive Coding Networks via Direct Kolen-Pollack Feedback Alignment

Davide Casnici; Martin Lefebvre; Justin Dauwels; Charlotte Frenkel

cs.LGFeb 17, 2026

Accelerated Predictive Coding Networks via Direct Kolen-Pollack Feedback Alignment

Authors: Davide Casnici, Martin Lefebvre, Justin Dauwels, Charlotte Frenkel

Organizations: Department of Microelectronics, Delft University of Technology, Delft, Netherlands.

Abstract

Predictive coding (PC) is a biologically inspired algorithm for training neural networks that relies only on local updates, allowing parallel learning across layers. However, practical implementations face two key limitations: error signals must still propagate from the output to early layers through multiple inference-phase steps, and feedback decays exponentially during this process, leading to vanishing updates in early layers. We propose direct Kolen-Pollack predictive coding (DKP-PC), which simultaneously addresses both feedback delay and exponential decay, yielding a more efficient and scalable variant of PC while preserving update locality. Leveraging direct feedback alignment and direct Kolen-Pollack algorithms, DKP-PC introduces learnable feedback connections from the output layer to all hidden layers, establishing a direct pathway for error transmission. This yields an algorithm that reduces the theoretical error propagation time complexity from O(L), with L being the network depth, to O(1), removing depth-dependent delay in error signals. Moreover, empirical results demonstrate that DKP-PC achieves performance at least comparable to, and often exceeding, that of standard PC, while offering improved latency and computational performance, supporting its potential for custom hardware-efficient implementations.

Figures & tables

Figure 1: DKP-PC embeds DKP within the PC framework to address the error feedback delay and exponential decay issues of PC. Blue arrows represent forward connections, red arrows represent feedback connections. Neural activities are shown as gray circles, with clamped values in darker gray;

x_{0}

denotes the input and

y

the target.

\mathcal{L}

is the loss function,

\delta_{\ell}

are the BP errors,

\tilde{\delta}_{\ell}

their approximations, and

\epsilon_{\ell}

the PC error neurons, represented as triangles. (A) BP propagates the global error sequentially. (B) DFA and DKP propagate the error directly from the output to each layer. (C) PC minimizes local errors through an inference phase, followed by a learning phase that updates weights. (D) DKP-PC employs direct feedback to deliver instantaneous error signals to every layer, accelerating error propagation during the PC inference phase while preserving the locality of weight updates.

Figure 2: Error propagation in PC (A) and DKP-PC (B) during the inference phase of a VGG-9 network trained on a single CIFAR-10 batch, at different magnitudes of the neural activity learning rate $\gamma$ . In (A), PC exhibits both an error decay problem, where the error magnitude decreases exponentially with network depth, and an error delay problem, as the error signal flows through the network sequentially, undermining the theoretical parallelism. White colour represents values equal to zero or below the numerical precision. In (B), DKP-PC mitigates both issues, generating a more uniform error signal across all layers at the start of neural activity optimization.

% Accuracy	DKP	PC	iPC	CN-PC	DKP-PC	BP
MLP
MNIST	$98.03^{\pm{0.10}}$	$98.26^{\pm{0.04}}$	$\mathbf{98.45^{\pm{0.09}}}$	$98.23^{\pm 0.09}$	$98.02^{\pm{0.09}}$	$98.29^{\pm{0.08}}$
FashionMNIST	$88.86^{\pm{0.13}}$	$89.58^{\pm{0.13}}$	$\mathbf{89.90^{\pm{0.06}}}$	$89.56^{\pm 0.05}$	$89.42^{\pm{0.25}}$	$89.48^{\pm{0.07}}$
VGG-7
CIFAR-10	$77.98^{\pm{0.39}}$	$81.91^{\pm{0.30}}$	$80.15^{\pm{0.18}}$	$\mathbf{88.40^{\pm 0.12}}$	$82.36^{\pm{0.18}}$	$89.91^{\pm{0.10}}$
CIFAR-100 (Top-1)	$36.96^{\pm{0.62}}$	$37.52^{\pm{2.60}}$	$43.99^{\pm{0.30}}$	$\mathbf{64.76^{\pm 0.17}}$	$50.42^{\pm{0.38}}$	$65.36^{\pm{0.15}}$

Table 1: Test accuracy in % (mean

\pm

standard deviation) averaged over 5 random seeds. Results for PC, iPC, CN-PC and BP are taken from Pinchetti et al. (2024) . The best results among local algorithms are highlighted in bold.

Seconds	DKP	PC	iPC	DKP-PC	BP
MLP
MNIST	$4.70^{\pm{0.10}}$	$4.71^{\pm{0.06}}$	$4.79^{\pm{0.09}}$	$4.74^{\pm{0.09}}$	$4.70^{\pm{0.06}}$
FashionMNIST	$4.62^{\pm{0.06}}$	$4.77^{\pm{0.13}}$	$5.07^{\pm{0.13}}$	$4.70^{\pm{0.14}}$	$4.62^{\pm{0.07}}$
VGG-7
CIFAR-10	$7.21^{\pm{0.26}}$	$31.39^{\pm{0.20}}$	$54.48^{\pm{0.12}}$	$11.13^{\pm{0.09}}$	$7.27^{\pm{0.19}}$
CIFAR-100	$7.11^{\pm{0.06}}$	$31.48^{\pm{0.17}}$	$54.69^{\pm{0.22}}$	$11.67^{\pm{0.04}}$	$7.15^{\pm{0.04}}$

Table 2: Training time per epoch in seconds (mean

\pm

standard deviation), averaged over 5 trials. All results were obtained via sequential execution of the algorithms.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Forward weight gradients alignment across layers of a VGG9-like CNN trained for 50 epochs on CIFAR-100. Each curve shows the cosine similarity between the instantaneous forward-weight gradient produced by DKP and DKP-PC algorithms, compared to the one computed with BP. All gradients exclude weight decay and momentum and are smoothed using an exponential moving average with a window of 100 batches. DKP (brown) displays positive but slow alignment with BP, progressively deteriorating with increasing distance from the output layer. DKP-PC (yellow) gradient is computed as sum of the gradients resulting from DKP and PC stages. It achieves consistently faster, higher, and more stable alignment across all layers compared to standard DKP. The light blue curve shows that disabling the PC forward-weight update in DKP-PC causes alignment to collapse in all layers, confirming its role in injecting alignment information into the forward weights. The blue curve, obtained by disabling the feedback-weight update in DKP-PC, demonstrates that the alignment and regularization terms introduced by the PC stage also improve the update of the feedback matrices, resulting in worse alignment when disabled.

Figure 4: Energy evolution of four-layer MLP networks on a Fashion-MNIST batch for three different neural activity learning rate magnitudes. DKP-PC and its incremental variant are shown in blue and light blue, respectively, while standard PC and iPC are represented in brown and yellow. Both DKP-PC variants start from higher energy values due to the immediate error term at every layer, and converge to levels similar to those of the standard PC and iPC networks. Although DKP-PC exhibits slower convergence due to the additional terms in the neural activity dynamics, its incremental version equals the convergence speed of iPC across all evaluated learning rates, suggesting that updating forward parameters during inference effectively compensates for the additional complexity introduced by DKP.

Figure 5: Test accuracy distributions over 30 trials are shown as a function of the total number of neural activity optimization steps. Blue boxplots correspond to a four-layer MLP trained on Fashion-MNIST with DKP-PC, while light blue boxplots show the same architecture trained with iDKP-PC. In line with PC theory, both methods display a positive correlation between the number of optimization steps and the final test accuracy.

	MLP	VGG-7	VGG-9
Conv. channels	–	[128x2, 256x2, 512x2]	[128x2, 256x2, 512x2]
Kernel sizes	–	[3, 3, 3, 3, 3, 3]	[3, 3, 3, 3, 3, 3]
Strides	–	[1, 1, 1, 1, 1, 1]	[1, 1, 1, 1, 1, 1]
Paddings	–	[1, 1, 1, 0, 1, 0]	[1, 1, 1, 1, 1, 1]
Pool window	–	$2\times 2$	$2\times 2$
Pool stride	–	2	2

Appendix

Table 3: Architectural details. FC size refers to the number of units in the fully-connected layers after the convolutional ones (if any).

	MLP		VGG-7		VGG-9
	MNIST	FMNIST	CIFAR-10	CIFAR-100	CIFAR-10	CIFAR-100	Tiny ImageNet
activation	gelu	gelu	gelu	tanh	leaky	leaky	gelu
fw-lr	$4.616\mathrm{e}{-4}$	$5.254\mathrm{e}{-4}$	$1.458\mathrm{e}{-4}$	$2.482\mathrm{e}{-4}$	$1.609\mathrm{e}{-4}$	$1.602\mathrm{e}{-4}$	$7.373\mathrm{e}{-5}$
fw-decay	$3.737\mathrm{e}{-2}$	$2.744\mathrm{e}{-5}$	$3.626\mathrm{e}{-4}$	$9.664\mathrm{e}{-2}$	$5.271\mathrm{e}{-2}$	$1.040\mathrm{e}{-2}$	$2.893\mathrm{e}{-5}$
fw-opt	adamw	adamw	adam	adam	adam	adam	adamw
i-lr	$1.068\mathrm{e}{-3}$	$8.297\mathrm{e}{-1}$	$5.655\mathrm{e}{-2}$	$1.036\mathrm{e}{-2}$	$1.113\mathrm{e}{-3}$	$1.169\mathrm{e}{-2}$	$3.136\mathrm{e}{-3}$

Appendix

Table 4: Hyperparameters employed for the DKP-PC networks in our experiments.

Figure 6: The first row compares the training time on a logarithmic scale, measured as the sum of the forward pass and the complete parameter updates, with the contribution of the forward pass illustrated as a red dashed line. Results are averaged over 20 samples. The second row compares the minimum FLOPs requirements, estimated from the core MAC operations. In all plots, for both PC and iPC, the number of inference-phase steps is assumed to equal the network depth. In contrast, for DKP-PC, only a single step is considered, as it has been empirically demonstrated to be enough to achieve comparable performance.

Predictive Coding

Predictive coding explains perception and learning through predictions, prediction errors, and updates to internal models.

See all 59 papers

Error Highways: Scaling Predictive Coding to Very Deep Networks
Jun 22, 2026Amirhossein Mohammadi, Alexander G. OrorbiaPredictive Coding Local Learning
SingularityNET Zug, Switzerland · Rochester Institute of Technology Rochester, NY, USA
Augmented Lagrangian Predictive Coding
May 29, 2026Jeffrey Seely, Julian GouldBackpropagation Predictive Coding
Sakana AI
Closed-form predictive coding via hierarchical Gaussian filters
May 19, 2026Aleksandrs Baskakovs, Sylvain Estebe, Kenneth Enevoldsen +3Predictive Coding Variational Inference
Center for Humanities Computing Aarhus University Aarhus, Denmark · Interacting Minds Center Aarhus University Aarhus, Denmark

Jun 22, 2026J/K move · Enter open · S save

Error Highways: Scaling Predictive Coding to Very Deep Networks

Amirhossein Mohammadi, Alexander G. Ororbia

SingularityNET Zug, Switzerland · Rochester Institute of Technology Rochester, NY, USA

Predictive Coding Local Learning

Open paper AskPDF arXiv

Predictive coding networks (PCNs) offer a biologically-plausible, local-learning alternative to back-propagation of errors (backprop). Nevertheless, they have remained largely confined to shallow architectures and evaluated on simple machine intelligence benchmarks. A central obstacle to scaling PCNs is that the learning signal decays rapidly as it propagates away from the clamped boundaries, leaving interior layers effectively unchanged. To directly counter this problem, we propose highway error propagation (HEP), a scheme that augments the free energy function underlying predictive coding (PC) by altering its neural structure with feedback matrices

V_{L\to i}

that couple selected hidden states directly to the clamped output error. Since this coupling is linear in the hidden state, the highway pathway delivers a correction at every inference step whose magnitude is independent of depth, in contrast to vanilla PC where the output error reaches the

i

-th hidden layer with attenuation that decays exponentially in depth. This bypasses the Jacobian chain while preserving the local PC synaptic update rule. On MNIST and Fashion-MNIST, we show that HEP effectively trains MLPs of up to 128 layers with accuracy that is robust with respect to depth.

Abstract

Predictive Coding

Explore similar work

Error Highways: Scaling Predictive Coding to Very Deep Networks

Augmented Lagrangian Predictive Coding

Closed-form predictive coding via hierarchical Gaussian filters