Error-Based Bidirectional Predictive Coding Enables Deep, 4-Bit Predictive Coding Networks

Garrett Kinman; Zeljko Zilic

cs.LGOct 11, 2026

Error-Based Bidirectional Predictive Coding Enables Deep, 4-Bit Predictive Coding Networks

Authors: Garrett Kinman, Zeljko Zilic

Organizations: Department of Electrical and Computer Engineering, McGill University, Montréal, Québec, Canada

Abstract

Predictive coding networks (PCNs) promise properties not offered by backpropagation-trained networks: learning from purely local information, a single network that both discriminates and generates, and flexible inference by energy minimization. However, their iterative inference is expensive, making hardware acceleration desirable, and a useful, efficient accelerator needs both depth and aggressive quantization. Canonical state-based PC delivers neither: its inference signal decays exponentially with depth, so deep models fail to train, and low precision makes it worse. We show that pairing two recent methods carries the full PCN property set into the deep, 4-bit regime: bidirectional PC (bPC), which unifies discrimination and generation, and the error-based reparameterization (ePC), which optimizes prediction errors rather than states during inference. Error optimization delivers the output signal to every layer in a single backward pass, removing the signal decay; one additional element, global-norm clipping of the error gradients, stabilizes the error dynamics at depth. The resulting ePC+bPC model trains stably up to at least eight layers and retains its full-precision accuracy and clean class-conditional generation under W4A4E4 quantization-aware training, across FP4 and three posit formats. On six-layer networks, where state-based bPC collapses to chance, ePC+bPC classifies more than 50 accuracy points above it after one epoch of training, with class-conditional generations 5-9x better in RMSE. 4-bit quantization is nearly free: full-precision and 4-bit accuracies differ by at most ~5 points, and by under two in the best-performing formats. We characterize the two failure modes the method overcomes, bounded signal decay in state-based PC and error divergence in unregularized error optimization, and discuss implications for a 4-bit-native PCN accelerator.

Figures & tables

Format	Positive levels	Range	Worst gap in $[0.125,2]$
FP4 E2M1	$0.5,\ 1,\ 1.5,\ 2,\ 3,\ 4,\ 6$	$\pm[0.5,6]$	$2\times$
Posit $\langle$ 4,0 $\rangle$	$0.25,\ 0.5,\ 0.75,\ 1,\ 1.5,\ 2,\ 4$	$\pm[0.25,4]$	$2\times$
Posit $\langle$ 4,1 $\rangle$	$0.0625,\ 0.25,\ 0.5,\ 1,\ 2,\ 4,\ 16$	$\pm[0.0625,16]$	$4\times$
Posit $\langle$ 4,2 $\rangle$	$0.0039,\ 0.0625,\ 0.125,\ 1,\ 2,\ 16,\ 256$	$\pm[0.0039,256]$	$8\times$

Table 1: The four 4-bit formats, given as their seven positive representable levels (each format also represents zero and the seven negatives, for 15 values in total). The last column gives the largest ratio between adjacent levels over

[\,0.125,\,2\,]

, the band holding most of the mass once tensors are RMS-normalized; larger means a coarser grid where the data sits. Posit

\langle

4,2

\rangle

buys its wide dynamic range with an eightfold gap there.

Figure 1: Test accuracy vs. network depth (mean

\pm

std over 5 seeds) on MNIST (top) and Fashion-MNIST (bottom), at full precision (left) and under W4A4E4 Posit

\langle

4,2

\rangle

(right). On both datasets and both precisions, state-based bPC collapses to chance for

L\geq 5

(halving already at

L\!=\!4

) as its inference signal decays with depth, while ePC+bPC stays far above it at every depth, with only a gentle decline at the deepest layers. ePC alone (discriminative; dashed) is an accuracy ceiling but cannot generate.

	Method	FP32	FP4 E2M1	Posit $\langle$ 4,2 $\rangle$	Posit $\langle$ 4,1 $\rangle$	Posit $\langle$ 4,0 $\rangle$
		Acc / RMSE	Acc / RMSE	Acc / RMSE	Acc / RMSE	Acc / RMSE
MNIST	bPC, state-based	$10.2$ / $0.45$	$12.0$ / $0.58$	$14.1$ / $0.50$	$13.5$ / $0.50$	$11.2$ / $0.50$
	ePC (no bidirect.)	$96.6$ / —	—	$95.5$ / —	—	—
	ePC+bPC (ours)	$\mathbf{82.8}$ / $\mathbf{0.09}$	$\mathbf{79.6}$ / $\mathbf{0.08}$	$\mathbf{77.5}$ / $\mathbf{0.09}$	$\mathbf{80.1}$ / $\mathbf{0.10}$	$\mathbf{82.4}$ / $\mathbf{0.09}$
Fashion	bPC, state-based	$10.0$ / $0.59$	$13.6$ / $0.66$	$12.5$ / $0.64$	$13.0$ / $0.61$	$12.3$ / $0.64$
	ePC (no bidirect.)	$85.0$ / —	—	$83.9$ / —	—	—

Table 2: Test accuracy (%) and generative RMSE (lower better) at depth

L\!=\!6

, mean

\pm

std over 5 seeds. ePC (discriminative) has no generative pathway and was run only at FP32 and Posit

\langle

4,2

\rangle

Figure 2: Class-conditional MNIST generations at depth $L\!=\!6$ . Top four rows: state-based bPC; bottom four rows: ePC+bPC; one row per number format. ePC+bPC generates clean digits at FP32 and all 4-bit formats; state-based bPC, unable to train at this depth, produces label-independent noise.

Figure 3: Class-conditional Fashion-MNIST generations at depth $L\!=\!6$ (same layout as Fig. 2 ). ePC+bPC renders recognizable garments across all precisions; state-based bPC collapses to noise.

Figure 4: Converged max-layer inference-error magnitude vs. depth (averaged over formats and seeds). State-based bPC errors stay bounded (

O(1)

); it fails by signal decay, not blow-up. Unregularized ePC+bPC errors diverge, growing from

\sim

0.06 at

L\!=\!2

\sim\!10^{3}

L\!=\!6

and

\sim\!10^{5}

L\!=\!8

. Global-norm clipping (ours) holds them at

\sim\!10^{-4}

across all depths.

Configuration	Test Acc (%)	Generation
state-based bPC (no ePC)	$14.1$	poor (RMSE 0.50)
ePC alone (no bidirectionality)	$95.5$	— (none)
ePC+bPC, no gradient clipping	$13.5$	diverged
ePC+bPC (full)	$\mathbf{77.5}$	good (RMSE 0.09)

Table 3: Ablation at

L\!=\!6

, W4A4E4 Posit

\langle

4,2

\rangle

(mean over 5 seeds).

Predictive Coding

Predictive coding explains perception and learning through predictions, prediction errors, and updates to internal models.

See all 59 papers

Jun 22, 2026

Error Highways: Scaling Predictive Coding to Very Deep Networks

Predictive coding networks (PCNs) offer a biologically-plausible, local-learning alternative to back-propagation of errors (backprop). Nevertheless, they have remained largely confined to shallow architectures and evaluated on simple machine intelligence benchmarks. A central obstacle to scaling PCNs is that the learning signal decays rapidly as it propagates away from the clamped boundaries, leaving interior layers effectively unchanged. To directly counter this problem, we propose highway error propagation (HEP), a scheme that augments the free energy function underlying predictive coding (PC) by altering its neural structure with feedback matrices

V_{L\to i}

that couple selected hidden states directly to the clamped output error. Since this coupling is linear in the hidden state, the highway pathway delivers a correction at every inference step whose magnitude is independent of depth, in contrast to vanilla PC where the output error reaches the

i

-th hidden layer with attenuation that decays exponentially in depth. This bypasses the Jacobian chain while preserving the local PC synaptic update rule. On MNIST and Fashion-MNIST, we show that HEP effectively trains MLPs of up to 128 layers with accuracy that is robust with respect to depth.

Amirhossein Mohammadi, Alexander G. Ororbia

SingularityNET Zug, Switzerland · Rochester Institute of Technology Rochester, NY, USA

Predictive Coding Local Learning

Ask PDF

May 19, 2026

Closed-form predictive coding via hierarchical Gaussian filters

Predictive coding (PC) offers a local and biologically grounded alternative to backpropagation in the training of artificial neural networks, yet to date, it remains slower, and performance degrades sharply as network depth increases. We trace both problems to a single simplification: current PC networks fix the precision matrix to the identity, discarding precision-weighted prediction errors that the variational derivation requires to be fast, local, and Bayesian. We close this gap by expressing predictive coding networks as deep hierarchical Gaussian filters (HGFs) and restore precision-weighted message passing, yielding dynamic uncertainty estimates and Hebbian-compatible update rules at every layer. The resulting networks can simultaneously learn activations, weights, and precisions under a single free-energy objective, with no global error signal, and resolve inference without requiring iterations or automatic differentiation. On FashionMNIST, our solution approaches backpropagation in epoch-level wall-clock cost while converging in fewer epochs, and outperforms it on online, data efficiency, and concept-drift tasks. We thus establish that closed-form variational inference with online precision learning provides a tractable foundation for deep predictive coding networks, retaining biological and interpretative advantages, without requiring iterative relaxation or global error signals.

Aleksandrs Baskakovs, Sylvain Estebe, Kenneth Enevoldsen +3

Center for Humanities Computing Aarhus University Aarhus, Denmark · Interacting Minds Center Aarhus University Aarhus, Denmark

Predictive Coding Variational Inference Local Learning

Ask PDF

May 29, 2026

Augmented Lagrangian Predictive Coding

Predictive coding (PC) is a local-learning alternative to backpropagation (BP), training deep networks via local energy-minimization dynamics rather than a global backward pass. We introduce Augmented Lagrangian Predictive Coding (PC-ALM), which maintains PC's inference budget but aligns each weight update toward BP by accumulating per-layer constraint errors into a layer-local Lagrange multiplier. In linear PC networks, PC-ALM converges to an equilibrium with exact BP gradients distributed across the network via only layer-local updates. We analyze PC-ALM in nonlinear PC networks up to depth 128 and show that it matches BP performance across all width-depth regimes, notably in deep narrow networks where PC underperforms. PC-ALM introduces recurrent dynamics in each layer's activations. Compared to PC's heat flow on a scalar energy, PC-ALM dynamics are driven by dual ascent on the augmented Lagrangian. We observe "ballistic" credit propagation across very deep networks, with credit signals evenly distributed across layers, compared to PC's slow, diffusive credit propagation. Beyond the algorithm itself, the augmented Lagrangian framework offers a generalization of PC, and may yield insights into how distributed systems could compute and propagate BP-like credit signals through purely local dynamics.

Jeffrey Seely, Julian Gould

Sakana AI

Backpropagation Predictive Coding Augmented Lagrangian Method Neural Network Training Dynamics Local Learning

Ask PDF

Abstract

Predictive Coding

Explore similar work

Error Highways: Scaling Predictive Coding to Very Deep Networks

Closed-form predictive coding via hierarchical Gaussian filters

Augmented Lagrangian Predictive Coding