Analog-Friendly Predictive Coding without Activation Derivatives

Francesco Innocenti

cs.LGSep 26, 2026

Analog-Friendly Predictive Coding without Activation Derivatives

Authors: Francesco Innocenti

Organizations: Brain Network Dynamics Unit University of Oxford, UK

Abstract

Predictive coding (PC) is a local, energy-based alternative to backpropagation (BP) whose iterative inference dynamics make it attractive for implementation on analog hardware. However, standard nonlinear PC requires evaluating the derivative of the activation function during both inference and learning, which can be difficult to realise physically. Here, we introduce \textit{activation-matched Bregman PC}, replacing standard squared-error energies with Bregman divergences matched to the activation function. This formulation eliminates activation derivatives and, when combined with inference via mirror descent, yields local inference and learning rules requiring only weighted sums, local prediction errors, state integration, and the activation function. In digital experiments, Bregman PC performs comparably to standard PC and BP on classification and generative tasks, while preserving characteristic learning dynamics of PC and its convergence to BP under stable large-model parameterisations. These results provide a more analog-friendly formulation of nonlinear PC while retaining its key computational properties.

Figures & tables

Panel of Figure 1 — Figure 1: Standard predictive coding versus activation-matched Bregman predictive coding. Both formulations use the same nonlinear prediction $\widehat{\bm{z}}^{\ell}=\phi(\bm{a}^{\ell})$ (see Table 1 for notation), but differ in the error measure used at each hidden layer. Standard PC uses a squared Euclidean error (Eq. 3 ), resulting in explicit activation derivatives ${\color[rgb]{0.75,0.375,0}\phi^{\prime}}$ in both inference and learning (Eq. 4 & Eq. 6 ). Bregman PC instead uses an activation-matched Bregman divergence (Eq. 18 ) satisfying $\nabla\Psi(\bm{z}^{\ell})=\phi^{-1}(\bm{z}^{\ell})$ . This removes the explicit activation derivative: inference evolves the dual state $\bm{u}^{\ell}$ (Eq. 22 ), with $\bm{z}^{\ell}=\phi(\bm{u}^{\ell})$ , while learning reduces to a local error-activity outer product (Eq. 24 ).

Symbol	Name	Description
$\bm{z}^{\ell}$	Activity / primal state	State of layer $\ell$ . The input and output activities are clamped to $\bm{x}$ and $\bm{y}$ , respectively, while hidden activities are inferred.
$\bm{u}^{\ell}$	Dual state	Dual coordinate of the activity, $\bm{u}^{\ell}=\phi^{-1}(\bm{z}^{\ell})$ , equivalently $\bm{z}^{\ell}=\phi(\bm{u}^{\ell})$ .
$\bm{a}^{\ell}$	Preactivation	Bottom-up linear input to layer $\ell$ , $\bm{a}^{\ell}=\bm{W}^{\ell}\bm{z}^{\ell-1}$ . At feedforward initialisation, $\bm{a}^{\ell}=\bm{u}^{\ell}$ .
$\widehat{\bm{z}}^{\ell}$	Predicted activity	Bottom-up prediction of the activity at layer $\ell$ , $\widehat{\bm{z}}^{\ell}=\phi(\bm{a}^{\ell})$ .
$\phi$	Activation function	Map $\phi:\mathbb{R}^{n}\rightarrow\phi(\mathbb{R}^{n})$ applied elementwise.
$\psi$	Scalar Bregman potential	Strictly convex scalar function matched to the activation through $\psi^{\prime}(z)=\phi^{-1}(z)$ .

Table 1: Key notation.

Figure 2: Constructing an activation-matched Bregman divergence for

\tanh

. (a) The activation function

\phi(a)=\tanh(a)

. (b) Its inverse,

\phi^{-1}(z)=\operatorname{arctanh}(z)

, which defines the derivative of the matching potential through

\psi^{\prime}(z)=\phi^{-1}(z)

(Eq. 10 ). (c) Integrating the inverse activation gives the matching potential

\psi(z)=z\operatorname{arctanh}(z)+\frac{1}{2}\log(1-z^{2})

, up to an additive constant. Since

\operatorname{arctanh}(z)

is strictly increasing,

\psi

is strictly convex. The Bregman divergence

D_{\psi}(z,\widehat{z})

is the gap between

\psi(z)

and the tangent-line approximation of

\psi

at the prediction

\widehat{z}=\phi(a)

. See Figures 7 - 8 for divergences with other activations.

Figure 3: Activation matching prevents attenuation of prediction errors. For

\tanh

, the standard squared error gives

\partial_{a}\mathcal{F}=-\varepsilon\phi^{\prime}(a)

, which is attenuated near saturation. The matched Bregman divergence contributes

1/\phi^{\prime}(a)

, cancelling this factor and yielding

\partial_{a}D_{\psi}=-\varepsilon

. Analogous results for other activations are shown in Figures 9 - 10 .

Figure 4: Mirror inference in primal and dual coordinates. Inference trajectories for a two-hidden-state Bregman PCN with

\phi=\tanh

, initialised from the same state and, in this example, converging to the same equilibrium. (a) In primal coordinates

\bm{z}=(z^{1},z^{2})

, Euclidean inference follows

d\bm{z}/d\tau=-\nabla_{\bm{z}}\mathcal{F}_{\mathrm{B}}

, whereas mirror inference follows the preconditioned flow

d\bm{z}/d\tau=-[\nabla^{2}\Psi(\bm{z})]^{-1}\nabla_{\bm{z}}\mathcal{F}_{\mathrm{B}}

, leading to a different trajectory through activity space. (b) In dual coordinates

\bm{u}=\nabla\Psi(\bm{z})=\phi^{-1}(\bm{z})

, the same mirror dynamics take the simpler form

d\bm{u}/d\tau=-\nabla_{\bm{z}}\mathcal{F}_{\mathrm{B}}

. Grey contours show the Bregman-PC energy in the corresponding coordinates. For other activations, see Figures 11 - 12 .

	Classification accuracy (%) $\uparrow$		Reconstruction MSE $[0,1]$ $\downarrow$
Algorithm	MNIST	Fashion-MNIST	MNIST	Fashion-MNIST
BP	$98.05\pm 0.10$	$88.92\pm 0.11$	$0.05310\pm 0.00003$	$0.05263\pm 0.00004$
Standard PC	$98.08\pm 0.08$	$88.87\pm 0.10$	$0.05320\pm 0.00003$	$0.05275\pm 0.00006$
Bregman PC	$97.57\pm 0.08$	$87.30\pm 0.24$	$0.05334\pm 0.00007$	$0.05326\pm 0.00006$

Table 2: Classification and generation performance of Bregman PC vs BP and standard PC. Validation classification accuracy and image-reconstruction MSE on MNIST and Fashion-MNIST (mean

\pm

SEM over 3 seeds). See Figures 13 - 14 for loss curves and examples of generated images, and § E for experimental details.

Figure 5: Bregman PC preserves established PC learning regimes. (a) “Saddle-to-saddle dynamics” for 5-layer $\tanh$ MLPs trained on MNIST with a small, near-origin initialisation. BP remains on a prolonged high-loss plateau, whereas standard PC and Bregman PC rapidly escape, with nearly overlapping loss trajectories. (b) Cosine similarity between Bregman-PC and BP parameter gradients after $100$ training steps for nonlinear residual networks trained on CIFAR-10 using the width- and depth-stable parameterisation of Innocenti et al. (2026) . Similarities are averaged over three random seeds and approach $1$ as width becomes large relative to depth. For more details, see § E .

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6: Candidate analog implementation of Bregman-PC inference. The capacitor voltage represents the dual state $u_{i}^{\ell}$ . A resistor connecting it to the bottom-up preactivation $a_{i}^{\ell}$ generates the local restoring current $(a_{i}^{\ell}-u_{i}^{\ell})/R$ , while a weighted-sum circuit supplies a feedback current proportional to $\sum_{j}W_{ji}^{\ell+1}\varepsilon_{j}^{\ell+1}$ . The same nonlinear transfer function can be used to obtain the activity $z_{i}^{\ell}=\phi(u_{i}^{\ell})$ and prediction $\widehat{z}_{i}^{\ell}=\phi(a_{i}^{\ell})$ . Kirchhoff’s current law reproduces the Bregman-PC inference dynamics in Eq. 22 up to an overall rescaling of time by $RC$ . Unlike standard nonlinear PC (Figure 1 ), the feedback pathway requires no explicit evaluation of the activation derivative.

Figure 7: Bregman divergence for the sigmoid function

\sigma

. (a) The activation function

\phi(a)=\sigma(a)

. (b) Its inverse,

\sigma^{-1}(z)=\operatorname{logit}(z)

. (c) Integrating the inverse activation gives the matching potential

\psi(z)=z\operatorname{arctanh}(z)+\frac{1}{2}\log(1-z^{2})

, up to an additive constant.

Figure 8: Bregman divergence for the identity function. (a) The activation function

\phi(a)=\operatorname{id}(a)

. (b) Its inverse,

\phi^{-1}(z)=z

. (c) Integrating the inverse activation gives the matching potential

\psi(z)=z\operatorname{arctanh}(z)+\frac{1}{2}\log(1-z^{2})

, up to an additive constant.

Figure 9: Activation matching for sigmoid. As in Figure 3 , the standard squared error gives

\partial_{a}\mathcal{F}=-\varepsilon\sigma^{\prime}(a)

, which is attenuated as the activation saturates, whereas the matched Bregman divergence yields

\partial_{a}D_{\psi}=-\varepsilon

Figure 10: Activation matching for a linear activation. For

\phi(a)=a

, we have

\phi^{\prime}(a)=1

, so the standard squared error and the matched Bregman divergence coincide, both yielding

\partial_{a}\mathcal{F}=\partial_{a}D_{\psi}=-\varepsilon

Figure 11: Mirror inference with sigmoid activation. Analogous to Figure 4 for

\tanh

, inference trajectories are shown for a two-hidden-state Bregman PCN with a sigmoid activation. Mirror inference follows a curvature-preconditioned gradient flow in primal coordinates and the corresponding simpler dynamics

d\bm{u}/d\tau=-\nabla_{\bm{z}}\mathcal{F}_{\mathrm{B}}

in dual coordinates.

Figure 12: Mirror inference with a linear activation. Analogous to Figure 4 , but for

\phi(a)=a

. In this case, the matched Bregman potential is quadratic (Figure 8 ), primal and dual coordinates coincide, and mirror inference reduces exactly to ordinary Euclidean gradient flow, yielding identical trajectories.

Figure 13: Classification learning curves on MNIST and Fashion-MNIST. Validation classification accuracy during training for BP, standard PC, and Bregman PC on MNIST and Fashion-MNIST. Final validation accuracies are reported in Table 2 .

Figure 14: Generated samples on MNIST and Fashion-MNIST. Representative images generated by models trained with BP, standard PC, and Bregman PC on MNIST and Fashion-MNIST. Corresponding reconstruction losses are reported in Table 2 .

Explore similar work

May 29, 2026cs.LG

Augmented Lagrangian Predictive Coding

Predictive coding (PC) is a local-learning alternative to backpropagation (BP), training deep networks via local energy-minimization dynamics rather than a global backward pass. We introduce Augmented Lagrangian Predictive Coding (PC-ALM), which maintains PC's inference budget but aligns each weight update toward BP by accumulating per-layer constraint errors into a layer-local Lagrange multiplier. In linear PC networks, PC-ALM converges to an equilibrium with exact BP gradients distributed across the network via only layer-local updates. We analyze PC-ALM in nonlinear PC networks up to depth 128 and show that it matches BP performance across all width-depth regimes, notably in deep narrow networks where PC underperforms. PC-ALM introduces recurrent dynamics in each layer's activations. Compared to PC's heat flow on a scalar energy, PC-ALM dynamics are driven by dual ascent on the augmented Lagrangian. We observe "ballistic" credit propagation across very deep networks, with credit signals evenly distributed across layers, compared to PC's slow, diffusive credit propagation. Beyond the algorithm itself, the augmented Lagrangian framework offers a generalization of PC, and may yield insights into how distributed systems could compute and propagate BP-like credit signals through purely local dynamics.

Jeffrey Seely, Julian Gould

Sakana AI

Backpropagation Predictive Coding Augmented Lagrangian Method Neural Network Training Dynamics Local Learning

Ask PDF

May 12, 2026cs.LG

Understanding Sample Efficiency in Predictive Coding

Predictive Coding (PC) is an influential account of cortical learning. Much of recent work has focused on comparing PC to Backpropagation (BP) to find whether PC offers any advantages. Small scale experiments show that PC enables learning that is more sample efficient and effective in many contexts, though a thorough theoretical understanding of the phenomena remains elusive. To address this, we quantify the efficiency of learning in BP and PC through a metric called ``target alignment'', which measures how closely the change in the output of the network is aligned to the output prediction error. We then derive and empirically validate analytical expressions for target alignment in Deep Linear Networks. We show that learning in PC is more efficient than BP, which is especially pronounced in deep, narrow and pre-trained networks. We also derive exact conditions for guaranteed optimal target alignment in PC and validate our findings through experiments. We study full training trajectories of linear and non-linear models, and find the predicted benefits of PC persist in practice even when some assumptions are violated. Overall, this work provides a mechanistic understanding of the higher learning efficiency observed for PC over BP in previous works, and can guide how PC should be parametrised to learn most effectively.

Gaspard Oliviers, Elene Lominadze, Rafal Bogacz

Nuffield Department of Clinical Neurosciences, University of Oxford, United Kingdom · MRC Centre of Research Excellence in Restorative Neural Dynamics, United Kingdom

Backpropagation Predictive Coding Deep Linear Networks Neural Network Training Dynamics Data-Efficient Learning

Ask PDF

May 19, 2026cs.LG

Closed-form predictive coding via hierarchical Gaussian filters

Predictive coding (PC) offers a local and biologically grounded alternative to backpropagation in the training of artificial neural networks, yet to date, it remains slower, and performance degrades sharply as network depth increases. We trace both problems to a single simplification: current PC networks fix the precision matrix to the identity, discarding precision-weighted prediction errors that the variational derivation requires to be fast, local, and Bayesian. We close this gap by expressing predictive coding networks as deep hierarchical Gaussian filters (HGFs) and restore precision-weighted message passing, yielding dynamic uncertainty estimates and Hebbian-compatible update rules at every layer. The resulting networks can simultaneously learn activations, weights, and precisions under a single free-energy objective, with no global error signal, and resolve inference without requiring iterations or automatic differentiation. On FashionMNIST, our solution approaches backpropagation in epoch-level wall-clock cost while converging in fewer epochs, and outperforms it on online, data efficiency, and concept-drift tasks. We thus establish that closed-form variational inference with online precision learning provides a tractable foundation for deep predictive coding networks, retaining biological and interpretative advantages, without requiring iterative relaxation or global error signals.

Aleksandrs Baskakovs, Sylvain Estebe, Kenneth Enevoldsen +3

Center for Humanities Computing Aarhus University Aarhus, Denmark · Interacting Minds Center Aarhus University Aarhus, Denmark

Predictive Coding Variational Inference Local Learning

Ask PDF