Analog-Friendly Predictive Coding without Activation Derivatives

Francesco Innocenti

cs.LGSep 26, 2026

Analog-Friendly Predictive Coding without Activation Derivatives

Authors: Francesco Innocenti

Organizations: Brain Network Dynamics Unit University of Oxford, UK

Abstract

Predictive coding (PC) is a local, energy-based alternative to backpropagation (BP) whose iterative inference dynamics make it attractive for implementation on analog hardware. However, standard nonlinear PC requires evaluating the derivative of the activation function during both inference and learning, which can be difficult to realise physically. Here, we introduce \textit{activation-matched Bregman PC}, replacing standard squared-error energies with Bregman divergences matched to the activation function. This formulation eliminates activation derivatives and, when combined with inference via mirror descent, yields local inference and learning rules requiring only weighted sums, local prediction errors, state integration, and the activation function. In digital experiments, Bregman PC performs comparably to standard PC and BP on classification and generative tasks, while preserving characteristic learning dynamics of PC and its convergence to BP under stable large-model parameterisations. These results provide a more analog-friendly formulation of nonlinear PC while retaining its key computational properties.

Figures & tables

Panel of Figure 1 — Figure 1: Standard predictive coding versus activation-matched Bregman predictive coding. Both formulations use the same nonlinear prediction $\widehat{\bm{z}}^{\ell}=\phi(\bm{a}^{\ell})$ (see Table 1 for notation), but differ in the error measure used at each hidden layer. Standard PC uses a squared Euclidean error (Eq. 3 ), resulting in explicit activation derivatives ${\color[rgb]{0.75,0.375,0}\phi^{\prime}}$ in both inference and learning (Eq. 4 & Eq. 6 ). Bregman PC instead uses an activation-matched Bregman divergence (Eq. 18 ) satisfying $\nabla\Psi(\bm{z}^{\ell})=\phi^{-1}(\bm{z}^{\ell})$ . This removes the explicit activation derivative: inference evolves the dual state $\bm{u}^{\ell}$ (Eq. 22 ), with $\bm{z}^{\ell}=\phi(\bm{u}^{\ell})$ , while learning reduces to a local error-activity outer product (Eq. 24 ).

Symbol	Name	Description
$\bm{z}^{\ell}$	Activity / primal state	State of layer $\ell$ . The input and output activities are clamped to $\bm{x}$ and $\bm{y}$ , respectively, while hidden activities are inferred.
$\bm{u}^{\ell}$	Dual state	Dual coordinate of the activity, $\bm{u}^{\ell}=\phi^{-1}(\bm{z}^{\ell})$ , equivalently $\bm{z}^{\ell}=\phi(\bm{u}^{\ell})$ .
$\bm{a}^{\ell}$	Preactivation	Bottom-up linear input to layer $\ell$ , $\bm{a}^{\ell}=\bm{W}^{\ell}\bm{z}^{\ell-1}$ . At feedforward initialisation, $\bm{a}^{\ell}=\bm{u}^{\ell}$ .
$\widehat{\bm{z}}^{\ell}$	Predicted activity	Bottom-up prediction of the activity at layer $\ell$ , $\widehat{\bm{z}}^{\ell}=\phi(\bm{a}^{\ell})$ .
$\phi$	Activation function	Map $\phi:\mathbb{R}^{n}\rightarrow\phi(\mathbb{R}^{n})$ applied elementwise.
$\psi$	Scalar Bregman potential	Strictly convex scalar function matched to the activation through $\psi^{\prime}(z)=\phi^{-1}(z)$ .

Table 1: Key notation.

Figure 2: Constructing an activation-matched Bregman divergence for

\tanh

. (a) The activation function

\phi(a)=\tanh(a)

. (b) Its inverse,

\phi^{-1}(z)=\operatorname{arctanh}(z)

, which defines the derivative of the matching potential through

\psi^{\prime}(z)=\phi^{-1}(z)

(Eq. 10 ). (c) Integrating the inverse activation gives the matching potential

\psi(z)=z\operatorname{arctanh}(z)+\frac{1}{2}\log(1-z^{2})

, up to an additive constant. Since

\operatorname{arctanh}(z)

is strictly increasing,

\psi

is strictly convex. The Bregman divergence

D_{\psi}(z,\widehat{z})

is the gap between

\psi(z)

and the tangent-line approximation of

\psi

at the prediction

\widehat{z}=\phi(a)

. See Figures 7 - 8 for divergences with other activations.

Figure 3: Activation matching prevents attenuation of prediction errors. For

\tanh

, the standard squared error gives

\partial_{a}\mathcal{F}=-\varepsilon\phi^{\prime}(a)

, which is attenuated near saturation. The matched Bregman divergence contributes

1/\phi^{\prime}(a)

, cancelling this factor and yielding

\partial_{a}D_{\psi}=-\varepsilon

. Analogous results for other activations are shown in Figures 9 - 10 .

Figure 4: Mirror inference in primal and dual coordinates. Inference trajectories for a two-hidden-state Bregman PCN with

\phi=\tanh

, initialised from the same state and, in this example, converging to the same equilibrium. (a) In primal coordinates

\bm{z}=(z^{1},z^{2})

, Euclidean inference follows

d\bm{z}/d\tau=-\nabla_{\bm{z}}\mathcal{F}_{\mathrm{B}}

, whereas mirror inference follows the preconditioned flow

d\bm{z}/d\tau=-[\nabla^{2}\Psi(\bm{z})]^{-1}\nabla_{\bm{z}}\mathcal{F}_{\mathrm{B}}

, leading to a different trajectory through activity space. (b) In dual coordinates

\bm{u}=\nabla\Psi(\bm{z})=\phi^{-1}(\bm{z})

, the same mirror dynamics take the simpler form

d\bm{u}/d\tau=-\nabla_{\bm{z}}\mathcal{F}_{\mathrm{B}}

. Grey contours show the Bregman-PC energy in the corresponding coordinates. For other activations, see Figures 11 - 12 .

	Classification accuracy (%) $\uparrow$		Reconstruction MSE $[0,1]$ $\downarrow$
Algorithm	MNIST	Fashion-MNIST	MNIST	Fashion-MNIST
BP	$98.05\pm 0.10$	$88.92\pm 0.11$	$0.05310\pm 0.00003$	$0.05263\pm 0.00004$
Standard PC	$98.08\pm 0.08$	$88.87\pm 0.10$	$0.05320\pm 0.00003$	$0.05275\pm 0.00006$
Bregman PC	$97.57\pm 0.08$	$87.30\pm 0.24$	$0.05334\pm 0.00007$	$0.05326\pm 0.00006$

Table 2: Classification and generation performance of Bregman PC vs BP and standard PC. Validation classification accuracy and image-reconstruction MSE on MNIST and Fashion-MNIST (mean

\pm

SEM over 3 seeds). See Figures 13 - 14 for loss curves and examples of generated images, and § E for experimental details.

Figure 5: Bregman PC preserves established PC learning regimes. (a) “Saddle-to-saddle dynamics” for 5-layer $\tanh$ MLPs trained on MNIST with a small, near-origin initialisation. BP remains on a prolonged high-loss plateau, whereas standard PC and Bregman PC rapidly escape, with nearly overlapping loss trajectories. (b) Cosine similarity between Bregman-PC and BP parameter gradients after $100$ training steps for nonlinear residual networks trained on CIFAR-10 using the width- and depth-stable parameterisation of Innocenti et al. (2026) . Similarities are averaged over three random seeds and approach $1$ as width becomes large relative to depth. For more details, see § E .

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6: Candidate analog implementation of Bregman-PC inference. The capacitor voltage represents the dual state $u_{i}^{\ell}$ . A resistor connecting it to the bottom-up preactivation $a_{i}^{\ell}$ generates the local restoring current $(a_{i}^{\ell}-u_{i}^{\ell})/R$ , while a weighted-sum circuit supplies a feedback current proportional to $\sum_{j}W_{ji}^{\ell+1}\varepsilon_{j}^{\ell+1}$ . The same nonlinear transfer function can be used to obtain the activity $z_{i}^{\ell}=\phi(u_{i}^{\ell})$ and prediction $\widehat{z}_{i}^{\ell}=\phi(a_{i}^{\ell})$ . Kirchhoff’s current law reproduces the Bregman-PC inference dynamics in Eq. 22 up to an overall rescaling of time by $RC$ . Unlike standard nonlinear PC (Figure 1 ), the feedback pathway requires no explicit evaluation of the activation derivative.

Figure 7: Bregman divergence for the sigmoid function

\sigma

. (a) The activation function

\phi(a)=\sigma(a)

. (b) Its inverse,

\sigma^{-1}(z)=\operatorname{logit}(z)

. (c) Integrating the inverse activation gives the matching potential

\psi(z)=z\operatorname{arctanh}(z)+\frac{1}{2}\log(1-z^{2})

, up to an additive constant.

Figure 8: Bregman divergence for the identity function. (a) The activation function

\phi(a)=\operatorname{id}(a)

. (b) Its inverse,

\phi^{-1}(z)=z

. (c) Integrating the inverse activation gives the matching potential

\psi(z)=z\operatorname{arctanh}(z)+\frac{1}{2}\log(1-z^{2})

, up to an additive constant.

Figure 9: Activation matching for sigmoid. As in Figure 3 , the standard squared error gives

\partial_{a}\mathcal{F}=-\varepsilon\sigma^{\prime}(a)

, which is attenuated as the activation saturates, whereas the matched Bregman divergence yields

\partial_{a}D_{\psi}=-\varepsilon

Figure 10: Activation matching for a linear activation. For

\phi(a)=a

, we have

\phi^{\prime}(a)=1

, so the standard squared error and the matched Bregman divergence coincide, both yielding

\partial_{a}\mathcal{F}=\partial_{a}D_{\psi}=-\varepsilon

Figure 11: Mirror inference with sigmoid activation. Analogous to Figure 4 for

\tanh

, inference trajectories are shown for a two-hidden-state Bregman PCN with a sigmoid activation. Mirror inference follows a curvature-preconditioned gradient flow in primal coordinates and the corresponding simpler dynamics

d\bm{u}/d\tau=-\nabla_{\bm{z}}\mathcal{F}_{\mathrm{B}}

in dual coordinates.

Figure 12: Mirror inference with a linear activation. Analogous to Figure 4 , but for

\phi(a)=a

. In this case, the matched Bregman potential is quadratic (Figure 8 ), primal and dual coordinates coincide, and mirror inference reduces exactly to ordinary Euclidean gradient flow, yielding identical trajectories.

Figure 13: Classification learning curves on MNIST and Fashion-MNIST. Validation classification accuracy during training for BP, standard PC, and Bregman PC on MNIST and Fashion-MNIST. Final validation accuracies are reported in Table 2 .

Figure 14: Generated samples on MNIST and Fashion-MNIST. Representative images generated by models trained with BP, standard PC, and Bregman PC on MNIST and Fashion-MNIST. Corresponding reconstruction losses are reported in Table 2 .

Explore similar work

Augmented Lagrangian Predictive Coding
May 29, 2026Jeffrey Seely, Julian GouldBackpropagation Predictive Coding
Sakana AI
Understanding Sample Efficiency in Predictive Coding
May 12, 2026Gaspard Oliviers, Elene Lominadze, Rafal BogaczBackpropagation Predictive Coding
Nuffield Department of Clinical Neurosciences, University of Oxford, United Kingdom · MRC Centre of Research Excellence in Restorative Neural Dynamics, United Kingdom
Closed-form predictive coding via hierarchical Gaussian filters
May 19, 2026Aleksandrs Baskakovs, Sylvain Estebe, Kenneth Enevoldsen +3Predictive Coding Variational Inference
Center for Humanities Computing Aarhus University Aarhus, Denmark · Interacting Minds Center Aarhus University Aarhus, Denmark

May 29, 2026J/K move · Enter open · S save

Augmented Lagrangian Predictive Coding

Jeffrey Seely, Julian Gould

Sakana AI

Backpropagation Predictive Coding Augmented Lagrangian Method Neural Network Training Dynamics Local Learning

Open paper AskPDF arXiv

Predictive coding (PC) is a local-learning alternative to backpropagation (BP), training deep networks via local energy-minimization dynamics rather than a global backward pass. We introduce Augmented Lagrangian Predictive Coding (PC-ALM), which maintains PC's inference budget but aligns each weight update toward BP by accumulating per-layer constraint errors into a layer-local Lagrange multiplier. In linear PC networks, PC-ALM converges to an equilibrium with exact BP gradients distributed across the network via only layer-local updates. We analyze PC-ALM in nonlinear PC networks up to depth 128 and show that it matches BP performance across all width-depth regimes, notably in deep narrow networks where PC underperforms. PC-ALM introduces recurrent dynamics in each layer's activations. Compared to PC's heat flow on a scalar energy, PC-ALM dynamics are driven by dual ascent on the augmented Lagrangian. We observe "ballistic" credit propagation across very deep networks, with credit signals evenly distributed across layers, compared to PC's slow, diffusive credit propagation. Beyond the algorithm itself, the augmented Lagrangian framework offers a generalization of PC, and may yield insights into how distributed systems could compute and propagate BP-like credit signals through purely local dynamics.

Abstract

Explore similar work

Augmented Lagrangian Predictive Coding

Understanding Sample Efficiency in Predictive Coding

Closed-form predictive coding via hierarchical Gaussian filters