Information Geometry of Message Passing

Mykola Lukashchuk; Kyrylo Yemets; Alex Ledbetter; İsmail Şenöz

cs.LGAug 16, 2026

Information Geometry of Message Passing

Authors: Mykola Lukashchuk, Kyrylo Yemets, Alex Ledbetter, İsmail Şenöz

Organizations: Eindhoven University of Technology · Lviv Polytechnic National University · Lazy Dynamics, Utrecht, the Netherlands

Abstract

We show that the natural-gradient stationary condition of variational inference has an edge-local form on a Forney-style factor graph. We start from the Bethe free energy and constrain a selected edge marginal to an exponential family. At a stationary point, the natural parameter of that edge equals the sum of two projected messages, one from each incident factor. Each projected message is the natural-gradient projection of the exact belief-propagation log-message at the current receiving marginal, or equivalently, the gradient of its expectation in the so-called mean coordinates. We call the resulting scheme natural-gradient message passing (NGMP). The rule is local; each edge may carry its own exponential family, and the message a factor sends depends on the marginal that receives it. Compared with variational message passing, NGMP keeps the part of the exact message that the receiving family can represent instead of averaging the factor under the neighboring beliefs. The two coincide when the uncertainty on the edges entering a non-conjugate factor vanishes, and NGMP is more accurate when that uncertainty persists, for example, along a partially observed latent chain or when parameters are filtered through successive data batches. Experiments on Poisson smoothing, heteroskedastic regression, and hourly ETTh forecasting confirm this and show that the gain appears mainly in uncertainty calibration.

Figures & tables

Panel of Figure 1 — Figure 1: Messages produced by belief propagation (BP) rely on the local factor $f_{b}$ and its other incoming messages. Variational message passing (VMP), including its structured variant, first forms a surrogate factor $\widetilde{f}_{b}$ by variational expectation of $f_{b}$ with respect to the current marginals of the out-of-cluster variables, then passes a BP message through this new factor; non-conjugate and projective VMP keep this flow and additionally project the resulting message at the receiving marginal ( Section 6 ). Expectation propagation (EP) instead forms a cavity distribution from the opposing side, restores the exact factor contribution to obtain a tilted marginal, projects that marginal onto the chosen form-constraint family via $\Pi_{\mathcal{E}_{i}}[\cdot]$ , and divides out the cavity to obtain the new site ( Minka, 2001 ) . Finally, natural-gradient message passing uses the receiving marginal to define the tangent space for natural-gradient projection of the exact BP log-message. The final message towards $z_{i}$ is obtained as the exponential of the resulting projected natural parameters $\eta_{b\to i}$ applied to the chosen family’s sufficient statistics $T_{i}(z_{i})$ .

Panel of Figure 2 — Figure 2: Forney-style Factor Graph representation of the factorization in equation 4 .

Panel of Figure 3 — Figure 3: Geometric view of the projected message. The exact BP message is $\mu_{a\to i}=\exp\{\ell_{a\to i}\}$ , a tangent in the ambient probability space $\mathcal{P}$ . The decomposition is drawn in log-message coordinates: $\ell_{a\to i}$ is split into the tangent component $\bm{\eta}_{a\to i}^{\top}T_{i}$ and the orthogonal residual $\ell_{a\to i}^{\perp}$ at the current marginal $q_{\lambda_{i}}$ . The actual finite message sent along the constrained edge is obtained by exponentiation of the tangent component, $\widehat{\mu}_{a\to i}\propto\exp\{\bm{\eta}_{a\to i}^{\top}T_{i}\}$ . The residual may remain in the functional Lagrange multipliers, but it is invisible to the finite-dimensional edge marginal. Making the ambient space picture fully rigorous would require a non-parametric density-manifold formulation; see Wang and Li (2020) for tangent and cotangent spaces of probability densities and Fisher–Rao/Wasserstein information metrics. We use the ambient manifold here informally to explain what is the finite-dimensional projection.

Panel of Figure 4 — Figure 4: Two equivalent implementations of the projected updates on two displayed observation slices of equation 39 ; both factor graphs are trees and the dots indicate continuation to later states. (a) Surrogate-model view. Each Poisson observation leaf is represented by a Gaussian pseudo-observation $\mathcal{N}^{\star}=\mathcal{N}(\widetilde{y}_{k}^{(t)}\mid z_{k},(\Lambda_{k}^{(t)})^{-1})$ , so ordinary Gaussian BP runs on the entire auxiliary tree. (b) NGMP message view. The original Poisson factors remain visible; each blue label is the projected observation message and the adjacent red marginal defines its tangent space. Inserting a Gaussian factor with the blue message’s natural parameters produces the corresponding leaf in panel (a). Applying this conversion independently at every $k$ gives equation 45 . The same idea generalizes per interface: a multi-interface factor sends a separate projected message through each constrained interface, as shown in Figure 5 .

Panel of Figure 5 — Figure 5: The two-observation instance of equation 46 and the message from the first observation toward $\tau$ . The Normal prior for $x$ and Gamma prior for $\tau$ are at the top; the two $\mathcal{N}(y_{i}\mid x,\tau^{-1})$ factors close the diamond loop. Solid edges carry $x$ and dashed edges carry $\tau$ . (a) VMP uses the previous global marginal $q^{(t-1)}(x)\propto p(x)\prod_{n=1}^{N}\widetilde{\mu}_{y_{n}\to x}^{(t-1)}(x)$ to compute $\widetilde{\mu}_{y_{1}\to\tau}^{(t)}$ . (b) NGMP instead uses the cavity $\mu_{x\setminus 1}^{(t)}$ and projects the resulting exact BP log-message at $q_{\tau}^{(t-1)}$ . The blue labels enumerate the four messages around the loop; their subscripts give the directions. The red labels are the receiving marginals at which the two blue projected messages are evaluated. The new $y_{1}\!\to\!\tau$ message changes the cavity entering the lower Normal, whose message toward $x$ changes the cavity returning to the upper Normal on the next sweep. The message toward $x$ is analogous.

Panel of Figure 6 — Figure 6: Surrogate-model interpretation of the multi-interface NGMP computation in Figure 5 b. (a) The original observation factor $\mathcal{N}(y\mid x,\tau^{-1})$ touches the $x$ and $\tau$ equality constraints and produces one projected message for each interface. (b) With those message parameters frozen, the factor’s contributions are represented by a Gaussian leaf $\mathcal{N}^{\star}$ on $x$ and a Gamma leaf $\mathcal{G}^{\star}$ on $\tau$ ; their observed ports denote the projected natural parameters. Neighboring factors $f,g,h,k$ keep their ordinary BP rules. This is the direct multi-interface analogue of Figure 4 a, except that one original factor yields one surrogate leaf per constrained interface.

Panel of Figure 7 — Figure 7: Marginal inclusive divergence in the isolated Normal mean–precision model: (a) $\mathbb{D}_{\mathrm{KL}}[p(x\mid y)\|q(x)]$ and (b) $\mathbb{D}_{\mathrm{KL}}[p(\tau\mid y)\|q(\tau)]$ (mean $\pm$ 95% CI over 20 instances; each instance draws a fresh true $(x,\tau)$ from the priors and generates its own stream of samples). For $\tau$ , VMP decays as $N^{-2}$ and NGMP as $N^{-3}$ ; for $x$ , NGMP is a roughly constant $2.3\times$ closer.

Panel of Figure 8 — Figure 8: Sunspot results at $50\%$ holdout. Top: a five-year window around the deepest held-out stretch of a representative mask, one method per panel on identical axes; lines are the plug-in predictive rates $\exp(\mathbb{E}[z_{k}]+\operatorname{Var}(z_{k})/2)$ , shaded bands push the pointwise $95\%$ credible intervals of $z_{k}$ through the exponential link, and diamonds are held-out counts. PVMP’s band keeps a fixed width inside the gap and its rate drifts confidently away from the held-out counts, while NGMP’s band widens toward the gap center and keeps covering them. Bottom: held-out negative log-likelihood (log scale) and posterior state variance against distance to the nearest observation over all 20 masks.

	NLL		RMSE
Held out	PVMP	NGMP	PVMP	NGMP
5%	4.534 $\pm$ 0.087	4.532 $\pm$ 0.085	13.918 $\pm$ 0.385	13.935 $\pm$ 0.378
10%	4.569 $\pm$ 0.058	4.566 $\pm$ 0.058	13.972 $\pm$ 0.260	13.995 $\pm$ 0.260
20%	4.672 $\pm$ 0.039	4.661 $\pm$ 0.037	14.429 $\pm$ 0.214	14.446 $\pm$ 0.208
50%	5.411 $\pm$ 0.162	4.930 $\pm$ 0.035	16.839 $\pm$ 0.471	15.789 $\pm$ 0.162

Table 1: Sunspot state-space model: held-out predictive metrics (mean

\pm

95% CI over 20 random masks). PVMP and NGMP agree for short gaps and separate when the

50\%

mask creates long uncertain stretches. The budget-matched NCVMP control is reported in Table 5 of Appendix D .

	NLL	RMSE	NLL	RMSE	batching penalty
	full batch		sequential
PVMP	$0.276\pm 0.031$	$0.588\pm 0.037$	$0.512\pm 0.044$	$0.583\pm 0.039$	$0.235\pm 0.030$
NGMP	$0.235\pm 0.031$	$0.591\pm 0.038$	$0.244\pm 0.031$	$0.607\pm 0.042$	$0.009\pm 0.010$

Table 2: Sequential heteroscedastic model over 20 paired seeds (mean

\pm

95% CI). The same 400 observations are fitted jointly or filtered in ten batches; the batching penalty is the resulting increase in held-out NLL.

Panel of Figure 9 — Figure 9: Factor graph and online schedule for the sequential heteroscedastic model. Panel (a) shows branch $o$ of the batch box; each softdot node is the soft dot-product factor equation 58 . The upper softdot is the log-precision path of equation 57 , with $b=w$ , $\phi=\psi(x_{o})$ , $\tau=c_{\mathrm{top}}$ , and $z=s_{o}$ ; the observation path below it is the same factor with the uncertain precision $\tau=e^{s_{o}}$ . The lateral stubs with ellipses at the equality nodes continue to the other observations $o^{\prime}\in\mathcal{B}_{b}\setminus\{o\}$ , so the same $w$ and $v$ are shared by all 40 branches. Panel (b) opens the double-bordered $\mathcal{N}_{\!\exp}$ composite and, inside it, the softdot box; the inner box is equation 58 drawn graphically. The deterministic dot node forms the dot product $b^{\top}\phi$ , the Gaussian $\mathcal{N}$ receives it as its mean, and the precision of that Gaussian is the exponentiated incoming score $\tau=e^{s}$ . In panel (a) the ports are $b=v$ , $\phi=\phi(x_{o})$ , $s=s_{o}$ , and $z=y_{o}$ . Panel (c) repeats the compact batch box for $\mathcal{B}_{1},\mathcal{B}_{2},\ldots,\mathcal{B}_{10}$ beneath the two shared-weight chains. Batch 1 starts from the model priors $p(w)p(v)$ ; after batch $b$ , the Gaussian marginals $q_{b}(w)q_{b}(v)$ are frozen and copied left-to-right as the next priors. A perpendicular stop bar marks that no message from a later batch is sent back into an earlier batch; the bar specifies the online schedule and is not an additional model factor. With exact state-independent Gaussian messages, this posterior-as-prior multiplication is ordinary associative Kalman-style filtering. Solid edges carry weights, features, and observations; dashed edges carry precisions; dash-dotted edges carry the latent log-precision scores.

Panel of Figure 10 — Figure 10: Posterior predictive bands after the ten-step filtering chain of Figure 9 , on one representative seed with identical axes. The orange dashed curve is PVMP and the blue solid curve is NGMP; the matching shaded regions are pointwise 95% posterior-predictive bands. Gray dash-dotted curves show the true mean, and gray points are the 400 training observations. Both methods resolve the oscillatory mean; under posterior reuse, PVMP’s noise-weight covariance collapses and its NLL rises, whereas NGMP remains close to its full-batch NLL. The full-batch fits of both methods are visually indistinguishable from panel (b); Figure 18 in Appendix D shows all four fits.

Panel of Figure 11 — Figure 11: Posterior predictive variance on the same representative seed as Figure 10 . The blue solid curve in panel (a) is NGMP and the orange long-dashed curves in panels (b)–(c) are PVMP; panel (b) uses all observations at once, panels (a) and (c) use the ten sequential batches. Each curve is the posterior mean total predictive variance, $\bar{S}(x)=V_{\mathrm{epi}}(x)+V_{\mathrm{alea}}(x)$ , where $V_{\mathrm{epi}}(x)=\phi(x)^{\top}\Sigma_{v}\phi(x)$ is epistemic variance from the uncertain mean weights and $V_{\mathrm{alea}}(x)=\mathbb{E}_{q}[e^{-s(x)}]$ is learned aleatoric variance. Shading gives a pointwise 95% posterior credible interval for the total. The purple dotted curve shows $V_{\mathrm{epi}}(x)$ alone, so its vertical gap to the orange or blue total is $V_{\mathrm{alea}}(x)$ . The gray dash-dotted curve is the benchmark’s true aleatoric variance $[0.45(x+0.5)]^{2}$ . These pointwise credible intervals describe posterior uncertainty for one fit and are distinct from the across-seed confidence intervals in Table 2 . All panels use identical logarithmic axes. NGMP’s full-batch fit is visually indistinguishable from panel (a); Figure 19 in Appendix D shows all four fits.

Panel of Figure 12 — Figure 12: One observation branch of the arbitrary-depth heteroscedastic hierarchy used in the regression experiment. Every level has its own Gaussian weights $w_{\ell}$ , shared across observation branches through the equality nodes, and its own clamped feature map $\psi_{\ell}(x_{o})$ . The top-level softdot is the soft dot-product factor equation 58 . Each level turns its incoming score into a precision and emits the score $s_{\ell,o}$ read by the level below, and the observation level turns the last score into the noise precision of $y_{o}$ . The double-bordered $\mathcal{N}_{\!\exp}$ composites are opened in Figure 9 (b). The experiments in this section use the depth-three instance. Edge styles are as in Figure 9 : solid for weights, features, and observations; dashed for precisions; dash-dotted for scores.

Method	Concrete	Energy	Boston	Power	Wine	Yacht
BBB	$3.3328\pm 0.0215$	$2.4229\pm 0.0350$	$2.6555\pm 0.0385$	$2.7785\pm 0.0143$	$\bm{0.9438\pm 0.0232}$	$2.6565\pm 0.0451$
dDVI	$3.0477\pm 0.0432$	$\bm{1.1920\pm 0.4625}$	$\bm{2.4646\pm 0.0921}$	$2.8238\pm 0.0183$	$\bm{0.9440\pm 0.0314}$	$0.4641\pm 0.1000$
DVI	$3.0406\pm 0.0438$	$\bm{1.0094\pm 0.2973}$	$\bm{2.4489\pm 0.0917}$	$2.8249\pm 0.0200$	$\bm{0.9448\pm 0.0307}$	$0.4543\pm 0.1096$
IVON ( $K=20$ )	$3.7966\pm 0.0082$	$3.2693\pm 0.0412$	$3.3284\pm 0.0855$	$3.7839\pm 0.0013$	$1.0430\pm 0.0156$	$3.6584\pm 0.0101$
NGMP	$\bm{2.9845\pm 0.0433}$	$\bm{0.9783\pm 0.0655}$	$2.6720\pm 0.0752$	$\bm{2.7610\pm 0.0226}$	$\bm{0.9571\pm 0.0293}$	$\bm{0.2782\pm 0.1437}$

Table 3: Test negative log likelihood (NLL) in original target units; lower is better. Values are means

\pm

approximate 95% confidence-interval half-widths over 20 paired splits. Bold values include the best point estimate within their confidence interval. BBB, dDVI, and DVI are the heteroscedastic variants; IVON is homoscedastic and uses

K=20

posterior network draws for its predictive mixture. The remaining homoscedastic variants are in Table 9 .

ETTh1
Method	96	192	336	720
RMSE
MoE – IVON, affine gate ( $K{=}1000$ )	$0.3832\pm 0.0090$	$0.3695\pm 0.0091$	$0.3672\pm 0.0084$	$0.5428\pm 0.0127$
MoE – IVON, ReLU gate ( $K{=}1000$ )	$0.3832\pm 0.0090$	$0.3697\pm 0.0091$	$0.3677\pm 0.0084$	$0.5334\pm 0.0125$
PGE – PVMP ( Lukashchuk et al., 2026 )	$0.3583\pm 0.0083$	$0.3386\pm 0.0085$	$\mathbf{0.3105\pm 0.0073}$	$0.3347\pm 0.0076$
PGE – NGMP (this work)	$\mathbf{0.3554\pm 0.0084}$	$\mathbf{0.3337\pm 0.0086}$	$\mathbf{0.3115\pm 0.0074}$	$\mathbf{0.3300\pm 0.0080}$

Table 4: ETTh forecasting in standardized OT units. Entries are test-set point estimates

\pm

approximate 95% normal confidence-interval half-widths over test predictions; RMSE intervals use the delta method. Lower is better. Among the methods shown, the lowest point estimate is bold; the second-lowest is also bold when its paired 95% normal confidence interval relative to the lowest, computed on identical test predictions, includes zero. For IVON,

K=1000

is the number of gate-weight samples drawn from the fitted variational posterior to approximate the posterior predictive. The predictive-mixture construction and complete experimental setup are described in Section E.2 ; the full results, including Adam-trained gates, are reported in Table 11 .

Figure 13: ETTh1 at $H=192$ : NGMP (solid blue) and PVMP (hatched orange), with 95% predictive intervals, over an illustrative one-week segment. The black solid line is the observed target. This panel illustrates the predictions only; the full-test RMSE and NLL comparison is reported in Table 4 .

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 14 — Figure 14: Convergence of the NGMP fixed-point iteration on the Poisson state-space model equation 39 for series simulated from the same model ( $\sigma^{2}=0.1$ , all counts observed) of length $N=100$ , $250$ (top row), $500$ , and $1000$ (bottom row), 20 seeds each, 200 variational iterations. BFE on the vertical axis is the Bethe free energy of the model per observation minus the value reached at convergence by the $\alpha=0.25$ setting on the same seed, so every seed converges to zero (log scale); curves are the geometric mean across seeds with a 95% confidence band. Orange: undamped fixed-point iteration ( $\alpha=1$ ). Blue: messages damped in natural coordinates with $\alpha=0.25$ . Green dashed: the damped heavy-ball setting used for the sunspot experiments ( $\alpha=0.5$ , $\beta=0.2$ ); its momentum step left the natural domain and the fit stopped on 3 of 20 seeds at $N=500$ and 9 of 20 at $N=1000$ (the undamped iteration failed on one seed at $N=1000$ ), and the curves average over the seeds that reached each iteration. Longer chains make larger excursions of the latent log-rate; the undamped map then stops converging while the damped map still does.

Panel of Figure 15 — Figure 15: Bethe free energy per observed count on the sunspot series (mean $\pm$ 95% CI across 20 masks, log scale) at $5\%$ and $10\%$ holdout (top row) and $20\%$ and $50\%$ holdout (bottom row). Both methods plateau within the 20-sweep budget: NGMP by sweep three to four, PVMP shortly after. Each PVMP trace evaluates its own variational objective; the NGMP trace is a surrogate diagnostic, since the local Gaussian surrogates are rebuilt between outer sweeps. The dotted trace is the budget-matched NCVMP control of Table 5 .

Figure 16: Bethe free energy per observation for the sequential heteroscedastic model (mean

\pm

95% CI over 20 seeds). Left: the full-batch fit against sweep (first two sweeps omitted for scale). Right: the final-sweep value of each filtering update against batch index. PVMP evaluates its own mean-field variational objective; the NGMP trace scores the same node energies through moment-matched joint

(\mu,s)

cluster marginals, so the two curves approximate different Bethe functionals and their levels are not directly comparable. The dotted trace is the budget-matched NCVMP control of Table 6 .

Figure 17: Noise-weight concentration

\log\det\Sigma_{w}

along the filtering chain (mean

\pm

95% CI over 20 seeds; dotted lines show full-batch values; lower means more certain). PVMP’s sequential endpoint coincides with its full-batch value, and the budget-matched NCVMP control ( Table 6 ) traces the identical curve; NGMP concentrates as evidence accumulates and ends near its full-batch level.

Panel of Figure 18 — Figure 18: Posterior predictive bands of all four fits on one representative seed with identical axes. Orange dashed curves are PVMP and blue solid curves are NGMP; the matching shaded regions are pointwise 95% posterior-predictive bands. Gray dash-dotted curves show the true mean, and gray points are the 400 training observations. Panels (a) and (c) use all observations at once, whereas panels (b) and (d) use the ten-step filtering chain of Figure 9 . Panels (a), (c), and (d) are visually indistinguishable; only PVMP’s sequential fit in panel (b) collapses.

Panel of Figure 19 — Figure 19: Posterior predictive variance of all four fits on the same representative seed as Figure 18 . Panels (a) and (c) use all observations at once, whereas panels (b) and (d) use ten sequential batches. The orange long-dashed curves in panels (a)–(b) are PVMP and the blue solid curves in panels (c)–(d) are NGMP. Each is the posterior mean total predictive variance, $\bar{S}(x)=V_{\mathrm{epi}}(x)+V_{\mathrm{alea}}(x)$ , where $V_{\mathrm{epi}}(x)=\phi(x)^{\top}\Sigma_{v}\phi(x)$ is epistemic variance from the uncertain mean weights and $V_{\mathrm{alea}}(x)=\mathbb{E}_{q}[e^{-s(x)}]$ is learned aleatoric variance. Shading gives a pointwise 95% posterior credible interval for the total. The purple dotted curve shows $V_{\mathrm{epi}}(x)$ alone, so its vertical gap to the orange or blue total is $V_{\mathrm{alea}}(x)$ . The gray dash-dotted curve is the benchmark’s true aleatoric variance $[0.45(x+0.5)]^{2}$ . All panels use identical logarithmic axes.

	NLL			RMSE
Held out	PVMP	NCVMP	NGMP	PVMP	NCVMP	NGMP
5%	4.534 $\pm$ 0.087	12.492 $\pm$ 0.546	4.532 $\pm$ 0.085	13.918 $\pm$ 0.385	26.441 $\pm$ 0.951	13.935 $\pm$ 0.378
10%	4.569 $\pm$ 0.058	12.998 $\pm$ 0.453	4.566 $\pm$ 0.058	13.972 $\pm$ 0.260	27.137 $\pm$ 0.768	13.995 $\pm$ 0.260
20%	4.672 $\pm$ 0.039	13.547 $\pm$ 0.410	4.661 $\pm$ 0.037	14.429 $\pm$ 0.214	27.616 $\pm$ 0.549	14.446 $\pm$ 0.208
50%	5.411 $\pm$ 0.162	16.116 $\pm$ 0.391	4.930 $\pm$ 0.035	16.839 $\pm$ 0.471	30.729 $\pm$ 0.507	15.789 $\pm$ 0.162

Appendix

Table 5: Sunspot state-space model with the budget-matched control (mean

\pm

95% CI over 20 random masks): the two converged methods of Table 1 alongside NCVMP, that is, PVMP with single-step projections.

	NLL	RMSE	NLL	RMSE	batching penalty
	full batch		sequential
PVMP	$0.276\pm 0.031$	$0.588\pm 0.037$	$0.512\pm 0.044$	$0.583\pm 0.039$	$0.235\pm 0.030$
NCVMP	$0.913\pm 0.015$	$0.577\pm 0.035$	$0.916\pm 0.017$	$0.576\pm 0.035$	$0.003\pm 0.005$
NGMP	$0.235\pm 0.031$	$0.591\pm 0.038$	$0.244\pm 0.031$	$0.607\pm 0.042$	$0.009\pm 0.010$

Appendix

Table 6: Sequential heteroscedastic model with the budget-matched NCVMP control (mean

\pm

95% CI over 20 paired seeds), extending Table 2 .

Data set	Samples	Input features
Concrete	1,030	8
Energy	768	8
Boston Housing	506	13
Power Plant	9,568	4
Wine Quality Red	1,599	11
Yacht	308	6

Appendix

Table 7: UCI regression data sets used in Section 8.1 . The sample count is the complete data set before each repeated holdout.

Method	Architecture	Model coefficients $P$	Posterior / propagation	Uncertainty parameters $U$	Noise
hoBBB	$d\!\to\!50\!\to\!50\!\to\!1$	2,851–3,301	Diag.-G	2,851–3,301	fixed
BBB	$d\!\to\!50\!\to\!50\!\to\!2$	2,902–3,352	Diag.-G	2,902–3,352	input
hodDVI	$d\!\to\!50\!\to\!1$	301–751	Diag.-G / 50	301–751	fixed
dDVI	$d\!\to\!50\!\to\!2$	352–802	Diag.-G / 50	352–802	input
hoDVI	$d\!\to\!50\!\to\!1$	301–751	Diag.-G / 1,275	301–751	fixed
DVI	$d\!\to\!50\!\to\!2$	352–802	Diag.-G / 1,275	352–802	input

Appendix

Table 8: Numerical model and uncertainty capacity in the UCI comparison. NGMP uses the hierarchy in Figure 12 with

L=2

: one mean-weight vector and two precision-weight vectors. Each vector has 1,000 random-feature coefficients,

d

direct-input coefficients, and one intercept. All ranges are exact over the six data sets.

Method	Concrete	Energy	Boston	Power	Wine	Yacht
hoBBB	$3.8613\pm 0.0059$	$3.2892\pm 0.0043$	$3.2488\pm 0.0157$	$3.7960\pm 0.0019$	$1.0336\pm 0.0173$	$3.6925\pm 0.0059$
BBB	$3.3328\pm 0.0215$	$2.4229\pm 0.0350$	$2.6555\pm 0.0385$	$2.7785\pm 0.0143$	$\bm{0.9438\pm 0.0232}$	$2.6565\pm 0.0451$
hodDVI	$3.8602\pm 0.0066$	$3.2970\pm 0.0027$	$3.2571\pm 0.0111$	$3.7918\pm 0.0011$	$1.0360\pm 0.0164$	$3.7236\pm 0.0097$
dDVI	$3.0477\pm 0.0432$	$\bm{1.1920\pm 0.4625}$	$\bm{2.4646\pm 0.0921}$	$2.8238\pm 0.0183$	$\bm{0.9440\pm 0.0314}$	$0.4641\pm 0.1000$
hoDVI	$3.8602\pm 0.0066$	$3.2971\pm 0.0027$	$3.2572\pm 0.0111$	$3.7918\pm 0.0011$	$1.0360\pm 0.0164$	$3.7238\pm 0.0097$
DVI	$3.0406\pm 0.0438$	$\bm{1.0094\pm 0.2973}$	$\bm{2.4489\pm 0.0917}$	$2.8249\pm 0.0200$	$\bm{0.9448\pm 0.0307}$	$0.4543\pm 0.1096$

Appendix

Table 9: Full test negative log likelihood (NLL) in original target units for all baseline variants; lower is better. Values are means

\pm

approximate 95% confidence-interval half-widths over 20 paired splits. Bold values include the best point estimate within their confidence interval.

Method	Concrete	Energy	Boston	Power	Wine	Yacht
hoBBB	$7.2822\pm 0.2072$	$2.8175\pm 0.1459$	$\bm{3.8392\pm 0.4243}$	$4.2831\pm 0.0703$	$\bm{0.6469\pm 0.0185}$	$3.4875\pm 0.4063$
BBB	$6.7086\pm 0.3117$	$2.7537\pm 0.1548$	$\bm{3.7959\pm 0.4387}$	$\bm{4.0251\pm 0.0737}$	$\bm{0.6470\pm 0.0176}$	$3.0494\pm 0.4150$
hodDVI	$6.7179\pm 0.2764$	$2.2486\pm 0.1159$	$\bm{3.4801\pm 0.3374}$	$4.3009\pm 0.0719$	$\bm{0.6484\pm 0.0190}$	$2.1779\pm 0.2301$
dDVI	$\bm{5.6591\pm 0.3352}$	$2.1088\pm 0.3088$	$\bm{3.4741\pm 0.4119}$	$4.1321\pm 0.0763$	$\bm{0.6463\pm 0.0198}$	$\bm{0.8491\pm 0.1387}$
hoDVI	$6.7179\pm 0.2766$	$2.2493\pm 0.1161$	$\bm{3.4802\pm 0.3375}$	$4.3009\pm 0.0720$	$\bm{0.6484\pm 0.0190}$	$2.1787\pm 0.2301$
DVI	$\bm{5.6654\pm 0.3249}$	$1.9434\pm 0.3353$	$\bm{3.4291\pm 0.4013}$	$4.1428\pm 0.0801$	$\bm{0.6469\pm 0.0197}$	$\bm{0.8528\pm 0.1533}$

Appendix

Table 10: Test root mean squared error (RMSE) in original target units for all baseline variants; lower is better. Values are means

\pm

approximate 95% confidence-interval half-widths over 20 paired splits. Bold values include the best point estimate within their confidence interval.

ETTh1
Method	96	192	336	720
RMSE
MoE – Adam, affine gate	$0.3836\pm 0.0090$	$0.3699\pm 0.0091$	$0.4152\pm 0.0099$	$0.4639\pm 0.0111$
MoE – Adam, ReLU gate	$0.3955\pm 0.0097$	$0.3699\pm 0.0091$	$0.4801\pm 0.0123$	$0.4642\pm 0.0112$
MoE – IVON, affine gate ( $K{=}1000$ )	$0.3832\pm 0.0090$	$0.3695\pm 0.0091$	$0.3672\pm 0.0084$	$0.5428\pm 0.0127$
MoE – IVON, ReLU gate ( $K{=}1000$ )	$0.3832\pm 0.0090$	$0.3697\pm 0.0091$	$0.3677\pm 0.0084$	$0.5334\pm 0.0125$

Appendix

Table 11: Full descriptive ETTh1 and ETTh2 RMSE and NLL results. Intervals are 95% descriptive confidence-interval half-widths computed per test origin; RMSE intervals use the delta method. Counts are 3446, 3426, 3398, and 3321 at horizons 96, 192, 336, and 720. The PGE–PVMP rows are taken from Lukashchuk et al. (2026) , computed on the same test split.

Explore similar work

May 28, 2026cs.LG

Composing Non-Conjugate Factor Graphs with Closed-Form Variational Inference

Stacking probabilistic building blocks into deeper architectures typically breaks closed-form inference. We show that closed-form inference can be preserved. We identify five factor-graph primitives: a bilinear factor, an exponential link, a Gamma prior, a Gaussian likelihood, and an equality node, and prove that any model composed from them admits closed-form variational message passing. The construction works because each primitive preserves a small set of message families: under mean-field factorization, messages on Gaussian variables remain Gaussian and messages on precision variables remain Gamma, while the only non-conjugate interface, the exponential link, remains tractable through the Gaussian moment-generating function and the sufficient statistics of the Gamma family. We demonstrate composition at increasing depth, from static ensembles through input-dependent gating to split-branch routing, and show that stacking routing layers encodes arbitrary decision trees, establishing universal function approximation with closed-form inference. Applied to ensemble time-series forecasting, the framework yields a Bayesian mixture of experts in which gating functions are inferred rather than learned, providing calibrated uncertainty over expert selection across five benchmark datasets.

Mykola Lukashchuk, Kyrylo Yemets, Wouter M. Kouw +4

Eindhoven University of Technology, the Netherlands · Lviv Polytechnic National University, Lviv, Ukraine · Lazy Dynamics, Utrecht, the Netherlands +1

Factor Graph Optimization Mixture-of-Experts Inference Probabilistic Graphical Models Variational Inference

Ask PDF

Sep 26, 2026cs.LG

Efficient Message Passing for Partial Differential Equation Priors

Prior information for real-world physical quantities is most elegantly expressed via partial differential equations (PDEs). In this paper, we propose a novel way to solve PDEs using probabilistic inference on a factor graph. In general, factor graphs provide a natural way to encode prior knowledge into a model as explicit factors; here, this knowledge is provided by a governing PDE, which narrows the solution space, while observed data further shape the posterior over the parameters. The approximate parameter posterior is inferred using message passing based on moment matching, without posterior sampling or global gradient-based optimization. We demonstrate our approach on the first-order advection and the second-order semi-linear Fisher-KPP equations, where it achieves predictive accuracy comparable to a standard baseline while providing structured predictive uncertainty. Moreover, the inferred posterior marginal means and uncertainty structure match more closely those obtained using Hamiltonian Monte Carlo than the evaluated mean-field variational inference baseline, while requiring up to 10x less training time in our experiments, with inference speed comparable to variational inference.

Anna Kazachkova, Leonhard Hennicke, Rainer Schlosser +1

Hasso Plattner Institute University of Potsdam Potsdam, Germany

PDE Solving Physics-Informed ML Bayesian Inference

Ask PDF

Sep 24, 2026cs.LG

Direct Message Approximation (DMA): A Consistency-Based Framework for Tractable Approximate Inference on Factor Graphs

Approximate message passing on factor graphs underlies two dominant families of probabilistic inference algorithms: expectation propagation (EP) and variational message passing (VMP). Both methods approximate the marginal at each factor edge, forcing an iterative round-robin schedule, risking negative-precision messages, and, for VMP, collapsing to point estimates at Dirac-delta factors. We introduce Direct Message Approximation (DMA), which approximates factor-to-variable messages directly rather than the marginal. For normalisable factors, we define a consistency condition (requiring exactness when all other incoming messages are Dirac deltas) to guide message construction. We prove a master theorem (proper messages, any graph) bounding marginal KL from message KL, with three structural corollaries: Dirac-input consistency, no EP-style inner-loop iteration, and no negative-precision messages. Further, we prove a complementary

O(1/r^2)

guarantee for the inherently improper backward message of the product factor, whose closed-form treatment has resisted prior work. As a concrete instantiation, we derive explicit DMA messages for the product and leaky-ReLU factors and assemble a Bayesian neural network (BNN) inference algorithm with one forward/backward sweep per training example and no gradient learning-rate hyperparameter, validating that the structural guarantees translate to predictive uncertainty that widens in data-sparse regions, including under model mismatch.

Ralf Herbrich, Rainer Schlosser, Jan Lemcke +11

Hasso Plattner Institute University of Potsdam Potsdam, Germany

Efficient Inference Uncertainty Quantification Bayesian Inference

Ask PDF