cs.LGAug 16, 2026
SaveInformation Geometry of Message Passing
Organizations: Eindhoven University of Technology · Lviv Polytechnic National University · Lazy Dynamics, Utrecht, the Netherlands
Abstract
We show that the natural-gradient stationary condition of variational inference has an edge-local form on a Forney-style factor graph. We start from the Bethe free energy and constrain a selected edge marginal to an exponential family. At a stationary point, the natural parameter of that edge equals the sum of two projected messages, one from each incident factor. Each projected message is the natural-gradient projection of the exact belief-propagation log-message at the current receiving marginal, or equivalently, the gradient of its expectation in the so-called mean coordinates. We call the resulting scheme natural-gradient message passing (NGMP). The rule is local; each edge may carry its own exponential family, and the message a factor sends depends on the marginal that receives it. Compared with variational message passing, NGMP keeps the part of the exact message that the receiving family can represent instead of averaging the factor under the neighboring beliefs. The two coincide when the uncertainty on the edges entering a non-conjugate factor vanishes, and NGMP is more accurate when that uncertainty persists, for example, along a partially observed latent chain or when parameters are filtered through successive data batches. Experiments on Poisson smoothing, heteroskedastic regression, and hourly ETTh forecasting confirm this and show that the gain appears mainly in uncertainty calibration.
Figures & tables
Figure 1: Messages produced by belief propagation (BP) rely on the local factor and its other incoming messages. Variational message passing (VMP), including its structured variant, first forms a surrogate factor by variational expectation of with respect to the current marginals of the out-of-cluster variables, then passes a BP message through this new factor; non-conjugate and projective VMP keep this flow and additionally project the resulting message at the receiving marginal ( Section 6 ). Expectation propagation (EP) instead forms a cavity distribution from the opposing side, restores the exact factor contribution to obtain a tilted marginal, projects that marginal onto the chosen form-constraint family via , and divides out the cavity to obtain the new site ( Minka, 2001 ) . Finally, natural-gradient message passing uses the receiving marginal to define the tangent space for natural-gradient projection of the exact BP log-message. The final message towards is obtained as the exponential of the resulting projected natural parameters applied to the chosen family’s sufficient statistics .
Figure 2: Forney-style Factor Graph representation of the factorization in equation 4 .
Figure 3: Geometric view of the projected message. The exact BP message is , a tangent in the ambient probability space . The decomposition is drawn in log-message coordinates: is split into the tangent component and the orthogonal residual at the current marginal . The actual finite message sent along the constrained edge is obtained by exponentiation of the tangent component, . The residual may remain in the functional Lagrange multipliers, but it is invisible to the finite-dimensional edge marginal. Making the ambient space picture fully rigorous would require a non-parametric density-manifold formulation; see Wang and Li (2020) for tangent and cotangent spaces of probability densities and Fisher–Rao/Wasserstein information metrics. We use the ambient manifold here informally to explain what is the finite-dimensional projection.
Figure 4: Two equivalent implementations of the projected updates on two displayed observation slices of equation 39 ; both factor graphs are trees and the dots indicate continuation to later states. (a) Surrogate-model view. Each Poisson observation leaf is represented by a Gaussian pseudo-observation , so ordinary Gaussian BP runs on the entire auxiliary tree. (b) NGMP message view. The original Poisson factors remain visible; each blue label is the projected observation message and the adjacent red marginal defines its tangent space. Inserting a Gaussian factor with the blue message’s natural parameters produces the corresponding leaf in panel (a). Applying this conversion independently at every gives equation 45 . The same idea generalizes per interface: a multi-interface factor sends a separate projected message through each constrained interface, as shown in Figure 5 .
Figure 5: The two-observation instance of equation 46 and the message from the first observation toward . The Normal prior for and Gamma prior for are at the top; the two factors close the diamond loop. Solid edges carry and dashed edges carry . (a) VMP uses the previous global marginal to compute . (b) NGMP instead uses the cavity and projects the resulting exact BP log-message at . The blue labels enumerate the four messages around the loop; their subscripts give the directions. The red labels are the receiving marginals at which the two blue projected messages are evaluated. The new message changes the cavity entering the lower Normal, whose message toward changes the cavity returning to the upper Normal on the next sweep. The message toward is analogous.
Figure 6: Surrogate-model interpretation of the multi-interface NGMP computation in Figure 5 b. (a) The original observation factor touches the and equality constraints and produces one projected message for each interface. (b) With those message parameters frozen, the factor’s contributions are represented by a Gaussian leaf on and a Gamma leaf on ; their observed ports denote the projected natural parameters. Neighboring factors keep their ordinary BP rules. This is the direct multi-interface analogue of Figure 4 a, except that one original factor yields one surrogate leaf per constrained interface.
Figure 7: Marginal inclusive divergence in the isolated Normal mean–precision model: (a) and (b) (mean 95% CI over 20 instances; each instance draws a fresh true from the priors and generates its own stream of samples). For , VMP decays as and NGMP as ; for , NGMP is a roughly constant closer.
Figure 8: Sunspot results at holdout. Top: a five-year window around the deepest held-out stretch of a representative mask, one method per panel on identical axes; lines are the plug-in predictive rates , shaded bands push the pointwise credible intervals of through the exponential link, and diamonds are held-out counts. PVMP’s band keeps a fixed width inside the gap and its rate drifts confidently away from the held-out counts, while NGMP’s band widens toward the gap center and keeps covering them. Bottom: held-out negative log-likelihood (log scale) and posterior state variance against distance to the nearest observation over all 20 masks.
| NLL | RMSE | |||
|---|---|---|---|---|
| Held out | PVMP | NGMP | PVMP | NGMP |
| 5% | 4.534 0.087 | 4.532 0.085 | 13.918 0.385 | 13.935 0.378 |
| 10% | 4.569 0.058 | 4.566 0.058 | 13.972 0.260 | 13.995 0.260 |
| 20% | 4.672 0.039 | 4.661 0.037 | 14.429 0.214 | 14.446 0.208 |
| 50% | 5.411 0.162 | 4.930 0.035 | 16.839 0.471 | 15.789 0.162 |
Table 1: Sunspot state-space model: held-out predictive metrics (mean 95% CI over 20 random masks). PVMP and NGMP agree for short gaps and separate when the mask creates long uncertain stretches. The budget-matched NCVMP control is reported in Table 5 of Appendix D .
| full batch | sequential | ||||
|---|---|---|---|---|---|
| NLL | RMSE | NLL | RMSE | batching penalty | |
| PVMP | |||||
| NGMP | |||||
Table 2: Sequential heteroscedastic model over 20 paired seeds (mean 95% CI). The same 400 observations are fitted jointly or filtered in ten batches; the batching penalty is the resulting increase in held-out NLL.
Figure 9: Factor graph and online schedule for the sequential heteroscedastic model. Panel (a) shows branch of the batch box; each softdot node is the soft dot-product factor equation 58 . The upper softdot is the log-precision path of equation 57 , with , , , and ; the observation path below it is the same factor with the uncertain precision . The lateral stubs with ellipses at the equality nodes continue to the other observations , so the same and are shared by all 40 branches. Panel (b) opens the double-bordered composite and, inside it, the softdot box; the inner box is equation 58 drawn graphically. The deterministic dot node forms the dot product , the Gaussian receives it as its mean, and the precision of that Gaussian is the exponentiated incoming score . In panel (a) the ports are , , , and . Panel (c) repeats the compact batch box for beneath the two shared-weight chains. Batch 1 starts from the model priors ; after batch , the Gaussian marginals are frozen and copied left-to-right as the next priors. A perpendicular stop bar marks that no message from a later batch is sent back into an earlier batch; the bar specifies the online schedule and is not an additional model factor. With exact state-independent Gaussian messages, this posterior-as-prior multiplication is ordinary associative Kalman-style filtering. Solid edges carry weights, features, and observations; dashed edges carry precisions; dash-dotted edges carry the latent log-precision scores.
Figure 10: Posterior predictive bands after the ten-step filtering chain of Figure 9 , on one representative seed with identical axes. The orange dashed curve is PVMP and the blue solid curve is NGMP; the matching shaded regions are pointwise 95% posterior-predictive bands. Gray dash-dotted curves show the true mean, and gray points are the 400 training observations. Both methods resolve the oscillatory mean; under posterior reuse, PVMP’s noise-weight covariance collapses and its NLL rises, whereas NGMP remains close to its full-batch NLL. The full-batch fits of both methods are visually indistinguishable from panel (b); Figure 18 in Appendix D shows all four fits.
Figure 11: Posterior predictive variance on the same representative seed as Figure 10 . The blue solid curve in panel (a) is NGMP and the orange long-dashed curves in panels (b)–(c) are PVMP; panel (b) uses all observations at once, panels (a) and (c) use the ten sequential batches. Each curve is the posterior mean total predictive variance, , where is epistemic variance from the uncertain mean weights and is learned aleatoric variance. Shading gives a pointwise 95% posterior credible interval for the total. The purple dotted curve shows alone, so its vertical gap to the orange or blue total is . The gray dash-dotted curve is the benchmark’s true aleatoric variance . These pointwise credible intervals describe posterior uncertainty for one fit and are distinct from the across-seed confidence intervals in Table 2 . All panels use identical logarithmic axes. NGMP’s full-batch fit is visually indistinguishable from panel (a); Figure 19 in Appendix D shows all four fits.
Figure 12: One observation branch of the arbitrary-depth heteroscedastic hierarchy used in the regression experiment. Every level has its own Gaussian weights , shared across observation branches through the equality nodes, and its own clamped feature map . The top-level softdot is the soft dot-product factor equation 58 . Each level turns its incoming score into a precision and emits the score read by the level below, and the observation level turns the last score into the noise precision of . The double-bordered composites are opened in Figure 9 (b). The experiments in this section use the depth-three instance. Edge styles are as in Figure 9 : solid for weights, features, and observations; dashed for precisions; dash-dotted for scores.
| Method | Concrete | Energy | Boston | Power | Wine | Yacht |
|---|---|---|---|---|---|---|
| BBB | ||||||
| dDVI | ||||||
| DVI | ||||||
| IVON ( ) | ||||||
| NGMP |
Table 3: Test negative log likelihood (NLL) in original target units; lower is better. Values are means approximate 95% confidence-interval half-widths over 20 paired splits. Bold values include the best point estimate within their confidence interval. BBB, dDVI, and DVI are the heteroscedastic variants; IVON is homoscedastic and uses posterior network draws for its predictive mixture. The remaining homoscedastic variants are in Table 9 .
| ETTh1 | ||||
|---|---|---|---|---|
| Method | 96 | 192 | 336 | 720 |
| RMSE | ||||
| MoE – IVON, affine gate ( ) | ||||
| MoE – IVON, ReLU gate ( ) | ||||
| PGE – PVMP ( Lukashchuk et al., 2026 ) | ||||
| PGE – NGMP (this work) | ||||
Table 4: ETTh forecasting in standardized OT units. Entries are test-set point estimates approximate 95% normal confidence-interval half-widths over test predictions; RMSE intervals use the delta method. Lower is better. Among the methods shown, the lowest point estimate is bold; the second-lowest is also bold when its paired 95% normal confidence interval relative to the lowest, computed on identical test predictions, includes zero. For IVON, is the number of gate-weight samples drawn from the fitted variational posterior to approximate the posterior predictive. The predictive-mixture construction and complete experimental setup are described in Section E.2 ; the full results, including Adam-trained gates, are reported in Table 11 .
Figure 13: ETTh1 at : NGMP (solid blue) and PVMP (hatched orange), with 95% predictive intervals, over an illustrative one-week segment. The black solid line is the observed target. This panel illustrates the predictions only; the full-test RMSE and NLL comparison is reported in Table 4 .
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 14: Convergence of the NGMP fixed-point iteration on the Poisson state-space model equation 39 for series simulated from the same model ( , all counts observed) of length , (top row), , and (bottom row), 20 seeds each, 200 variational iterations. BFE on the vertical axis is the Bethe free energy of the model per observation minus the value reached at convergence by the setting on the same seed, so every seed converges to zero (log scale); curves are the geometric mean across seeds with a 95% confidence band. Orange: undamped fixed-point iteration ( ). Blue: messages damped in natural coordinates with . Green dashed: the damped heavy-ball setting used for the sunspot experiments ( , ); its momentum step left the natural domain and the fit stopped on 3 of 20 seeds at and 9 of 20 at (the undamped iteration failed on one seed at ), and the curves average over the seeds that reached each iteration. Longer chains make larger excursions of the latent log-rate; the undamped map then stops converging while the damped map still does.
Figure 15: Bethe free energy per observed count on the sunspot series (mean 95% CI across 20 masks, log scale) at and holdout (top row) and and holdout (bottom row). Both methods plateau within the 20-sweep budget: NGMP by sweep three to four, PVMP shortly after. Each PVMP trace evaluates its own variational objective; the NGMP trace is a surrogate diagnostic, since the local Gaussian surrogates are rebuilt between outer sweeps. The dotted trace is the budget-matched NCVMP control of Table 5 .
Figure 16: Bethe free energy per observation for the sequential heteroscedastic model (mean 95% CI over 20 seeds). Left: the full-batch fit against sweep (first two sweeps omitted for scale). Right: the final-sweep value of each filtering update against batch index. PVMP evaluates its own mean-field variational objective; the NGMP trace scores the same node energies through moment-matched joint cluster marginals, so the two curves approximate different Bethe functionals and their levels are not directly comparable. The dotted trace is the budget-matched NCVMP control of Table 6 .
Figure 17: Noise-weight concentration along the filtering chain (mean 95% CI over 20 seeds; dotted lines show full-batch values; lower means more certain). PVMP’s sequential endpoint coincides with its full-batch value, and the budget-matched NCVMP control ( Table 6 ) traces the identical curve; NGMP concentrates as evidence accumulates and ends near its full-batch level.
Figure 18: Posterior predictive bands of all four fits on one representative seed with identical axes. Orange dashed curves are PVMP and blue solid curves are NGMP; the matching shaded regions are pointwise 95% posterior-predictive bands. Gray dash-dotted curves show the true mean, and gray points are the 400 training observations. Panels (a) and (c) use all observations at once, whereas panels (b) and (d) use the ten-step filtering chain of Figure 9 . Panels (a), (c), and (d) are visually indistinguishable; only PVMP’s sequential fit in panel (b) collapses.
Figure 19: Posterior predictive variance of all four fits on the same representative seed as Figure 18 . Panels (a) and (c) use all observations at once, whereas panels (b) and (d) use ten sequential batches. The orange long-dashed curves in panels (a)–(b) are PVMP and the blue solid curves in panels (c)–(d) are NGMP. Each is the posterior mean total predictive variance, , where is epistemic variance from the uncertain mean weights and is learned aleatoric variance. Shading gives a pointwise 95% posterior credible interval for the total. The purple dotted curve shows alone, so its vertical gap to the orange or blue total is . The gray dash-dotted curve is the benchmark’s true aleatoric variance . All panels use identical logarithmic axes.
| NLL | RMSE | |||||
|---|---|---|---|---|---|---|
| Held out | PVMP | NCVMP | NGMP | PVMP | NCVMP | NGMP |
| 5% | 4.534 0.087 | 12.492 0.546 | 4.532 0.085 | 13.918 0.385 | 26.441 0.951 | 13.935 0.378 |
| 10% | 4.569 0.058 | 12.998 0.453 | 4.566 0.058 | 13.972 0.260 | 27.137 0.768 | 13.995 0.260 |
| 20% | 4.672 0.039 | 13.547 0.410 | 4.661 0.037 | 14.429 0.214 | 27.616 0.549 | 14.446 0.208 |
| 50% | 5.411 0.162 | 16.116 0.391 | 4.930 0.035 | 16.839 0.471 | 30.729 0.507 | 15.789 0.162 |
Table 5: Sunspot state-space model with the budget-matched control (mean 95% CI over 20 random masks): the two converged methods of Table 1 alongside NCVMP, that is, PVMP with single-step projections.
| full batch | sequential | ||||
|---|---|---|---|---|---|
| NLL | RMSE | NLL | RMSE | batching penalty | |
| PVMP | |||||
| NCVMP | |||||
| NGMP | |||||
Table 6: Sequential heteroscedastic model with the budget-matched NCVMP control (mean 95% CI over 20 paired seeds), extending Table 2 .
| Data set | Samples | Input features |
|---|---|---|
| Concrete | 1,030 | 8 |
| Energy | 768 | 8 |
| Boston Housing | 506 | 13 |
| Power Plant | 9,568 | 4 |
| Wine Quality Red | 1,599 | 11 |
| Yacht | 308 | 6 |
Table 7: UCI regression data sets used in Section 8.1 . The sample count is the complete data set before each repeated holdout.
| Method | Architecture | Model coefficients | Posterior / propagation | Uncertainty parameters | Noise |
|---|---|---|---|---|---|
| hoBBB | 2,851–3,301 | Diag.-G | 2,851–3,301 | fixed | |
| BBB | 2,902–3,352 | Diag.-G | 2,902–3,352 | input | |
| hodDVI | 301–751 | Diag.-G / 50 | 301–751 | fixed | |
| dDVI | 352–802 | Diag.-G / 50 | 352–802 | input | |
| hoDVI | 301–751 | Diag.-G / 1,275 | 301–751 | fixed | |
| DVI | 352–802 | Diag.-G / 1,275 | 352–802 | input |
Table 8: Numerical model and uncertainty capacity in the UCI comparison. NGMP uses the hierarchy in Figure 12 with : one mean-weight vector and two precision-weight vectors. Each vector has 1,000 random-feature coefficients, direct-input coefficients, and one intercept. All ranges are exact over the six data sets.
| Method | Concrete | Energy | Boston | Power | Wine | Yacht |
|---|---|---|---|---|---|---|
| hoBBB | ||||||
| BBB | ||||||
| hodDVI | ||||||
| dDVI | ||||||
| hoDVI | ||||||
| DVI |
Table 9: Full test negative log likelihood (NLL) in original target units for all baseline variants; lower is better. Values are means approximate 95% confidence-interval half-widths over 20 paired splits. Bold values include the best point estimate within their confidence interval.
| Method | Concrete | Energy | Boston | Power | Wine | Yacht |
|---|---|---|---|---|---|---|
| hoBBB | ||||||
| BBB | ||||||
| hodDVI | ||||||
| dDVI | ||||||
| hoDVI | ||||||
| DVI |
Table 10: Test root mean squared error (RMSE) in original target units for all baseline variants; lower is better. Values are means approximate 95% confidence-interval half-widths over 20 paired splits. Bold values include the best point estimate within their confidence interval.
| ETTh1 | ||||
|---|---|---|---|---|
| Method | 96 | 192 | 336 | 720 |
| RMSE | ||||
| MoE – Adam, affine gate | ||||
| MoE – Adam, ReLU gate | ||||
| MoE – IVON, affine gate ( ) | ||||
| MoE – IVON, ReLU gate ( ) | ||||
Table 11: Full descriptive ETTh1 and ETTh2 RMSE and NLL results. Intervals are 95% descriptive confidence-interval half-widths computed per test origin; RMSE intervals use the delta method. Counts are 3446, 3426, 3398, and 3321 at horizons 96, 192, 336, and 720. The PGE–PVMP rows are taken from Lukashchuk et al. (2026) , computed on the same test split.