Generalized Matheron Variational Implicit Processes

Luis A. Ortega; Andrés R. Masegosa; Thomas D. Nielsen

cs.LGOct 6, 2026

Generalized Matheron Variational Implicit Processes

Authors: Luis A. Ortega, Andrés R. Masegosa, Thomas D. Nielsen

Organizations: Aalborg University

Abstract

Implicit-process priors specify distributions over functions through sample-forward mechanisms such as Bayesian neural networks and stochastic simulators, but their function-space densities are typically unavailable. We introduce Generalized Matheron Variational Implicit Processes (GMVIP), a pathwise variational family for posterior inference with such priors. For Gaussian-process priors, GMVIP recovers the standard inducing-variable variational GP construction; for general implicit priors, its empirical covariance construction preserves the prior mean and covariance in the population limit. GMVIP constructs posterior samples by drawing a function from the prior and applying a correction anchored at a set of inducing inputs. The effect of this correction away from the inducing inputs is determined directly from prior samples, allowing the posterior to retain the structure and variability of the original implicit process. The (surrogate) prior and variational posterior use the same pathwise construction and differ only in the distribution of whitened inducing coefficients, yielding a tractable coefficient-space Kullback-Leibler divergence. Experiments on regression, classification, and forecasting with simulator-defined and retrieval-conditioned empirical trajectory priors show that GMVIP is broadly competitive with existing methods.

Figures & tables

Figure 1: Pathwise construction of a GMVIP sample. Starting from a prior draw $g\sim p_{0}$ , we evaluate the path at inducing inputs $\mathbf{Z}$ and form the residual between the sampled replacement values $\bm{\mu}+\mathbf{D}\mathbf{a}$ and $g(\mathbf{Z})$ . The Matheron operator $\bm{\Psi}_{\mathbf{Z}}$ propagates this residual across the input domain. Adding the resulting smooth correction to $g$ yields a sample $f$ that matches the replacement values at $\mathbf{Z}$ .

RMSE	Boston	Concrete	Energy	Kin8nm	Naval ( $\times 10^{-4}$ )	Power	Protein	Wine Red	Yacht
MAP	$4.34\pm 1.41$	$5.45\pm 0.57$	$0.52\pm 0.05$	$0.082\pm 0.001$	$4.04\pm 1.97$	$3.94\pm 0.13$	$4.62\pm 0.07$	$0.69\pm 0.01$	$0.52\pm 0.20$
MFVI	$3.93\pm 0.70$	$6.23\pm 0.51$	$0.62\pm 0.09$	$0.091\pm 0.002$	$147.61\pm 0.90$	$4.12\pm 0.17$	$4.94\pm 0.08$	${\color[rgb]{0,0.5,0.5}\mathbf{0.62\pm 0.03}}$	$0.68\pm 0.10$
VIP	$6.49\pm 1.27$	$5.76\pm 0.97$	$0.58\pm 0.09$	$0.080\pm 0.003$	$2.59\pm 0.96$	${\color[rgb]{0.75,0,0.25}\mathbf{3.76\pm 0.07}}$	$4.37\pm 0.06$	$0.79\pm 0.05$	$0.84\pm 0.30$
FBNN	$4.17\pm 1.27$	${\color[rgb]{0,0.5,0.5}\mathbf{5.21\pm 0.52}}$	${\color[rgb]{0,0.5,0.5}\mathbf{0.51\pm 0.05}}$	$0.086\pm 0.001$	$3.95\pm 0.47$	$4.05\pm 0.16$	$4.81\pm 0.05$	$0.65\pm 0.04$	${\color[rgb]{0.75,0,0.25}\mathbf{0.23\pm 0.06}}$
SIP	${\color[rgb]{0.75,0,0.25}\mathbf{3.22\pm 0.99}}$	$5.30\pm 0.27$	$0.52\pm 0.04$	${\color[rgb]{0,0.5,0.5}\mathbf{0.071\pm 0.001}}$	${\color[rgb]{0,0.5,0.5}\mathbf{1.48\pm 0.23}}$	$3.84\pm 0.11$	$4.62\pm 0.05$	$0.66\pm 0.03$	$0.41\pm 0.15$
TFSVI	$6.39\pm 0.63$	$5.97\pm 0.85$	$0.59\pm 0.15$	$0.081\pm 0.002$	$3.95\pm 0.81$	$3.90\pm 0.13$	$4.61\pm 0.05$	$0.73\pm 0.02$	$0.83\pm 0.38$

Table 1: UCI regression metrics (

\downarrow

). Entries are mean

\pm

standard deviation across five seeds. Best mean per dataset is shown in purple ; second-best mean is shown in teal .

Figure 2: Lotka–Volterra forecasting with an implicit ODE prior. Black points are noisy observations. Curves and bands show posterior predictions from the final scheduled optimization iterate.

Method	RMSE $\downarrow$	NLL $\downarrow$	CRPS $\downarrow$	Cov 90	ODE residual $\downarrow$
Prior predictive	$0.63\pm 0.06$	${\color[rgb]{0,0.5,0.5}\mathbf{0.90\pm 0.11}}$	$0.36\pm 0.04$	${\color[rgb]{0.75,0,0.25}\mathbf{0.90\pm 0.03}}$	${\color[rgb]{0.75,0,0.25}\mathbf{\approx 10^{-6}}}$
GMVIP sur. prior	$0.63\pm 0.06$	$1.95\pm 0.93$	$0.37\pm 0.03$	$0.93\pm 0.02$	$0.31\pm 0.00$
VIP	${\color[rgb]{0,0.5,0.5}\mathbf{0.50\pm 0.08}}$	$11.32\pm 5.01$	${\color[rgb]{0,0.5,0.5}\mathbf{0.31\pm 0.05}}$	$0.72\pm 0.06$	$0.65\pm 0.03$
FTIP	$0.57\pm 0.08$	$1.04\pm 0.22$	$0.34\pm 0.05$	$0.82\pm 0.04$	$0.62\pm 0.03$
GMVIP	${\color[rgb]{0.75,0,0.25}\mathbf{0.32\pm 0.06}}$	${\color[rgb]{0.75,0,0.25}\mathbf{0.51\pm 0.29}}$	${\color[rgb]{0.75,0,0.25}\mathbf{0.20\pm 0.04}}$	${\color[rgb]{0,0.5,0.5}\mathbf{0.90\pm 0.04}}$	${\color[rgb]{0,0.5,0.5}\mathbf{0.31\pm 0.02}}$

Table 2: Mean

\pm

standard error Lotka–Volterra results on the held-out test interval

t\in(20,30]

across 20 target trajectories. VIP and FTIP use

S=20

. VIP is trained for 800 iterations; FTIP uses a 400-iteration VIP warm start followed by 400 fine-tuning iterations. Best results are shown in purple and second-best results in teal . Coverage is ranked by proximity to the nominal

0.90

level.

Figure 3: Electricity-load forecasting with a retrieval-conditioned empirical prior. Methods observe the first 24 hours and forecast the following 24 hours. The panels show VIP, FTIP, and GMVIP on the same target window. Black points are observations, the black curve is the held-out target, and blue curves and bands show posterior predictions.

Method	RMSE $\downarrow$	NLL $\downarrow$	CRPS $\downarrow$	CQM $\downarrow$	Cov. 80%	Cov. 90%
Analog prior	$55.45\pm 15.91$	${\color[rgb]{0,0.5,0.5}\mathbf{5.31\pm 0.44}}$	$30.73\pm 9.04$	${\color[rgb]{0,0.5,0.5}\mathbf{0.12\pm 0.01}}$	$74.18\pm 2.14$	$85.96\pm 1.70$
VIP	$60.93\pm 18.61$	$99.19\pm 55.22$	$40.60\pm 12.83$	$0.38\pm 0.01$	$20.94\pm 1.16$	$26.26\pm 1.42$
FTIP	$68.02\pm 22.10$	$10.03\pm 4.17$	$41.28\pm 13.99$	$0.17\pm 0.01$	$57.25\pm 2.03$	$68.06\pm 1.99$
Empirical Gaussian	${\color[rgb]{0.75,0,0.25}\mathbf{48.44\pm 15.27}}$	$6.63\pm 0.99$	${\color[rgb]{0,0.5,0.5}\mathbf{28.37\pm 9.06}}$	$0.13\pm 0.01$	${\color[rgb]{0.75,0,0.25}\mathbf{79.32\pm 2.16}}$	${\color[rgb]{0.75,0,0.25}\mathbf{86.62\pm 1.78}}$
GMVIP	${\color[rgb]{0,0.5,0.5}\mathbf{49.06\pm 15.06}}$	${\color[rgb]{0.75,0,0.25}\mathbf{4.35\pm 0.14}}$	${\color[rgb]{0.75,0,0.25}\mathbf{27.48\pm 8.74}}$	${\color[rgb]{0.75,0,0.25}\mathbf{0.11\pm 0.01}}$	${\color[rgb]{0,0.5,0.5}\mathbf{76.22\pm 2.11}}$	${\color[rgb]{0,0.5,0.5}\mathbf{86.53\pm 1.73}}$

Table 3: Electricity-load results across

75

seed–target windows. Entries are means

\pm

standard errors across windows. Coverage and its standard error are reported in percentage points. Best results are shown in purple and second-best results in teal . Coverage is ranked by proximity to its nominal level; lower is better otherwise.

Metric	Dataset	MFVI	VIP	FBNN	SIP	TFSVI	FTIP	GMVIP
RMSE	Year	${\color[rgb]{0,0.5,0.5}\mathbf{8.95\pm 0.01}}$	$9.17\pm 0.01$	${\color[rgb]{0.75,0,0.25}\mathbf{8.91\pm 0.02}}$	$9.06\pm 0.01$	$9.05\pm 0.01$	$9.15\pm 0.01$	$8.98\pm 0.02$
RMSE	Airline	$37.87\pm 0.12$	$38.03\pm 0.16$	$37.90\pm 0.09$	$37.99\pm 0.07$	${\color[rgb]{0,0.5,0.5}\mathbf{37.72\pm 0.13}}$	$38.04\pm 0.12$	${\color[rgb]{0.75,0,0.25}\mathbf{37.62\pm 0.10}}$
NLL	Year	${\color[rgb]{0,0.5,0.5}\mathbf{3.61\pm 0.00}}$	$3.64\pm 0.00$	${\color[rgb]{0.75,0,0.25}\mathbf{3.60\pm 0.00}}$	$3.62\pm 0.00$	$3.62\pm 0.00$	$3.63\pm 0.00$	${\color[rgb]{0,0.5,0.5}\mathbf{3.61\pm 0.00}}$
NLL	Airline	${\color[rgb]{0.75,0,0.25}\mathbf{5.07\pm 0.00}}$	$5.09\pm 0.01$	${\color[rgb]{0.75,0,0.25}\mathbf{5.07\pm 0.00}}$	${\color[rgb]{0,0.5,0.5}\mathbf{5.08\pm 0.00}}$	$5.09\pm 0.00$	$5.09\pm 0.01$	${\color[rgb]{0.75,0,0.25}\mathbf{5.07\pm 0.01}}$
CRPS	Year	${\color[rgb]{0,0.5,0.5}\mathbf{4.717\pm 0.007}}$	$4.859\pm 0.010$	${\color[rgb]{0.75,0,0.25}\mathbf{4.675\pm 0.012}}$	$4.785\pm 0.010$	$4.777\pm 0.012$	$4.845\pm 0.004$	$4.740\pm 0.014$
CRPS	Airline	$17.67\pm 0.13$	$17.79\pm 0.22$	$17.70\pm 0.17$	$17.71\pm 0.08$	${\color[rgb]{0.75,0,0.25}\mathbf{17.47\pm 0.13}}$	$17.79\pm 0.17$	${\color[rgb]{0,0.5,0.5}\mathbf{17.61\pm 0.04}}$

Table 4: Large regression metrics (

\downarrow

). Entries are mean

\pm

standard deviation across seeds. Best method is shown in purple ; second-best is shown in teal .

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Synthetic one-dimensional regression. Black points denote observed data. Each panel shows the predictive behavior of one method on the same input grid. MAP gives a deterministic fit, while the Bayesian and function-space methods represent posterior uncertainty through predictive samples with learned Gaussian noise.

Method	RMSE $\downarrow$	NLL $\downarrow$	CRPS $\downarrow$	CQM $\downarrow$	Cov. 80%	Cov. 90%
Analog prior	$17.46\ [12.80,25.41]$	${\color[rgb]{0,0.5,0.5}\mathbf{4.30\ [4.06,4.70]}}$	$8.92\ [6.49,13.78]$	${\color[rgb]{0.75,0,0.25}\mathbf{0.093\ [0.077,0.112]}}$	${\color[rgb]{0,0.5,0.5}\mathbf{78.13\ [73.96,84.38]}}$	${\color[rgb]{0.75,0,0.25}\mathbf{91.67\ [88.54,92.71]}}$
VIP	$22.94\ [14.13,27.18]$	$10.05\ [8.60,11.64]$	$15.41\ [9.70,18.26]$	$0.384\ [0.365,0.396]$	$20.83\ [17.71,22.92]$	$25.00\ [21.88,29.17]$
FTIP	$21.20\ [15.75,28.02]$	$4.88\ [4.27,5.37]$	$12.35\ [8.81,17.57]$	$0.153\ [0.132,0.174]$	$57.29\ [54.17,61.46]$	$68.75\ [66.67,73.96]$
Empirical Gaussian	${\color[rgb]{0,0.5,0.5}\mathbf{16.99\ [11.14,25.10]}}$	$4.36\ [4.02,5.06]$	${\color[rgb]{0,0.5,0.5}\mathbf{8.39\ [6.06,14.05]}}$	$0.108\ [0.070,0.151]$	$84.38\ [80.21,87.50]$	${\color[rgb]{0.75,0,0.25}\mathbf{91.67\ [89.58,93.75]}}$
GMVIP	${\color[rgb]{0.75,0,0.25}\mathbf{16.20\ [11.14,24.85]}}$	${\color[rgb]{0.75,0,0.25}\mathbf{4.08\ [3.79,4.54]}}$	${\color[rgb]{0.75,0,0.25}\mathbf{8.14\ [5.93,13.52]}}$	${\color[rgb]{0,0.5,0.5}\mathbf{0.102\ [0.072,0.112]}}$	${\color[rgb]{0.75,0,0.25}\mathbf{80.21\ [77.08,84.38]}}$	${\color[rgb]{0,0.5,0.5}\mathbf{92.71\ [89.58,93.75]}}$

Appendix

Table 5: Electricity-load results across

75

seed–target windows. Entries are medians with

95\%

percentile-bootstrap confidence intervals from

100{,}000

resamples. Coverage is reported in percentage points and ranked by proximity to its nominal level. Lower is better otherwise.

Figure 5: Original and surrogate prior paths for the published synthetic configurations. Thin curves are sampled functions, thick curves are sample means, and shaded regions are pointwise

90\%

intervals. VIP uses

S=20

; GMVIP uses

M=256

inducing inputs and

B=1024

operator-bank paths. The VIP mean and path distribution visibly depend on the particular finite basis, whereas the GMVIP residual paths retain the qualitative variability of the original BNN prior.

Distribution	Configuration	$\operatorname{SW}_{2}$	$\overline{W}_{1}$	$\max_{x}W_{1}$	$e_{\mu}$	$e_{\mathbf{K}}$	Energy	$\operatorname{MMD}_{\mathrm{RBF}}^{2}$
True-prior split	$R_{\mathrm{ev}}=2048$	$0.064\pm 0.004$	$0.045\pm 0.005$	$0.066\pm 0.007$	$0.034\pm 0.008$	$0.056\pm 0.010$	$0.0032\pm 0.0003$	$0.0011\pm 0.0001$
VIP surrogate	$S=20$	$0.253\pm 0.032$	$0.214\pm 0.030$	$0.312\pm 0.029$	$0.195\pm 0.030$	$0.369\pm 0.073$	$0.0317\pm 0.0088$	$0.0117\pm 0.0028$
GMVIP surrogate	$M=256,\ B=1024$	$\bm{0.072\pm 0.004}$	$\bm{0.049\pm 0.003}$	$\bm{0.069\pm 0.006}$	$\bm{0.040\pm 0.005}$	$\bm{0.073\pm 0.009}$	$\bm{0.0056\pm 0.0001}$	$\bm{0.0022\pm 0.0001}$

Appendix

Table 6: Surrogate-prior fidelity at the published synthetic configurations (

\downarrow

). Entries are mean

\pm

standard error across five seeds. The true-prior split is the finite-sample noise floor rather than a competing method. Bold marks the closer of the two nontrivial surrogates.

Figure 6: Pointwise marginal

W_{1}

for the published VIP and GMVIP configurations. Curves and bands show mean

\pm

standard error across five seeds. The GMVIP surrogate follows the true-prior split floor across

[-5,5]

; the VIP finite-basis surrogate has a much larger discrepancy throughout the interval.

Coefficient dim.	$\operatorname{SW}_{2}$		$\overline{W}_{1}$
$S=M$	VIP	GMVIP	VIP	GMVIP
$8$	$0.367\pm 0.043$	$\bm{0.083\pm 0.005}$	$0.303\pm 0.039$	$\bm{0.059\pm 0.007}$
$20$	$0.253\pm 0.032$	$\bm{0.078\pm 0.004}$	$0.214\pm 0.030$	$\bm{0.055\pm 0.007}$
$32$	$0.189\pm 0.037$	$\bm{0.077\pm 0.003}$	$0.144\pm 0.031$	$\bm{0.049\pm 0.001}$
$64$	$0.144\pm 0.018$	$\bm{0.081\pm 0.005}$	$0.103\pm 0.014$	$\bm{0.054\pm 0.006}$
$128$	$0.119\pm 0.022$	$\bm{0.074\pm 0.006}$	$0.086\pm 0.024$	$\bm{0.049\pm 0.005}$

Appendix

Table 7: Core distances under matched coefficient dimension (

\downarrow

). GMVIP uses

B=1024

. Entries are mean

\pm

standard error across five seeds; bold marks the smaller distance in each pair.

Figure 7: Joint sliced

W_{2}

and average marginal

W_{1}

against matched coefficient dimension. Error bars are standard errors across five seeds; the dashed line and gray band show the true-prior split mean and standard error. VIP approaches the original prior as its sampled-function basis grows, whereas GMVIP is near the finite-sample floor throughout the sweep.

Figure 8: Standardized mean RMSE and relative covariance error under matched coefficient dimension. GMVIP remains close to the finite-sample floor because it combines an empirical prior-covariance correction with a fresh residual prior path. VIP moment errors decrease with basis size but remain larger over the tested range.

Figure 9: Energy distance and multi-bandwidth RBF

\operatorname{MMD}_{\mathrm{RBF}}^{2}

under matched coefficient dimension. These metrics use a deterministic

512

-sample prefix. They reproduce the Wasserstein conclusion: GMVIP stays close to the true-split floor, while VIP requires a much larger basis to reduce the discrepancy.

Bank $B$	$\operatorname{SW}_{2}$	$\overline{W}_{1}$	$e_{\mu}$	$e_{\mathbf{K}}$	Energy	$\operatorname{MMD}_{\mathrm{RBF}}^{2}$
$256$	$0.093\pm 0.013$	$0.065\pm 0.011$	$0.058\pm 0.017$	$0.105\pm 0.020$	$0.0083\pm 0.0018$	$0.0028\pm 0.0006$
$512$	$0.106\pm 0.008$	$0.083\pm 0.009$	$0.071\pm 0.011$	$0.116\pm 0.022$	$0.0081\pm 0.0018$	$0.0029\pm 0.0006$
$1024$	$0.072\pm 0.004$	$0.049\pm 0.003$	$0.040\pm 0.005$	$0.073\pm 0.009$	$0.0056\pm 0.0001$	$0.0022\pm 0.0001$
$2048$	$\bm{0.066\pm 0.005}$	$\bm{0.043\pm 0.005}$	$\bm{0.035\pm 0.007}$	$\bm{0.059\pm 0.011}$	$\bm{0.0044\pm 0.0011}$	$\bm{0.0015\pm 0.0005}$

Appendix

Table 8: GMVIP operator-bank sensitivity at

M=256

(

\downarrow

). Entries are mean

\pm

standard error across five seeds.

Figure 10: GMVIP joint and marginal prior discrepancy as a function of operator-bank size at

M=256

. The dashed line and gray band denote the true-prior split floor. At

B=2048

, the error bars overlap the floor at the resolution of these sample-based diagnostics, although the smaller-bank trend is non-monotone.

Metric	Dataset	MAP	MFVI	VIP	FBNN	SIP	FTIP	GMVIP
NLL	FashionMNIST	$0.287\pm 0.012$	$0.296\pm 0.014$	$0.282\pm 0.015$	${\color[rgb]{0.75,0,0.25}\mathbf{0.261\pm 0.016}}$	$0.277\pm 0.009$	${\color[rgb]{0,0.5,0.5}\mathbf{0.274\pm 0.017}}$	${\color[rgb]{0.75,0,0.25}\mathbf{0.261\pm 0.015}}$
NLL	CIFAR10	$1.099\pm 0.054$	$1.189\pm 0.023$	$1.190\pm 0.081$	${\color[rgb]{0,0.5,0.5}\mathbf{1.093\pm 0.065}}$	${\color[rgb]{0.75,0,0.25}\mathbf{1.039\pm 0.065}}$	$1.344\pm 0.109$	${\color[rgb]{0,0.5,0.5}\mathbf{1.093\pm 0.025}}$
Error	FashionMNIST	$0.102\pm 0.004$	$0.108\pm 0.006$	$0.099\pm 0.006$	${\color[rgb]{0,0.5,0.5}\mathbf{0.093\pm 0.005}}$	$0.094\pm 0.005$	$0.096\pm 0.005$	${\color[rgb]{0.75,0,0.25}\mathbf{0.092\pm 0.005}}$
Error	CIFAR10	$0.374\pm 0.018$	$0.424\pm 0.010$	$0.372\pm 0.012$	$0.350\pm 0.009$	${\color[rgb]{0.75,0,0.25}\mathbf{0.342\pm 0.010}}$	$0.374\pm 0.016$	${\color[rgb]{0,0.5,0.5}\mathbf{0.348\pm 0.010}}$
ECE	FashionMNIST	$0.020\pm 0.006$	${\color[rgb]{0.75,0,0.25}\mathbf{0.016\pm 0.003}}$	${\color[rgb]{0,0.5,0.5}\mathbf{0.019\pm 0.004}}$	$0.020\pm 0.005$	$0.024\pm 0.009$	$0.021\pm 0.007$	$0.021\pm 0.005$
ECE	CIFAR10	${\color[rgb]{0,0.5,0.5}\mathbf{0.069\pm 0.015}}$	${\color[rgb]{0.75,0,0.25}\mathbf{0.045\pm 0.011}}$	$0.121\pm 0.023$	$0.105\pm 0.017$	$0.071\pm 0.033$	$0.165\pm 0.019$	$0.102\pm 0.004$

Appendix

Table 9: Bayesian-head LeNet classification metrics (

\downarrow

). Entries are mean

\pm

standard deviation across five seeds. Best results are shown in purple ; second-best are shown in teal .

Method	Accuracy (%) $\uparrow$	NLL $\downarrow$	ECE $\downarrow$
MAP	$93.93\pm 0.02$	$0.1855\pm 0.0001$	$0.0092\pm 0.0001$
MFVI	$93.80\pm 0.06$	$0.1872\pm 0.0001$	$0.0088\pm 0.0005$
VIP	${\color[rgb]{0,0.5,0.5}\mathbf{94.42\pm 0.05}}$	${\color[rgb]{0,0.5,0.5}\mathbf{0.1700\pm 0.0008}}$	$0.0093\pm 0.0021$
FBNN	$93.94\pm 0.03$	$0.1851\pm 0.0001$	$0.0092\pm 0.0003$
SIP	$93.05\pm 0.03$	$0.2107\pm 0.0021$	${\color[rgb]{0.75,0,0.25}\mathbf{0.0057\pm 0.0011}}$
TFSVI	$93.94\pm 0.02$	$0.1851\pm 0.0001$	$0.0105\pm 0.0005$

Appendix

Table 10: Frozen-CLIP CIFAR-10 test results with linear classifier heads. Entries are mean

\pm

standard deviation over three seeds. Accuracy is in percentage points; NLL and ECE are temperature calibrated. Best mean is shown in purple and second-best mean in teal .

	Round 1 (in distribution)				Round 2 (shifted)
Method	Acc. (%) $\uparrow$	NLL $\downarrow$	ECE $\downarrow$	AURC $\downarrow$	Acc. (%) $\uparrow$	NLL $\downarrow$	ECE $\downarrow$	AURC $\downarrow$
MAP	$57.61\pm 0.03$	$0.8985\pm 0.0001$	$0.0437\pm 0.0003$	$0.2706\pm 0.0002$	$55.74\pm 0.08$	$0.9335\pm 0.0001$	$0.0351\pm 0.0037$	$0.3262\pm 0.0002$
MFVI	$57.23\pm 0.13$	$0.9031\pm 0.0003$	$0.0433\pm 0.0004$	$0.2749\pm 0.0005$	$55.19\pm 0.08$	$0.9374\pm 0.0004$	${\color[rgb]{0,0.5,0.5}\mathbf{0.0319\pm 0.0004}}$	$0.3284\pm 0.0005$
VIP	${\color[rgb]{0,0.5,0.5}\mathbf{59.30\pm 0.21}}$	${\color[rgb]{0,0.5,0.5}\mathbf{0.8796\pm 0.0005}}$	${\color[rgb]{0.75,0,0.25}\mathbf{0.0385\pm 0.0006}}$	${\color[rgb]{0,0.5,0.5}\mathbf{0.2530\pm 0.0006}}$	${\color[rgb]{0.75,0,0.25}\mathbf{57.96\pm 0.29}}$	${\color[rgb]{0,0.5,0.5}\mathbf{0.9209\pm 0.0023}}$	$0.0372\pm 0.0035$	${\color[rgb]{0,0.5,0.5}\mathbf{0.3168\pm 0.0005}}$
FBNN	$58.48\pm 0.40$	$0.8888\pm 0.0027$	${\color[rgb]{0,0.5,0.5}\mathbf{0.0388\pm 0.0007}}$	$0.2615\pm 0.0030$	$56.81\pm 0.28$	$0.9261\pm 0.0011$	$0.0417\pm 0.0045$	$0.3205\pm 0.0005$
SIP	$57.21\pm 0.29$	$0.9073\pm 0.0017$	$0.0418\pm 0.0019$	$0.2779\pm 0.0020$	$55.83\pm 0.20$	$0.9385\pm 0.0020$	${\color[rgb]{0.75,0,0.25}\mathbf{0.0296\pm 0.0011}}$	$0.3274\pm 0.0009$

Appendix

Table 11: Frozen-CLIP DynaSent results with linear three-class heads. Round 1 is the in-distribution test set and Round 2 is the adversarially collected shifted test set. Entries are mean

\pm

standard deviation; SIP has two completed seeds and all other methods have three. Accuracy is in percentage points; NLL and ECE are temperature calibrated. Lower is better except for accuracy. AURC is the area under the risk–coverage curve. Best and second-best means are highlighted as above.

RMSE	Boston	Concrete	Energy	Kin8nm	Naval ( $\times 10^{-4}$ )	Power	Protein	Wine Red	Yacht
MAP	$4.34\pm 1.41$	$5.45\pm 0.57$	$0.52\pm 0.05$	$0.082\pm 0.001$	$4.04\pm 1.97$	$3.94\pm 0.13$	$4.62\pm 0.07$	$0.69\pm 0.01$	$0.52\pm 0.20$
MFVI	$3.93\pm 0.70$	$6.23\pm 0.51$	$0.62\pm 0.09$	$0.091\pm 0.002$	$147.61\pm 0.90$	$4.12\pm 0.17$	$4.94\pm 0.08$	${\color[rgb]{0,0.5,0.5}\mathbf{0.62\pm 0.03}}$	$0.68\pm 0.10$
VIP	$6.49\pm 1.27$	$5.76\pm 0.97$	$0.58\pm 0.09$	$0.080\pm 0.003$	$2.59\pm 0.96$	${\color[rgb]{0.75,0,0.25}\mathbf{3.76\pm 0.07}}$	$4.37\pm 0.06$	$0.79\pm 0.05$	$0.84\pm 0.30$
FBNN	$4.17\pm 1.27$	${\color[rgb]{0,0.5,0.5}\mathbf{5.21\pm 0.52}}$	${\color[rgb]{0,0.5,0.5}\mathbf{0.51\pm 0.05}}$	$0.086\pm 0.001$	$3.95\pm 0.47$	$4.05\pm 0.16$	$4.81\pm 0.05$	$0.65\pm 0.04$	${\color[rgb]{0.75,0,0.25}\mathbf{0.23\pm 0.06}}$
SIP	${\color[rgb]{0.75,0,0.25}\mathbf{3.22\pm 0.99}}$	$5.30\pm 0.27$	$0.52\pm 0.04$	${\color[rgb]{0,0.5,0.5}\mathbf{0.071\pm 0.001}}$	${\color[rgb]{0,0.5,0.5}\mathbf{1.48\pm 0.23}}$	$3.84\pm 0.11$	$4.62\pm 0.05$	$0.66\pm 0.03$	$0.41\pm 0.15$
TFSVI	$6.39\pm 0.63$	$5.97\pm 0.85$	$0.59\pm 0.15$	$0.081\pm 0.002$	$3.95\pm 0.81$	$3.90\pm 0.13$	$4.61\pm 0.05$	$0.73\pm 0.02$	$0.83\pm 0.38$

Appendix

Table 12: UCI regression metrics (

\downarrow

). Entries are mean

\pm

standard deviation across seeds. Frozen-prior variants are omitted. Best mean per dataset is shown in purple ; second-best mean is shown in teal . Naval RMSE and CRPS are reported in units of

10^{-4}

RMSE	Boston	Concrete	Energy	Kin8nm	Naval ( $\times 10^{-4}$ )	Power	Protein	Wine red	Yacht
Grid	${\color[rgb]{0,0.5508,0.5859}\mathbf{3.26\pm 0.55}}$	$4.75\pm 0.55$	$0.52\pm 0.06$	$0.071\pm 0.001$	$2.26\pm 0.53$	$3.82\pm 0.12$	$4.49\pm 0.03$	${\color[rgb]{0,0.5508,0.5859}\mathbf{0.62\pm 0.03}}$	$0.49\pm 0.27$
Random subset	${\color[rgb]{0.8477,0.1055,0.375}\mathbf{3.22\pm 0.85}}$	${\color[rgb]{0,0.5508,0.5859}\mathbf{4.27\pm 0.54}}$	${\color[rgb]{0,0.5508,0.5859}\mathbf{0.51\pm 0.05}}$	${\color[rgb]{0,0.5508,0.5859}\mathbf{0.069\pm 0.001}}$	${\color[rgb]{0,0.5508,0.5859}\mathbf{1.45\pm 0.34}}$	${\color[rgb]{0,0.5508,0.5859}\mathbf{3.79\pm 0.12}}$	${\color[rgb]{0,0.5508,0.5859}\mathbf{4.32\pm 0.04}}$	$0.62\pm 0.03$	${\color[rgb]{0,0.5508,0.5859}\mathbf{0.41\pm 0.15}}$
$k$ -means	$3.32\pm 0.97$	${\color[rgb]{0.8477,0.1055,0.375}\mathbf{4.24\pm 0.74}}$	${\color[rgb]{0.8477,0.1055,0.375}\mathbf{0.51\pm 0.05}}$	${\color[rgb]{0.8477,0.1055,0.375}\mathbf{0.069\pm 0.001}}$	${\color[rgb]{0.8477,0.1055,0.375}\mathbf{1.36\pm 0.19}}$	${\color[rgb]{0.8477,0.1055,0.375}\mathbf{3.78\pm 0.12}}$	${\color[rgb]{0.8477,0.1055,0.375}\mathbf{4.32\pm 0.04}}$	${\color[rgb]{0.8477,0.1055,0.375}\mathbf{0.60\pm 0.03}}$	${\color[rgb]{0.8477,0.1055,0.375}\mathbf{0.32\pm 0.11}}$
NLL	Boston	Concrete	Energy	Kin8nm	Naval	Power	Protein	Wine red	Yacht
Grid	${\color[rgb]{0.8477,0.1055,0.375}\mathbf{2.83\pm 0.41}}$	$2.96\pm 0.12$	$0.76\pm 0.06$	$-1.23\pm 0.01$	$-6.36\pm 0.10$	$2.76\pm 0.03$	$2.92\pm 0.01$	${\color[rgb]{0,0.5508,0.5859}\mathbf{0.98\pm 0.07}}$	$0.96\pm 1.48$
Random subset	${\color[rgb]{0,0.5508,0.5859}\mathbf{2.87\pm 0.58}}$	${\color[rgb]{0,0.5508,0.5859}\mathbf{2.84\pm 0.12}}$	${\color[rgb]{0.8477,0.1055,0.375}\mathbf{0.74\pm 0.07}}$	${\color[rgb]{0,0.5508,0.5859}\mathbf{-1.25\pm 0.02}}$	${\color[rgb]{0,0.5508,0.5859}\mathbf{-7.08\pm 0.06}}$	${\color[rgb]{0,0.5508,0.5859}\mathbf{2.75\pm 0.03}}$	${\color[rgb]{0,0.5508,0.5859}\mathbf{2.88\pm 0.01}}$	$1.01\pm 0.10$	${\color[rgb]{0,0.5508,0.5859}\mathbf{0.35\pm 0.32}}$

Appendix

Table 13: Inducing-location initialization ablation on the nine UCI regression datasets (

\downarrow

). Entries are mean

\pm

standard deviation across five seeds. The best mean per dataset is shown in purple ; the second-best is shown in teal . Naval RMSE and CRPS are reported in units of

10^{-4}

RMSE	Boston	Concrete	Energy	Kin8nm	Naval ( $\times 10^{-4}$ )	Power	Protein	Wine red	Yacht
$M=16$	$3.50\pm 1.30$	$4.75\pm 0.44$	$0.51\pm 0.05$	$0.076\pm 0.002$	$1.98\pm 0.19$	$3.94\pm 0.12$	$4.57\pm 0.03$	${\color[rgb]{0,0.5508,0.5859}\mathbf{0.61\pm 0.03}}$	${\color[rgb]{0,0.5508,0.5859}\mathbf{0.46\pm 0.14}}$
$M=32$	${\color[rgb]{0,0.5508,0.5859}\mathbf{3.48\pm 0.92}}$	${\color[rgb]{0,0.5508,0.5859}\mathbf{4.39\pm 0.54}}$	$0.52\pm 0.05$	$0.072\pm 0.001$	$1.76\pm 0.16$	$3.86\pm 0.14$	$4.47\pm 0.04$	$0.61\pm 0.03$	$0.60\pm 0.22$
$M=64$	${\color[rgb]{0.8477,0.1055,0.375}\mathbf{3.24\pm 1.34}}$	$4.55\pm 0.40$	${\color[rgb]{0,0.5508,0.5859}\mathbf{0.51\pm 0.05}}$	${\color[rgb]{0,0.5508,0.5859}\mathbf{0.070\pm 0.001}}$	${\color[rgb]{0,0.5508,0.5859}\mathbf{1.56\pm 0.46}}$	${\color[rgb]{0,0.5508,0.5859}\mathbf{3.84\pm 0.12}}$	${\color[rgb]{0,0.5508,0.5859}\mathbf{4.37\pm 0.04}}$	$0.61\pm 0.04$	$0.58\pm 0.36$
$M=128$	$3.51\pm 1.19$	${\color[rgb]{0.8477,0.1055,0.375}\mathbf{4.27\pm 0.73}}$	${\color[rgb]{0.8477,0.1055,0.375}\mathbf{0.51\pm 0.05}}$	${\color[rgb]{0.8477,0.1055,0.375}\mathbf{0.068\pm 0.001}}$	${\color[rgb]{0.8477,0.1055,0.375}\mathbf{1.54\pm 0.38}}$	${\color[rgb]{0.8477,0.1055,0.375}\mathbf{3.76\pm 0.12}}$	${\color[rgb]{0.8477,0.1055,0.375}\mathbf{4.29\pm 0.05}}$	${\color[rgb]{0.8477,0.1055,0.375}\mathbf{0.60\pm 0.02}}$	${\color[rgb]{0.8477,0.1055,0.375}\mathbf{0.40\pm 0.24}}$
NLL	Boston	Concrete	Energy	Kin8nm	Naval	Power	Protein	Wine red	Yacht
$M=16$	${\color[rgb]{0,0.5508,0.5859}\mathbf{3.40\pm 1.77}}$	$2.94\pm 0.11$	$0.75\pm 0.05$	$-1.16\pm 0.03$	$-6.54\pm 0.06$	$2.79\pm 0.03$	$2.94\pm 0.01$	${\color[rgb]{0,0.5508,0.5859}\mathbf{0.95\pm 0.07}}$	${\color[rgb]{0.8477,0.1055,0.375}\mathbf{0.46\pm 0.16}}$

Appendix

Table 14: Number of inducing locations ablation on the nine UCI regression datasets (

\downarrow

). Entries are mean

\pm

standard deviation across five seeds. The best mean per dataset is shown in purple ; the second-best is shown in teal . Naval RMSE and CRPS are reported in units of

10^{-4}

RMSE	Boston	Concrete	Energy	Kin8nm	Naval ( $\times 10^{-4}$ )	Power	Protein	Wine red	Yacht
$B=32$	$4.15\pm 1.28$	$4.47\pm 0.89$	$0.51\pm 0.06$	$0.073\pm 0.003$	${\color[rgb]{0.8477,0.1055,0.375}\mathbf{1.29\pm 0.17}}$	$3.81\pm 0.12$	$4.36\pm 0.05$	$0.68\pm 0.01$	$0.39\pm 0.17$
$B=64$	$3.71\pm 1.17$	${\color[rgb]{0,0.5508,0.5859}\mathbf{4.37\pm 0.47}}$	$0.51\pm 0.05$	$0.071\pm 0.001$	${\color[rgb]{0,0.5508,0.5859}\mathbf{1.32\pm 0.20}}$	${\color[rgb]{0,0.5508,0.5859}\mathbf{3.78\pm 0.12}}$	$4.33\pm 0.05$	$0.65\pm 0.02$	$0.44\pm 0.22$
$B=128$	$3.86\pm 1.25$	$4.37\pm 0.70$	$0.52\pm 0.04$	$0.070\pm 0.002$	$1.39\pm 0.18$	$3.78\pm 0.14$	${\color[rgb]{0,0.5508,0.5859}\mathbf{4.31\pm 0.05}}$	$0.65\pm 0.01$	$0.76\pm 0.88$
$B=256$	${\color[rgb]{0,0.5508,0.5859}\mathbf{3.68\pm 1.35}}$	$4.40\pm 0.60$	${\color[rgb]{0.8477,0.1055,0.375}\mathbf{0.51\pm 0.05}}$	${\color[rgb]{0.8477,0.1055,0.375}\mathbf{0.069\pm 0.001}}$	$1.52\pm 0.19$	${\color[rgb]{0.8477,0.1055,0.375}\mathbf{3.77\pm 0.13}}$	${\color[rgb]{0.8477,0.1055,0.375}\mathbf{4.31\pm 0.04}}$	${\color[rgb]{0,0.5508,0.5859}\mathbf{0.64\pm 0.03}}$	${\color[rgb]{0,0.5508,0.5859}\mathbf{0.33\pm 0.12}}$
$B=512$	${\color[rgb]{0.8477,0.1055,0.375}\mathbf{3.32\pm 0.97}}$	${\color[rgb]{0.8477,0.1055,0.375}\mathbf{4.24\pm 0.74}}$	${\color[rgb]{0,0.5508,0.5859}\mathbf{0.51\pm 0.05}}$	${\color[rgb]{0,0.5508,0.5859}\mathbf{0.069\pm 0.001}}$	$1.36\pm 0.19$	$3.78\pm 0.12$	$4.32\pm 0.04$	${\color[rgb]{0.8477,0.1055,0.375}\mathbf{0.60\pm 0.03}}$	${\color[rgb]{0.8477,0.1055,0.375}\mathbf{0.32\pm 0.11}}$
NLL	Boston	Concrete	Energy	Kin8nm	Naval	Power	Protein	Wine red	Yacht

Appendix

Table 15: Prior-bank size ablation on the nine UCI regression datasets (

\downarrow

). Entries are mean

\pm

standard deviation across five seeds. The best mean per dataset is shown in purple ; the second-best is shown in teal . Naval RMSE and CRPS are reported in units of

10^{-4}

RMSE	Boston	Concrete	Energy	Kin8nm	Naval ( $\times 10^{-4}$ )	Power	Protein	Wine Red	Yacht
Full	$\bm{3.32\pm 0.97}$	$\bm{4.24\pm 0.74}$	$0.51\pm 0.05$	$\bm{0.069\pm 0.001}$	$1.36\pm 0.19$	$3.78\pm 0.12$	$4.32\pm 0.04$	$\bm{0.60\pm 0.03}$	$\bm{0.32\pm 0.11}$
Induc. only	$4.09\pm 1.10$	$5.39\pm 0.65$	$\bm{0.45\pm 0.03}$	$0.085\pm 0.003$	$\bm{1.16\pm 0.25}$	$\bm{3.72\pm 0.08}$	$\bm{4.22\pm 0.05}$	$0.87\pm 0.05$	$0.46\pm 0.15$
NLL	Boston	Concrete	Energy	Kin8nm	Naval	Power	Protein	Wine Red	Yacht
Full	$\bm{3.11\pm 1.03}$	$\bm{2.82\pm 0.16}$	$\bm{0.75\pm 0.09}$	$\bm{-1.26\pm 0.02}$	$-7.12\pm 0.04$	$2.75\pm 0.03$	$2.88\pm 0.01$	$\bm{0.94\pm 0.08}$	$\bm{0.18\pm 0.22}$
Induc. only	$2083.56\pm 1408.57$	$5.57\pm 1.00$	$1.49\pm 0.43$	$-1.02\pm 0.04$	$\bm{-7.58\pm 0.12}$	$\bm{2.73\pm 0.02}$	$\bm{2.86\pm 0.01}$	$26.66\pm 2.95$	$6.16\pm 5.43$
CRPS	Boston	Concrete	Energy	Kin8nm	Naval ( $\times 10^{-4}$ )	Power	Protein	Wine Red	Yacht

Appendix

Table 16: GMVIP path ablation on UCI regression (

\downarrow

). Entries are mean

\pm

standard deviation across five seeds. The full-path model retains the residual prior path, whereas the inducing-only variant removes it. The best mean per dataset is shown in bold. Naval RMSE and CRPS are reported in units of

10^{-4}

RMSE	Boston	Concrete	Energy	Kin8nm	Naval ( $\times 10^{-4}$ )	Power	Protein	Wine Red	Yacht
VIP (tunable)	$6.49\pm 1.27$	$5.76\pm 0.97$	$0.58\pm 0.09$	$0.080\pm 0.003$	$2.59\pm 0.96$	$\bm{3.76\pm 0.07}$	$4.37\pm 0.06$	$0.79\pm 0.05$	$0.84\pm 0.30$
VIP (frozen)	$6.70\pm 1.26$	$11.79\pm 0.90$	$4.97\pm 0.59$	$0.234\pm 0.007$	$96.20\pm 8.64$	$7.01\pm 0.94$	$5.52\pm 0.05$	$0.70\pm 0.04$	$10.79\pm 2.21$
FTIP (tunable)	$6.68\pm 1.58$	$6.02\pm 1.08$	$0.55\pm 0.07$	$0.079\pm 0.002$	$3.36\pm 1.58$	$3.77\pm 0.11$	$4.36\pm 0.05$	$0.82\pm 0.03$	$1.17\pm 0.52$
FTIP (frozen)	$6.71\pm 1.27$	$11.77\pm 0.86$	$4.97\pm 0.60$	$0.234\pm 0.006$	$90.54\pm 9.39$	$7.02\pm 0.93$	$5.53\pm 0.06$	$0.70\pm 0.04$	$10.80\pm 2.22$
GMVIP (tunable)	$\bm{3.32\pm 0.97}$	$\bm{4.24\pm 0.74}$	$\bm{0.51\pm 0.05}$	$\bm{0.069\pm 0.001}$	$\bm{1.36\pm 0.19}$	$3.78\pm 0.12$	$\bm{4.32\pm 0.04}$	$\bm{0.60\pm 0.03}$	$\bm{0.32\pm 0.11}$
GMVIP (frozen)	$3.86\pm 0.97$	$7.34\pm 0.45$	$2.80\pm 0.13$	$0.144\pm 0.005$	$4.10\pm 0.56$	$4.01\pm 0.11$	$4.64\pm 0.05$	$0.60\pm 0.03$	$4.11\pm 1.07$

Appendix

Table 17: Frozen-prior ablation for VIP, FTIP, and GMVIP on UCI regression (

\downarrow

). Entries are mean

\pm

standard deviation across five matched seeds. Tunable variants optimize the BNN prior jointly with the remaining model parameters, whereas frozen variants keep the prior fixed at initialization. The best mean per dataset among the compared variants is shown in bold. Naval RMSE and CRPS are reported in units of

10^{-4}

Method	Representation size	RMSE $\downarrow$	NLL $\downarrow$	CRPS $\downarrow$	Cov 90	ODE residual $\downarrow$
VIP	$S=20$	$0.50\pm 0.08$	$11.32\pm 5.01$	$0.31\pm 0.05$	$0.72\pm 0.06$	$0.65\pm 0.03$
VIP	$S=64$	$0.37\pm 0.08$	$6.20\pm 5.87$	$0.23\pm 0.05$	$0.87\pm 0.04$	$0.69\pm 0.02$
VIP	$S=128$	$0.36\pm 0.09$	$7.93\pm 7.62$	$0.23\pm 0.06$	${\color[rgb]{0,0.5,0.5}\mathbf{0.89\pm 0.04}}$	$0.71\pm 0.03$
VIP	$S=256$	$0.35\pm 0.09$	$9.97\pm 9.71$	${\color[rgb]{0,0.5,0.5}\mathbf{0.22\pm 0.06}}$	$0.89\pm 0.04$	$0.69\pm 0.03$
FTIP	$S=20$	$0.57\pm 0.08$	$1.04\pm 0.22$	$0.34\pm 0.05$	$0.82\pm 0.04$	${\color[rgb]{0,0.5,0.5}\mathbf{0.62\pm 0.03}}$
FTIP	$S=64$	$0.43\pm 0.13$	$1.80\pm 1.41$	$0.29\pm 0.10$	$0.92\pm 0.04$	$0.67\pm 0.02$

Appendix

Table 18: Sensitivity of VIP and FTIP to the number

S

of sampled prior basis functions in the Lotka–Volterra experiment. Entries are means

\pm

standard errors across 20 target trajectories on the test interval

t\in(20,30]

. VIP is trained for 800 iterations. FTIP is warm-started from a separate 400-iteration VIP fit at the same

S

and then fine-tuned for 400 additional iterations. GMVIP uses 96 inducing locations and is included as a reference. Coverage is ranked by proximity to the nominal

90\%

level; lower is better otherwise. Best and second-best results, determined using unrounded values, are highlighted.

Method	Setup	Train / step	Eval time	CUDA memory	Latent dim.	Prior samples
VIP	0.04s	0.006s	0.247s	28 / 48 MB	20	–
FTIP	0.16s	0.023s	0.216s	84 / 112 MB	20	512
FBNN	0.03s	0.044s	0.247s	176 / 290 MB	512	512
TFSVI	4.09s	0.067s	0.232s	249 / 378 MB	512	–
SIP	2.34s	0.021s	0.357s	626 / 672 MB	100	512
GMVIP	1.17s	0.031s	0.489s	626 / 672 MB	100	512

Appendix

Table 19: GPU efficiency diagnostics on Airline. Setup includes model construction and method-specific preparation; train time is mean wall-clock time per measured optimizer step after warmup. Memory reports peak CUDA allocated/reserved memory.

Explore similar work

Jun 1, 2026cs.LG

Flow-Transformed Implicit Processes for Function-Space Variational Inference

Implicit-process priors define distributions over functions through flexible generative mechanisms, making them attractive for Bayesian function-space modelling. However, performing posterior inference with such priors is challenging because their induced function-space distributions are typically not available in closed form. One practical strategy is to approximate the prior using a finite collection of sampled functions, and then represent posterior functions as learned combinations of these samples. Existing approaches commonly place a Gaussian variational distribution over the combination weights. While tractable, this choice limits the shapes of posterior uncertainty that can be represented, especially when the true posterior is asymmetric, heavy-tailed, or multimodal. We propose Flow-Transformed Implicit Processes (FTIP), a variational inference method that makes this finite-dimensional function-space approximation more expressive. Instead of using a Gaussian distribution over the combination weights, FTIP uses a normalizing flow to define a richer variational distribution. This induces a flexible posterior distribution over functions while preserving tractable optimization. We train the model using a Black-Box α objective, allowing us to compare mass-covering and mode-seeking variational behaviour. Experiments show that FTIP captures asymmetric and multimodal posterior structure in function space that Gaussian coefficient approximations tend to smooth or collapse.

Luis A. Ortega, Andrés R. Masegosa, Thomas D. Nielsen

Aalborg University

Variational Inference Generative Processes

Ask PDF

Oct 2, 2026stat.ML

Amortized Structured Stochastic Variational Inference for Gaussian Process Latent Variable Models

Many machine learning methods aim to approximate the lower-dimensional manifold on which the data lives. A desirable feature of such methods is that they should capture the epistemic uncertainty of this learned manifold. One model that achieves this is the Gaussian Process Latent Variable Model, in which a Gaussian Process (GP) mapping from the latent space provides an estimate of the uncertainty of the manifold. However, the effectiveness of this uncertainty estimation is limited by the mean-field variational approximation between the GP inducing points and the latent variables. In this work, we apply Amortized Structured Stochastic Variational Inference to allow the variational posterior for the latent space to be conditionally dependent on the value of the inducing points. We demonstrate that this more flexible variational posterior improves several metrics relating to the reconstruction of points on the data manifold.

Maksym Tretiakov, Sarah Filippi, Vincent Fortuin +3

LMU Munich · Munich Center for Machine Learning (MCML) · Imperial College London +3

Variational Inference Latent Variable

Ask PDF

Oct 18, 2024stat.ML

Predictive variational inference: Learn the predictively optimal posterior distribution

Vanilla variational inference finds an optimal approximation to the Bayesian posterior distribution, but even the exact Bayesian posterior is often not meaningful under model misspecification. We propose predictive variational inference (PVI): a general inference framework that seeks and samples from an optimal posterior density such that the resulting posterior predictive distribution is as close to the true data generating process as possible, while this closeness is measured by multiple scoring rules. By optimizing the objective, the predictive variational inference is generally not the same as, or even attempting to approximate, the Bayesian posterior, even asymptotically. Rather, we interpret it as implicit hierarchical expansion. Further, the learned posterior uncertainty detects heterogeneity of parameters among the population, enabling automatic model diagnosis. This framework applies to both likelihood-exact and likelihood-free models. We demonstrate its application in real data examples.

Jinlin Lai, Antonio Linero, Yuling Yao

College of Information and Computer Sciences, University of Massachusetts Amherst · Department of Statistics and Data Sciences, University of Texas, Austin

Posterior Predictive Distribution Variational Inference Posterior

Ask PDF