Variational Learning of Fractional Posteriors

Kian Ming A. Chai; Edwin V. Bonilla

cs.LGMar 29, 2026

Variational Learning of Fractional Posteriors

Authors: Kian Ming A. Chai, Edwin V. Bonilla

Organizations: DSO National Laboratories, Singapore · CSIRO’s Data61, Australia

Abstract

We introduce a novel one-parameter variational objective that lower bounds the data evidence and enables the estimation of approximate fractional posteriors. We extend this framework to hierarchical construction and Bayes posteriors, offering a versatile tool for probabilistic modelling. We demonstrate two cases where gradients can be obtained analytically and a simulation study on mixture models showing that our fractional posteriors can be used to achieve better calibration compared to posteriors from the conventional variational bound. When applied to variational autoencoders (VAEs), our approach attains higher evidence bounds and enables learning of high-performing approximate Bayes posteriors jointly with fractional posteriors. We show that VAEs trained with fractional posteriors produce decoders that are better aligned for generation from the prior.

Figures & tables

	$\kappa$	$\ell$	$\kappa$	$\ell$	bound
	$\mu_{1}$		$\mu_{2}$
$\mathcal{L}_{0.1}$	$1.0000$	$0.8515$	$1.0000$	$0.8987$	$-832.5$
$\mathcal{L}_{0.3}$	$0.9994$	$0.4924$	$0.9988$	$0.5200$	$-833.3$
$\mathcal{L}_{0.5}$	$0.9876$	$0.3816$	$0.9860$	$0.4029$	$-834.0$
$\mathcal{L}_{0.7}$	$0.9694$	$0.3225$	$0.9606$	$0.3406$	$-834.4$
$\mathcal{L}_{0.9}$	$0.9438$	$0.2845$	$0.9334$	$0.3004$	$-834.8$

Table 1: Calibration study of GMM at

\alpha=0.05

significance. The empirical coverages

\kappa

(higher is better) and average interval lengths

\ell

(shorter better) for each of the

\left\{\mu_{1},\mu_{2}\right\}

means are shown. The last column gives the bounds to

\mathcal{L}_{\mathrm{evd}}

(higher better). The first set of results is from modelling with different

\gamma

s (

\mathcal{L}_{1.0}

is ELBO). Results for

\gamma\in\left\{0.2,0.4,0.6,0.8\right\}

are omitted for brevity, but the trend remains. The second set is from conflated models combining the means from ELBO and the variances from

\mathcal{L}_{\gamma}

. The third set is from calibrations of using the first set of results. The

\gamma

values for

R_{\ell}

and

R_{\kappa}

are

0.785

and

0.798

			Test using Objective			Test using ELBO
Objective	$\gamma$	Train (Total)	Total	data	div	Total	data	div
$\mathcal{L}_{\gamma}$	1.0	$1614.3{\scriptscriptstyle\pm 11.0}$	$1583.2{\scriptscriptstyle\pm 14.6}$	$1588.5{\scriptscriptstyle\pm 14.5}$	$5.3{\scriptscriptstyle\pm 0.2}$	$1583.2{\scriptscriptstyle\pm 14.6}$	$1588.5{\scriptscriptstyle\pm 14.5}$	$5.3{\scriptscriptstyle\pm 0.2}$
	0.9	$1648.5{\scriptscriptstyle\pm 5.1~}$	$1639.3{\scriptscriptstyle\pm 4.5~}$	$1641.9{\scriptscriptstyle\pm 4.8~}$	$2.6{\scriptscriptstyle\pm 0.4}$	$1452.6{\scriptscriptstyle\pm 48.0}$	$1455.0{\scriptscriptstyle\pm 48.0}$	$2.4{\scriptscriptstyle\pm 0.3}$
	0.5	$1675.9{\scriptscriptstyle\pm 5.0~}$	$1672.8{\scriptscriptstyle\pm 5.9~}$	$1674.7{\scriptscriptstyle\pm 6.0~}$	$1.9{\scriptscriptstyle\pm 0.3}$	$1318.7{\scriptscriptstyle\pm 42.1}$	$1320.1{\scriptscriptstyle\pm 42.2}$	$1.4{\scriptscriptstyle\pm 0.2}$
	0.1	$1680.1{\scriptscriptstyle\pm 2.9~}$	$1677.2{\scriptscriptstyle\pm 3.4~}$	$1679.5{\scriptscriptstyle\pm 3.4~}$	$2.3{\scriptscriptstyle\pm 0.3}$	$1322.8{\scriptscriptstyle\pm 49.0}$	$1324.2{\scriptscriptstyle\pm 49.1}$	$1.3{\scriptscriptstyle\pm 0.2}$
$\mathcal{L}^{\mathrm{h}}_{\gamma}$	1.0	$1639.6{\scriptscriptstyle\pm 14.6}$	$1609.6{\scriptscriptstyle\pm 20.6}$	$1614.6{\scriptscriptstyle\pm 20.5}$	$5.0{\scriptscriptstyle\pm 0.3}$	$1609.7{\scriptscriptstyle\pm 20.6}$	$1614.6{\scriptscriptstyle\pm 20.5}$	$5.0{\scriptscriptstyle\pm 0.3}$

Table 2: Average log-evidences (higher better) over data samples, and its breakdown for VAE on MNIST data sets. We give the mean and three standard deviations of these averages over ten experimental runs. For Monte Carlo averages, 1,024 samples are used (

32\times 32

for semi-implicit posteriors). For

\gamma=1.0

, the figures are the same under Test using Objective and Test using ELBO . For the first eight rows, the columns under Test using ELBO are solely for diagnostics to understand the learnt posteriors using the same metrics: they are not performance measures. For

\mathcal{L}^{\mathrm{bh}}_{\gamma}

, we show the addition of the KL divergence and Rényi divergence for Test using Objective ; and for Test using ELBO we evaluate the Bayes posterior

r

Panel of Figure 1 — Figure 1: We train VAEs on the Fashion-MNIST dataset using $\mathcal{L}_{\gamma}$ for different $\gamma$ s. We obtain mean images from the decoding latent variables that are systematically sampled by coordinate-wise inverse-CDF (standard normal) transform from a unit square. Fig. b shows the images using the Bayes posterior (learnt with ELBO), and Fig. c shows those using a fractional posterior very close to the prior. The last image is the heat map of the corresponding prior densities.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Description	Notation	Expression
Log-evidence	$\mathcal{L}_{\mathrm{evd}}$	$\displaystyle\log p(D)=\log\int p(z)p(D\|z){\mathrm{d}z}$
ELBO (evidence lower bound)	$\mathcal{L}_{\mathrm{ELBO}}$	$\displaystyle\int q(z)\log p(D\|z){\mathrm{d}z}-\int q(z)\log\frac{q(z)}{p(z)}{\mathrm{d}z}$
Weighted KL divergence ( $\beta$ -VAE objective)	$\mathcal{L}^{\beta}_{\beta}$	$\displaystyle\int q(z)\log p(D\|z){\mathrm{d}z}-\beta\int q(z)\log\frac{q(z)}{p(z)}{\mathrm{d}z}$
Variational Rényi bound	$\mathcal{L}^{\mathrm{R}}_{\alpha}$	$\displaystyle\frac{1}{1-\alpha}\log\int q(z)\left(\frac{p(D\|z)\,p(z)}{q(z)}\right)^{1-\alpha}{\mathrm{d}z}$
Our primary bound	$\mathcal{L}_{\gamma}$	$\displaystyle\frac{1}{1-\gamma}\log\int\tilde{q}(z)p(D\|z)^{1-\gamma}{\mathrm{d}z}-\frac{\gamma}{1-\gamma}\log\int\tilde{q}(z)\left(\frac{\tilde{q}(z)}{p(z)}\right)^{1/\gamma-1}{\mathrm{d}z}$
$\bullet$ with hierarchical fractional posterior	$\mathcal{L}^{\mathrm{h}}_{\gamma}$	$\displaystyle\begin{multlined}\frac{1}{1-\gamma}\log\iint\tilde{q}(z\|u)q(u)p(D\|z)^{1-\gamma}{\mathrm{d}z}{\mathrm{d}u}\hskip 113.81102pt\\[-8.61108pt] -\frac{\gamma}{1-\gamma}\log\iint\tilde{q}(z\|u)q(u)\left(\frac{\tilde{q}(z\|u)}{p(z)}\right)^{1/\gamma-1}{\mathrm{d}z}\,{\mathrm{d}u}\end{multlined}$

Appendix

Table 3: List of lower bounds. Some are expressed differently from the main paper to ease comparison among the bounds. The

\beta

-VAE objective is the weighted KL divergence ( Knoblauch et al., 2022 , §B.3.1) .

Table 4: Calibration study of the Gaussian mixture model at

\alpha=0.05

significance level, for different settings.

Panel of Figure 2 — Figure 2: Differences between the precisions of the fractional posteriors for the first component of the mixture model.

$\gamma$	$\mathcal{L}_{\gamma}$	$\widehat{\mathcal{L}_{\mathrm{evd}}}$	CV	$\mathcal{L}_{\gamma}$	$\widehat{\mathcal{L}_{\mathrm{evd}}}$	CV	$\mathcal{L}_{\gamma}$	$\widehat{\mathcal{L}_{\mathrm{evd}}}$	CV	$\mathcal{L}_{\gamma}$	$\mathcal{L}_{\mathrm{evd}}$	CV
	$n=400$ , $\mu_{i}=\pm 2$			$n=30$ , $\mu_{i}=\pm 2$			$n=400$ , $\mu_{i}=\pm 1/2$			$n=400$ , 4 components
$0.1$	$-813.0$	$-806.9$	$2.2$	$-65.7$	$-60.2$	$2.0$	$-602.1$	$-593.1$	$84.0$	$-805.2$	$-786.3$	$87.6$
$0.2$	$-812.9$	$-806.9$	$1.5$	$-64.6$	$-60.2$	$1.3$	$-601.0$	$-593.4$	$23.7$	$-801.8$	$-786.3$	$83.0$
$0.3$	$-813.2$	$-806.9$	$1.1$	$-64.5$	$-60.2$	$1.0$	$-600.9$	$-593.4$	$19.8$	$-801.1$	$-786.2$	$62.8$
$0.4$	$-813.5$	$-806.9$	$1.0$	$-64.5$	$-60.2$	$0.8$	$-601.0$	$-593.4$	$31.5$	$-801.1$	$-786.4$	$36.8$
$0.5$	$-813.7$	$-806.9$	$0.9$	$-64.6$	$-60.2$	$0.6$	$-601.2$	$-593.4$	$24.3$	$-801.2$	$-786.6$	$19.4$

Appendix

Table 5: Importance sampling to estimate the log-evidence

\widehat{\mathcal{L}_{\mathrm{evd}}}

of Gaussian mixture models using approximate posteriors. We use the four experimental settings from sections 5.1 and C.1.2 . For each setting, and for ten values of

\gamma

(including

\gamma=1.0

for ELBO), we have the bound

\mathcal{L}_{\gamma}

from the variational optimisation,

\widehat{\mathcal{L}_{\mathrm{evd}}}

, and the coefficient of variation CV of the weights used to compute

\widehat{\mathcal{L}_{\mathrm{evd}}}

			Test using Objective			Test using ELBO
Objective	$\gamma$	Train (Total)	Total	data	div	Total	data	div
$\mathcal{L}^{\mathrm{bh}}_{\gamma}$	0.9	$1636.2{\scriptscriptstyle\pm 11.5}$	$1608.8{\scriptscriptstyle\pm 23.3}$	$1613.4{\scriptscriptstyle\pm 23.0}$	$4.0{\scriptscriptstyle\pm 0.2}+0.6{\scriptscriptstyle\pm 0.3}$	$1607.7{\scriptscriptstyle\pm 23.9}$	$1613.4{\scriptscriptstyle\pm 23.0}$	$5.7{\scriptscriptstyle\pm 1.0}$
	0.5	$1635.2{\scriptscriptstyle\pm 10.5}$	$1608.0{\scriptscriptstyle\pm 25.4}$	$1612.7{\scriptscriptstyle\pm 25.2}$	$4.3{\scriptscriptstyle\pm 0.2}+0.4{\scriptscriptstyle\pm 0.2}$	$1607.4{\scriptscriptstyle\pm 25.8}$	$1612.7{\scriptscriptstyle\pm 25.3}$	$5.3{\scriptscriptstyle\pm 0.6}$
	0.1	$1635.5{\scriptscriptstyle\pm 12.0}$	$1607.5{\scriptscriptstyle\pm 16.1}$	$1612.4{\scriptscriptstyle\pm 16.1}$	$4.6{\scriptscriptstyle\pm 0.1}+0.3{\scriptscriptstyle\pm 0.2}$	$1607.3{\scriptscriptstyle\pm 16.2}$	$1612.4{\scriptscriptstyle\pm 16.1}$	$5.1{\scriptscriptstyle\pm 0.2}$
$\mathcal{L}^{\mathrm{bh}}_{\gamma}$ -alt	0.9	$1639.4{\scriptscriptstyle\pm 9.4~}$	$1609.1{\scriptscriptstyle\pm 21.1}$	$1613.6{\scriptscriptstyle\pm 21.0}$	$4.0{\scriptscriptstyle\pm 0.1}+0.6{\scriptscriptstyle\pm 0.2}$	$1608.0{\scriptscriptstyle\pm 21.6}$	$1613.6{\scriptscriptstyle\pm 21.0}$	$5.6{\scriptscriptstyle\pm 0.8}$
	0.5	$1639.4{\scriptscriptstyle\pm 10.6}$	$1608.2{\scriptscriptstyle\pm 24.6}$	$1612.9{\scriptscriptstyle\pm 24.5}$	$4.3{\scriptscriptstyle\pm 0.1}+0.4{\scriptscriptstyle\pm 0.2}$	$1607.6{\scriptscriptstyle\pm 24.8}$	$1612.9{\scriptscriptstyle\pm 24.5}$	$5.3{\scriptscriptstyle\pm 0.5}$

Appendix

Table 6: Average log-evidences (higher better) over data samples, and its breakdown for VAE on MNIST data sets, for

\mathcal{L}^{\mathrm{bh}}_{\gamma}

and

\mathcal{L}^{\mathrm{bh}}_{\gamma}

-alt. We give the mean and three standard deviations of these averages over ten experimental runs. For Monte Carlo averages, 1,024 samples are used (

32\times 32

for semi-implicit posteriors). We show the addition of the KL divergence and Rényi’s divergence for Test using Objective ; and for Test using ELBO we evaluate the Bayes posterior

r

		Test using Objective			Test using ELBO
$\gamma$	Train (Total)	Total	data	div	Total	data	div
(reference) 1.0	$1616.814$	$1591.401$	$1596.695$	$5.294$	$1591.407$	$1596.700$	$5.293$
1.0	$1580.006$	$1558.188$	$1562.939$	$4.751$	$1558.189$	$1562.940$	$4.751$
0.9	$1625.566$	$1620.629$	$1624.170$	$3.541$	$1551.088$	$1554.442$	$3.354$
0.5	$1638.652$	$1636.510$	$1639.442$	$2.933$	$1451.295$	$1453.317$	$2.022$
0.1	$1641.655$	$1639.472$	$1642.944$	$3.472$	$1427.995$	$1429.871$	$1.876$

Appendix

Table 7: Log-evidences (higher better) over data samples for a single run, and its breakdown for VAE on MNIST data sets, for

\mathcal{L}_{\gamma}

where the decoders are fixed to be the same as that optimised for ELBO (first row in table). For Monte Carlo averages, 1,024 samples are used.

Panel of Figure 3 — Figure 3: Mean images from decoded latent variables obtained by coordinate-wise inverse-CDF (standard normal) transform from a unit square. For the last row, the first image are samples from the test set. The quality of the images are visually similar across $\gamma$ s and posterior families, and they are all not as sharp as the real test images. The VAE is trained on the MNIST dataset.

Panel of Figure 4 — Figure 4: Means of the explicit posteriors for 5,000 sampled MNIST test images, colour-coded by the class labels. All axes ranges from $-4$ to $4$ .

Panel of Figure 5 — Figure 5: 5,000 samples from the posteriors of the MNIST test images. For $\mathcal{L}^{\mathrm{bh}}$ and $\mathcal{L}^{\mathrm{bh}}$ -alt, the Bayes posteriors are used. All axes ranges from $-4$ to $4$ .

Panel of Figure 6 — Figure 6: 500 samples from the implicit posteriors for 16 MNIST test images: one small square is for one image. All axes ranges from $0$ to $1$ , which is the range of the samples by design.

		Generalised Variance						Total Variation
Objective	$\gamma$		Median	Max	Mean	CV	Skew		Median	Max	Mean	CV	Skew
$\mathcal{L}^{\mathrm{h}}$	$1.0$		$0$	$\prescript{}{5.4}{-3}$	$\prescript{}{3.2}{-6}$	$\prescript{}{3.0}{1}$	$\prescript{}{4.5}{1}$		$0$	$\prescript{}{8.2}{-1}$	$\prescript{}{2.0}{-3}$	$\prescript{}{1.1}{1}$	$\prescript{}{1.8}{1}$
	$0.9$		$\prescript{}{4.8}{-8}$	$\prescript{}{1.9}{-1}$	$\prescript{}{1.3}{-3}$	6.8	$\prescript{}{1.1}{1}$		$\prescript{}{3.0}{-3}$	1.1	$\prescript{}{5.5}{-2}$	2.3	3.6
	$0.5$		$0$	$\prescript{}{1.9}{-1}$	$\prescript{}{1.8}{-3}$	5.5	$\prescript{}{9.3}{~}$		$\prescript{}{9.8}{-3}$	1.2	$\prescript{}{1.0}{-1}$	1.9	2.6
	$0.1$		$0$	$\prescript{}{1.5}{-1}$	$\prescript{}{1.3}{-3}$	6.4	$\prescript{}{9.7}{~}$		$\prescript{}{3.8}{-5}$	1.0	$\prescript{}{5.9}{-2}$	2.3	2.9
$\mathcal{L}^{\mathrm{bh}}$	$0.9$		$0$	$\prescript{}{1.1}{-2}$	$\prescript{}{1.5}{-6}$	$\prescript{}{7.4}{1}$	$\prescript{}{9.7}{1}$		$\prescript{}{4.5}{-9}$	$\prescript{}{4.5}{-1}$	$\prescript{}{1.3}{-3}$	9.9	$\prescript{}{1.6}{1}$

Appendix

Table 8: Descriptive statistics, over the test images, of the generalised variance and total variation of the transformed implicit distributions. Figures less than

10^{-10}

are treated as zero. For a number

a\times 10^{b}

, we express it as

\prescript{}{a}{b}

, except that we use

0

when

a=0

and a when

b=0

Panel of Figure 7 — Figure 7: Mean images from decoded latent variables obtained by coordinate-wise inverse-CDF (standard normal) transform from a unit square. The VAE is trained on the Fashion-MNIST dataset. The top row uses the bound introduced in this paper; the bottom row (sans the first figure) uses the objective from $\beta$ -VAE.

$\dim(z)$	Obj.	$\gamma\text{ or }\beta$	Train (Total)	Total	data	div	Total	data	div	FID
				Test using Objective			Test using ELBO
2	$\mathcal{L}_{\gamma}$	$1.0$	$1152.30$	$1118.15$	$1123.11$	$4.96$	$1118.15$	$1123.11$	$4.96$	$~83.5$
		$10^{-1}$	$1180.08$	$1171.38$	$1173.97$	$2.59$	$~902.79$	$~904.56$	$1.77$	$~69.5$
		$10^{-3}$	$1179.68$	$1171.05$	$1173.91$	$2.87$	$~915.56$	$~917.30$	$1.74$	$~67.8$
		$10^{-5}$	$1183.97$	$1175.29$	$1178.04$	$2.75$	$~853.45$	$~855.16$	$1.71$	$~68.8$
	$\mathcal{L}^{\beta}_{\beta}$	$5.0$	$1135.83$	$1103.29$	$1121.04$	$17.76$	$1117.50$	$1121.05$	$3.55$	$~78.4$

Appendix

Table 9: Average log-evidences (higher better) over data samples, and its breakdown for VAE on Fashion-MNIST data sets. For Monte Carlo averages, 1,024 samples are used. For

\gamma=1.0

, the figures are the same under Test using Objective and Test using ELBO . The columns under Test using ELBO are solely for diagnostics to understand the learnt posteriors using the same metrics: they are not performance measures. For ease of comparison, we also add the last column of FID scores of the 10,000 generated images.

Explore similar work

Predictive variational inference: Learn the predictively optimal posterior distribution
Oct 18, 2024Jinlin Lai, Antonio Linero, Yuling YaoModel Misspecification Variational Inference
College of Information and Computer Sciences, University of Massachusetts Amherst · Department of Statistics and Data Sciences, University of Texas, Austin
Theoretical Study on the Evidential Learning-based Variational Autoencoder
Aug 19, 2026Ge WangVariational Autoencoders Variational Inference
Rensselaer Polytechnic Institute, Troy, New York, USA
Variational objectives for amortized Bayesian inference in inverse problems: The role of posterior conditioning
Sep 21, 2026Abhishek Srivastava, Arijit Hazra, Rajesh DubbakuAmortized Inference Bayesian Inverse Problems
Department of Mechanical Engineering, Indian Institute of Technology Palakkad · Department of Mathematics, Indian Institute of Technology Palakkad

Oct 18, 2024J/K move · Enter open · S save

Predictive variational inference: Learn the predictively optimal posterior distribution

Jinlin Lai, Antonio Linero, Yuling Yao

College of Information and Computer Sciences, University of Massachusetts Amherst · Department of Statistics and Data Sciences, University of Texas, Austin

Model Misspecification Variational Inference

Open paper AskPDF arXiv

Vanilla variational inference finds an optimal approximation to the Bayesian posterior distribution, but even the exact Bayesian posterior is often not meaningful under model misspecification. We propose predictive variational inference (PVI): a general inference framework that seeks and samples from an optimal posterior density such that the resulting posterior predictive distribution is as close to the true data generating process as possible, while this closeness is measured by multiple scoring rules. By optimizing the objective, the predictive variational inference is generally not the same as, or even attempting to approximate, the Bayesian posterior, even asymptotically. Rather, we interpret it as implicit hierarchical expansion. Further, the learned posterior uncertainty detects heterogeneity of parameters among the population, enabling automatic model diagnosis. This framework applies to both likelihood-exact and likelihood-free models. We demonstrate its application in real data examples.

Abstract

Explore similar work

Predictive variational inference: Learn the predictively optimal posterior distribution

Theoretical Study on the Evidential Learning-based Variational Autoencoder

Variational objectives for amortized Bayesian inference in inverse problems: The role of posterior conditioning