Towards Reliable Simulation-based Inference

Arnaud Delaunoy

Figure 1.1 : Visualization of Box’s loop [ 21 , 17 ] . The loop iterates over three main elements: model building, model’s hidden quantities inference, and model criticism. Iterating over those three elements aims to build better scientific models, taking into account model criticism feedback.

Figure 2.1 : Illustration of the experiment of a feather falling. A gravitational force

\mathbf{F}=m\ g\ \mathbf{e}

is applied on the feather and the fall time is measured.

Figure 2.2 : Histogram of samples obtained by running an MCMC algorithm on the feather falling model with measured time

4.68\text{s}

,

6.97\text{s}

and

9.69\text{s}

for heights of

3\text{m}

,

5\text{m}

and

7\text{m}

respectively.

Figure 2.3 : Approximate posterior density obtained by running a variational inference algorithm on the feather falling model with measured time

4.68\text{s}

,

6.97\text{s}

and

9.69\text{s}

for heights of

3\text{m}

,

5\text{m}

and

7\text{m}

respectively. The plotted variational distribution is a log normal distribution with parameters

\mu=-1.6873

and

\sigma=0.0829

.

Inputs:	gravity constant $g$ , drag coefficient $C_{D}$ , height $h$ , mass $m$ and
	measurement error standard deviation $\sigma_{\text{measurement}}$ .
Outputs:	The measured fall time.
hyper-parameters:	Duration of a step $\Delta t$

Algorithm 3.1 Feather falling simulator.

Figure 3.1 : Example of feather falling simulations using various gravity and friction parameters. The feather is dropped from the same height in all those configurations, and the feather position is reported at various time steps.

Figure 3.2 : Bayesian simulation-based inference overview

Figure 3.3 : Simulation-based inference with histograms. For both configurations

\bm{\theta}=\bm{\theta}_{1}

and

\bm{\theta}=\bm{\theta}_{2}

, samples are drawn from the simulator. Those samples are then used to construct histograms approximating the densities

p(\bm{x}_{0}|\bm{\theta}_{1})

and

p(\bm{x}_{o}|\bm{\theta}_{2})

.

Inputs:	distance function $d$ , distance threshold $\epsilon$ , number of simulations n_sim
	and real observation $\bm{x}_{o}$ .
Outputs:	The retained parameters, approximately sampled from the posterior.

Algorithm 3.2 Approximate Bayesian Computation (ABC)

Figure 3.4 : Visualization of a diffusion model applied on the two moons simulation-based inference benchmark. A backward process is learned to inverse a forward diffusion process. Applying the backward process to noise can then recover samples from the posterior distribution. Figure reproduced from [ 145 ] .

Figure 3.5 : Approximate posteriors for the feather falling simulator with measured falling time of

4.68\text{s}

,

6.97\text{s}

and

9.69\text{s}

for heights

3\text{m}

,

5\text{m}

and

7\text{m}

respectively.

Inputs:	A prior $p(\bm{\theta})$ , a simulator $p(\bm{x}\|\bm{\theta})$ and a target $\bm{x}_{o}$
Outputs:	An approximate posterior $\hat{p}(\bm{\theta}\|\bm{x})$ with focused training on $\bm{x}_{o}$
Hyperparameter:	Number rounds n_rounds and number of samples per round n_samples

Algorithm 3.3 Sequential simulation-based inference

Figure 3.6 : Visual inspection of the posterior quality. Either the approximate posterior density

\hat{p}(\bm{\theta}|\bm{x})

or samples from it are shown. The black star represents the parameters

\bm{\theta}^{*}

that were used to generate the synthetic test observation.

Figure 4.1 : A classifier-based metric measures the divergence between posterior approximations and a ground truth by means of evaluating the classifier’s discriminative performance through Area Under the Receiver Operating Characteristics curve ( auroc ). The metric indicates that both the overconfident and the conservative approximations are equally accurate as it yields auroc = 0.7 for both of them. From an inference perspective however, the conservative approximation is more suitable because it produces credible regions larger than they should be.

Figure 4.2 : Evolution of the expected coverage w.r.t the simulation budget. A perfectly calibrated posterior has an expected coverage probability equal to the credibility level and produces a diagonal line. Conservative estimators on the other hand produce curves above the diagonal and overconfident models underneath. All algorithms can lead to non-conservative estimators. This pathology tends to be accentuated for small simulation budgets and non-amortized methods. Finally, the computationally prohibitive results indicate that the computational requirements did not allow for a coverage analysis. In the case of SNL, this was mostly due to high dimensional observables. For the astronomy benchmarks, the simulation model was simply too expensive to reasonably evaluate coverage for non-amortized methods.

Figure 4.3 : Evolution of the predicted log posterior probability of the nominal parameters with respect to the simulation budget. We evaluate the log posterior probability on

100,000

test samples for amortized algorithms and

300

for non-amortized algorithms. We report the

0.25;0.5;0.75

quantiles of this quantity over the test set. We observe that the predictive performance increases in general with the simulation budget. Exceptions are the MG1 benchmark with the SNRE algorithm and the lowest quantile of the streams benchmark.

Figure 4.4 : Analysis of coverage between ensemble and individual models w.r.t the various simulation budgets. The blue line represents the mean expected coverage of individual models over

5

runs, the shaded area represents its standard deviation. The black line represents the expected coverage of a single ensemble composed of 5 models. We observe that deep ensembles consistently have a higher expected coverage probability compared to the average individual model. A similar effect is not always observed with bagging, indicated by the red line. Ensembles are only evaluated for amortized approaches such as NPE and NRE.

Figure 4.5 : Evolution of the empirical expected coverage of deep ensembles with respect to ensemble size for various confidence levels. The results are obtained by training ratio estimators ( nre ) on the slcp benchmark. A positive effect is observed in terms of empirical expected coverage and ensemble size, i.e., a larger ensemble size correlates with a larger empirical expected coverage. This is unsurprising, because a larger ensemble is expected to capture more of the uncertainty that stems from the training procedure.

Figure 4.6 : Evolution of the empirical expected coverage of bagging ensembles with respect to ensemble size for various confidence levels. The results are obtained by training ratio estimators ( nre ) on the slcp benchmark. A positive effect is observed in terms of empirical expected coverage and ensemble size, i.e., a larger ensemble size correlates with a larger empirical expected coverage. This is unsurprising, because a larger ensemble is expected to capture more of the uncertainty that stems from the training procedure.

Inputs:	Implicit generative model $p(\bm{x}\|\bm{\theta})$ (simulator) and prior $p(\bm{\theta})$
Outputs:	Approximate classifier $\hat{d}_{\psi}(\bm{\theta},\bm{x})$ parameterized by $\psi$
hyper-parameters:	Balancing condition strength $\lambda$ (default = 100) and batch-size $n$

Algorithm 5.1 Training algorithm for Balanced Neural Ratio Estimation (BNRE).

	SLCP	M/G/1	Weinberg	Lotka-V.	Spatial SIR	GW
Embedding network	None	None	None	CNN	Resnet-18	CNN
Embedding layers	/	/	/	$8$	/	$13$
Embedding channels	/	/	/	$8$	/	$16$
Convolution type	/	/	/	Conv1D	Conv2D	Dilated Conv1D
Head network	MLP	MLP	MLP	MLP	MLP	MLP
Head layers	$6$	$6$	$6$	$3$	$3$	$3$

Table 5.1 : Architectures and training hyper-parameters

Figure 5.1 : Expected coverage for increasing simulation budgets. A perfectly calibrated posterior has an expected coverage probability equal to the nominal coverage probability and hence produces a diagonal line. A conservative estimator has an expected coverage curve at or above the diagonal line, while an overconfident estimator produces curves below the diagonal line. The diagnostic therefore provides an immediate visual interpretation. We observe that NRE can produce overconfident estimators, while BNRE always produces coverage curves above the diagonal line and therefore the desired behaviour: conservative posterior approximations. The means over

5

runs are reported.

Figure 5.2 : Coverage AUC measures the integrated signed area between the expected coverage curve and the diagonal. A perfectly calibrated posterior has an expected coverage probability equal to the nominal coverage probability, producing a diagonal line and has a coverage AUC of

0

, as shown on the left subplot. A conservative estimator on the other hand has a coverage AUC larger than

0

and an overconfident estimator smaller than

0

. We observe that while NRE can produce coverage AUC both below or above

0

, BNRE always produces a coverage AUC larger than

0

, implying that its posterior approximations are conservative on average. The means over

5

runs are reported. A complete overview, including standard deviations, are provided in Section 20 .

Figure 5.3 : Expected value

\mathbb{E}_{p(\bm{\theta},\bm{x})}\big[\log\hat{p}(\bm{\theta}|\bm{x})\big]

of the approximate log posterior density of the nominal parameters with respect to the simulation budget. We observe that BNRE produces log posterior densities lower than NRE. This shows that enforcing the balancing condition to have more reliable posterior approximates comes at the price of a small loss in information gain. However, BNRE improves over the prior and eventually converges towards NRE as the simulation budget increases. Solid lines represent the mean over

5

runs and shaded areas represent the standard deviation.

Figure 5.4 : Comparison between NRE and BNRE in terms of expected coverage, bias and variance on the Weinberg benchmark. On the left side, the coverage is shown with respect to the simulation budget represented by the colormap. The bias and variance are represented on the right side of the plot. BNRE is run with

\lambda=100

. Consistent with our previous observations in Figure 5.3 , we observe that the gap in both bias and variance reduces as the simulation budget increases. Furthermore, in contrast with NRE, the posterior approximations of BNRE are tending towards being increasingly calibrated while at the same time being conservative. Solid lines represent the mean over

5

runs and shaded areas represent the standard deviation.

Figure 5.5 : Effect of the hyper-parameter

\lambda

for a fixed simulation budget of

1024

. The first plot from left to right shows the evolution of the approximate posterior for a given observation at a fixed

\bm{\theta}^{*}

, indicated by the red vertical line. This approximate posterior is compared to NRE trained on a large simulation budget, shown in yellow and serving as a proxy for the true posterior. The second plot illustrates the empirical expected coverage. The third plot provides a summarized view of the second plot using the coverage AUC as summary statistic. The fourth plot shows that classifiers are becoming increasingly more balanced as

\lambda

increases. In addition, the plots show that

\lambda

is directly tied to the statistical performance and reliability of the posterior approximations. Classifiers trained with small

\lambda

’s are associated with (relatively) tight posteriors and overconfident approximations, while classifiers trained with larger values of

\lambda

are increasingly more dispersed and conservative until the posterior approximations reduce to the prior due to inflated statistical noise of the Monte Carlo estimation of the balancing condition. Furthermore, the expected coverage plot shows the estimator is almost perfectly calibrated and implicitly balanced. Immediately visible from the various posterior approximations in the leftmost subplot, is the fact that BNRE produces overconfident and biased approximations in the presence of a small simulation budget and a small

\lambda

, indicated by their dark blue color. However, the balancing condition can be applied to the underlying estimator to improve its reliability by increasing

\lambda

. Ideally,

\lambda

should be as small as possible to maximize predictive performance, while at the same time remain sufficiently large to guarantee coverage. From the 3 th subplot from the left, in this particular problem setting, that happens at the point where the coverage AUC transitions from being negative to positive (

\lambda\approx 25.0

).

Figure 5.6 : Coverage AUC measures the integrated signed area between the expected coverage curve and the diagonal. A perfectly calibrated posterior has an expected coverage probability equal to the nominal coverage probability, producing a diagonal line and has a coverage AUC of

0

, as shown on the left subplot. A conservative estimator on the other hand has a coverage AUC larger than

0

and an overconfident estimator smaller than

0

. We observe that while NRE can produce coverage AUC both below or above

0

, BNRE always produces a coverage AUC larger than

0

, implying that its posterior approximations are conservative. Solid lines represent the mean over

5

runs and shaded areas represent the standard deviation.

Figure 5.7 : Evolution of the bias and variance w.r.t. the simulation budget. The bias and variance are estimated as described in Section 17 and are scaled to account for the prior’s spread, permitting a direct comparison between the benchmarks. Marginals are considered when dealing with multidimensional parameter spaces. Those are denoted by an index following the benchmark name.

	SLCP	Weinberg	Lotka-V.	Spatial SIR	Two Moons
Embedding network	None	None	CNN	CNN	None
Embedding layers	/	/	$8$	$8$	/
Embedding channels	/	/	$8$	$16$	/
Convolution type	/	/	Conv1D	Conv2D	/
Classifier Head	MLP	MLP	MLP	MLP	MLP
Classifier layers	$6$	$6$	$6$	$6$	$6$

Table 6.1 : Architectures and training hyper-parameters

Figure 6.1 : Empirical expected coverage for increasing simulation budgets. A perfectly calibrated surrogate has an expected coverage probability equal to the nominal coverage probability and produces a diagonal line. A conservative surrogate has an expected coverage curve at or above the diagonal line. An overconfident surrogate produces curves below the diagonal line. Balanced algorithms tend to produce conservative surrogates. 5 runs are performed for each simulation budget with the median

\hat{\alpha}

at each nominal credibility reported.

Figure 6.2 : Nominal log posterior for increasing simulation budgets. 5 runs are performed for each simulation budget. Solid lines represent the median and the shaded areas represent the minimum and maximum. Larger values are desirable.

Figure 6.3 : Balancing error for increasing simulation budgets. 5 runs are performed for each simulation budget. Solid lines represent the median and the shaded areas represent the minimum and maximum. Smaller values are desirable.

Figure 7.1 : Visualization of the prior tuned to match the GP prior on the SLCP benchmark. Left: examples of posterior functions sampled from the tuned prior over neural network’s weights. Right: expected coverage of the prior Bayesian model average with the tuned prior and normal priors for varying standard deviations.

Figure 7.2 : Comparison of simulation-based inference methods through the nominal log probability and coverage area under the curve. The higher the nominal log probability, the more performant the method is. A calibrated posterior approximation exhibits a coverage AUC of

0

. A positive coverage AUC indicates conservativeness, and a negative coverage AUC indicates overconfidence. 3 runs are performed, and the median is reported. The plot at the top shows the results for NPE simulation-based inference methods, and the one at the bottom shows NRE methods.

Figure 7.3 : Examples of

95\%

highest posterior density regions obtained with various algorithms and simulation budgets on the SLCP benchmark for a single observation. The black star represents the ground truth used to generate the observation.

Figure 7.4 : Comparison of posterior approximations obtained using a prior tuned to match the Gaussian process-based prior and using independent normal priors on weights with zero means and various standard deviations on the SLCP benchmark. 3 runs are performed, and the median is reported.

Figure 7.5 : Quantification of the different forms of uncertainties captured by the different NPE-based methods on the SLCP benchmark. 3 runs are performed, and the median is reported.

Figure 7.6 : Comparison of the posterior approximations obtained with and without a Bayesian neural network on the cosmological application. First plot: An example observation: particles representing galaxies in a synthetic universe. Second plot: example of

95\%

highest posterior density regions for increasing simulation budgets. The black star represents the ground truth used to generate the observation. Third plot: Expected coverage with and without using a Bayesian neural network for increasing simulation budgets. Fourth plot: The nominal log posterior.

Figure 7.7 : Coverage of different NPE simulation-based inference methods. A calibrated posterior approximation exhibits a coverage AUC of

0

. A coverage curve above the diagonal indicates conservativeness and a curve below the diagonal indicates overconfidence. 3 runs are performed, and the median is reported.

Figure 7.8 : Coverage of different NRE simulation-based inference methods. A calibrated posterior approximation exhibits a coverage AUC of

0

. A coverage curve above the diagonal indicates conservativeness and a curve below the diagonal indicates overconfidence. 3 runs are performed, and the median is reported.

Figure 7.9 : Quantification of the different forms of uncertainties captured by the different NPE-based methods. 3 runs are performed, and the median is reported.

Figure 7.10 : Quantification of the different forms of uncertainties captured by the different NRE-based methods. 3 runs are performed, and the median is reported.

Figure 7.11 : Comparison of different NPE simulation-based inference methods through the nominal log probability and coverage area under the curve. The higher the nominal log probability, the more performant the method is. A calibrated posterior approximation exhibits a coverage AUC of

0

. A positive coverage AUC indicates conservativeness, and a negative coverage AUC indicates overconfidence. 3 runs are performed. The median run is reported in plain, and the shaded lines represent the other two runs.

Figure 7.12 : Comparison of different NRE simulation-based inference methods through the nominal log probability and coverage area under the curve. The higher the nominal log probability, the more performant the method is. A calibrated posterior approximation exhibits a coverage AUC of

0

. A positive coverage AUC indicates conservativeness, and a negative coverage AUC indicates overconfidence. 3 runs are performed. The median run is reported in plain, and the shaded lines represent the other two runs.

Figure 8.1 : Comparison of high posterior density credible intervals obtained using various parameterizations. We consider that one parametrization is the exponential of the other. The blue lines represent the p.d.f. and the black lines represent the

95\%

high posterior density credible intervals.

Towards Reliable Simulation-based Inference

Abstract

Figures & tables

Explore similar work

An Introduction to Bayesian and Frequentist Simulation-Based Inference with Machine Learning

Conservative neural posterior estimation via distributionally robust training

Sample-Size Extrapolation in Simulation-Based Inference via Statistical Scaling