Amortized Inference of Multi-Modal Posteriors using Likelihood-Weighted Normalizing Flows
Authors: Rajneil Baruah
Organizations: Department of Physics, SEAS, Bennett University, Greater Noida, Uttar Pradesh, India, 201310 · Department of Physics, Bangabasi Evening College, Kolkata, West Bengal, 700011
We present a technique for posterior estimation using Normalizing Flows trained with likelihood-weighted importance sampling, the special case of neural importance sampling in which the proposal is the prior. This approach allows for the efficient inference of theoretical parameters in high-dimensional inverse problems without the need for posterior training samples. We implement the method on multi-modal benchmark tasks in 2D and 3D to check for the efficacy. A critical observation of our study is the impact of the base distribution on the modelled posteriors. We find that standard unimodal base distributions leave spurious probability bridges between well-separated modes. We demonstrate that initializing the flow with a Gaussian Mixture Model that matches the number of target modes significantly improves reconstruction fidelity, as measured by some distance and divergence metrics. Finally, we apply this method to a curated problem in heavy flavour physics --- the extraction of the Wolfenstein parameters from the CP asymmetry in B0→J/ψK0; it is multimodal, non-Gaussian, and asymmetric in its mode weights --- and compare the results against a well-converged Markov Chain Monte Carlo reference and an exact reference computed by quadrature, using different metrics. Used as a proposal for importance sampling, the trained flow removes its remaining bias and gives the evidence.
Figures & tables
Method
Objective
VI
Eq[logp(D∣θ)]−DKL(q∥p(θ))
IWVI
Eq[logK1∑kq(θk)p(D∣θk)p(θk)]
SBI
Ep(θ,x)[−logqϕ(θ∣x)]
NIS
Er(θ)[r(θ)p(D∣θ)p(θ)logqϕ(θ)] , r built from qϕ
LW-NF
Ep(θ)[p(D∣θ)logqϕ(θ)]
Table 1: Comparison of objective functions across methods. LW-NF performs likelihood-weighted density estimation rather than evidence maximisation or conditional density learning. It is the special case of NIS in which the proposal r is the prior.
Figure 1: The workflow, including training the NF and then sampling from the base to pass it through the flow
Figure 2: Ground truth densities for the 2D benchmark tasks. From left to right: Single-mode, Two-mode, and Three-mode posteriors constructed from Gaussian mixtures.
Figure 3: Modelled posterior distributions qϕ(θ) generated by the Normalizing Flow. While the single-mode posterior (Left) is captured accurately, the multi-modal cases (Center, Right) exhibit spurious “bridges” connecting the modes.
Target Posterior
DKL
W1
W1 (floor)
Single-Mode
<10−3
0.0287
0.0486
Two-Mode
0.0143
0.1154
0.1009
Three-Mode
0.0290
0.1352
0.0845
Table 2: Quantitative comparison of posterior reconstruction with the same model (single-Gaussian base) against three target posteriors of increasing multimodality. The noise floor is the same W1 computed between two disjoint halves of the reference sample; it is the value that two samples of the identical distribution produce at this sample size, and a reported distance is meaningful only insofar as it exceeds it. The single-mode reconstruction is at the floor, i.e. statistically indistinguishable from the truth, while the multimodal cases sit clearly above it.
Figure 4: Isosurface visualization of the ground truth 3-dimensional posterior distribution, displaying three distinct modes in the parameter space. The vertical axes share identical limits and tick spacing across all panels.
Figure 5: 2D Marginal projections of the modelled 3D posterior. The flow captures the location of the modes, but leaves bridges between them with the unimodal base distribution. The vertical axes share identical limits and tick spacing across all panels
Figure 6: Comparison of learned posteriors from Normalizing Flows with different numbers of distinct modes in the base distribution. From Left to Right, the models have 1 base mode( Model−2D1 ), 2 base modes( Model−2D2 ) and 3 base modes( Model−2D3 ).
Figure 7: Comparison of learned posterior from Normalizing Flows with different no. of distinct modes in the base distribution. From Left to Right, the models have 1 base mode( Model−3D1 ), 2 base modes( Model−3D2 ) and 3 base modes( Model−3D3 ). The top and bottom rows are the projections on x-y plane and x-z plane, respectively.
Model
W1
SW1
DKL
W1 (floor)
Model-2D1
0.1352
0.1291
0.0358
0.1070
Model-2D2
0.2372
0.2797
0.0216
0.1367
Model-2D3
0.0787
0.0948
0.0124
0.0914
Table 3: Performance comparison between different 2D models when the true posterior has three modes. Model-2D 3 , whose number of base components matches the number of modes of the true posterior, is the best on all three metrics; its W1 is at the noise floor, i.e. its reconstruction is statistically indistinguishable from the truth at this sample size.
Figure 8: Comparison of learned posterior from Normalizing Flows with different no. of distinct modes in the base distribution, when the true posterior is a non-Gaussian distribution. From Left to Right, the models have 1 base mode(Model-nonGauss 1 ), 2 base modes(Model-nonGauss 2 ) and 3 base modes(Model-nonGauss 3 ). The top and bottom rows are the projections on x-y plane and x-z plane, respectively.
Model
W1
SW1
DKL
W1 (floor)
Model-nonGauss 1
1.6173
1.5800
0.1699
0.3340
Model-nonGauss 2
4.4237
4.4310
0.0722
0.3517
Model-nonGauss 3
0.3732
0.3788
0.0322
0.2291
Table 4: Performance comparison between the three 3D models when the true posterior is non-Gaussian with three modes. Model-nonGauss 3 is again the best on every metric, matching the trend of table 3 ; the overall performance is, however, poorer than in the Gaussian case, and all three sit above the noise floor. The reference sample for this target contains a substantial fraction of repeated rows (it was generated by resampling with replacement), which inflates the effective precision of density-based estimators; the W1 and SW1 columns are unaffected.
Figure 9: The probability density plot of the observables in 1-D( ACP=sin2β ) and 2-D( (ACP,∣A∣2) )
Parameter
ρˉ
ηˉ
∣Vcb∣
g
Prior
[−0.10,0.97]
[0.20,0.50]
[0.020,0.070]
[10,40]
Table 5: Uniform prior bounds for the 4D problem. The ρˉ range contains both solutions, ρˉ1≈0.143 and ρˉ2≈0.857 .
Figure 10: One-dimensional marginal posteriors for ρˉ and ηˉ comparing MCMC and likelihood-weighted normalizing flows.
Figure 11: Two-dimensional credible regions from MCMC and likelihood-weighted normalizing flows in the (ρˉ,ηˉ) and (ρˉ,g) planes.
Method
f1
f2
JS↓
W1↓
Exact (quadrature)
0.811
0.189
−
−
MCMC (ref.)
0.809
0.191
−
−
LW-NF k=1
0.824±0.001
0.176±0.001
0.0013±0.0002
0.0075±0.0013
LW-NF k=2
0.819±0.006
0.181±0.006
0.0012±0.0003
0.0067±0.0021
Table 6: Mode fractions, with the modes separated at ρˉ=0.5 , and mean marginal Jensen–Shannon ( JS ) and Wasserstein-1 ( W1 ) distances to the well-converged MCMC reference. The exact values are computed by quadrature ( section 6.5.2 ). Reference-relative metrics are not given for the references themselves ( − ).
Method
dMah(1)↓
dMah(2)↓
dTrMah(1)↓
dTrMah(2)↓
Exact (quadrature)
−
−
0.124
0.364
MCMC (ref.)
−
−
0.128
0.361
LW-NF k=1
0.050±0.009
0.117±0.030
0.135±0.015
0.381±0.054
LW-NF k=2
0.053±0.022
0.108±0.013
0.124±0.014
0.341±0.038
Table 7: Mode-location metrics. dMah(i) measures the displacement of the reconstructed mode centroid from the MCMC reference centroid in units of the reference per-mode covariance, while dTrMah(i) measures the displacement from the injected parameter values. The truth-based distance of the exact posterior is not zero, because the prior displaces the mean of each mode; it is the value that a faithful method should reproduce.
Quantity
Exact (quadrature)
LW-NF k=2 with IS
f1 , split at ρˉ=0.5
0.8111
0.8100±0.0011
f1 , split at β=π/4
0.8220
0.8210±0.0011
lnZ
−9.4989
−9.5032±0.0019
Neff / draws inside the prior
−
118672/192581=0.62
Table 8: Mode fractions by the two definitions of section 6.4 , and the logarithm of the evidence, from the exact quadrature and from a single k=2 flow used as the proposal of eq. 31 with α=0.1 . The evidence is given up to the normalisation constant of the Gaussian likelihood, which is the same in both columns. The uncertainties are statistical and follow from Neff .
Figure 12: Marginal posteriors of ρˉ (left) and ηˉ (right) from a k=2 flow after the importance-sampling correction of eq. 31 (red), compared with the exact marginals computed by quadrature (black, dashed). The red curves are Gaussian kernel density estimates of the corrected samples, reflected at the edges of the prior. The dotted line on the right marks ηˉ≈0.449 , where the first solution reaches the lower edge ρˉ=−0.10 of the prior.
Neural posterior estimation learns probability distributions over unknown model parameters from observed data, often using simulations when likelihoods are unavailable.
School of Mathematics and Statistics, Xi’an Jiaotong University, Xi’an 710049, China · Academy of Mathematics and Systems Sciences, Chinese Academy of Sciences, Beijing, 100190, China