Proximal-IMH: Proximal Posterior Proposals for Independent Metropolis-Hastings with Approximate Operators

Youguang Chen; George Biros

cs.LGFeb 24, 2026

Proximal-IMH: Proximal Posterior Proposals for Independent Metropolis-Hastings with Approximate Operators

Authors: Youguang Chen, George Biros

Organizations: Oden Institute for Computational Engineering and Sciences, The University of Texas at Austin, Austin, Texas, USA

Abstract

We consider the problem of sampling from a posterior distribution arising in Bayesian inverse problems in science, engineering, and imaging. Our method belongs to the family of independence Metropolis-Hastings (IMH) sampling algorithms, which are common in Bayesian inference. Relying on the existence of an approximate posterior distribution that is cheaper to sample from but may have significant bias, we introduce Proximal-IMH, a scheme that removes this bias by correcting samples from the approximate posterior through an auxiliary optimization problem. This yields a local adjustment that trades off adherence to the exact model against stability around the approximate reference point. For idealized settings, we prove that the proximal correction tightens the match between approximate and exact posteriors, thereby improving acceptance rates and mixing. The method applies to both linear and nonlinear input-output operators and is particularly suitable for inverse problems where exact posterior sampling is too expensive. We present numerical experiments including multimodal and data-driven priors with nonlinear input-output operators. The results show that Proximal-IMH reliably outperforms existing IMH variants.

Figures & tables

Figure 1: Sensitivity of the expected KL divergence of the Approx , Latent , and Proximal proposals (relative to the Exact posterior ) for different noise levels, operator errors, observation ratios, and dimensions. Details of the experimental setup are given in Table 2 .

Figure 2: Comparison of sampling performance in the bimodal test. Rows correspond to Tests I–III. From left to right, panels show Metropolis–Hastings acceptance rates for three IMH methods, convergence of the relative mean error for the IMH methods together with NUTS and MALA, and histograms of the projected samples

w^{\top}x

for two levels of operator error. In the histogram panels, Approx-NUTS denotes samples drawn from the Approx posterior

\pi_{a}(x\mid y)

. For Proximal-IMH , we set

\beta=\sigma^{2}

. Results in the left panel are averaged over 5 independent trials for each method.

Figure 3: Operator discrepancy in the bimodal test. The quantities

\|\mathbf{I}-\widetilde{{\bf F}}^{-1}{\bf F}\|

and

\|\mathbf{I}-{\bf K}^{-1}\|

are reported for Latent-IMH and Proximal-IMH , respectively, where

{\bf K}

is defined in Equation 7 . For Proximal-IMH , the hyperparameter is set to

\beta=\sigma^{2}

Figure 4: Effect of

\beta

on Proximal-IMH in the bimodal test. Values in parentheses represent

\|{\bf A}-\widetilde{{\bf A}}\|/\|{\bf A}\|

. Blue curves show acceptance rates (left y-axis), and pink curves show relative mean errors after

10^{5}

MH steps (right y-axis).

Panel of Figure 5 — Figure 5: Ground truth, posterior mean and MAP estimates for the MNIST inverse problem in Tests IV–VII . The MAP estimates are obtained from different random initial guesses for the optimization problem. We observe that in several cases the reconstructed MAP point is wrong. This is due to the nonconvexity of solving for a MAP point.

Figure 6: Sampling performance on MNIST imaging inverse problems: convergence of the relative mean error and relative second-moment error for different sampling methods. Dashed lines indicate the bias induced by the Approx posterior

\pi_{a}(x\mid y)

. For Proximal-IMH , we set

\beta=\sigma^{2}

. Results in the convergence plots are reported over 5 independent trials.

Figure 7: Effect of the hyperparameter

\beta

on Proximal-IMH in the inverse problem in MNIST. Values in parentheses represent

\|{\bf A}-\widetilde{{\bf A}}\|/\|{\bf A}\|

. Blue curves show acceptance rates (left y-axis), and pink curves show relative mean errors after

10^{5}

MH steps (right y-axis). The results are not sensitive to

\beta

Panel of Figure 8 — Figure 8: Nonlinear Helmholtz inverse problem. Ground truth, MAP estimate, and posterior mean for the acoustic scattering test, together with convergence of the relative mean error.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Posterior	Distribution	Mean	Covariance	Pseudo-inverse
Exact	$\pi(x\mid y)=\mathcal{N}(\mu,\bm{\Sigma})$	$\mu={\bf A}^{\dagger}y$	$\bm{\Sigma}={\bf I}-{\bf A}^{\dagger}{\bf A}$	${\bf A}^{\dagger}={\bf A}^{\top}\left({\bf A}{\bf A}^{\top}+\sigma^{2}{\bf I}\right)^{-1}$
Approx	$\pi_{a}(x\mid y)=\mathcal{N}(\mu_{a},\bm{\Sigma}_{a})$	$\mu_{a}={\bf A}_{a}^{\dagger}y$	$\bm{\Sigma}_{a}={\bf I}-{\bf A}_{a}^{\dagger}\widetilde{{\bf A}}$	${\bf A}_{a}^{\dagger}=\widetilde{{\bf A}}^{\top}\left(\widetilde{{\bf A}}\widetilde{{\bf A}}^{\top}+\sigma^{2}{\bf I}\right)^{-1}$
Latent	$\pi_{l}(x\mid y)=\mathcal{N}(\mu_{l},\bm{\Sigma}_{l})$	$\mu_{l}={\bf A}_{l}^{\dagger}y$	$\bm{\Sigma}_{l}={\bf F}^{-1}\widetilde{{\bf F}}\bm{\Sigma}_{a}\widetilde{{\bf F}}^{\top}{\bf F}^{-\top}$	${\bf A}_{l}^{\dagger}={\bf F}^{-1}\widetilde{{\bf F}}{\bf A}_{a}^{\dagger}$
Proximal	$\pi_{p}(x\mid y)=\mathcal{N}(\mu_{p},\bm{\Sigma}_{p})$	$\mu_{p}={\bf A}_{p}^{\dagger}y$	$\bm{\Sigma}_{p}={\bf K}\bm{\Sigma}_{a}{\bf K}^{\top}$	${\bf A}_{p}^{\dagger}={\bf K}{\bf A}_{a}^{\dagger}$

Appendix

Table 1: Distributions of Exact , Approx , Latent and Proximal posteriors for Gaussian prior

p(x)=\mathcal{N}({\bf 0},{\bf I})

and noise

q(e)=\mathcal{N}({\bf 0},\sigma^{2}{\bf I})

. For Proximal posterior ,

{\bf K}=({\bf A}^{\top}{\bf A}+\beta{\bf I})^{-1}({\bf A}^{\top}\widetilde{{\bf A}}+\beta{\bf I})

Effects	$\log_{10}\mathrm{SNR}$	$\frac{\\|{\bf F}-\widetilde{{\bf F}}\\|}{\\|{\bf F}\\|}$	$d_{y}/d$	$d$
Noise level	0.5 —4.0	6%	0.2	500
Operator error	2.5	2% —21%	0.2	500
Observation ratio	2.5	6%	0.05 —0.5	500
Dimension	2.5	6%	0.2	100 —2000

Appendix

Table 2: Experimental setup for comparing

\mathbb{D}_{a}

\mathbb{D}_{l}

and

\mathbb{D}_{p}

Figure 9: Histogram of the projection of prior samples in the bimodal test (see Equation 21 ).

$d_{z}$	$d_{x}$	$d_{u}$	$\widetilde{d}_{u}$	$d_{y}$
128	784 ( $=28^{2}$ )	$10^{4}$ ( $=100^{2}$ )	$\{35^{2},\,20^{2}\}$	100

Appendix

Table 3: Dimensions of parameters in Tests VI and VII in Section 4.2 .

Panel of Figure 10 — Figure 10: Matrix constructions of ${\bf A}$ and $\widetilde{{\bf A}}$ in Tests VI and VII in Section 4.2 . Left: ${\bf A}={\bf O}\,{\bf F}\,{\bf P}$ . Right: $\widetilde{{\bf A}}={\bf O}\,{\bf P}_{u}\,\widetilde{{\bf F}}\,{\bf P}_{x}$ . In our setting $d_{y}<d_{x}<d_{u}$ , and $\widetilde{d}_{u}<d_{u}$ .

Figure 11: Jacobian log-determinant diagnostics for the Gauss–Newton correction. Each row corresponds to one test case. Left: Sorted values of the log determinant of the Jacobian of a single Gauss–Newton transformation, evaluated at 2000 randomly selected posterior samples from

\pi_{a}(x\mid y)

. Right: Order statistics of ratios of Jacobian determinants between 2000 randomly selected pairs of posterior samples from

\pi_{a}(x\mid y)

, i.e.,

\det J_{\mathrm{GN}}(x_{i})/\det J_{\mathrm{GN}}(x_{j})

. The numbers shown in the legends indicate the relative operator approximation error

\|{\bf A}-\widetilde{{\bf A}}\|/\|{\bf A}\|

. In the right panels, dashed lines denote the 5%–95% quantile range.

$d_{x}$	$d_{u}$	$\widetilde{d}_{u}$	$d_{y}$
1024 ( $=32^{2}$ )	16,384 ( $=128^{2}$ )	4,096 ( $=64^{2}$ )	512

Appendix

Table 4: Dimensions of parameters in the nonlinear Helmholtz test in Section 4.3 .

Figure 12: Effect of preconditioning on GMRES convergence for the nonlinear Helmholtz test.

Panel of Figure 13 — Figure 13: Wavefields $\{u_{i}=\mathcal{L}(x)^{-1}f_{i}\}_{i=1}^{4}$ for the nonlinear Helmholtz test. Each column corresponds to a distinct source $f_{i}$ . Top: Ground-truth wavefields. Middle: Wavefields obtained from the MAP estimate. Bottom: Posterior mean wavefields computed from posterior samples.

Explore similar work

Oct 1, 2026

Flow Matching for Fast Posterior Sampling in Bayesian Inverse Problems

Sampling from the posterior is the central task of computational Bayesian inverse problems. The standard workhorse in Bayesian inference - Markov chain Monte Carlo (MCMC) - is sequential, yields correlated samples, and must be rerun for each observation. Conditional flow matching offers an amortized alternative: a transport map, trained once on joint samples of parameter and data, that yields independent approximate posterior samples for any observation at negligible online cost, without new likelihood evaluations. We give a careful, MCMC-literate assessment of flow matching for PDE-based inverse problems with function-valued parameters. Exploiting the flow's tractable density, we derive computable accuracy estimates of the underlying approximate posterior in total-variation distance and Kullback-Leibler divergence and, moreover, propose a hybrid sampler that is asymptotically exact by Metropolization. We validate the accuracy estimates and demonstrate the amortization in several numerical examples, including electrical impedance tomography and a likelihood-free Lotka-Volterra model.

Jan Blechschmidt, Oliver G. Ernst, Moritz Poguntke +1

Posterior Sampling Bayesian Inverse Problems Amortized Inference Conditional Flow Matching PDE Inverse Problems

Ask PDF

May 28, 2026

When, why, and how do diffusion posterior samplers fail? A finite-sample lens

Diffusion models have excellent capacity to model complex distributions of natural data, which has made them a popular and effective choice for posterior sampling in imaging inverse problems. Existing methods can incorporate any measurement model at inference time but must use an inexact approximation for the likelihood at intermediate timesteps for computational tractability. Although these approximations can often work well empirically, their downstream effect on the sampled posterior is poorly understood and can result in unexplained failures. To understand when, why, and how these likelihood approximations propagate to erroneous posterior distributions, we introduce a finite-sample perspective on posterior sampling that approximates the posterior to arbitrary precision as training set size tends towards infinity, for any forward model and prior distribution. Using this finite-sample lens, we observe that popular posterior sampling approximations tend to under- or over-estimate the spread of the posterior at intermediate timesteps, causing downstream consequences including sensitivity to early stopping time, inaccurate relative weighting of posterior modes, and hallucination, both of prior modes that are not in the posterior and likelihood modes that are not supported by the prior. Moreover, we find that the cause of these posterior errors requires neither a nonlinear measurement model nor a multimodal posterior, but can arise solely due to a multimodal prior and inaccurate posterior spread at intermediate sampling times. Our finite-sample posterior sampling approach is agnostic to the type of likelihood approximation and the type of (linear or nonlinear) forward model, and can thus serve as a drop-in diagnostic to evaluate the accuracy and failure modes of existing and future posterior samplers.

Benjamin A. Burns, Sara Fridovich-Keil

Georgia Institute of Technology

Bayesian Inverse Problems Posterior Sampling Diffusion-Based Inverse Problems Diffusion Posterior Sampling

Ask PDF

Mar 20, 2026

Alternating Diffusion for Proximal Sampling with Zeroth Order Queries

This work introduces a new approximate proximal sampler that operates solely with zeroth-order information of the potential function. Prior theoretical analyses have revealed that proximal sampling corresponds to alternating forward and backward iterations of the heat flow. The backward step was originally implemented by rejection sampling, whereas we directly simulate the dynamics. Unlike diffusion-based sampling methods that estimate scores via learned models or by invoking auxiliary samplers, our method treats the intermediate particle distribution as a Gaussian mixture, thereby yielding a Monte Carlo score estimator from directly samplable distributions. Theoretically, when the score estimation error is sufficiently controlled, our method inherits the exponential convergence of proximal sampling under isoperimetric conditions on the target distribution. In practice, the algorithm avoids rejection sampling, permits flexible step sizes, and runs with a deterministic runtime budget. Numerical experiments demonstrate that our approach converges rapidly to the target distribution, driven by interactions among multiple particles and by exploiting parallel computation.

Hirohane Takagi, Atsushi Nitanda

Graduate School of Information Science and Technology, The University of Tokyo, Japan · Institute of High Performance Computing, Agency for Science, Technology and Research, Singapore · Centre for Frontier AI Research, Agency for Science, Technology and Research, Singapore +1

Diffusion Sampling

Ask PDF