Native-latent guidance is a recent paradigm for solving inverse problems with latent diffusion models. It replaces repeated evaluations of the image-space forward model, each requiring a decoder pass, with efficient guidance computed using a learned latent-space surrogate. However, existing methods apply guidance uniformly across latent dimensions, ignoring that measurements are informative only along certain directions and that the reliability of model predictions varies across inputs and timesteps. We propose ANaLOG, a framework for efficient uncertainty-aware guidance with pretrained latent diffusion models. ANaLOG models uncertainty by learning an anisotropic, input- and time-dependent covariance that is integrated into the guidance mechanism to emphasize reliable directions and downweight uncertain ones. We theoretically analyze this framework in a linear model setting and prove that anisotropic, uncertainty-aware weighting is necessary for correct sampling, whereas isotropic guidance induces sampling errors. Experiments across five challenging inverse problems show that ANaLOG improves perceptual reconstruction quality over existing methods while preserving efficiency.
Figures & tables
Figure 1: Left: Representative results of our method on the Gaussian deblurring, 8x super-resolution, and nonlinear JPEG decompression tasks. Right: ANaLOG models anisotropic covariance during guidance without expensive decoding steps. On a Gaussian deblurring task, ANaLOG preserves identity-relevant detail that isotropic approximations wash out.
Efficient Diffusion
Efficient Guidance
Uncertainty- Aware
Pixel-space
✗
✓
✓
LDPS, PSLD
✓
✗
✗
STSL
✓
✗
✓
SILO
✓
✓
✗
ANaLOG (Ours)
✓
✓
✓
Table 1: Comparison of posterior sampling methods across three desiderata.
Figure 2: Guidance pathways at each sampling step.
Figure 3: Visualizing our theorem bounds.
Figure 4: Qualitative comparison on FFHQ deconvolution tasks. Best viewed zoomed in.
Figure 5: Learned variance maps and ablation of covariance structure. (a) Task-dependent variance maps show the model learns distinct spatial patterns for each inverse problem. (b) Progressively richer covariance models improve reconstruction quality.
Gaussian Blur
4x Super-Resolution
8x Super-Resolution
Method
Time (s)
FID ↓
LPIPS ↓
PSNR ↑
FID ↓
LPIPS ↓
PSNR ↑
FID ↓
LPIPS ↓
PSNR ↑
FFHQ
Decode
LDPS
52.2
84.8
0.564 (.004)
21.48 (.16)
68.6
0.554 (.003)
20.70 (.11)
55.5
0.536 (.003)
21.24 (.08)
PSLD
62.2
70.8
0.535 (.003)
22.65 (.14)
53.5
0.505 (.003)
22.52 (.10)
54.0
0.513 (.003)
21.68 (.08)
STSL
433
41.7
0.419 (.002)
27.36 (.07)
39.7
0.415 (.002)
26.88 (.10)
38.6
0.439 (.002)
25.58 (.08)
Native
SILO
24.0
47.3
0.367 (.002)
25.03 (.06)
38.8
0.331 (.002)
26.26 (.06)
35.0
0.356 (.002)
25.81 (.06)
ANaLOG
23.7
29.8
0.326 (.002)
27.34 (.09)
27.0
0.303 (.002)
27.56 (.08)
38.8
0.357 (.002)
25.39 (.05)
Table 2: Quantitative results on deconvolution tasks on the FFHQ and ImageNet datasets. Bolded entries indicate best performance, underlined entries indicate second best performance, and subscripts denote standard error of mean. We bold multiple rows in the event of approximately tied results. Gaussian deblurring is the representative task for timings since timings do not vary significantly across the different forward operators.
Box Inpainting
JPEG
Method
Time (s)
FID ↓
LPIPS ↓
PSNR ↑
FID ↓
LPIPS ↓
PSNR ↑
FFHQ
Decode
LDPS
52.2
68.2
0.205 (.001)
16.61 (.12)
101.0
0.612 (.003)
17.83 (.14)
PSLD
62.2
62.7
0.196 (.001)
17.09 (.12)
N/A
N/A
N/A
STSL
433
63.1
0.191 (.001)
18.19 (.13)
63.5
0.502 (.003)
22.09 (.17)
Native
SILO
24.0
32.8
0.142 (.001)
22.59 (.08)
68.3
0.471 (.002)
20.87 (.06)
ANaLOG
23.7
22.8
0.135 (.001)
21.31 (.11)
46.3
0.393 (.002)
23.91 (.06)
Table 3: Quantitative results on center box inpainting and JPEG decompression on the FFHQ dataset. Note PSLD does not apply to nonlinear inverse problems.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Commuting diagram for the latent forward operator. The Noble identity guarantees that encoding after degradation ( E∘A , right then down) equals latent degradation after encoding ( H∘E , down then right), so the diagram commutes: E(Ax0)=HE(x0)=Hz0 .
Figure 7: KL lower bound ( 18 ) vs. measurement noise σy at t=0.05 for different spectral decay rates α . The isotropic penalty depends non-trivially on σy : at low noise the bound is set by the operator’s frequency response, while at high noise the encoder’s whitening ( σy2/λω ) dominates.
Figure 8: Covariance eigenvalues cω(t) at t=0.05 for different forward operators ( α=2.5 , σy=0.001 ). Eigenvalues are plotted against nonnegative frequency magnitude ∣ω∣ . Each operator produces a distinct spectral profile. The identity is nearly flat; blurs suppress high frequencies progressively; 4× SR introduces a sharp cutoff. An isotropic covariance cannot adapt to these task-dependent structures.
Layer
Operation
Output shape
# Params
Input
f=[fagg∥eσ]
385×64×64
—
Conv-1
Conv2d(385→128,3×3,pad=1)
128×64×64
443,648
SiLU
128×64×64
—
Conv-2
Conv2d(128→32,3×3,pad=1)
32×64×64
36,896
SiLU
32×64×64
—
Conv-3
Conv2d(32→4,1×1)
4×64×64
132
Appendix
Table 4: Architecture of the log-variance head hϕ . Input is the pre-head feature f∈RB×385×64×64 (aggregated UNet features fagg∈R384×64×64 concatenated with the sigma embedding map eσ∈R1×64×64 ). Output is per-pixel logσ2∈RB×4×64×64 .
Figure 9: Gaussian Blur task on FFHQ dataset.
Figure 10: 4x Super-Resolution task on FFHQ dataset.
Figure 11: 8x Super-Resolution task on FFHQ dataset.
Figure 12: JPEG Decompression task on FFHQ dataset.
Table 6: Per-coordinate calibration of the learned diagonal covariance Σθ on FFHQ ( N=256 held-out images per task). E[z2] , z and std(z) are pooled over all latent coordinates. The remaining columns characterise the distribution of Varb[zi] over the 4×64×64=16,384 latent coordinates, and the per-coordinate residual bias ∣zi∣ averaged over coordinates. The sampling-noise floor on Varb[zi] at N=256 is ≈0.089 .
Figure 14: Calibration of the learned diagonal covariance Σθ on FFHQ, all five tasks overlaid ( gb , sr4 , sr8 , ip , jpeg ; N=256 held-out images each). (a) Distribution of the per-coordinate empirical variance Varb[zi] over the 16,384 latent coordinates; the dashed line marks the calibrated value 1 . All five tasks are sharply peaked at 1 . (b) Reliability diagram on log-log axes: coordinates binned into 12 quantiles by mean predicted variance; the dashed line is y=x . All five tasks track y=x across roughly two orders of magnitude of predicted variance, indicating that the variance head is calibrated not only on average but conditional on its own predictions.
Figure 15: Sensitivity to the guidance scale hyperparameter on FFHQ dataset.
Guidance
FID ↓
LPIPS ↓
PSNR ↑
Stop-gradient (ours)
61.7
0.317
27.47
Full covariance gradient
60.9
0.325
27.44
Appendix
Table 7: Effect of stopping gradients through the learned covariance.
Pretrained diffusion models are effective priors for Bayesian inverse problems, but posterior sampling with these priors is often costly because data-consistency guidance is applied throughout the full reverse trajectory. Existing methods have shown that vector-Jacobian products through the denoiser can sometimes be avoided, yet they typically still rely on dense guidance through the full trajectory or expensive inner solves. We introduce Sparse Scheduled Diffusion Guidance for Inverse Problems (Spin), a solver that avoids starting posterior sampling from pure noise. Spin first samples from a posterior time-marginal at an intermediate timestep t∗, and then uses that state as a warm start for a guided reverse diffusion process. At guidance time, instead of enforcing the measurement constraint at every denoising step, Spin applies lightweight corrections only at scheduled timesteps where the denoiser can still clean up artifacts. The resulting procedure decouples prior refinement from data consistency: the prior supplies denoising, while lightweight pixel-space optimization enforces the measurement constraint without backpropagation through the denoiser or decoder. Across linear and nonlinear inverse problems on FFHQ and ImageNet, Spin achieves competitive reconstruction quality with a substantially better runtime--memory profile, running 2x faster on pixel-space models and up to 50x faster on latent diffusion models, with lower memory costs.
Vision-Language Latent Diffusion Models (LDMs) (Rombach et al., 2022) provide powerful generative priors for inverse problems. However, existing LDM-based inverse solvers typically require a large number of neural function evaluations (NFEs) and backpropagation through large pretrained components, leading to substantial computational costs and, in some cases, degraded reconstruction quality. We propose a unified Euclidean-Wasserstein-2 gradient-flow framework that jointly performs posterior sampling and prompt optimization in the latent space through a single flow that aligns the prior and posterior with the observed data. Combined with few-step latent text-to-image models, this formulation enables low-NFE inference without backpropagation through autoencoders. Experiments across several canonical imaging inverse problems show that our method achieves state-of-the-art performance with significantly reduced computational cost.
Alessio Spagnoletti, Tim Y. J. Wang, Marcelo Pereyra +1
Laboratoire MAP5, UMR 8145, Université Paris Cité, CNRS · Department of Mathematics, Imperial College London · Heriot-Watt University, MACS & Maxwell Institute
Generative diffusion models can provide powerful prior probability models for inverse problems in imaging, but existing implementations suffer from two key limitations: (i) the prior density is represented implicitly, and (ii) they rely on likelihood approximations that introduce sampling biases. We address these challenges by introducing a new energy-based model trained for denoising with a covariance-based regularization term that enforces consistency across different measurement conditions. The trained model can compute normalized posterior densities for diverse linear inverse problems, without additional retraining or fine tuning. In addition to preserving the sampling capabilities of diffusion models, this enables previously unavailable capabilities: energy-guided adaptive sampling that adjusts schedules on-the-fly, unbiased Metropolis-Hastings correction steps, and blind estimation of the degradation operator via Bayes rule. We validate the method on multiple datasets (ImageNet, CelebA, AFHQ) and tasks (inpainting, deblurring), demonstrating competitive or superior performance to established baselines.
Nicolas Zilberstein, Santiago Segarra, Eero Simoncelli +1
Rice University, Houston, TX, USA · Flatiron Institute, New York, NY, USA · New York University, New York, NY, USA