Sparse Scheduled Diffusion Guidance for Inverse Problems
Organizations: MBZUAI, Abu Dhabi, United Arab Emirates · Institute of Foundation Models · EPITA, Paris, France
Abstract
Pretrained diffusion models are effective priors for Bayesian inverse problems, but posterior sampling with these priors is often costly because data-consistency guidance is applied throughout the full reverse trajectory. Existing methods have shown that vector-Jacobian products through the denoiser can sometimes be avoided, yet they typically still rely on dense guidance through the full trajectory or expensive inner solves. We introduce Sparse Scheduled Diffusion Guidance for Inverse Problems (Spin), a solver that avoids starting posterior sampling from pure noise. Spin first samples from a posterior time-marginal at an intermediate timestep , and then uses that state as a warm start for a guided reverse diffusion process. At guidance time, instead of enforcing the measurement constraint at every denoising step, Spin applies lightweight corrections only at scheduled timesteps where the denoiser can still clean up artifacts. The resulting procedure decouples prior refinement from data consistency: the prior supplies denoising, while lightweight pixel-space optimization enforces the measurement constraint without backpropagation through the denoiser or decoder. Across linear and nonlinear inverse problems on FFHQ and ImageNet, Spin achieves competitive reconstruction quality with a substantially better runtime--memory profile, running 2x faster on pixel-space models and up to 50x faster on latent diffusion models, with lower memory costs.
Figures & tables
| Method | Inference Objective | Guidance Density | VJP ? | VJP / ? | Assumptions on |
|---|---|---|---|---|---|
| DPS , PGDM | Score-based posterior approximation | Dense | Yes | Yes † | General / known |
| DAPS | Decoupled posterior annealing | Moderate + inner MCMC | No | Often † | General |
| RED-Diff , DDS | Variational / Krylov data consistency | Dense / iterative solve | No | No | General / linear |
| DiffPIR , PnP-DM | Plug-and-play / proximal posterior | Dense + inner loops | No | No | General |
| PSLD , ReSample , P2L , TReg , LATINO-PRO | Latent/text/consistency-model inverse solvers | Moderate–dense + inner solve | Varies | Varies † | Varies |
| FlowDPS , PnP-Flow , FLAIR | Flow / rectified-flow prior guidance | Dense / iterative solve | Varies | Often † | General / linear |
| LPIPS FFHQ | ||||||||
| Task | Spin | DAPS | Red-Diff | PNP-DM | DPS | DDNM | PGDM | Diffpir |
| Gaussian Deblur | 0.17 | 0.19 | 0.25 | 0.20 | 0.16 | 0.84 | 0.87 | – |
| Motion Deblur | 0.13 | 0.19 | 0.20 | 0.21 | 0.21 | – | – | – |
| SR ( 4) | 0.20 | 0.19 | 0.44 | 0.21 | 0.22 | 0.36 | 0.30 | – |
| SR ( 16) | 0.35 | 0.45 | 0.59 | 0.60 | 0.36 | 0.52 | 0.42 | – |
| Box Inpainting | 0.14 | 0.12 | 0.18 | 0.55 | 0.20 | 0.18 | 0.17 | 0.14 |
| LPIPS ImageNet | ||||||||
| Task | Spin | DAPS | Red-Diff | PNP-DM | DPS | DDNM | PGDM | Diffpir |
| Gaussian Deblur | 0.34 | 0.41 | 0.44 | 0.47 | 0.35 | 0.42 | 1.06 | – |
| Motion Deblur | 0.28 | 0.40 | 0.35 | 0.50 | 0.41 | – | – | – |
| SR ( 4) | 0.39 | 0.41 | 0.53 | 0.49 | 0.47 | 0.27 | 0.63 | – |
| SR ( 16) | 0.68 | 0.80 | 0.82 | 0.88 | 0.58 | 0.68 | 0.59 | – |
| Box Inpainting | 0.32 | 0.31 | 0.36 | 0.71 | 0.46 | 0.28 | 0.28 | 0.28 |
Appendix figures & tables35 assets
Supplementary material from the paper’s appendix.
Appendix
| Method | Target | Kernel / DC step | VJP den.? | VJP enc/dec? | Notes |
|---|---|---|---|---|---|
| DPS , PGDM | |||||
| [ 3 , 33 ] | Posterior via DPS-style score approximation | Reverse diffusion + per-step likelihood guidance | Yes | Yes † | General ; PGDM uses known/generalized ; high memory/NFE |
| DAPS , RED-Diff | |||||
| [ 40 , 22 ] | Decoupled / variational posterior | Annealed MCMC / variational update without denoiser VJPs | No | Often † | Inner sampling or optimization; DAPS latent variants may use encoder/decoder gradients |
| DDNM , DDRM | |||||
| [ 38 , 17 ] | Linear posterior / projection | Denoise + pseudo-inverse / SVD projection (DC) | No | No | Needs linear ; DDNM uses pseudo-inverse, DDRM uses SVD |
| Strategy | Weight Function | Parameters |
|---|---|---|
| Uniform | — | |
| Linear | — | |
| Polynomial | ||
| Exponential | ||
| Gaussian | ||
| Beta | , |
| Schedule | Deblur | Motion Deblur | SR4 |
|---|---|---|---|
| Uniform | 0.772 | 0.768 | 0.772 |
| Linear | 0.692 | 0.673 | 0.691 |
| Polynomial | 0.818 | 0.754 | 0.785 |
| Exponential | 0.823 | 0.808 | 0.805 |
| Beta | 0.759 | 0.523 | 0.717 |
| Gaussian | 0.825 | 0.824 | 0.827 |
| Schedule | Deblur | Motion Deblur | SR4 |
|---|---|---|---|
| Uniform | 26.42 | 26.24 | 26.38 |
| Linear | 23.63 | 23.04 | 23.48 |
| Polynomial | 28.78 | 27.93 | 28.18 |
| Exponential | 28.71 | 28.52 | 28.40 |
| Beta | 27.84 | 24.70 | 27.19 |
| Gaussian | 28.72 | 28.57 | 28.66 |
| Sampler | Mean dist. to truth | Mean data loss |
|---|---|---|
| Spin (ours) | 0.1431 | 2.79e-05 |
| pgdm | 0.3041 | 0.0184 |
| daps | 0.3108 | 0.1391 |
| dps | 0.3498 | 0.1260 |
| ddnm | 0.3819 | 0.0870 |
| SSIM FFHQ | ||||||||
| Task | Spin | DAPS | Red-Diff | PNP-DM | DPS | DDNM | PGDM | Diffpir |
| Gaussian Deblur | 0.83 | 0.81 | 0.81 | 0.77 | 0.72 | 0.03 | 0.14 | – |
| Motion Deblur | 0.82 | 0.79 | 0.74 | 0.75 | 0.65 | – | – | – |
| SR ( 4) | 0.79 | 0.80 | 0.63 | 0.75 | 0.67 | 0.69 | 0.56 | – |
| SR ( 16) | 0.63 | 0.55 | 0.59 | 0.50 | 0.50 | 0.62 | 0.42 | – |
| Box Inpainting | 0.76 | 0.80 | 0.70 | 0.55 | 0.72 | 0.73 | 0.70 | 0.82 |
| PSNR FFHQ | ||||||||
| Task | Spin | DAPS | Red-Diff | PNP-DM | DPS | DDNM | PGDM | Diffpir |
| Gaussian Deblur | 29.0 | 28.2 | 28.7 | 25.2 | 24.7 | 7.8 | 13.2 | – |
| Motion Deblur | 28.1 | 27.0 | 27.8 | 24.7 | 22.2 | – | – | – |
| SR ( 4) | 28.2 | 27.5 | 26.1 | 24.7 | 22.8 | 26.9 | 24.6 | – |
| SR ( 16) | 21.6 | 17.8 | 21.2 | 16.3 | 17.9 | 21.6 | 18.4 | – |
| Box Inpainting | 21.7 | 22.4 | 21.6 | 12.5 | 20.9 | 22.5 | 21.1 | 22.6 |
| SSIM ImageNet | ||||||||
| Task | Spin | DAPS | Red-Diff | PNP-DM | DPS | DDNM | PGDM | Diffpir |
| Gaussian Deblur | 0.77 | 0.68 | 0.69 | 0.63 | 0.57 | 0.60 | 0.07 | – |
| Motion Deblur | 0.72 | 0.66 | 0.65 | 0.60 | 0.49 | – | – | – |
| SR ( 4) | 0.71 | 0.67 | 0.59 | 0.60 | 0.51 | 0.74 | 0.27 | – |
| SR ( 16) | 0.58 | 0.43 | 0.45 | 0.40 | 0.34 | 0.50 | 0.21 | – |
| Box Inpainting | 0.72 | 0.73 | 0.36 | 0.51 | 0.58 | 0.76 | 0.61 | 0.78 |
| PSNR ImageNet | ||||||||
| Task | Spin | DAPS | Red-Diff | PNP-DM | DPS | DDNM | PGDM | Diffpir |
| Gaussian Deblur | 27.3 | 25.4 | 25.8 | 24.5 | 22.6 | 23.7 | 9.7 | – |
| Motion Deblur | 26.9 | 24.7 | 25.5 | 23.4 | 20.4 | – | – | – |
| SR ( 4) | 26.6 | 25.1 | 24.5 | 23.8 | 20.9 | 26.4 | 18.2 | – |
| SR ( 16) | 20.8 | 16.9 | 19.7 | 15.1 | 16.4 | 20.4 | 15.4 | – |
| Box Inpainting | 18.3 | 18.2 | 18.0 | 12.6 | 17.1 | 19.2 | 16.4 | 18.70 |
| Task group | |||
|---|---|---|---|
| Deblurring (Gaussian, Motion) | 10 | ||
| Super-resolution (SR4, SR16) | |||
| Inpainting (Box, Half) | 50 | ||
| JPEG / Phase Retrieval / HDR |
| Task group | ||||
|---|---|---|---|---|
| Deblurring (Gaussian, Motion) | 3000 | |||
| Super-resolution (SR4, SR16) | 300 | |||
| Inpainting (Box, Half) | 0.44 | 800 | ||
| JPEG / Phase Retrieval / HDR | 3.0 |
Diffusion-Based Inverse Problems
Diffusion-based inverse problems use generative priors to recover images and physical fields from incomplete, noisy, or indirect measurements.
See all 141 papers