Image Deblending with Diffusion Models
Organizations: Reichman University, Herzliya, Israel
Abstract
Image deblending is the task of decomposing a composite image back into its source images, which is an ill-posed problem, as a single blended image can be decomposed into multiple plausible source layers. While diffusion models are often used to solve inverse problems, they typically address single-output reconstruction problems. In this work, we formulate deblending as a joint inverse problem, employing diffusion posterior sampling to simultaneously reconstruct two source images, demonstrating that a single diffusion prior is expressive enough to disentangle two full images at once rather than just one. Drawing inspiration from human perception, where semantic understanding helps disentangle visual inconsistencies, we propose a training-free framework that leverages text prompts to constrain this solution space. We address the blind deblending task through a two-stage architecture: text-guided latent diffusion performs semantic separation using prompts as high-level cues, followed by a pixel-space refinement to recover fine details. Stochastic sampling produces multiple plausible decompositions that can be ranked without ground truth. Extensive experiments show that our training-free method achieves state-of-the-art results on unseen domains and extends to spatially-varying alpha maps, while remaining competitive with supervised baselines on standard scalar blending benchmarks.
Figures & tables
| In-Distribution (Seen Domains) | Generalization (Unseen Domains) | |||||||||||||
| Fruits | Animals | Vehicles | Flowers | |||||||||||
| Scenario | Method | TF | PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS |
| Known Scalar | DAD | ✗ | ||||||||||||
| GIP | ✗ | |||||||||||||
| BIDeN | ✗ | |||||||||||||
| LCD | ✗ | |||||||||||||
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
| Sample 1 | Sample 2 | |||||||
| Scenario | Method | CFG ( ) | PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS |
| Known Scalar | Baseline | 0.0 | ||||||
| Cross-neg. (Ours) | 1.0 | |||||||
| 2.0 | ||||||||
| 3.0 | ||||||||
| Standard CFG | 1.0 | |||||||
| Prompts | PSNR | SSIM | LPIPS |
| None ( ) | 14.98 | 0.650 | 0.310 |
| Auto (Qwen2-VL) | 15.63 | 0.658 | 0.296 |
| Manual | 16.31 | 0.700 | 0.249 |
| Domain | Scenario | Selector | PSNR | SSIM | LPIPS |
| Seen | Known scalar | CLIP | 20.7634 | 0.8074 | 0.1493 |
| PickScore | 21.3595 | 0.8134 | 0.1418 | ||
| Unknown scalar | CLIP | 19.1261 | 0.7440 | 0.2078 | |
| PickScore | 19.7825 | 0.7634 | 0.1902 | ||
| Spatially varying | CLIP | 17.5881 | 0.7189 | 0.2261 | |
| PickScore | 18.0521 | 0.7325 | 0.2118 |
| Scenario | PSNR | SSIM | LPIPS | |
| Known scalar | 1 | 21.08 | 0.769 | 0.236 |
| 3 | 21.57 | 0.802 | 0.198 | |
| 16 | 22.97 | 0.829 | 0.180 | |
| Unknown scalar | 1 | 19.90 | 0.735 | 0.232 |
| 3 | 20.51 | 0.762 | 0.234 | |
| 16 | 22.02 | 0.783 | 0.201 |
| Scenario | Selector | PSNR | SSIM | LPIPS |
| Comparison (300 pairs) | ||||
| Known Scalar | CLIP | 14.42 | 0.515 | 0.537 |
| PickScore | 14.58 | 0.521 | 0.535 | |
| Unknown Scalar | CLIP | 14.66 | 0.511 | 0.545 |
| PickScore | 14.69 | 0.516 | 0.552 | |
| Spatially- Varying | CLIP | 13.68 | 0.457 | 0.597 |
| 2 | 4 | 6 | 8 | 10 | |
| 4 | 14.520/.565/.357 | 14.923/.597/.335 | 15.131/.631/.312 | 15.426/.637/.312 | 15.529/.651/.290 |
| 6 | 14.816/.587/.325 | 15.252/.618/.301 | 15.515/.647/.279 | 15.765/.658/.267 | 15.794/.661/.266 |
| 8 | 15.560/.625/.295 | 16.000/.656/.270 | 16.282/.671/.259 | 16.737/.690/.241 | 16.868 / .695 /.239 |
| 10 | 15.326/.631/.280 | 15.705/.659/.259 | 15.964/.674/.250 | 16.259/.689/.236 | 16.445/ .695/.232 |
| Scenario | Stage | PSNR | SSIM | LPIPS |
| Known Scalar | 1 | 17.3872 | 0.5910 | 0.3175 |
| 2 | 21.3595 | 0.8134 | 0.1418 | |
| Unknown Scalar | 1 | 16.7536 | 0.5668 | 0.3450 |
| 2 | 19.7825 | 0.7634 | 0.1902 | |
| Spatially- Varying | 1 | 15.6474 | 0.5453 | 0.3634 |
| 2 | 18.0521 | 0.7325 | 0.2118 |
| GT matte | Estimator | PSNR | SSIM | LPIPS | MAE |
| Polynomial | Quadratic | 14.18 | 0.504 | 0.586 | 0.180 |
| Grid | 14.52 | 0.518 | 0.565 | 0.142 | |
| Gaussian | Quadratic | 13.91 | 0.500 | 0.599 | 0.265 |
| Grid | 14.11 | 0.507 | 0.603 | 0.237 | |
| Reflection-like | Quadratic | 13.52 | 0.482 | 0.625 | 0.242 |
| Grid | 13.66 | 0.496 | 0.616 | 0.216 |
Diffusion Models for Image Restoration
Diffusion models for image restoration use generative priors to denoise, deblur, upscale, and reconstruct images from incomplete or corrupted measurements.
See all 202 papers