Diffusion models have shown promising results in free-form inpainting. Recent studies based on refined diffusion samplers or novel architectural designs led to realistic results and high data consistency. However, random initialization seed (noise) adopted in vanilla diffusion process may introduce mismatched semantic information in masked regions, leading to biased inpainting results, e.g., low consistency and low coherence with the other unmasked area. To address this issue, we propose the Initial Seed refined Diffusion Model (IS-Diff), a completely training-free approach incorporating distributional harmonious seeds to produce harmonious results. Specifically, IS-Diff employs initial seeds sampled from unmasked areas to imitate the masked data distribution, thereby setting a promising direction for the diffusion procedure. Moreover, a dynamic selective refinement mechanism is proposed to detect severe unharmonious inpaintings in intermediate latent and adjust the strength of our initialization prior dynamically. We validate our method on both standard and large-mask inpainting tasks using the CelebA-HQ, ImageNet, and Places2 datasets, demonstrating its effectiveness across all metrics compared to state-of-the-art inpainting methods.
Figures & tables
Fig. 1 : Existing issues with current diffusion-based models and our proposed training-free method to tackle these problems. We use grayscale distribution maps to demonstrate the distributional discrepancy between inpainting result and reference. (a) Contemporary free-form inpainting techniques (DDNM [ Wang_Yu_Zhang_2022 ] in this figure) often fail to create convincing semantic content and maintain cohesive textures because of distributional discrepancy between random initialization and the source image. (b) IS-Diff mitigates these challenges by utilizing information from the source image to close the distribution gap.
Fig. 2 : Qualitative examples of different inpainting methods from the ImageNet 1K [ Russakovsky_Deng_Su_Krause_Satheesh_Ma_Huang_Karpathy_Khosla_Bernstein_et_al._2015 ] dataset and the CelebA 1K [ Liu_Luo_Wang_Tang_2015 ] dataset. “DSR” denotes our dynamic selective refinement, “PP” denotes PowerPaint [ zhuang2023task ] , and “SI” denotes Stable Diffusion Inpainting [ Rombach_Blattmann_Lorenz_Esser_Ommer_2022 ] . Compared with various diffusion-based baselines, our plug-and-play method achieves more harmonious results on all types of masks.
Fig. 3 : Overview of our approach. (a) for the primary seed initialization, masked regions are initialized with unmasked data distribution. (b) we calculate distribution cross-entropy of the intermediate variables of diffusion to implement selective iterative generation, and adjust the initialization strength through feedback, resulting in efficient and powerful inpainting.
Fig. 4 : We analyze different initializations to explore how distribution differences inside and outside the mask affect inpainting results. “KL” denotes the KL divergence between the initialization distribution and the real distribution. The results indicate that initializations with similar distributions have a positive impact on inpainting. When initialized with pure noise, the generated contents tend to be hollow and lack structural consistency. As the KL divergence decreases, the inpainting results gradually become clearer, with the main subject better aligned and more harmonious. In particular, when directly sampling from the distribution of the unmasked regions, the inpainting results reach the highest quality.
Fig. 5 : Qualitative comparison with state-of-the-art methods on CelebA-HQ. “SI” denotes Stable Diffusion In painting, and “PP” denotes PowerPaint.
Fig. 6 : Qualitative comparison with state-of-the-art methods on ImageNet. “SI” denotes Stable Diffusion Inpainting, “PP” denotes PowerPaint.
Methods
Wide
Half
Expand
Time Cost (s/image)↓
LPIPS↓
FID↓
LPIPS↓
FID↓
LPIPS↓
FID↓
SI [CVPR 22] [ Rombach_Blattmann_Lorenz_Esser_Ommer_2022 ]
0.273
70.44
0.385
66.66
0.763
139.76
5
SI + HP [ICLR 25] [ manukyan2023hd ]
0.249
64.82
0.463
92.17
0.781
170.18
15
SI + Ours
0.238
54.91
0.381
61.32
0.676
123.36
3
PP [ECCV 24] [ zhuang2023task ]
0.263
63.83
0.397
91.62
0.779
178.45
3
PP + Ours
0.254
53.85
0.373
68.69
0.707
160.62
3
TABLE I: Comparison with state-of-the-art methods on ImageNet 1K. The LPIPS and FID scores are reported. Lower is better. Bold fonts represent the best result. “HP” denotes HD Painter while “PP” denotes PowerPaint.
Methods
Wide
Half
Expand
Time Cost (s/image)↓
LPIPS↓
FID↓
LPIPS↓
FID↓
LPIPS↓
FID↓
SI [CVPR 22] [ Rombach_Blattmann_Lorenz_Esser_Ommer_2022 ]
0.237
107.07
0.323
91.08
0.617
165.18
5
SI + HP [ICLR 25] [ manukyan2023hd ]
0.207
66.30
0.458
164.69
0.686
230.50
15
SI + Ours
0.191
63.22
0.303
52.88
0.609
132.64
3
PP [ECCV 24] [ zhuang2023task ]
0.223
101.90
0.267
53.32
0.691
276.22
3
PP + Ours
0.207
74.96
0.254
42.84
0.667
262.46
3
TABLE II: Comparison with state-of-the-art methods on CelebA-HQ 1K. The LPIPS and FID scores are reported. Lower is better. Bold fonts represent the best result. “HP” denotes HD Painter while “PP” denotes PowerPaint. As reported by Lugmayr et al. [ Lugmayr_Danelljan_Romero_Yu_Timofte_Van_Gool_2022 ] , LPIPS is not an suitable metric for large masks because diverse solutions different from ground truth are possible.
TABLE III : User Study Results. We collect 600 responses from 30 participants to compare the visual effects of IS-Diff against state-of-the-art diffusion-based methods on CelebA and ImageNet. The table shows that our method receives the majority of votes. Bold fonts represent the best result.
Fig. 7 : Results on challenging scenarios including complicated textures, non-object-centric, and object-truncated scenes. Our method showcasing better inpainting quality and more reasonable content.
Fig. 8 : Qualitative comparison with state-of-the-art methods on Places2. “SI” denotes Stable Diffusion Inpainting.
TABLE IV: Comparison with state-of-the-art methods on high resolution dataset. We use 512×512 images from Places2 dataset. The LPIPS and FID scores are reported. Lower is better. Bold fonts represent the best result. “HP” denotes HD Painter while “PP” denotes PowerPaint.
Method
CLIP Score ↑
SSIM ↑
Stable Diffusion Inpainting
30.1
0.708
Stable Diffusion Inpainting + Ours
30.7
0.726
TABLE V : Quantitative comparison with Stable Diffusion Inpainting on EditBench.
Fig. 9 : Qualitative visualization results of IS-Diff applied to the flow-matching-based model FLUX-Inpaint. Compared with the baseline model, IS-Diff improves the contextual compatibility of the generated regions and reduces visually inconsistent artifacts.
Methods
Wide
Half
Expand
LPIPS↓
FID↓
LPIPS↓
FID↓
LPIPS↓
FID↓
Baseline (w/o resample)
0.160
31.68
0.351
55.58
0.780
127.69
Ours (w/o DSR)
0.155
30.02
0.340
50.47
0.777
122.63
Ours
0.128
22.01
0.328
39.94
0.721
121.17
TABLE VI : Quantitative ablation study on the ImageNet dataset. Here, “DSR” denotes the process of dynamic selective refinement. Bold fonts represent the best result. The quantitative results show that our method can achieve better performance even without DSR.
Fig. 10 : Diverse inpainting results on ImageNet with “half” masks. The right half of image is masked.
Methods
L2 Distance of two runs ↑
Baseline (DDNM)
0.2273
Ours
0.3073
TABLE VII: Quantitative comparison of L2 distance between results from multiple runs with the baseline model. Higher is better.
Fig. 11 : Comparison of different values of Δt .
Methods
Expand
Inference time
LPIPS↓
FID↓
(s/image)↓
RePaint
0.564
99.21
21
RePaint ( T =250)
0.564
86.22
50
RePaint ( r =15)
0.563
102.51
27
Ours ( n =3)
0.547
60.86
4
Ours ( n =4)
0.539
57.82
5
TABLE VIII: Experiment with more time conditions under the “Expand” setting on CelebA-HQ. Quantitative metrics and average time consumption are reported. Here, n denotes the iteration number of iterative refinement, T denotes the sampling steps and r represents the repeat time of resampling in Repaint [ Lugmayr_Danelljan_Romero_Yu_Timofte_Van_Gool_2022 ] . If not specified, we set n to 1, T to 100, and r to 10 for RePaint.
K
LPIPS ↓
FID ↓
1
0.352
63.41
3
0.340
58.79
5
0.328
39.94
7
0.329
39.47
TABLE IX: Ablation on number of GMM components K on ImageNet (Half masks).
tc
0.4T
0.6T
0.8T
LPIPS ↓
0.342
0.328
0.343
FID ↓
43.89
39.94
45.43
TABLE X: Sensitivity analysis of tc on ImageNet (Half masks).
ϵ
LPIPS ↓
FID ↓
1.0
0.327
39.73
1.5
0.327
39.79
2.0
0.328
39.85
2.5
0.328
39.94
3.0
0.334
41.04
TABLE XI: Sensitivity analysis of DCE threshold ϵ on ImageNet (Half masks).
Configuration
LPIPS ↓
FID ↓
LPIPS
0.342
54.70
DCE
0.328
39.94
TABLE XII: Ablation on alternatives of DCE.
Fig. 12 : Failure Cases. Our method and other baselines usually fail to generate reasonable content when facing marginal occlusion of the foreground objects.
Image inpainting reconstructs missing or corrupted image regions using surrounding context, learned visual priors, and optional structural or reference guidance.
We study image inpainting with generative diffusion models. Existing methods typically either train dedicated task-specific models, or adapt a pretrained diffusion model separately for each masked image at deployment. We introduce a middle-ground model, termed Amortized Inpainting with Diffusion (AID), which keeps a pretrained diffusion backbone fixed, trains a small reusable guidance module offline, and then reuses it across masked images without per-instance optimization. We formulate it as a deterministic guidance problem with a supervised terminal objective. To make this problem learnable in high dimensions, we derive an auxiliary Gaussian formulation and prove that solving this randomized problem recovers the optimal deterministic guidance field. This bridge yields a principled continuous-time actor--critic algorithm for learning the guidance module in a fully data-driven manner. Empirically, on AFHQv2 and FFHQ under the pixel EDM pipeline and on ImageNet under the latent EDM2 pipeline, AID consistently improves the quality--speed trade-off over strong fixed-backbone and amortized inpainting baselines across multiple mask types, while adding less than one percent trainable overhead.
Yilie Huang, Xun Yu Zhou
Department of Industrial Engineering and Operations Research, Columbia University, New York, NY 10027, USA. · Department of Industrial Engineering and Operations Research & Data Science Institute, Columbia University, New York, NY 10027, USA.
Recent diffusion-based models achieve photorealism in image inpainting but require many sampling steps, limiting practical use. Few-step text-to-image models offer faster generation, but naively applying them to inpainting yields poor harmonization and artifacts between the background and inpainted region. We trace this cause to random Gaussian noise initialization, which under low function evaluations causes semantic misalignment and reduced fidelity. To overcome this, we propose InverFill, a one-step inversion method tailored for inpainting that injects semantic information from the input masked image into the initial noise, enabling high-fidelity few-step inpainting. Instead of training inpainting models, InverFill leverages few-step text-to-image models in a blended sampling pipeline with semantically aligned noise as input, significantly improving vanilla blended sampling and even matching specialized inpainting models at low NFEs. Moreover, InverFill does not require real-image supervision and only adds minimal inference overhead. Extensive experiments show that InverFill consistently boosts baseline few-step models, improving image quality and text coherence without costly retraining or heavy iterative optimization.
Duc Vu, Kien Nguyen, Ngan Nguyen +5
Qualcomm AI Research · Posts & Telecommunications Inst. of Tech., Vietnam
Face inpainting with diffusion models has recently achieved impressive visual quality, yet preserving identity fidelity under significant occlusion and conflicting text guidance remains a major challenge. To address this issue, we present Reference Semantic Inpainting for Face (ReSem-Face), a cascaded diffusion framework that introduces an explicit identity-conditioned semantic prior for multi-reference face inpainting. Our approach distills representative identity features from multiple references to reconstruct missing semantic regions, which then guide the diffusion process through a multi-stream conditioning architecture. This design provides strong semantic constraints when pixels are absent and stabilizes identity reconstruction while remaining compatible with prompt-driven edits. Experiments on CelebAHQ-IDI-5 and VGGFace2 demonstrate that ReSem-Face yields more reliable identity-preserving completion under severe semantic masks and improves text-controlled editing quality compared with representative baselines.
Feng Ding, Shuhuai Xie, Yue Zhou +3
Nanchang University · Shenzhen University · Huizhong University +1