IS-Diff: Improving Diffusion-Based Inpainting with Better Initial Seed

Yongzhe Lyu; Yu Wu; Yutian Lin; Bo Du

cs.CVSep 15, 2025

IS-Diff: Improving Diffusion-Based Inpainting with Better Initial Seed

Authors: Yongzhe Lyu, Yu Wu, Yutian Lin, Bo Du

Organizations: School of Computer Science, Wuhan University, Wuhan 430072, China

Abstract

Diffusion models have shown promising results in free-form inpainting. Recent studies based on refined diffusion samplers or novel architectural designs led to realistic results and high data consistency. However, random initialization seed (noise) adopted in vanilla diffusion process may introduce mismatched semantic information in masked regions, leading to biased inpainting results, e.g., low consistency and low coherence with the other unmasked area. To address this issue, we propose the Initial Seed refined Diffusion Model (IS-Diff), a completely training-free approach incorporating distributional harmonious seeds to produce harmonious results. Specifically, IS-Diff employs initial seeds sampled from unmasked areas to imitate the masked data distribution, thereby setting a promising direction for the diffusion procedure. Moreover, a dynamic selective refinement mechanism is proposed to detect severe unharmonious inpaintings in intermediate latent and adjust the strength of our initialization prior dynamically. We validate our method on both standard and large-mask inpainting tasks using the CelebA-HQ, ImageNet, and Places2 datasets, demonstrating its effectiveness across all metrics compared to state-of-the-art inpainting methods.

Figures & tables

Panel of Figure 1 — Fig. 1 : Existing issues with current diffusion-based models and our proposed training-free method to tackle these problems. We use grayscale distribution maps to demonstrate the distributional discrepancy between inpainting result and reference. (a) Contemporary free-form inpainting techniques (DDNM [ Wang_Yu_Zhang_2022 ] in this figure) often fail to create convincing semantic content and maintain cohesive textures because of distributional discrepancy between random initialization and the source image. (b) IS-Diff mitigates these challenges by utilizing information from the source image to close the distribution gap.

Fig. 2 : Qualitative examples of different inpainting methods from the ImageNet 1K [ Russakovsky_Deng_Su_Krause_Satheesh_Ma_Huang_Karpathy_Khosla_Bernstein_et_al._2015 ] dataset and the CelebA 1K [ Liu_Luo_Wang_Tang_2015 ] dataset. “DSR” denotes our dynamic selective refinement, “PP” denotes PowerPaint [ zhuang2023task ] , and “SI” denotes Stable Diffusion Inpainting [ Rombach_Blattmann_Lorenz_Esser_Ommer_2022 ] . Compared with various diffusion-based baselines, our plug-and-play method achieves more harmonious results on all types of masks.

Fig. 3 : Overview of our approach. (a) for the primary seed initialization, masked regions are initialized with unmasked data distribution. (b) we calculate distribution cross-entropy of the intermediate variables of diffusion to implement selective iterative generation, and adjust the initialization strength through feedback, resulting in efficient and powerful inpainting.

Fig. 4 : We analyze different initializations to explore how distribution differences inside and outside the mask affect inpainting results. “KL” denotes the KL divergence between the initialization distribution and the real distribution. The results indicate that initializations with similar distributions have a positive impact on inpainting. When initialized with pure noise, the generated contents tend to be hollow and lack structural consistency. As the KL divergence decreases, the inpainting results gradually become clearer, with the main subject better aligned and more harmonious. In particular, when directly sampling from the distribution of the unmasked regions, the inpainting results reach the highest quality.

Fig. 5 : Qualitative comparison with state-of-the-art methods on CelebA-HQ. “SI” denotes Stable Diffusion In painting, and “PP” denotes PowerPaint.

Fig. 6 : Qualitative comparison with state-of-the-art methods on ImageNet. “SI” denotes Stable Diffusion Inpainting, “PP” denotes PowerPaint.

Methods	Wide		Half		Expand		Time Cost (s/image)↓
Methods	LPIPS↓	FID↓	LPIPS↓	FID↓	LPIPS↓	FID↓	Time Cost (s/image)↓
SI [CVPR 22] [ Rombach_Blattmann_Lorenz_Esser_Ommer_2022 ]	0.273	70.44	0.385	66.66	0.763	139.76	5
SI + HP [ICLR 25] [ manukyan2023hd ]	0.249	64.82	0.463	92.17	0.781	170.18	15
SI + Ours	0.238	54.91	0.381	61.32	0.676	123.36	3
PP [ECCV 24] [ zhuang2023task ]	0.263	63.83	0.397	91.62	0.779	178.45	3
PP + Ours	0.254	53.85	0.373	68.69	0.707	160.62	3

TABLE I: Comparison with state-of-the-art methods on ImageNet 1K. The LPIPS and FID scores are reported. Lower is better. Bold fonts represent the best result. “HP” denotes HD Painter while “PP” denotes PowerPaint.

Methods	Wide		Half		Expand		Time Cost (s/image)↓
Methods	LPIPS↓	FID↓	LPIPS↓	FID↓	LPIPS↓	FID↓	Time Cost (s/image)↓
SI [CVPR 22] [ Rombach_Blattmann_Lorenz_Esser_Ommer_2022 ]	0.237	107.07	0.323	91.08	0.617	165.18	5
SI + HP [ICLR 25] [ manukyan2023hd ]	0.207	66.30	0.458	164.69	0.686	230.50	15
SI + Ours	0.191	63.22	0.303	52.88	0.609	132.64	3
PP [ECCV 24] [ zhuang2023task ]	0.223	101.90	0.267	53.32	0.691	276.22	3
PP + Ours	0.207	74.96	0.254	42.84	0.667	262.46	3

TABLE II: Comparison with state-of-the-art methods on CelebA-HQ 1K. The LPIPS and FID scores are reported. Lower is better. Bold fonts represent the best result. “HP” denotes HD Painter while “PP” denotes PowerPaint. As reported by Lugmayr et al. [ Lugmayr_Danelljan_Romero_Yu_Timofte_Van_Gool_2022 ] , LPIPS is not an suitable metric for large masks because diverse solutions different from ground truth are possible.

Methods	CelebA-HQ			ImageNet
Methods	Wide	Half	Expand	Wide	Half	Expand
DDNM [ Wang_Yu_Zhang_2022 ]	34.4%	16.9%	24.9%	21.3%	25%	22.6%
RePaint [ Lugmayr_Danelljan_Romero_Yu_Timofte_Van_Gool_2022 ]	16.8%	23.5%	36.3%	21.8%	19.9%	25.1%
Ours	48.8%	59.6%	38.8%	56.9%	55.1 %	52.3 %

TABLE III : User Study Results. We collect 600 responses from 30 participants to compare the visual effects of IS-Diff against state-of-the-art diffusion-based methods on CelebA and ImageNet. The table shows that our method receives the majority of votes. Bold fonts represent the best result.

Fig. 7 : Results on challenging scenarios including complicated textures, non-object-centric, and object-truncated scenes. Our method showcasing better inpainting quality and more reasonable content.

Fig. 8 : Qualitative comparison with state-of-the-art methods on Places2. “SI” denotes Stable Diffusion Inpainting.

Methods	Wide		Half		Expand		Time Cost (s/image)↓
Methods	LPIPS↓	FID↓	LPIPS↓	FID↓	LPIPS↓	FID↓	Time Cost (s/image)↓
SI’ [CVPR 22] [ Rombach_Blattmann_Lorenz_Esser_Ommer_2022 ]	0.212	8.23	0.363	13.81	0.689	45.13	5
SI + HP’ [ICLR 25] [ manukyan2023hd ]	0.234	12.43	0.469	74.57	0.779	105.46	15
SI + PP’ [ECCV 24] [ zhuang2023task ]	0.267	25.09	0.423	64.14	0.711	87.64	3
SI + Ours	0.181	6.82	0.350	12.01	0.678	31.31	3

TABLE IV: Comparison with state-of-the-art methods on high resolution dataset. We use

512\times 512

images from Places2 dataset. The LPIPS and FID scores are reported. Lower is better. Bold fonts represent the best result. “HP” denotes HD Painter while “PP” denotes PowerPaint.

Method	CLIP Score $\uparrow$	SSIM $\uparrow$
Stable Diffusion Inpainting	30.1	0.708
Stable Diffusion Inpainting + Ours	30.7	0.726

TABLE V : Quantitative comparison with Stable Diffusion Inpainting on EditBench.

Fig. 9 : Qualitative visualization results of IS-Diff applied to the flow-matching-based model FLUX-Inpaint. Compared with the baseline model, IS-Diff improves the contextual compatibility of the generated regions and reduces visually inconsistent artifacts.

Methods	Wide		Half		Expand
Methods	LPIPS↓	FID↓	LPIPS↓	FID↓	LPIPS↓	FID↓
Baseline (w/o resample)	0.160	31.68	0.351	55.58	0.780	127.69
Ours (w/o DSR)	0.155	30.02	0.340	50.47	0.777	122.63
Ours	0.128	22.01	0.328	39.94	0.721	121.17

TABLE VI : Quantitative ablation study on the ImageNet dataset. Here, “DSR” denotes the process of dynamic selective refinement. Bold fonts represent the best result. The quantitative results show that our method can achieve better performance even without DSR.

Fig. 10 : Diverse inpainting results on ImageNet with “half” masks. The right half of image is masked.

Methods	L2 Distance of two runs $\uparrow$
Baseline (DDNM)	0.2273
Ours	0.3073

TABLE VII: Quantitative comparison of L2 distance between results from multiple runs with the baseline model. Higher is better.

Fig. 11 : Comparison of different values of

\Delta t

Methods	Expand		Inference time
Methods	LPIPS↓	FID↓	(s/image)↓
RePaint	0.564	99.21	21
RePaint ( $T$ =250)	0.564	86.22	50
RePaint ( $r$ =15)	0.563	102.51	27
Ours ( $n$ =3)	0.547	60.86	4
Ours ( $n$ =4)	0.539	57.82	5

TABLE VIII: Experiment with more time conditions under the “Expand” setting on CelebA-HQ. Quantitative metrics and average time consumption are reported. Here,

n

denotes the iteration number of iterative refinement,

T

denotes the sampling steps and

r

represents the repeat time of resampling in Repaint [ Lugmayr_Danelljan_Romero_Yu_Timofte_Van_Gool_2022 ] . If not specified, we set

n

to 1,

T

to 100, and

r

to 10 for RePaint.

$K$	LPIPS $\downarrow$	FID $\downarrow$
1	0.352	63.41
3	0.340	58.79
5	0.328	39.94
7	0.329	39.47

TABLE IX: Ablation on number of GMM components

K

on ImageNet (Half masks).

$t_{c}$	0.4T	0.6T	0.8T
LPIPS $\downarrow$	0.342	0.328	0.343
FID $\downarrow$	43.89	39.94	45.43

TABLE X: Sensitivity analysis of

t_{c}

on ImageNet (Half masks).

$\epsilon$	LPIPS $\downarrow$	FID $\downarrow$
1.0	0.327	39.73
1.5	0.327	39.79
2.0	0.328	39.85
2.5	0.328	39.94
3.0	0.334	41.04

TABLE XI: Sensitivity analysis of DCE threshold

\epsilon

on ImageNet (Half masks).

$Configuration$	LPIPS $\downarrow$	FID $\downarrow$
LPIPS	0.342	54.70
DCE	0.328	39.94

TABLE XII: Ablation on alternatives of DCE.

Fig. 12 : Failure Cases. Our method and other baselines usually fail to generate reasonable content when facing marginal occlusion of the foreground objects.

Image Inpainting

Image inpainting reconstructs missing or corrupted image regions using surrounding context, learned visual priors, and optional structural or reference guidance.

See all 92 papers

Image AID via continuous-time reinforcement learning
May 13, 2026Yilie Huang, Xun Yu ZhouDiffusion Model Guidance Masked Diffusion Models
Department of Industrial Engineering and Operations Research, Columbia University, New York, NY 10027, USA. · Department of Industrial Engineering and Operations Research & Data Science Institute, Columbia University, New York, NY 10027, USA.
InverFill: One-Step Inversion for Enhanced Few-Step Diffusion Inpainting
Mar 24, 2026Duc Vu, Kien Nguyen, Ngan Nguyen +5Image Inpainting T2I Diffusion Models
Qualcomm AI Research · Posts & Telecommunications Inst. of Tech., Vietnam
When Diffusion Models Forget Who You Are: Identity Preservation in Face Inpainting under Large Occlusions
Aug 5, 2026Feng Ding, Shuhuai Xie, Yue Zhou +3Diffusion Model Guidance Image Inpainting
Nanchang University · Shenzhen University · Huizhong University +1

May 13, 2026J/K move · Enter open · S save

Image AID via continuous-time reinforcement learning

Yilie Huang, Xun Yu Zhou

Department of Industrial Engineering and Operations Research, Columbia University, New York, NY 10027, USA. · Department of Industrial Engineering and Operations Research & Data Science Institute, Columbia University, New York, NY 10027, USA.

Diffusion Model Guidance Masked Diffusion Models Continuous-Time RL Image Inpainting RL for Diffusion Models

Open paper AskPDF arXiv

We study image inpainting with generative diffusion models. Existing methods typically either train dedicated task-specific models, or adapt a pretrained diffusion model separately for each masked image at deployment. We introduce a middle-ground model, termed Amortized Inpainting with Diffusion (AID), which keeps a pretrained diffusion backbone fixed, trains a small reusable guidance module offline, and then reuses it across masked images without per-instance optimization. We formulate it as a deterministic guidance problem with a supervised terminal objective. To make this problem learnable in high dimensions, we derive an auxiliary Gaussian formulation and prove that solving this randomized problem recovers the optimal deterministic guidance field. This bridge yields a principled continuous-time actor--critic algorithm for learning the guidance module in a fully data-driven manner. Empirically, on AFHQv2 and FFHQ under the pixel EDM pipeline and on ImageNet under the latent EDM2 pipeline, AID consistently improves the quality--speed trade-off over strong fixed-backbone and amortized inpainting baselines across multiple mask types, while adding less than one percent trainable overhead.

Abstract

Image Inpainting

Explore similar work

Image AID via continuous-time reinforcement learning

InverFill: One-Step Inversion for Enhanced Few-Step Diffusion Inpainting

When Diffusion Models Forget Who You Are: Identity Preservation in Face Inpainting under Large Occlusions