InverFill: One-Step Inversion for Enhanced Few-Step Diffusion Inpainting

Duc Vu; Kien Nguyen; Ngan Nguyen; Trong-Tung Nguyen; Phong Nguyen; Khoi Nguyen; Cuong Pham; Anh Tran

cs.CVMar 24, 2026

InverFill: One-Step Inversion for Enhanced Few-Step Diffusion Inpainting

Authors: Duc Vu, Kien Nguyen, Ngan Nguyen, Trong-Tung Nguyen, Phong Nguyen, Khoi Nguyen, Cuong Pham, Anh Tran

Organizations: Qualcomm AI Research · Posts & Telecommunications Inst. of Tech., Vietnam

Abstract

Recent diffusion-based models achieve photorealism in image inpainting but require many sampling steps, limiting practical use. Few-step text-to-image models offer faster generation, but naively applying them to inpainting yields poor harmonization and artifacts between the background and inpainted region. We trace this cause to random Gaussian noise initialization, which under low function evaluations causes semantic misalignment and reduced fidelity. To overcome this, we propose InverFill, a one-step inversion method tailored for inpainting that injects semantic information from the input masked image into the initial noise, enabling high-fidelity few-step inpainting. Instead of training inpainting models, InverFill leverages few-step text-to-image models in a blended sampling pipeline with semantically aligned noise as input, significantly improving vanilla blended sampling and even matching specialized inpainting models at low NFEs. Moreover, InverFill does not require real-image supervision and only adds minimal inference overhead. Extensive experiments show that InverFill consistently boosts baseline few-step models, improving image quality and text coherence without costly retraining or heavy iterative optimization.

Figures & tables

Figure 1: InverFill enhances few-step inpainting by generating semantically aligned inverted noise latents, while adding as little as 0.06 seconds of overhead on a single NVIDIA A100 40GB GPU.

Figure 2: Inversion Network Training: We train an inversion network, $\mathbf{F}_{\theta}$ , to invert a masked image to a noise latent ${\hat{z}_{T}}$ such that, after blending with random Gaussian noise to form ${\hat{z}}_{T}^{\text{blend}}$ , the latent enables high-fidelity, well-harmonized reconstruction of the original image.

Figure 3: Inpainting Pipeline: The inversion network extracts the latent ${\hat{z}_{T}}$ from a masked image, which is blended with random noise to form $\hat{z}_{T}^{\text{blend}}$ and then fed into a few-step inpainting pipeline to generate the final image. (Zoom in for details)

Figure 4: Failure of BLD in few-step models (SDXL-Turbo, 4 steps) is illustrated in Column 3 and corrected by our method in Column 4 . (Zoom in for details)

Figure 5: Effects of the proposed Re-Blending operation during training, without $\bm{\mathcal{L}_{\text{reg}}}$ . (Zoom in for details)

Figure 6: Ablation Study on $\mathcal{L}_{\text{reg}}$ . Without the regularization loss, the model fails to preserve the background during reconstruction and produces blurred, low-detail outputs. With the loss, the background is well preserved and fine image details are restored.

Type	Method	NFEs	BrushBench				MagicBrush				Runtime $\downarrow$
Type	Method	NFEs	IR $\uparrow$	HPSv2.1 $\uparrow$	AS $\uparrow$	CLIP $\uparrow$	IR $\uparrow$	HPSv2.1 $\uparrow$	AS $\uparrow$	CLIP $\uparrow$	(seconds)
	SANA-Sprint 0.6B	2	10.73	26.27	6.03	27.23	2.65	23.57	5.35	25.70	0.37
	SANA-Sprint 0.6B + InverFill	2	11.76	26.74	6.13	27.30	3.07	23.78	5.42	25.77	0.43 (+0.06)
	SANA-Sprint 0.6B	4	10.85	26.20	6.07	27.14	2.52	23.58	5.35	25.50	0.45
	SANA-Sprint 0.6B + InverFill	4	11.58	26.79	6.17	27.17	2.93	23.82	5.43	25.63	0.51 (+0.06)
	SDXL Turbo	4	11.60	27.26	6.06	27.21	3.42	24.09	5.45	25.76	0.66

Table 1: Quantitative comparison of InverFill against few-step and multi-step diffusion inpainting baselines on BrushBench and MagicBrush. NFEs denotes the number of function evaluations.

\uparrow

indicates that higher is better,

\downarrow

indicates that lower is better. IR scores are scaled by

10

, and HPSv2.1 scores are scaled by

10^{2}

Figure 7: Our method achieves qualitative results comparable to multi-step SDXL-Inpainting and is on par with BrushNet (4 steps), as shown in Columns 7 and 8 . Notably, this performance is obtained using only text prompts during training, whereas competing methods rely on full text–image–mask supervision. Moreover, integrating our approach with BrushNet further enhances semantic coherence.

Method	IR $\uparrow$	HPSv2.1 $\uparrow$	AS $\uparrow$	CLIP $\uparrow$
w/o $\mathcal{L}_{reg}$	11.22	25.49	6.06	27.26
w/ $\mathcal{L}_{reg}$	11.51	26.03	6.10	27.28

Table 2: Effects of

\bm{\mathcal{L}_{\text{reg}}}

on SANA-Sprint 0.6B [ 3 ] with 2 NFEs on BrushBench [ 15 ] . All models were evaluated at 5000 iterations.

Method	NFEs	IR ×10 $\uparrow$	HPS ${}_{\times 10^{2}}$ $\uparrow$	AS $\uparrow$	CLIP $\uparrow$
SANA-Sprint 0.6B	2	7.53	25.81	6.09	28.10
SANA-Sprint 0.6B + InverFill	2	8.55	26.49	6.19	28.26
SANA-Sprint 0.6B	4	7.32	25.81	6.11	28.15
SANA-Sprint 0.6B + InverFill	4	8.54	26.50	6.21	28.29
SDXL Turbo	4	8.09	26.85	6.12	28.25
SDXL Turbo + InverFill	4	9.00	27.11	6.12	28.70

Table 3: Quantitative comparison of InverFill against few-step and multi-step diffusion inpainting baselines on BrushBench with enhanced prompts .

\uparrow

indicates that higher is better.

Method	$\lambda_{\text{noise}}$	$\lambda_{\text{image}}$	$\lambda_{\text{reg}}$	$\lambda_{\text{adv}}$	IR $\uparrow$	HPSv2.1 $\uparrow$	AS $\uparrow$	CLIP $\uparrow$
Sec. 4.3	2.0	1.0	0	0	11.20	25.50	6.02	27.23
	1.0	2.0	0	0	11.02	25.48	6.04	27.18
	1.0	1.0	0	0	11.22	25.49	6.06	27.26
Sec. 4.5	1.0	1.0	0.25	0	11.23	25.36	6.07	27.26
	1.0	1.0	0.5	0	11.51	26.03	6.10	27.28
	1.0	1.0	1.0	0	11.47	25.39	6.08	27.27

Table 4: Ablation study on key hyperparameters for each component. The best setting from each block is propagated to the next.

Method	IR $\uparrow$	HPSv2.1 $\uparrow$	AS $\uparrow$	CLIP $\uparrow$
$\mathcal{L}_{\text{SDS}}$ [ 34 ]	11.29	25.31	6.08	27.25
$\mathcal{L}_{reg}$ (Ours)	11.51	26.03	6.10	27.28

Table 5: Quantitative comparison between the SDS loss

\mathcal{L}_{\text{SDS}}

and our Gaussian regularization loss

\mathcal{L}_{reg}

. For a fair evaluation, both methods are tested on SANA-Sprint 0.6B using 2 NFEs.

Figure 8: Qualitative comparison between our proposed regularization loss ( $\mathcal{L}_{{reg}}$ ) and the Score Distillation Sampling (SDS) loss ( $\mathcal{L}_{\text{SDS}}$ ) from SwiftEdit [ 34 ] . This visualization shows that our $\mathcal{L}_{reg}$ is crucial for preserving the original image content, while using $\mathcal{L}_{\text{SDS}}$ leads to significant information loss and poor reconstruction.

Method	IR $\uparrow$	HPSv2.1 $\uparrow$	AS $\uparrow$	CLIP $\uparrow$	Runtime (seconds) $\downarrow$
DDIMInv ( w/o Blending) (50 steps) + SDXL-Turbo	4.06	21.93	5.30	26.47	4.18
DDIMInv ( w/ Blending) (50 steps) + SDXL-Turbo	12.07	27.19	6.04	27.39	4.32
InverFill (Ours) + SDXL-Turbo	12.34	27.42	6.08	27.74	0.74

Table 6: Quantitative comparison of one-step InverFill versus the 50-step DDIM inversion baseline on BrushBench. For a fair comparison, we run SDXL-Turbo with 4 NFEs.

InverFill
Method	IR ×10 $\uparrow$	HPS ${}_{\times 10^{2}}$ $\uparrow$	AS $\uparrow$	CLIP $\uparrow$
Baseline [ 34 ]	7.93	24.79	5.96	26.40
+ Re-Blending ( Sec. 4.4 )	11.22	25.49	6.06	27.26
+ Gaussian Reg. ( Sec. 4.5 )	11.51	26.03	6.10	27.28
+ LADD ( Sec. 4.6 )	11.76	26.74	6.13	27.30

Table 7: Ablation of components on SANA-Sprint 0.6B (2 NFEs).

InverFill
Method	IR ×10 $\uparrow$	HPS ${}_{\times 10^{2}}$ $\uparrow$	AS $\uparrow$	CLIP $\uparrow$
Baseline [ 34 ]	10.60	25.44	6.03	26.63
+ Re-Blending ( Sec. 4.4 )	11.29	26.16	6.03	27.23
+ Gaussian Reg. ( Sec. 4.5 )	12.10	27.12	6.06	27.64
+ LADD ( Sec. 4.6 )	12.34	27.42	6.08	27.74

Table 8: Ablation of components on SDXL-Turbo (4 NFEs).

Figure 9: Visualization of DDIM Inversion results. The masked regions are not encoded, producing smooth, null-like areas in the inverted noise and causing loss of content.

Figure 10: Qualitative comparison between InverFill and DDIM Inversion. InverFill achieves substantially better scene harmonization and semantic consistency.

Method	NFEs	FFHQ / DIV2K / BrushBench
		FID $\downarrow$	IR ×10 $\uparrow$	HPS ${}_{\times 10^{2}}$ $\uparrow$	AS $\uparrow$	CLIP $\uparrow$	LPIPS $\downarrow$	SSIM $\uparrow$
SANA-Sprint 0.6B	2	27.12	2.81 / 5.22	22.42 / 26.78	5.12 / 5.77	23.55 / 28.26	0.184 / 0.193 / 0.144	0.704 / 0.572 / 0.769
SANA-Sprint 0.6B + InverFill	2	26.53	5.27 / 5.87	23.26 / 27.17	5.31 / 5.89	23.65 / 28.43	0.172 / 0.182 / 0.138	0.719 / 0.575 / 0.771
SANA-Sprint 0.6B	4	27.32	2.66 / 5.25	22.50 / 26.83	5.17 / 5.79	23.84 / 28.31	0.184 / 0.192 / 0.140	0.706 / 0.573 / 0.774
SANA-Sprint 0.6B + InverFill	4	26.42	5.27 / 5.83	23.32 / 27.15	5.37 / 5.92	23.88 / 28.38	0.169 / 0.181 / 0.134	0.708 / 0.574 / 0.774
SDXL Turbo	4	26.32	7.37 / 4.71	25.73 / 26.81	5.67 / 5.92	25.24 / 28.21	0.269 / 0.292 / 0.139	0.626 / 0.454 / 0.813

Table 9: Quantitative results on FFHQ, DIV2K, and BrushBench. Red , Blue , and Black denote scores on FFHQ , DIV2K , and BrushBench , respectively.

Figure 11: Quantitative analysis of inversion effects. We report LPIPS in background regions at intermediate timesteps and JSD with respect to the Gaussian prior. Lower values indicate better alignment. Red bars denote results without InverFill, while Green bars denote results with InverFill.

Figure 12: Representative failure cases of our method. While InverFill improves overall coherence, it may produce color inconsistencies between the inpainted regions and the background.

Figure 13: More qualitative comparison on BrushBench (Zoom in for best view)

Figure 14: More qualitative comparison on BrushBench (Zoom in for best view)

Image Inpainting

Image inpainting reconstructs missing or corrupted image regions using surrounding context, learned visual priors, and optional structural or reference guidance.

See all 92 papers

Training-free image inversion for one-step diffusion models
May 31, 2026Tao Wu, Senmao Li, Yaxing Wang +3Diffusion Model Inversion Text-Guided Image Editing
Computer Vision Center, Universitat Autonoma de Barcelona, Barcelona, 08193, Spain · Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), Abu Dhabi, United Arab Emirates · Jilin University, 130012, Changchun, Jilin, China +3
Training-Free Inpainting Across Domains with a Frozen Text-to-Image Diffusion Model
Sep 1, 2026Zhenhuan Wang, Fengyi YuanDiffusion Model Guidance Cross-Domain Generalization
School of Data Science, The Chinese University of Hong Kong, Shenzhen · School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen
SONIC: Spectral Optimization of Noise for Inpainting with Consistency
Nov 25, 2025Seungyeon Baek, Erqun Dong, Shadan Namazifard +2Diffusion Model Inversion Image Inpainting
University of British Columbia, Canada · Google DeepMind, USA

May 31, 2026J/K move · Enter open · S save

Training-free image inversion for one-step diffusion models

Tao Wu, Senmao Li, Yaxing Wang, Shiqi Yang+2

Computer Vision Center, Universitat Autonoma de Barcelona, Barcelona, 08193, Spain · Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), Abu Dhabi, United Arab Emirates · Jilin University, 130012, Changchun, Jilin, China+3

Diffusion Model Inversion Text-Guided Image Editing One-Step Generative Modeling Diffusion-Based Image Editing

Open paper AskPDF arXiv

In this work, we introduce a novel training-free inversion (TFinv) framework for one-step diffusion models,addressing key challenges in real image inversion and editing. We first identify two critical factors hamperingreal-image inversion and editing: (1) Initial Latent Editability, which is related to the distance between theinitial noise and the ideal Gaussian distribution, and (2) Caption Gap, which means the alignment betweentext captions and image representations. Both factors influence inversion efficiency and the editability ofone-step diffusion models. Then, we propose two novel techniques: iterative noise alignment (iterNA), whichminimizes the distribution gap to align with the normal Gaussian distribution, and suffix learning (suffL),which enhances text-to-image caption alignment by introducing learned suffix prompt tokens. These techniquesenable precise inversion of input images into their initial noise representations and facilitate image editing.Furthermore, we propose a mask-based editing technique for localized edits while preserving backgroundintegrity. Comprehensive experiments on the PIE-Bench dataset validate that our method TFinv not onlyachieves state-of-the-art performance in one-step diffusion editing, but also significantly outperforms existingmultistep approaches in efficiency. The code is available at https://github.com/tttao-uwu/TFinv.git.

Abstract

Image Inpainting

Explore similar work

Training-free image inversion for one-step diffusion models

Training-Free Inpainting Across Domains with a Frozen Text-to-Image Diffusion Model

SONIC: Spectral Optimization of Noise for Inpainting with Consistency