Latent Bias Alignment for High-Fidelity Diffusion Inversion in Real-World Image Reconstruction and Manipulation

Weiming Chen; Qifan Liu; Siyi Liu; Yushun Tang; Yijia Wang; Zhihan Zhu; Zhihai He

cs.CVMar 25, 2026

Latent Bias Alignment for High-Fidelity Diffusion Inversion in Real-World Image Reconstruction and Manipulation

Authors: Weiming Chen, Qifan Liu, Siyi Liu, Yushun Tang, Yijia Wang, Zhihan Zhu, Zhihai He

Organizations: the Department of Electrical and Electronic Engineering, Southern University of Science and Technology, Shenzhen 518055, China. · Undergraduate School of Artificial Intelligence, Shenzhen Polytechnic University, Shenzhen 518055, China. · Huawei Technologies Co., Ltd., Shenzhen, China. · the Pengcheng Lab, Shenzhen, China.

Abstract

Recent research has shown that text-to-image diffusion models are capable of generating high-quality images guided by text prompts. But can they be used to generate or approximate real-world images from the seed noise? This is known as the diffusion inversion problem, which serves as a fundamental building block for bridging diffusion models and real-world scenarios. However, existing diffusion inversion methods often suffer from low reconstruction quality or weak robustness. Two major challenges need to be carefully addressed: (1) the misalignment between the inversion and generation trajectories during the diffusion process, and (2) the mismatch between the diffusion inversion process and the VQ autoencoder (VQAE) reconstruction. To address these challenges, we introduce a latent bias vector at each inversion step, which is learned to reduce the misalignment between inversion and generation trajectories. We refer to this strategy as Latent Bias Optimization (LBO). Furthermore, we perform an approximate joint optimization of the diffusion inversion and VQAE reconstruction processes by learning to adjust the image latent representation, which serves as the connecting interface between them. We refer to this technique as Image Latent Boosting (ILB). Extensive experimental results demonstrate that the proposed method significantly improves the image reconstruction quality of the diffusion model, as well as the performance of downstream tasks, including image editing and rare concept generation.

Figures & tables

Fig. 1: The schematic illustration of the proposed Latent Bias Optimization (LBO) strategy, which learns the latent bias vector at each inversion step to align the inversion and generation trajectories, thereby seeking an optimal inversion trajectory for the given real-world image.

Fig. 2: A comparison of four different types of diffusion inversion methods.

Fig. 3: The overview of the proposed Image Latent Boosting (ILB) technique.

Method	PSNR ( $\uparrow$ )	SSIM ( $\uparrow$ )	LPIPS ( $\downarrow$ )	Additional
Method	PSNR ( $\uparrow$ )	SSIM ( $\uparrow$ )	LPIPS ( $\downarrow$ )	Parameters
Stable Diffusion v1.5 (SD1)
VQAE (upper bound)	26.75	0.7699	0.0512	0
DDIM Inversion	14.10	0.4480	0.5210	0
NPI	24.00	0.7208	0.1141	0
NTI	25.81	0.7566	0.0765	2.81 M

TABLE I: Comparison of different methods across three variants of diffusion models on the COCO 2017 validation set.

$\mathcal{L}_{\rm{con}}$			$\mathcal{L}_{\rm{reg}}$	PSNR ( $\uparrow$ )	SSIM ( $\uparrow$ )	LPIPS ( $\downarrow$ )
$\mathcal{L}_{\rm{L1}}$	$\mathcal{L}_{\rm{SSIM}}$	$\mathcal{L}_{\rm{LPIPS}}$	$\mathcal{L}_{\rm{reg}}$	PSNR ( $\uparrow$ )	SSIM ( $\uparrow$ )	LPIPS ( $\downarrow$ )
✗	✗	✗	✗	26.65	0.7680	0.0547
✓	✗	✗	✗	29.16	0.8287	0.1055
✓	✓	✗	✗	28.44	0.8456	0.1101
✓	✓	✓	✗	28.08	0.8310	0.0312
✓	✓	✓	✓	28.12	0.8328	0.0344

TABLE II: Ablation study on the loss components of ILB.

Fig. 4: Visual comparisons of diffusion inversion reconstruction results with and without the regularization term $\mathcal{L}_{\rm{reg}}$ .

Fig. 5: Visualization of diffusion inversion reconstruction results under different ILB consistency loss settings.

Inversion Method	Structure Distance $\downarrow$	Unedited Fidelity		CLIP Similarity
Inversion Method	Structure Distance $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	Whole $\uparrow$	Edited $\uparrow$
DDIM Inv.	0.0705	17.82	0.7141	25.18	22.35
FPI	0.0222	23.57	0.8109	24.33	21.57
AIDI	0.0220	23.65	0.8112	24.33	21.56
NTI	0.0176	25.87	0.8396	24.85	22.01
ExactDPM	0.0336	22.55	0.7922	24.52	21.74

TABLE III: Comparison results on the image editing task.

Method	C1	C2	C3	C4	C5
SD1	0.8521	0.3852	0.7033	0.5083	0.6463
NAO-SS	0.8847	0.5143	0.7120	0.5304	0.6610
NAO-SS (w/ LBI)	0.8923	0.5366	0.7502	0.5869	0.6977

TABLE IV: Quantitative results on rare concept generation across five rare concepts from the ImageNet-1K dataset.

Fig. 6: Qualitative results of image editing using P2P integrating with the proposed LBI method.

Fig. 7: Qualitative results of the rare concept generation task. We present the performance for three additional rare concepts: tiger cat , pay phone , and oxygen mask .

Explore similar work

Dual Inversion for Text-to-Image Diffusion Models: From Both Prompt and Noise Perspectives
Jul 29, 2026Xiaolong Liu, Junjian Li, Yuan Xiao +4Diffusion Model Inversion Diffusion-Based Inverse Problems
Geely Hangzhou, China · City University of Macau Macau, China
Training-free image inversion for one-step diffusion models
May 31, 2026Tao Wu, Senmao Li, Yaxing Wang +3Diffusion Model Inversion Text-Guided Image Editing
Computer Vision Center, Universitat Autonoma de Barcelona, Barcelona, 08193, Spain · Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), Abu Dhabi, United Arab Emirates · Jilin University, 130012, Changchun, Jilin, China +3
ResetEdit: Precise Text-guided Editing of Generated Image via Resettable Starting Latent
Apr 28, 2026Hanyi Wang, Han Fang, Zheng Wang +2Diffusion Model Inversion Text-Guided Image Editing
Shanghai Jiao Tong University China · University of Science and Technology of China China · Wuhan University China +1

Jul 29, 2026J/K move · Enter open · S save

Dual Inversion for Text-to-Image Diffusion Models: From Both Prompt and Noise Perspectives

Xiaolong Liu, Junjian Li, Yuan Xiao, Jiaqi Deng+3

Geely Hangzhou, China · City University of Macau Macau, China

Diffusion Model Inversion Diffusion-Based Inverse Problems Diffusion-Based Image Editing Controllable Image Generation T2I Diffusion Models

Open paper AskPDF arXiv

Prompt inversion, as a typical reverse engineering technique, enables text-to-image (T2I) diffusion models to generate the desired target images without extensive prompt engineering. However, existing prompt inversion methods suffer from significant limitations: (1) gradient-based methods are unstable and uninterpretable, often resulting in generated images with severe artifacts; (2) gradient-free methods yield human-readable prompts but still fail to preserve visual fidelity due to the lack of fine-grained detail alignment. We contend that the limitations stem from treating prompt inversion as a sufficient condition for reverse engineering, ignoring the critical role of the latent noise that encodes structural information. Consequently, we propose Dualin (Dual inversion), a two-stage method that jointly recovers both the semantic prompt and latent noise of the target image. In the first stage, we integrate vision-language model, CLIP and large language model to invert a faithful, human-interpretable hard prompt. In the second stage, unconditional DDIM inversion reconstructs the exact latent noise of the target image, guaranteeing the consistency at the structural information level. Theoretically, we prove that the inverted noise enables flexible image editing without re-optimization. Extensive experiments on diverse datasets demonstrate that Dualin simultaneously generates high-quality inverted prompts and achieves state-of-the-art image fidelity. Additionally, Dualin can establish a robust foundation for the precise and controllable image editing.

Abstract

Explore similar work

Dual Inversion for Text-to-Image Diffusion Models: From Both Prompt and Noise Perspectives

Training-free image inversion for one-step diffusion models

ResetEdit: Precise Text-guided Editing of Generated Image via Resettable Starting Latent