PANDORA: Pixel-wise Attention Dissolution and Latent Guidance for Zero-Shot Object Removal
Authors: Dinh-Khoi Vo, Van-Loc Nguyen, Tam V. Nguyen, Minh-Triet Tran, Trung-Nghia Le
Organizations: University of Science, Ho Chi Minh City, Vietnam · Vietnam National University, Ho Chi Minh City, Vietnam · University of Dayton, Ohio, United States
Removing objects from natural images is challenging due to difficulty of synthesizing semantically coherent content while preserving background integrity. Existing methods often rely on fine-tuning, prompt engineering, or inference-time optimization, yet still suffer from texture inconsistency, rigid artifacts, weak foreground-background disentanglement, and poor scalability for multi-object removal. We propose a novel zero-shot object removal framework, namely PANDORA, that operates directly on pre-trained text-to-image diffusion models, requiring no fine-tuning, prompts, or optimization. We propose Pixel-wise Attention Dissolution to remove object by nullifying the most correlated attention keys for masked pixels, effectively eliminating the object from self-attention flow and allowing background context to dominate reconstruction. We further introduce Localized Attentional Disentanglement Guidance to steer denoising toward latent manifolds favorable to clean object removal. Together, these components enable precise, non-rigid, prompt-free, and scalable multi-object erasure in a single pass. Experiments demonstrate superior visual fidelity and semantic plausibility compared to state-of-the-art methods. The project page is available at https://vdkhoi20.github.io/PANDORA.
Figures & tables
Fig. 1 : Our PANDORA enables prompt-free, fine-tuning-free object removal across various scenarios in a single forward. Without requiring training or textual prompts, our approach handles diverse and challenging removal settings, from a single object to multiple similar or distinct targets and even densely packed similar objects, while preserving background fidelity and structural consistency.
Fig. 2 : Overview of our proposed pipeline with an intuitive illustration of each module. The image is inverted into noise with intermediate latents stored and injected into BPA and PAD to preserve background and dissolve objects, respectively. Specifically, BPA restricts background queries to background regions, while PAD operates at the pixel level to constrain object queries to unrelated regions. Finally, LADG steers denoising away from masked object regions for seamless synthesis.
Fig. 3 : Qualitative comparison across diverse object removal scenarios, including single-object, multi-object, and mass similar-object removal (top to bottom). From left to right: the original image with mask and results from different methods. The last five columns correspond to zero-shot approaches.
Zero-shot methods (no retraining, SD 2.1 backbone)
TABLE I : Quantitative comparison of fine-tuned and zero-shot methods. PANDORA consistently achieves the best object removal quality with competitive background realism, without any retraining or textual prompts, demonstrating strong generalization across bdifferent backbones. Removing LADG slightly reduces removal quality, while removing PAD causes a significant degradation.
Method
UPR
Fine-tuning-based methods
PowerPaint [ 23 ]
4.50
LaMa [ 19 ]
6.75
SD2-Inpaint [ 16 ]
6.25
SD2-Inpaint-w/ prompt [ 16 ]
7.00
Zero-shot methods (no training required)
TABLE II : User study results showing the User Preference Rate (UPR) as the percentage of times each method was chosen as the best. PANDORA is consistently favored by users.
Fig. 4 : From left to right, mask is 20% smaller, 10% smaller, tightly aligned, 10% larger, and 20% larger than object.
Suppressed Percentile
FID ↓
LPIPS ↓
MSE ↓
CLIP ↑
1%
42.32
0.1837
0.0175
24.547
3%
44.07
0.1871
0.0180
24.553
5%
44.98
0.1895
0.0184
24.570
15%
48.08
0.1969
0.0198
24.596
25%
49.84
0.2015
0.0204
24.600
TABLE III : Ablation study on percentile threshold used in PAD (SD 1.5 backbone).