cs.CVFeb 27, 2026

Towards Source-Aware Object Swapping with Initial Noise Perturbation

Authors: Jiahui Zhan, Xianbing Sun, Xiangnan Zhu, Yikun Ji, Ruitong Liu, Liqing Zhang, Jianfu Zhang

Organizations: Shanghai Jiao Tong University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Object swapping aims to replace a source object in a scene with a reference object while preserving object fidelity, scene fidelity, and object-scene harmony. Existing methods either require per-object finetuning and slow inference or rely on extra paired data that mostly depict the same object across contexts, forcing models to rely on background cues rather than learning cross-object alignment. We propose SourceSwap, a self-supervised and source-aware framework that learns cross-object alignment. Our key insight is to synthesize high-quality pseudo pairs from any image via a frequency-separated perturbation in the initial-noise space, which alters appearance while preserving pose, coarse shape, and scene layout, requiring no videos, multi-view data, or additional images. We then train a dual U-Net with full-source conditioning and a noise-free reference encoder, enabling direct inter-object alignment, zero-shot inference without per-object finetuning, and lightweight iterative refinement. We further introduce SourceBench, a high-quality benchmark with higher resolution, more categories, and richer interactions. Experiments demonstrate that SourceSwap achieves superior fidelity, stronger scene preservation, and more natural harmony, and it transfers well to edits such as subject-driven refinement and face swapping.

Figures & tables

Figure 2: Motivation of initial-noise perturbation. Existing inpainting-based methods rely on paired data from multi-view capture, videos, retrieval, or cropping, which are costly, blurry, or cannot model meaningful appearance changes of distinct objects. Our approach generates high-quality pseudo pairs from single images, producing clear yet coherent object variations while keeping the background intact.
Figure 3: Overview of SourceSwap . The pipeline consists of two phases: (1) Initial-noise perturbation, which converts any source image IsI_{s} into a pseudo pair by transforming its initial latent zTz_{T} to the frequency domain and locally permuting high-frequency components within the object mask to alter appearance while preserving coarse structure and scene context; and (2) Source-aware training, which uses these pseudo pairs to train a dual U-Net with full-source conditioning and a clean-reference encoder, enabling efficient learning of inter-object alignment and zero-shot inference. A lightweight iterative refinement step further improves object fidelity.
Figure 4: Left: decoded DDIM latents at different timesteps and their low and high frequency components. The high-frequency component retains most of the identity and structural cues, while the low-frequency component is coarse. Right: ablations during pseudo pair construction. Permuting all components severely distorts the object; permuting only the low-frequency component yields small changes; fixing the low-frequency and permuting only the high-frequency component produces coherent and meaningful appearance edits.
Figure 5: Pseudo pairs generated by initial‑noise perturbation. Only high‑frequency latents inside the mask are permuted. The images show appearance variation with preserved structure and scene harmony.
Figure 6: Iterative refinement progressively enhances object fidelity by replacing the source image with the output from each inference, recovering fine-grained details such as the vase’s color gradients and the plate’s texture.
Figure 7: Qualitative comparison with state-of-the-art methods on our SourceBench (please zoom in for a detailed view).
BenchmarkMin. Resolution# Class# Object# Image# Pairs
SRSR
DreamEditBench [ 19 ]168×154168\times 15422220302201581574
SourceBench700×525700\times 525543492283492281554
Table 1: Statistics of SourceBench and DreamEditBench. “S” denotes source and “R” denotes reference. With a comparable number of pairs, SourceBench contains more object categories/instances and higher resolution than DreamEditBench [ 19 ] .
Figure 8: Qualitative comparison with state-of-the-art methods on DreamEditBench (please zoom in for a detailed view).
Figure 9: Quantitative comparisons. Lower DreamSim and LPIPS scores indicate better performance, corresponding to points closer to the lower-left region in (a) and (b).
MethodDreamEditBenchSourceBench
Paint-by-Example [ 41 ]94.44%84.31%
AnyDoor [ 5 ]83.33%68.32%
MimicBrush [ 4 ]91.67%82.35%
TIGIC [ 17 ]97.23%96.04%
Diptych Prompting [ 32 ]62.86%65.69%
PhotoSwap [ 8 ]88.89%-
Table 2: Evaluation of object-scene harmony using MLLM. Each entry indicates how many times ChatGPT‑5 [ 23 ] preferred our method over the competing approaches.
Figure 10: Quantitative ablation study. “Full model” denotes our complete setting; results closer to the lower-left corner are better.
Figure 11: Qualitative ablation study on different variants.
Baselines# Training DataData Type
Paint-by-Example [ 41 ]∼ ⁣190000k\sim\!190000kImages
AnyDoor [ 5 ]∼ ⁣410k\sim\!410kImages & Videos
MimicBrush [ 4 ]∼ ⁣10100k\sim\!10100kImages & Videos
ObjectMate [ 38 ]∼ ⁣600000k\sim\!600000kMulti-view Images
IMPRINT [ 35 ]∼ ⁣1627k\sim\!1627kMulti-view Images & Videos
Ours∼ ⁣40k\sim\!40kImages
Table 3: Training data scale of learning-based baselines. Prior methods rely on large and expensive datasets (often videos or multi-view images), whereas SourceSwap requires only 40K single images for fine-tuning.
Figure 12: Ablation on training size. Performance saturates at 40K samples, indicating that SourceSwap learns effectively from small-scale single-image training
 Test-time Tuning
  PhotoSwapInstantSwapSwapAnything 
Time 128.85s(+751.97s)23.48s(+751.97s)127.39s(+751.97s) 
 Tuning-free
  TIGICDiptychPrompt 
Time 12.84s124.63s 
 Learning-based
Table 4: Inference time comparison across test-time tuning, tuning-free, and learning-based methods. For test-time tuning methods, we also report (in parentheses) the time required to train the personalized DreamBooth model. For our method, we report 1–4 rounds of inference and adopt 2 rounds as the final setting. Learning-based methods are substantially faster, and SourceSwap achieves the best inference speed among them.
Figure 13: Qualitative effect of iterative refinement. One round ( k=1k=1 ) captures the overall structure but misses surface colors; two rounds ( k=2k=2 ) significantly improve color consistency, and further rounds ( k=3,4k=3,4 ) yield only marginal visual gains.
 k=1k=1k=2k=2k=3k=3k=4k=4
DreamSim0.46530.44280.43660.4405
Table 5: Effect of iterative refinement rounds kk on object fidelity. As kk increases, fidelity improves and peaks at k=3k=3 , while k=4k=4 shows a slight drop in DreamSim.
Figure 14: Comparison between the precise source mask MsM_{s} and the bounding-box mask MboxM_{box} . We train two variants by conditioning the denoising U-Net on MsM_{s} and MboxM_{box} , respectively. Strictly constraining the object shape with MsM_{s} often causes undesirable deformations, whereas MboxM_{box} relaxes the shape constraint and enables the object to retain attributes from the reference.
Figure 15: Failure case caused by ambiguous localization. Segmentation tools such as Grounded SAM [ 28 ] tend to extract all instances ( e.g . , multiple lemons), leading the model to swap every segmented object rather than the intended one.
Figure 16: SourceSwap is capable of refining suboptimal outputs from other subject-driven generation methods, without additional retraining. It consistently improves the outputs of AnyDoor through its refinement capability.
Figure 17: SourceSwap is capable of handling multi-object swapping. In this example, all three pillows are successfully swapped and blend naturally with the scene.
BaselinesObjectBackgroundHarmony ↑\uparrow
Fidelity ↑\uparrowFidelity ↑\uparrow
Paint-by-Example [ 41 ]1.633.372.25
AnyDoor [ 5 ]2.742.412.11
MimicBrush [ 4 ]2.203.122.44
TIGIC [ 17 ]1.731.951.53
Diptych Prompting [ 32 ]2.622.683.15
Table 6: User study results on three aspects (object fidelity (object details), scene fidelity (background preservation), and object–scene harmony) using a five-point Likert scale (1–5, higher is better).
Figure 18: Example of the questionnaire used in the user study. Participants rate each result on object fidelity (object details), scene fidelity (scene preservation), and object–scene harmony using a five-point Likert scale.
Figure 19: Examples of categories and source-reference pairs in SourceBench. Scene images feature rich backgrounds and complex interactions, while reference images provide diverse object appearances. Note that the same image may participate in multiple pairs.
Figure 20: Qualitative comparison with baselines on DreamEditBench.
Figure 21: Qualitative comparison with baselines on SourceBench.
More in this topic

Image Editing

Image editing changes selected visual content while preserving the parts of an image that should remain unchanged.

See all 315 papers

Explore similar work

CardsList
Dec 10, 2025

DirectSwap: Paired, Mask-Free Video Head Swapping with Full-Reference Evaluation

Head swapping replaces an entire head while preserving pose, expression, body motion, and scene. Progress is limited by the lack of cross-identity paired videos: real footage cannot provide different identities performing exactly the same motion, leaving the task without paired supervision or frame-aligned ground truth. Existing methods therefore rely on same-identity masked reconstruction, which restricts supervision to predefined editable regions. To address this, we introduce an identity-expression decoupled synthesis pipeline that constructs expression-synchronized cross-identity video pairs from real footage. Expression-bearing facial regions are retained under small clip-consistent geometric perturbations, while the surrounding head is regenerated with a new identity as synthesized swapping input. These pairs form a cross-identity benchmark with frame-aligned real-world video as ground-truth targets, enabling full-reference evaluation of identity, expression, pose, reconstruction fidelity, and temporal stability. This yields HeadSwapBench, the first cross-identity paired dataset for video head swapping, supporting both training (20,278 videos) and benchmarking (1,040 videos). With the cross-identity supervision enabled by this paired dataset, we propose DirectSwap, a mask-free video head swapping training paradigm. Under otherwise identical settings, this formulation outperforms head- and rectangle-masked same-identity reconstruction, particularly for bidirectional head-silhouette changes. At inference, driving-output divergence and emergent reference attention estimate the edit support, allowing unchanged non-head content to be restored from the driving video without external segmentation or additional training. On 1,040 clips from 104 unseen subjects, the resulting model demonstrates that the proposed paired supervision supports effective whole-head swapping.

Yanan Wang, Shengcai Liao, Panwen Hu +4

Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), UAE · United Arab Emirates University (UAEU), Al Ain, UAE · Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), Abu Dhabi, UAE +1

Video EditingVideo Generation Evaluation
AskPDF
Jul 23, 2026

Adaptive Identity Anchoring: Closed-Loop Keyframe Placement for Synthetic Paired Supervision in Video Face Swapping

Video face swapping has no natural paired supervision: no real footage exists of one person's face performing another person's video. The strongest current answer, DreamID-V's SyncID-Pipe, mints pairs by replacing the identity in exactly two frames of a real clip -- the first and the last -- and regenerating the rest from a pose sequence alone. Pose carries no appearance evidence of the swapped-in identity, so over long clips, occlusions, and extreme pose excursions the synthesized identity has a long unanchored span on which to drift; no published ablation examines anchor count or placement. We propose Adaptive Identity Anchoring (AIA): (i) generalize the synthesizer to arbitrary anchor sets, architecturally natural for diffusion-forcing-style transformers where conditioning on a frame is clamping its tokens to zero noise; (ii) place anchors by a closed feedback loop that scores every generated frame against the real reference identity and inserts an image-face-swapped anchor at the worst-scoring frame until the pair passes a threshold or exhausts a budget; (iii) reuse the loop's verdict as an automatic data filter. A second pathology, the beauty-filter look of over-smoothed skin, has the same root cause: micro-texture, like identity, is priced by none of the pipeline's objectives. We therefore pair AIA with Reality-Referenced Texture Restoration: matched re-graining from each real frame's non-face regions, band-split transfer of sub-identity micro-texture from the real footage, and a second, spectral acceptance channel refereed by the footage's own spectrum. Identity-anchor density, we argue, is a controllable quality dial, and we specify falsifiable experiments -- drift-versus-gap curves, uniform-versus-adaptive placement at matched budgets, student training on AIA-minted data, and texture ablations with a human beauty-filter study -- that would validate or refute the proposal.

Logan Robbins

Independent researcher.

Synthetic Data AugmentationTemporal Consistency in Video Generation
AskPDF
Mar 6, 2026

PICS: Pairwise Image Compositing with Spatial Interactions

Despite strong single-turn performance, diffusion-based image compositing often struggles to preserve coherent spatial relations in pairwise or sequential edits, where subsequent insertions may overwrite previously generated content and disrupt physical consistency. We introduce PICS, a self-supervised composition-by-decomposition paradigm that composes objects in parallel while explicitly modeling the compositional interactions among (fully-/partially-)visible objects and background. At its core, an Interaction Transformer employs mask-guided Mixture-of-Experts to route background, exclusive, and overlap regions to dedicated experts, with an adaptive α-blending strategy that infers a compatibility-aware fusion of overlapping objects while preserving boundary fidelity. To further enhance robustness to geometric variations, we incorporate geometry-aware augmentations covering both out-of-plane and in-plane pose changes of objects. Our method delivers superior pairwise compositing quality and substantially improved stability, with extensive evaluations across virtual try-on, indoor, and street scene settings showing consistent gains over state-of-the-art baselines. Code and data are available at https://github.com/RyanHangZhou/PICS

Hang Zhou, Xinxin Zuo, Sen Wang +1

University of Alberta · Concordia University

Image EditingCompositional Image Generation
AskPDF