With the growing success of diffusion models in computer vision, we explore their potential for same-identity facial replacement in photographs. Specifically, we propose a diffusion-based method, built on top of a pre-trained text-to-image model, that takes as input a portrait image of a person and a second reference image of the same individual, potentially captured under different conditions. The goal is to seamlessly replace the input face with the reference face, while keeping the background intact. Surprisingly, despite the clear real-world utility of this task, no recently published work has directly addressed face replacement in this specific setting. To support this goal, we construct a large dataset of image pairs depicting the same person under varying facial expressions and poses. Experimental results demonstrate that our approach produces more realistic and identity-consistent results than existing face reenactment models.
Figures & tables
Figure 1: We present PicturePerfect , a diffusion-based identity preserving facial replacement approach. Prior methods like FaceAdapter [ 20 ] tend to alter the facial identity and lose realism on both in-distribution (ID) ( left ) and out-of-distribution (OOD) ( right ) face replacement tasks.
Figure 2: PicturePerfect network overview: Given a source ( s ) and target ( tg ) image pair, we extract the driving image ( d ) by segmenting the face from the target image. To simulate real-world variations, we apply data augmentations such as scaling, rotation, translation, and relighting. The image triplet is then encoded using the SDXL [ 42 ] VAE to obtain latent representations and concatenated spatially.
Figure 3: PicturePerfect dataset samples. For each neutral face (top row), we generate 20 expressive variants using a face animation model [ 19 ] . For visualization purposes, we display three randomly sampled expressive variants per neutral face.
Figure 4: Face Replacement ( in-distribution ). Our method produces high-quality, realistic reconstructions with stronger identity preservation by leveraging both the source and masked driving image (red inset), outperforming other approaches for diverse identities.
Figure 5: Face Replacement ( out-of-distribution ). Our method demonstrates strong generalization for a wide range of facial poses, expressions, identities, and lighting conditions. Moreover, it effectively generates reenacted source images that preserve the subject’s identity. Rows 1-3 showcase results on authentic image-pairs from the PicturePerfect test-set, while row 4 contains results on synthetic images.
Replacement out-of-distribution (OOD)
PicturePerfect (Real)
VoxCeleb2 (LivePortrait)
Methods
CSIM ↑
FID ↓
AED ↓
APD ↓
AGD ↓
CSIM ↑
FID ↓
AED ↓
APD ↓
AGD ↓
DAM
0.350
55.16
2.61
0.2222
15.29
0.246
38.51
2.91
0.2472
15.59
FADM
0.342
74.98
2.66
0.2252
16.51
0.219
61.76
3.01
0.2490
16.45
IP-Adapter
0.275
39.16
2.45
0.2666
16.59
0.214
23.68
3.14
0.2761
23.13
FaceAdapter
0.343
—
2.53
0.2076
11.31
0.207
—
3.06
0.3104
14.61
Table 1: Quantitative comparison with prior approaches using ArcFace CSIM [ 12 ] , FID, pose (APD) [ 31 ] , expression (AED) [ 31 ] , gaze (AGD) [ 5 ] errors, LPIPS [ 70 ] , and PSNR. Best scores are highlighted in blue , and second-best are underlined . FFHQ and VoxCeleb2 samples are generated with LivePortrait, while PicturePerfect uses curated real-world pairs. Baselines with unreliable FID scores are marked as (—).
Figure 6: User study: Replacement ID and OOD. Participants were asked to mark {source, output} pairs as “real” or “fake” based on perceptual quality. We report the proportion of images marked as “real” for each approach. The error bars represent the standard error of the mean (SEM).
Figure 7: Ablation study: Effects of Relighting. Without relighting, lighting artifacts from the driving image often leak into the output. Incorporating random relighting into the data augmentation pipeline produces significantly more realistic and consistent illumination.
Figure 8: Ablation study. We evaluate replacing spatial to channel-wise concatenation, swapping SDXL to SDv1.5, as well as disabling relighting, all data augmentations and inner face extraction.
Replacement out-of-distribution (OOD)
PicturePerfect (Real)
VoxCeleb2 (LivePortrait)
Methods
CSIM ↑
FID ↓
AED ↓
APD ↓
AGD ↓
CSIM ↑
FID ↓
AED ↓
APD ↓
AGD ↓
Ours
0.631
28.08
2.50
0.2262
9.87
0.579
12.44
2.78
0.2366
12.78
Channel-wise
0.618
80.70
3.37
0.3113
8.47
0.548
62.26
3.69
0.3418
11.80
SDv1.5
0.655
30.98
2.50
0.2164
9.71
0.576
12.96
2.84
0.2413
13.19
Table 2: Ablation study. We compare spatial versus channel-wise concatenation of the input images and assess the performance with SDv1.5 as the backbone UNet architecture [ 45 ] . We report the main ablation results here and defer the remaining variants to the supplementary material.
Figure 9: Real-world simulation of composite picture editing (“burst” mode reconstruction). Each one of the source faces is individually edited using the corresponding target face as the driving image. The model automatically adjusts for orientation, head pose, and blending issues to naturally fit the source composite photograph.
Figure 10: Comparing our facial replacement with instruction-based image editing approaches [ 8 , 7 ] . Note that, our approach uses a masked driving image (show in red inset), while the image editing approaches do not require a driving image. Our approach makes realistic edits following the driving image, while the image editing methods have no control over the facial edits, leading to unrealistic changes to personal identity.
Figure 11: Comparing our facial replacement with multimodal-based editing approach [ 48 ] . Note that, our approach uses a masked driving image (showed in red inset), while the SeeDream4.0 uses the full driving image. Our approach makes realistic edits following the driving image, while the image editing method has quality output but has tendency to change person’s identity.
Figure 12: Although our proposed approach achieves more realistic facial replacement compared to prior methods, it still exhibits limitations in certain areas, particularly around the eyes and hair.
Replacement in-distribution (ID)
FFHQ
Methods
CSIM ↑
FID ↓
AED ↓
APD ↓
AGD ↓
LPIPS ↓
PSNR ↑
DAM
0.557 ± 0.0034
25.28
2.91 ± 0.0128
0.2796 ± 0.0022
11.89 ± 0.1247
0.0380 ± 0.0003
21.37 ± 0.0491
FADM
0.542 ± 0.0034
43.24
2.97 ± 0.0138
0.2849 ± 0.0022
13.28 ± 0.1278
0.0453 ± 0.0004
19.76 ± 0.0464
IP-Adapter
0.312 ± 0.0018
17.22
3.01 ± 0.0116
0.2910 ± 0.0022
16.28 ± 0.1839
0.0835 ± 0.0004
14.78 ± 0.0312
FaceAdapter
0.429 ± 0.0064
122.47
2.97 ± 0.0327
0.2439 ± 0.0049
10.37 ± 0.3383
0.1252 ± 0.0009
10.32 ± 0.0824
Table 3: We quantitatively compare our method with prior approaches ( ± Standard Error of Mean) on replacement-ID testsets FFHQ [ 26 ] and VoxCeleb [ 37 , 11 ] .
Figure 13: Metrics like PSNR and LPIPS often favor blurry, low-quality reconstructions due to sensitivity to slight misalignments, failing to align with human judgment.
Figure 14: Ablation: Robustness to data augmentations. Our solution demonstrates the ability to mitigate the effects of different data augmentations, resulting in more stable and visually appealing results.
Figure 15: Additional face replacement (in-distribution) results
Figure 16: Additional face replacement (out-of-distribution) results
Replacement out-of-distribution (OOD)
PicturePerfect (Real)
VoxCeleb2 (LivePortrait)
Methods
CSIM ↑
FID ↓
AED ↓
APD ↓
AGD ↓
CSIM ↑
FID ↓
AED ↓
APD ↓
AGD ↓
Ours
0.631
28.08
2.50
0.2262
9.87
0.579
12.44
2.78
0.2366
12.78
Channel-wise
0.618
80.70
3.37
0.3113
8.47
0.548
62.26
3.69
0.3418
11.80
SDv1.5
0.655
30.98
2.50
0.2164
9.71
0.576
12.96
2.84
0.2413
13.19
No Relighting
0.759
26.92
2.43
0.2116
8.67
0.685
11.91
2.83
0.2465
13.00
Table 4: Additional ablation study of our identity-preserving face replacement framework. We compare the final model against variants without relighting, without data augmentation, and using the driving image as-is. While some ablations achieve higher reconstruction-oriented metrics, these settings introduce shortcut cues that can artificially increase similarity to the target image and therefore do not reflect robust identity-preserving face replacement.
Figure 17: Experiment with cross-identity facial replacement where the identity along with the expression from the driving image gets transferred to source.
Figure 18: Experiment with cross-identity facial replacement where only the expression from the driving image gets transferred to source while keeping the identity unaltered.
Figure 19: PicturePerfect dataset samples. For each neutral face (top row), we generate 20 expressive variants using a face animation model [ 19 ] . For visualization purposes, we display four randomly sampled expressive variants per neutral face.
Figure 20: PicturePerfect Dataset generation process overview: Neutral looking faces are detected and extracted from source videos, which are guided via driving videos to generate expressive animated faces using LivePortrait [ 19 ] .
Figure 21: Overview of some example driving videos used to animate the neutral faces with LivePortrait [ 19 ] .
Figure 22: Random samples from the dataset. The PicturePerfect dataset features a wide diversity of backgrounds, lighting conditions, ethnicities, clothing styles, poses, and facial expressions.
Generative diffusion models have revolutionized facial image synthesis, yet robust identity preservation in high resolution outputs remains a critical challenge. This issue is especially vital for security systems, biometric authentication, and privacy sensitive applications, where any drift in identity integrity can undermine trust and functionality. We introduce Diff-ID, a diffusion based framework that enforces identity consistency while delivering photorealistic quality. Central to our approach is a custom 210K image dataset synthesized from CelebA-HQ, FFHQ, and LAION-Face and captioned via a fine tuned BLIP model to bolster identity awareness during training. Diff-ID integrates ArcFace and CLIP embeddings through a dual cross attention adapter within a fine tuned Stable Diffusion UNet. To further reinforce identity fidelity, we propose a pseudo discriminator loss based on ArcFace cosine similarity with exponential timestep weighting. Experiments on held out and unseen faces show that Diff-ID does not exceed InstantID in raw ArcFace Face Similarity, but achieves substantially lower FID and the strongest FIQ based identity--realism trade off among the evaluated methods. We also present a unified DDIM based morphing pipeline that enables qualitative facial interpolation without per identity fine tuning. We further argue that identity preservation and photorealism should be evaluated jointly rather than in isolation, as high identity similarity alone does not guarantee realistic outputs. To make this trade off explicit, we report Face Image Quality (FIQ) as a complementary ratio based score that combines identity similarity and perceptual realism while keeping FS and FID as the primary metrics.
Taimoor Rizwan, Sara Atito, Muhammad Awais +2
Centre for Vision, Speech and Signal Processing (CVSSP) University of Surrey Guildford, United Kingdom · School of Artificial Intelligence and Computer Science Jiangnan University Wuxi, China
Face swapping aims to optimize realistic facial image generation by leveraging the identity of a source face onto a target face while preserving pose, expression, and context. However, existing methods, especially GAN-based methods, often struggle to balance identity preservation and visual realism due to limited controllability and mode collapse. In this paper, we introduce CA-IDD (Cross-Attention Guided Identity-Conditional Diffusion), the first diffusion-based face swapping approach that integrates multi-modal guidance comprising gaze, identity, and facial parsing through multi-scale cross-attention. Precomputed identity embeddings are incorporated into the denoising process via hierarchical attention layers, resulting in accurate and consistent identity transfer. To improve semantic coherence and visual quality, we use expert-guided supervision, with facial parsing and gaze-consistency modules. Unlike GAN-based or implicit-fusion methods, our diffusion framework provides stable training, robust generalization, and spatially adaptive identity alignment, allowing for fine-grained regional control across pose and expression variations. CA-IDD achieves an FID of 11.73, exceeding established baselines such as FaceShifter and MegaFS. Qualitative results also reveal improved identity retention across diverse poses, establishing CA-IDD as a strong foundation for future diffusion-based face editing.
Multi-person identity-preserving generation requires binding multiple reference faces to specified locations under a text prompt. Strong identity/layout conditions often trigger copy-paste shortcuts and weaken prompt-driven controllability. We present AnyPhoto, a diffusion-transformer finetuning framework with (i) a RoPE-aligned location canvas plus location-aligned token pruning for spatial grounding, (ii) AdaLN-style identity-adaptive modulation from face-recognition embeddings for persistent identity injection, and (iii) identity-isolated attention to prevent cross-identity interference. Training combines conditional flow matching with an embedding-space face similarity loss, together with reference-face replacement and location-canvas degradations to discourage shortcuts. On MultiID-Bench, AnyPhoto improves identity similarity while reducing copy-paste tendency, with gains increasing as the number of identities grows. AnyPhoto also supports prompt-driven stylization with accurate placement, showing great potential application value.