Multimodal editing large models have demonstrated powerful editing capabilities across diverse tasks. However, a persistent and long-standing limitation is the decline in facial identity (ID) consistency during realistic portrait editing. Due to the human eye's high sensitivity to facial features, such inconsistency significantly hinders the practical deployment of these models. Current facial ID preservation methods struggle to achieve consistent restoration of both facial identity and edited element IP due to Cross-source Distribution Bias and Cross-source Feature Contamination. To address these issues, we propose EditedID, an Alignment-Disentanglement-Entanglement framework for robust identity-specific facial restoration. By systematically analyzing diffusion trajectories, sampler behaviors, and attention properties, we introduce three key components: 1) Adaptive mixing strategy that aligns cross-source latent representations throughout the diffusion process. 2) Hybrid solver that disentangles source-specific identity attributes and details. 3) Attentional gating mechanism that selectively entangles visual elements. Extensive experiments show that EditedID achieves state-of-the-art performance in preserving original facial ID and edited element IP consistency. As a training-free and plug-and-play solution, it establishes a new benchmark for practical and reliable single/multi-person facial identity restoration in open-world settings, paving the way for the deployment of multimodal editing large models in real-person editing scenarios. The code is available at https://github.com/NDYBSNDY/EditedID.
Figures & tables
Figure 1: EditedID Optimized Multimodal Editing Large Models (Academic/Industrial): ID consistency capability. ID-Sim (same ID >0.7 , green border; otherwise red) indicates identity preservation in challenging scenarios: Non-focused, Multi-view, Occluded Faces, Partial-ID Multi-persons.
Figure 2: Enhanced by EditedID: ID preservation in Multimodal Editing Large Models. Comparison of repair effects and innovations. DiffBIR pre-repair triggered by Facial Corruption (FC) severity.
Figure 7: Qualitative evaluation of EditedID vs. Industrial/Academic facial reconstruction models with training cost comparison (data size, GPU). Edited image from six multimodal editing large models. Yellow boxes indicate edited elements (IP) that need to be preserved in the repaired ID.
Figure 14: Quantitative evaluation of reconstruction consistency (PSNR) under different adaptive mixing weight λ0 settings. The consistency threshold between reconstructed and original images is 25dB (higher PSNR indicates better quality). Results demonstrate that λ0 = 0.04 achieves optimal performance across different diffusion steps.
Figure 15: Qualitative evaluation of reconstruction consistency (PSNR) under different adaptive mixing weight λ0 settings. Optimal performance is achieved at λ0 = 0.04.
Figure 16: Impact of different diffusion steps and DPM-Solver++ proportions on the reconstruction results. [s1,s2] : starting and ending diffusion steps for DPM-Solver++ invocation. Results show that consistent input-reconstruction alignment is maintained within just 6 diffusion steps when applying DPM-Solver++ at [0,3] .
Figure 17: Inpainting results under different fusion weight w^ across various scenarios. w^ : fusion weight for overlapping regions of original identity image I1 ; 1- w^ : fusion weight for overlapping regions of edited image I2 . Green color indicates physically plausible ranges (e.g., hat above hair), and red color indicates implausible ranges (e.g., hair above hat). Total diffusion steps are 6.
Model
ID-Sim↑
CLIP-S↑
I-Reward↑
Full Model
0.73
28.14
1.82
w/o Alignment
0.57
26.92
1.72
w/o Disentanglement
0.63
26.48
1.69
w/o Entanglement
0.61
25.33
1.78
Baseline
0.21
19.32
1.03
Appendix
Table 7: Quantitative ablation study results
Method
ID Consistency↑
Facial Fidelity↑
IP Preservation↑
Rosberg et al. (2023)
3.34 ± 0.31
3.83 ± 0.07
4.19 ± 0.03
Ye et al. (2023a)
3.16 ± 0.97
2.13 ± 0.94
2.11 ± 0.33
Zhao et al. (2023)
3.77 ± 0.52
3.62 ± 0.83
3.41 ± 0.43
Han et al. (2024)
3.61 ± 0.02
3.51 ± 0.40
3.48 ± 0.45
Lin et al. (2024)
3.11 ± 0.90
4.02 ± 0.75
3.21 ± 0.70
Wang et al. (2025)
4.40 ± 0.33
3.55 ± 0.02
3.62 ± 0.73
Appendix
Table 8: The participants were asked to rate: (1) ID Consistency, (2) Facial Fidelity, and (3) IP Preservation. The perfect score is 5.
Method
Diffusion Steps
Total Time (s)
GPU Mem (GB)
Prompt-to-Prompt
50
∼ 8.9
∼ 13.6
Null-text Inversion
50
∼ 56.5
∼ 16.2
EditedID (ours)
6
∼ 7.0
∼ 16.5
Appendix
Table 9: Approximate inference cost of different methods for reference.
Layer
Resolution
Replace Self-attn (SSIM)
Replace Cross-attn (CLIP-S)
SD1.4
SDXL
SD1.4
SDXL
-
-
0.22
0.24
15.22
15.14
Down
32×32
0.62
0.67
19.70
21.32
16×16
0.58
0.65
18.51
20.09
Mid
8×8
0.52
0.56
16.74
19.21
Up
16×16
0.63
0.72
20.37
23.11
Appendix
Table 10: Attention mechanism analysis.
Method
Cumulative MSE ( ×10−3) ↓
maxJump ( ×10−3 )↓
latent L2 ( ×10−2 )↓
PSNR (dB)↑
DDIM-only
18.5 ± 3.2
1.9 ± 0.6
1.7 ± 0.4
28.6 ± 0.9
DPM-only
24.2 ± 4.5
3.7 ± 0.9
2.5 ± 0.6
27.2 ± 0.8
Hybrid-fragmented
47.1 ± 5.0
8.9 ± 2.2
4.3 ± 0.8
20.1 ± 1.1
Hybrid-global (ours)
17.8 ± 2.6
2.1 ± 0.5
1.8 ± 0.3
29.0 ± 0.9
Appendix
Table 11: Stability of the hybrid solver.
Method
ID-Sim
ImageReward
Clean → EditedID (w/o DiffBIR)
0.74 ± 0.03
1.82 ± 0.05
Degraded → DiffBIR only
0.36 ± 0.07
1.65 ± 0.08
Degraded → DiffBIR → EditedID
0.71 ± 0.04
1.78 ± 0.05
Appendix
Table 12: Impact of DiffBIR preprocessing on identity preservation and image quality.
Editing Attribute
EditedID Architecture
IP (CLIP-S ≥ 0.75)
ID (ID-Sim ≥ 0.70)
Single
full
92%
94%
w/o gating
78%
74%
Dual
full
87%
85%
w/o gating
69%
66%
Three
full
84%
81%
w/o gating
58%
62%
Appendix
Table 13: Effect of attentional gating in multi-region editing.
Unified multimodal models (UMMs) have recently demonstrated powerful instruction-based image editing capabilities, while also raising serious concerns about the unauthorized manipulation of personal portraits. We investigate a novel and practical problem: protecting facial identities against unauthorized editing of UMMs. Existing diffusion-based and VLM-based protection methods often become ineffective because they typically disrupt only a single visual branch. To understand this limitation, we conduct a feature-level analysis of the understanding and generation branches in unified image editing models. Our observations show that the structural agreement between these two branches is closely related to successful image editing. When only one branch is distorted, the model may still recover identity information from the other branch. Based on this, we propose Cross-Branch Conflict as a Shield (CCS), a unified adversarial protection framework. CCS jointly drives the ViT and VAE representations away from their clean counterparts. It also uses a linear Centered Kernel Alignment (CKA) objective to disrupt the structural consistency between the two branches. By degrading reliable identity information in both visual pathways and inducing incompatible cross-branch representations, CCS effectively prevents UMMs from recovering consistent facial identity cues during editing. Extensive experiments suggest that CCS consistently provides stronger protection in suppressing identity-preserving edits. Codes are in the supplementary material.
Weiwei Tan, Junxian Li, Rui Wang +3
Institute of Information Engineering, Chinese Academy of Sciences · Shanghai Jiao Tong University · Zhejiang University +1
Generative diffusion models have revolutionized facial image synthesis, yet robust identity preservation in high resolution outputs remains a critical challenge. This issue is especially vital for security systems, biometric authentication, and privacy sensitive applications, where any drift in identity integrity can undermine trust and functionality. We introduce Diff-ID, a diffusion based framework that enforces identity consistency while delivering photorealistic quality. Central to our approach is a custom 210K image dataset synthesized from CelebA-HQ, FFHQ, and LAION-Face and captioned via a fine tuned BLIP model to bolster identity awareness during training. Diff-ID integrates ArcFace and CLIP embeddings through a dual cross attention adapter within a fine tuned Stable Diffusion UNet. To further reinforce identity fidelity, we propose a pseudo discriminator loss based on ArcFace cosine similarity with exponential timestep weighting. Experiments on held out and unseen faces show that Diff-ID does not exceed InstantID in raw ArcFace Face Similarity, but achieves substantially lower FID and the strongest FIQ based identity--realism trade off among the evaluated methods. We also present a unified DDIM based morphing pipeline that enables qualitative facial interpolation without per identity fine tuning. We further argue that identity preservation and photorealism should be evaluated jointly rather than in isolation, as high identity similarity alone does not guarantee realistic outputs. To make this trade off explicit, we report Face Image Quality (FIQ) as a complementary ratio based score that combines identity similarity and perceptual realism while keeping FS and FID as the primary metrics.
Taimoor Rizwan, Sara Atito, Muhammad Awais +2
Centre for Vision, Speech and Signal Processing (CVSSP) University of Surrey Guildford, United Kingdom · School of Artificial Intelligence and Computer Science Jiangnan University Wuxi, China
Text-conditioned image editing has greatly benefitted from the advancements in Image Diffusion Models. However, extending these techniques to facial video editing introduces challenges in preserving facial identity throughout the source video and ensuring consistency of the edited subject across frames. In this paper, we introduce DiffMagicFace, a unique video editing framework that integrates two fine-tuned models for text and image control. These models operate concurrently during inference to produce video frames that maintain identity features while seamlessly aligning with the editing semantics. To ensure the consistency of the edited videos, we develop a dataset comprising images showcasing various facial perspectives for each edited subject. The creation of a data set is achieved through rendering techniques and the subsequent application of optimization algorithms. Remarkably, our approach does not depend on video datasets but still delivers high-quality results in both consistency and content. The excellent effect holds even for complex tasks like talking head videos and distinguishing closely related categories. The videos edited using our framework exhibit parity with videos that are made using traditional rendering software. Through comparative analysis with current state-of-the-art methods, our framework demonstrates superior performance in both visual appeal and quantitative metrics.