Text-to-image diffusion models can learn and reproduce an individual's identity from only a few reference photos, creating risks of unauthorized and misleading content generation. Proactive defenses address this risk by adding adversarial perturbations to facial images before release, thereby disrupting identity acquisition during downstream personalization. However, existing methods typically rely on dense L∞-bounded perturbations, whose accumulated distortions can become visible around facial structures. Recent methods improve protected-image quality, but often at the cost of reduced protection effectiveness. In this paper, we propose Sparse Encoder Attack on Latent Variance (SEAL), a sparse adversarial protection framework that modifies only 5% of the spatial locations in a facial image. SEAL combines a differentiable top-k masking scheme that jointly optimizes perturbation locations and values with a variance-oriented objective over the frozen VAE encoder's latent distribution to disrupt identity-consistent representations. A saliency-guided dispersion penalty further discourages perturbation concentration in perceptually salient facial regions. Experiments on CelebA-HQ and VGGFace2 show that SEAL achieves the best protected-image quality among the compared methods, exceeding 41 dB PSNR with 0.97 SSIM and at most 0.07 LPIPS, while reducing identity similarity to at most 0.15 after diffusion personalization. These results demonstrate that sparse location optimization enables strong identity protection while preserving high visual quality. Code and data are available at https://github.com/Lvxan/SEAL
Figures & tables
Figure 1: By modifying only a small fraction of pixels, SEAL preserves the visual appearance of the protected face image while disrupting identity reproduction after diffusion personalization. Compared with existing protection methods, SEAL achieves both high image quality and strong protection effectiveness.
Figure 2: Overview of SEAL . Given a clean image x , SEAL jointly optimizes a sparse binary mask m and a bounded perturbation δ to disrupt the VAE latent variance. A saliency-guided dispersion regularizer discourages the selected pixels from clustering in perceptually salient facial regions.
Figure 3: Qualitative comparison. Each method shows the protected image, a magnified crop with its perturbation pattern, and DreamBooth generations under two prompts. P2: a dslr portrait of sks person ; P3: a photo of sks person looking at the mirror .
Image Quality
Protection (P1)
Protection (P2)
Dataset
Method
SSIM ↑
PSNR ↑
LPIPS ↓
DISTS ↓
BRISQUE ↑
SER-FIQ ↓
FDFR ↑
ISM ↓
BRISQUE ↑
SER-FIQ ↓
FDFR ↑
ISM ↓
VGGFace2
No Def.
–
–
–
–
17.34
0.75
0.01
0.67
8.60
0.72
0.01
0.49
Anti-DB
0.90
34.52
0.21
0.18
35.26
0.35
0.32
0.35
40.28
0.40
0.43
0.27
Mist
0.74
28.62
0.38
0.38
20.59
0.47
0.12
0.22
32.84
0.44
0.23
0.19
SDS(-)
0.74
29.87
0.31
0.34
40.84
0.75
0.00
0.62
47.16
0.77
0.00
0.54
MYOPIA
0.92
34.35
0.11
0.22
31.25
0.72
0.00
0.55
54.28
0.77
0.02
0.51
Table 1: Comparison of protection methods on VGGFace2 and CelebA-HQ. ↑ / ↓ indicates higher/lower is better. Best in bold , second best underlined . P1: a photo of sks person ; P2: a dslr portrait of sks person .
Figure 4: Cross-model and cross-method transferability on CelebA-HQ. Bars are rescaled to a common range (taller = better). Prompt: a photo of sks person .
Ratio
BRISQUE ↑
SER-FIQ ↓
FDFR ↑
100%
67.64
0.07
0.93
75%
59.78
0.12
0.88
50%
48.71
0.31
0.69
25%
30.51
0.59
0.33
Table 2: Protection under different protected-image ratios on CelebA-HQ. Prompt: a photo of sks person .
Method
GPU Mem. (GB)
Time (s)
Anti-DB
∼ 17
∼ 420
Mist
∼ 11
∼ 120
SDS(-)
∼ 9
∼ 70
MYOPIA
∼ 18
∼ 590
VCPro
∼ 20
∼ 100
SEAL
∼ 9
∼ 70
Table 3: Efficiency comparison. Time and GPU memory are measured for protecting one four-image identity set at 512×512 resolution on a single NVIDIA A100.
Top- k
Saliency
LA
F-LPIPS ↓
LPIPS ↓
ISM ↓
w/o Saliency
✓
✓
0.05
0.06
0.02
Random Mask
✓
✓
0.03
0.05
0.68
MSE Loss
✓
✓
0.04
0.07
0.37
LE Loss
✓
✓
0.03
0.06
0.25
Full model
✓
✓
✓
0.03
0.07
0.07
Table 4: Component ablation on CelebA-HQ. Prompt: a photo of sks person .
Figure 5: Effect of the saliency-guided dispersion penalty. (a-b) Without the penalty, selected locations concentrate around the eyes and nose. (c-d) With the penalty, the selected locations are distributed more broadly across less salient regions.
Figure 6: Trade-off between protected-image distortion (LPIPS) and protection effectiveness (ISM) across sparsity ratios. The default setting r=5% provides a balanced operating point. Prompt: a photo of sks person .
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
ID
Class
w
ID
Class
w
0
Background
0.00
10
Nose
0.90
1
Skin
0.80
11
Mouth
0.90
2
L. eyebrow
0.70
12
Upper lip
0.90
3
R. eyebrow
0.70
13
Lower lip
0.90
4
L. eye
1.00
14
Neck
0.50
5
R. eye
1.00
15
Necklace
0.10
Appendix
Table 5: Complete BiSeNet class-to-weight mapping used by the saliency-guided dispersion penalty. Higher weights w more strongly penalize perturbation concentration in that region.
Image Quality
Protection
r (%)
SSIM ↑
LPIPS ↓
BRISQUE ↑
SER-FIQ ↓
FDFR ↑
ISM ↓
2
0.99
0.02
26.68
0.67
0.16
0.62
3
0.98
0.04
43.62
0.33
0.61
0.33
4
0.98
0.05
53.26
0.26
0.73
0.28
5
0.97
0.07
80.27
0.06
0.97
0.07
8
0.95
0.10
90.93
0.12
0.85
0.08
Appendix
Table 6: Effect of sparsity ratio r on CelebA-HQ. Prompt: a photo of sks person .
Figure 7: Hyperparameter sensitivity on CelebA-HQ (prompt: a photo of sks person ). LPIPS ( ↓ , perceptual cost) and ISM ( ↓ , protection) versus (a) the dispersion weight λ , (b) the density threshold τ , and (c) the number of optimization steps T ; the default setting is highlighted in each panel.
Setting
Target
FDFR ↑
ISM ↓
Surrogate
SD 2.1
0.97
0.07
Cross-backbone
SD 1.5
0.99
0.03
SDXL
0.69
0.56
Cross-method
LoRA
0.67
0.26
Textual Inversion
0.91
0.03
Appendix
Table 7: Quantitative cross-model and cross-method transferability on CelebA-HQ (prompt: a photo of sks person ). Protection is optimized on the SD 2.1 surrogate and evaluated on unseen backbones and personalization methods.
Figure 8: Cross-model and cross-method transferability on CelebA-HQ and VGGFace2. Columns: clean image, our protected image, and personalized generations under the surrogate (SD 2.1), two unseen backbones (SD 1.5, SDXL), and two unseen personalization methods (LoRA, Textual Inversion). All generations use the prompt “a photo of sks person” . The protected identity fails to be reproduced across all settings.
Default
λ=0
Transform
Param
BRISQUE ↑
FDFR ↑
ISM ↓
BRISQUE ↑
FDFR ↑
ISM ↓
None
–
82.34
0.89
0.09
102.16
1.00
0.00
JPEG
q=85
15.15
0.03
0.66
14.33
0.02
0.63
q=75
17.39
0.03
0.66
17.21
0.03
0.62
q=55
22.39
0.03
0.66
22.41
0.03
0.65
Blur
r=1
37.55
0.08
0.67
37.03
0.10
0.65
Appendix
Table 8: Image pre-processing on VGGFace2 for the default and λ=0 settings. Prompt: a photo of sks person .
Protection (P3)
Protection (P4)
Dataset
Method
BRISQUE ↑
SER-FIQ ↓
FDFR ↑
ISM ↓
BRISQUE ↑
SER-FIQ ↓
FDFR ↑
ISM ↓
VGGFace2
No Def.
20.26
0.58
0.02
0.42
22.14
0.53
0.26
0.22
Anti-DB
35.44
0.27
0.36
0.24
39.26
0.28
0.43
0.11
Mist
27.19
0.26
0.48
0.12
28.04
0.36
0.66
0.09
SDS(-)
46.72
0.58
0.07
0.40
23.40
0.51
0.23
0.19
MYOPIA
50.93
0.59
0.04
0.40
36.44
0.45
0.38
0.19
Appendix
Table 9: Protection effectiveness under additional prompts on VGGFace2 and CelebA-HQ. P3: a photo of sks person looking at the mirror ; P4: a photo of sks person in front of eiffel tower ; P5: a close-up photo of sks person, high details ; P6: a photo of sks person stand in front of the table holding a knife .
Figure 9: Per-method qualitative comparison on three identities. For each method we show the protected image, a zoomed crop, the perturbation residual, and two DreamBooth generations: Gen P1 ( “a photo of sks person” ) and Gen P4 ( “a photo of sks person in front of eiffel tower” ). Compared with baselines, our perturbation is the least visible in the zoom and residual while still preventing identity reproduction.
Figure 10: Additional qualitative results of our method across identities from CelebA-HQ and VGGFace2. Columns: clean image, and DreamBooth generations under six prompts. P1: “a photo of sks person” ; P2: “a dslr portrait of sks person” ; P3: “a photo of sks person looking at the mirror” ; P4: “a photo of sks person in front of eiffel tower” ; P5: “a close-up photo of sks person, high details” ; P6: “a photo of sks person stand in front of the table holding a knife” . Protection holds consistently across identities and prompts.
Recent advances in diffusion models have enabled high-fidelity, identity-preserving image generation for personalized applications such as digital avatars and virtual try-on systems. However, their reliance on sensitive facial reference images raises growing privacy concerns. Existing defense mechanisms are primarily designed for training-based personalization and struggle to generalize to emerging training-free approaches, due to fundamental differences in their identity integration paradigms. To bridge this gap, we propose IDGuardian-the first generalizable and model-agnostic identity protection framework capable of defending against both training-based and training-free personalization methods. IDGuardian abstracts the personalization process into two critical stages: identity extraction and identity injection. It then introduces crafted adversarial perturbations to simultaneously disrupt both stages. Specifically, it degrades the identity features extracted by external encoders and establishes an adversarial conceptual bridge that misdirects the generative trajectory away from the target identity. Extensive experiments show that IDGuardian effectively protects identity across various personalization pipelines and model architectures, while remaining robust to post-processing, adaptive attacks, and cross-dataset generalization. Our code is available at https://github.com/OpenAscent-L/IDGuardian.git.
Lizhi Xiong, Jun Li, Ziqiang Li +2
Nanjing University of Information Science and Technology
Photos of faces uploaded online are vulnerable to malicious actors who can scrape facial images from online sources and intrude on personal privacy via unauthorized use of facial recognition models. This paper presents FaceCloak, a novel personalized face privacy protection system, which can generate defensive identity-specific universal face privacy masks from a single image of a user, causing facial recognition to fail. FaceCloak introduces a three-stage personalized face perturbation learning methodology: (1) It generates a small set of high-variety synthetic face images of a person based on a single image of the person. (2) It learns face cloaking by adding more protection to key facial-identity leakage regions through iterative perturbation generation over the small set of synthetic images, effectively shifting a user's identity embedding towards a distant anchor identity and away from a similar one. (3) It generates a personalized identity-protective mask in the form of pixel-wise cloaking, which is light-weight and can be efficiently applied to any facial image of a user while maintaining good perceptual quality. Extensive experiments on three popular face datasets across ten recognition models show the effectiveness of FaceCloak compared to 29 other existing representative methods. Code is available at https://github.com/zacharyyahn/FaceCloak
Zachary Yahn, Fatih Ilhan, Tiansheng Huang +5
Georgia Institute of Technology, Atlanta, Georgia, USA
Personalized text-to-image diffusion models can reproduce a person from a handful of photographs. Image protection perturbs each image before publication so that fine-tuning on it fails, but it optimizes every perturbation from scratch, so its cost grows with the number of images. We present the protective diffusion model (PDM), which is trained once and samples the perturbation of a new image over many denoising steps. PDM distills an iterative teacher and is then post-trained with the target residual projection (TRP) loss as a reward, a stronger objective than existing ones. PDM outperforms nine baselines on faces and artworks and nearly matches its teacher at a tenth of the cost.