Text-to-image diffusion models can learn and reproduce an individual's identity from only a few reference photos, creating risks of unauthorized and misleading content generation. Proactive defenses address this risk by adding adversarial perturbations to facial images before release, thereby disrupting identity acquisition during downstream personalization. However, existing methods typically rely on dense L∞-bounded perturbations, whose accumulated distortions can become visible around facial structures. Recent methods improve protected-image quality, but often at the cost of reduced protection effectiveness. In this paper, we propose Sparse Encoder Attack on Latent Variance (SEAL), a sparse adversarial protection framework that modifies only 5% of the spatial locations in a facial image. SEAL combines a differentiable top-k masking scheme that jointly optimizes perturbation locations and values with a variance-oriented objective over the frozen VAE encoder's latent distribution to disrupt identity-consistent representations. A saliency-guided dispersion penalty further discourages perturbation concentration in perceptually salient facial regions. Experiments on CelebA-HQ and VGGFace2 show that SEAL achieves the best protected-image quality among the compared methods, exceeding 41 dB PSNR with 0.97 SSIM and at most 0.07 LPIPS, while reducing identity similarity to at most 0.15 after diffusion personalization. These results demonstrate that sparse location optimization enables strong identity protection while preserving high visual quality. Code and data are available at https://github.com/Lvxan/SEAL
Figures & tables
Figure 1: By modifying only a small fraction of pixels, SEAL preserves the visual appearance of the protected face image while disrupting identity reproduction after diffusion personalization. Compared with existing protection methods, SEAL achieves both high image quality and strong protection effectiveness.
Figure 2: Overview of SEAL . Given a clean image x , SEAL jointly optimizes a sparse binary mask m and a bounded perturbation δ to disrupt the VAE latent variance. A saliency-guided dispersion regularizer discourages the selected pixels from clustering in perceptually salient facial regions.
Figure 3: Qualitative comparison. Each method shows the protected image, a magnified crop with its perturbation pattern, and DreamBooth generations under two prompts. P2: a dslr portrait of sks person ; P3: a photo of sks person looking at the mirror .
Image Quality
Protection (P1)
Protection (P2)
Dataset
Method
SSIM ↑
PSNR ↑
LPIPS ↓
DISTS ↓
BRISQUE ↑
SER-FIQ ↓
FDFR ↑
ISM ↓
BRISQUE ↑
SER-FIQ ↓
FDFR ↑
ISM ↓
VGGFace2
No Def.
–
–
–
–
17.34
0.75
0.01
0.67
8.60
0.72
0.01
0.49
Anti-DB
0.90
34.52
0.21
0.18
35.26
0.35
0.32
0.35
40.28
0.40
0.43
0.27
Mist
0.74
28.62
0.38
0.38
20.59
0.47
0.12
0.22
32.84
0.44
0.23
0.19
SDS(-)
0.74
29.87
0.31
0.34
40.84
0.75
0.00
0.62
47.16
0.77
0.00
0.54
MYOPIA
0.92
34.35
0.11
0.22
31.25
0.72
0.00
0.55
54.28
0.77
0.02
0.51
Table 1: Comparison of protection methods on VGGFace2 and CelebA-HQ. ↑ / ↓ indicates higher/lower is better. Best in bold , second best underlined . P1: a photo of sks person ; P2: a dslr portrait of sks person .
Figure 4: Cross-model and cross-method transferability on CelebA-HQ. Bars are rescaled to a common range (taller = better). Prompt: a photo of sks person .
Ratio
BRISQUE ↑
SER-FIQ ↓
FDFR ↑
100%
67.64
0.07
0.93
75%
59.78
0.12
0.88
50%
48.71
0.31
0.69
25%
30.51
0.59
0.33
Table 2: Protection under different protected-image ratios on CelebA-HQ. Prompt: a photo of sks person .
Method
GPU Mem. (GB)
Time (s)
Anti-DB
∼ 17
∼ 420
Mist
∼ 11
∼ 120
SDS(-)
∼ 9
∼ 70
MYOPIA
∼ 18
∼ 590
VCPro
∼ 20
∼ 100
SEAL
∼ 9
∼ 70
Table 3: Efficiency comparison. Time and GPU memory are measured for protecting one four-image identity set at 512×512 resolution on a single NVIDIA A100.
Top- k
Saliency
LA
F-LPIPS ↓
LPIPS ↓
ISM ↓
w/o Saliency
✓
✓
0.05
0.06
0.02
Random Mask
✓
✓
0.03
0.05
0.68
MSE Loss
✓
✓
0.04
0.07
0.37
LE Loss
✓
✓
0.03
0.06
0.25
Full model
✓
✓
✓
0.03
0.07
0.07
Table 4: Component ablation on CelebA-HQ. Prompt: a photo of sks person .
Figure 5: Effect of the saliency-guided dispersion penalty. (a-b) Without the penalty, selected locations concentrate around the eyes and nose. (c-d) With the penalty, the selected locations are distributed more broadly across less salient regions.
Figure 6: Trade-off between protected-image distortion (LPIPS) and protection effectiveness (ISM) across sparsity ratios. The default setting r=5% provides a balanced operating point. Prompt: a photo of sks person .
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
ID
Class
w
ID
Class
w
0
Background
0.00
10
Nose
0.90
1
Skin
0.80
11
Mouth
0.90
2
L. eyebrow
0.70
12
Upper lip
0.90
3
R. eyebrow
0.70
13
Lower lip
0.90
4
L. eye
1.00
14
Neck
0.50
5
R. eye
1.00
15
Necklace
0.10
Appendix
Table 5: Complete BiSeNet class-to-weight mapping used by the saliency-guided dispersion penalty. Higher weights w more strongly penalize perturbation concentration in that region.
Image Quality
Protection
r (%)
SSIM ↑
LPIPS ↓
BRISQUE ↑
SER-FIQ ↓
FDFR ↑
ISM ↓
2
0.99
0.02
26.68
0.67
0.16
0.62
3
0.98
0.04
43.62
0.33
0.61
0.33
4
0.98
0.05
53.26
0.26
0.73
0.28
5
0.97
0.07
80.27
0.06
0.97
0.07
8
0.95
0.10
90.93
0.12
0.85
0.08
Appendix
Table 6: Effect of sparsity ratio r on CelebA-HQ. Prompt: a photo of sks person .
Figure 7: Hyperparameter sensitivity on CelebA-HQ (prompt: a photo of sks person ). LPIPS ( ↓ , perceptual cost) and ISM ( ↓ , protection) versus (a) the dispersion weight λ , (b) the density threshold τ , and (c) the number of optimization steps T ; the default setting is highlighted in each panel.
Setting
Target
FDFR ↑
ISM ↓
Surrogate
SD 2.1
0.97
0.07
Cross-backbone
SD 1.5
0.99
0.03
SDXL
0.69
0.56
Cross-method
LoRA
0.67
0.26
Textual Inversion
0.91
0.03
Appendix
Table 7: Quantitative cross-model and cross-method transferability on CelebA-HQ (prompt: a photo of sks person ). Protection is optimized on the SD 2.1 surrogate and evaluated on unseen backbones and personalization methods.
Figure 8: Cross-model and cross-method transferability on CelebA-HQ and VGGFace2. Columns: clean image, our protected image, and personalized generations under the surrogate (SD 2.1), two unseen backbones (SD 1.5, SDXL), and two unseen personalization methods (LoRA, Textual Inversion). All generations use the prompt “a photo of sks person” . The protected identity fails to be reproduced across all settings.
Default
λ=0
Transform
Param
BRISQUE ↑
FDFR ↑
ISM ↓
BRISQUE ↑
FDFR ↑
ISM ↓
None
–
82.34
0.89
0.09
102.16
1.00
0.00
JPEG
q=85
15.15
0.03
0.66
14.33
0.02
0.63
q=75
17.39
0.03
0.66
17.21
0.03
0.62
q=55
22.39
0.03
0.66
22.41
0.03
0.65
Blur
r=1
37.55
0.08
0.67
37.03
0.10
0.65
Appendix
Table 8: Image pre-processing on VGGFace2 for the default and λ=0 settings. Prompt: a photo of sks person .
Protection (P3)
Protection (P4)
Dataset
Method
BRISQUE ↑
SER-FIQ ↓
FDFR ↑
ISM ↓
BRISQUE ↑
SER-FIQ ↓
FDFR ↑
ISM ↓
VGGFace2
No Def.
20.26
0.58
0.02
0.42
22.14
0.53
0.26
0.22
Anti-DB
35.44
0.27
0.36
0.24
39.26
0.28
0.43
0.11
Mist
27.19
0.26
0.48
0.12
28.04
0.36
0.66
0.09
SDS(-)
46.72
0.58
0.07
0.40
23.40
0.51
0.23
0.19
MYOPIA
50.93
0.59
0.04
0.40
36.44
0.45
0.38
0.19
Appendix
Table 9: Protection effectiveness under additional prompts on VGGFace2 and CelebA-HQ. P3: a photo of sks person looking at the mirror ; P4: a photo of sks person in front of eiffel tower ; P5: a close-up photo of sks person, high details ; P6: a photo of sks person stand in front of the table holding a knife .
Figure 9: Per-method qualitative comparison on three identities. For each method we show the protected image, a zoomed crop, the perturbation residual, and two DreamBooth generations: Gen P1 ( “a photo of sks person” ) and Gen P4 ( “a photo of sks person in front of eiffel tower” ). Compared with baselines, our perturbation is the least visible in the zoom and residual while still preventing identity reproduction.
Figure 10: Additional qualitative results of our method across identities from CelebA-HQ and VGGFace2. Columns: clean image, and DreamBooth generations under six prompts. P1: “a photo of sks person” ; P2: “a dslr portrait of sks person” ; P3: “a photo of sks person looking at the mirror” ; P4: “a photo of sks person in front of eiffel tower” ; P5: “a close-up photo of sks person, high details” ; P6: “a photo of sks person stand in front of the table holding a knife” . Protection holds consistently across identities and prompts.