No Way To Steal My Face: Proactive Defense Against Identity-Preserving Personalized Generation

Lizhi Xiong; Jun Li; Ziqiang Li; Weiwei Jiang; Zhangjie Fu

cs.CVOct 14, 2026

No Way To Steal My Face: Proactive Defense Against Identity-Preserving Personalized Generation

Authors: Lizhi Xiong, Jun Li, Ziqiang Li, Weiwei Jiang, Zhangjie Fu

Organizations: Nanjing University of Information Science and Technology

Abstract

Recent advances in diffusion models have enabled high-fidelity, identity-preserving image generation for personalized applications such as digital avatars and virtual try-on systems. However, their reliance on sensitive facial reference images raises growing privacy concerns. Existing defense mechanisms are primarily designed for training-based personalization and struggle to generalize to emerging training-free approaches, due to fundamental differences in their identity integration paradigms. To bridge this gap, we propose IDGuardian-the first generalizable and model-agnostic identity protection framework capable of defending against both training-based and training-free personalization methods. IDGuardian abstracts the personalization process into two critical stages: identity extraction and identity injection. It then introduces crafted adversarial perturbations to simultaneously disrupt both stages. Specifically, it degrades the identity features extracted by external encoders and establishes an adversarial conceptual bridge that misdirects the generative trajectory away from the target identity. Extensive experiments show that IDGuardian effectively protects identity across various personalization pipelines and model architectures, while remaining robust to post-processing, adaptive attacks, and cross-dataset generalization. Our code is available at https://github.com/OpenAscent-L/IDGuardian.git.

Figures & tables

Figure 1: Existing defense methods become ineffective in training-free personalization due to paradigm discrepancies and fusion diversity.

Figure 2: Overview of IDGuardian. The framework centers on two complementary stages of personalization: identity extraction and identity injection . To disrupt identity extraction, Cross-Encoder Identity Field Confounding decorrelates visual identity embeddings across feature spaces. To weaken identity injection, Guidance-Flow Identity Deflection redirects the denoising trajectory toward identity-neutral regions. Identity loss is computed using embeddings from the original ( $\mathbf{I}_{\text{FaceNet/CLIP}}$ ) and protected ( $\mathbf{P}_{\text{FaceNet/CLIP}}$ ) images, while distributional deviation is induced via adversarial conceptual bridge during the denoising process.

Figure 3: Visualization of Conceptual Bridges Constructed via Noise Prediction Differences in Diffusion Sampling.

Method	Ip-Adapter		Ip-Ada + XL		Blip-Diffu		InfiniteID		Photomaker		Dreambooth		InfiniteYou
Method	ISM $\downarrow$	FQ $\downarrow$	ISM $\downarrow$	FQ $\downarrow$	ISM $\downarrow$	FQ $\downarrow$	ISM $\downarrow$	FQ $\downarrow$	ISM $\downarrow$	FQ $\downarrow$	ISM $\downarrow$	FQ $\downarrow$	ISM $\downarrow$	FQ $\downarrow$
No Protected	0.382	0.334	0.582	0.456	0.336	0.331	0.675	0.495	0.327	0.456	0.525	0.332	0.673	0.493
Anti-DB [ 43 ]	0.307	0.305	0.548	0.455	0.271	0.357	0.620	0.496	0.272	0.288	0.433	0.319	0.594	0.467
SimAC [ 44 ]	0.295	0.312	0.401	0.451	0.305	0.363	0.591	0.484	0.248	0.296	0.390	0.343	0.571	0.459
AdvDM [ 19 ]	0.286	0.349	0.398	0.438	0.260	0.355	0.542	0.490	0.227	0.293	0.337	0.192	0.563	0.459
ACE [ 50 ]	0.327	0.363	0.536	0.453	0.275	0.363	0.576	0.499	0.252	0.287	0.420	0.215	0.635	0.461

Table 1: Comparison of identity protection against baseline methods. ISM is the average identity similarity (ArcFace, FaceNet, VGGFace); FQ is the FaceQNet quality score. “No Protected” denotes generation w/o defense. Best results are in bold.

	At-DB	SAC	ADM	ACE	IDP-P	IDP	IDGuar
PSNR	31.63	32.15	30.76	31.61	32.49	32.10	32.19
SSIM	0.701	0.772	0.735	0.753	0.840	0.805	0.842
Time	312.1	347.3	55.3	90.0	4089	0.42	47.0

Table 2: Comparison of protected image quality (PSNR (dB), SSIM) and time cost (s) across different methods. At-DB denotes Anti-DreamBooth , SAC denotes SimAC , ADM denotes AdvDM , ACE denotes ACE , IDP-P denotes the PGD-optimized variant of IDProtector , IDP denotes IDProtector , and IDGuar denotes our proposed IDGuardian .

Method	Ip-Adapter		Ip-Ada + XL		Blip-Diffu		InfiniteID		Photomaker		Dreambooth		InfiniteYou
Method	ISM $\downarrow$	FQ $\downarrow$	ISM $\downarrow$	FQ $\downarrow$	ISM $\downarrow$	FQ $\downarrow$	ISM $\downarrow$	FQ $\downarrow$	ISM $\downarrow$	FQ $\downarrow$	ISM $\downarrow$	FQ $\downarrow$	ISM $\downarrow$	FQ $\downarrow$
No Protected	0.382	0.334	0.582	0.456	0.336	0.331	0.675	0.495	0.327	0.456	0.525	0.332	0.673	0.493
JPEG	0.141	0.221	0.340	0.454	0.215	0.323	0.384	0.492	0.135	0.274	0.331	0.254	0.322	0.454
Blur	0.220	0.321	0.376	0.467	0.247	0.290	0.412	0.503	0.147	0.275	0.405	0.281	0.316	0.448
Noise	0.149	0.250	0.339	0.450	0.185	0.346	0.482	0.502	0.148	0.281	0.400	0.358	0.356	0.453
Impress	0.132	0.250	0.329	0.427	0.240	0.335	0.462	0.494	0.183	0.276	0.174	0.248	0.293	0.454

Table 3: Robustness evaluation against common image post-processing operations, including JPEG compression, Gaussian blur , and Gaussian noise , and attacks targeting active defenses such as Impress .

Method	Ip-Adapter		Ip-Ada + XL		Blip-Diffu		InfiniteID		Photomaker		Dreambooth		InfiniteYou
Method	ISM $\downarrow$	FQ $\downarrow$	ISM $\downarrow$	FQ $\downarrow$	ISM $\downarrow$	FQ $\downarrow$	ISM $\downarrow$	FQ $\downarrow$	ISM $\downarrow$	FQ $\downarrow$	ISM $\downarrow$	FQ $\downarrow$	ISM $\downarrow$	FQ $\downarrow$
No Protected	0.382	0.334	0.582	0.456	0.336	0.331	0.675	0.495	0.327	0.456	0.525	0.332	0.673	0.493
CLIP	0.243	0.313	0.217	0.473	0.287	0.368	0.577	0.514	0.264	0.301	0.504	0.331	0.531	0.473
Facenet	0.246	0.319	0.222	0.479	0.288	0.372	0.583	0.508	0.268	0.308	0.503	0.330	0.531	0.466
CLIP + Facenet	0.175	0.271	0.167	0.469	0.255	0.378	0.536	0.513	0.213	0.292	0.497	0.298	0.477	0.468
IDGuar (Ours)	0.036	0.282	0.126	0.450	0.220	0.351	0.444	0.494	0.142	0.273	0.336	0.220	0.301	0.452

Table 4: Ablation study of identity loss combinations and guidance directions , where G adv denotes guidance solely toward the adversarial identity distribution and G clean denotes guidance solely away from the clean identity distribution.

Figure 4: Visualization of identity suppression effectiveness for IDGuardian and comparison methods across different personalization settings.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

No Protected			JPEG (85)			Blur (5 × 5, 1.5)			Noise (100)
Arc $\downarrow$	Face $\downarrow$	VGG $\downarrow$	Arc $\downarrow$	Face $\downarrow$	VGG $\downarrow$	Arc $\downarrow$	Face $\downarrow$	VGG $\downarrow$	Arc $\downarrow$	Face $\downarrow$	VGG $\downarrow$
0.4985	0.6899	0.5573	0.3209	0.3983	0.3006	0.3317	0.4443	0.3520	0.3078	0.3970	0.3113
No Protected			JPEG (70)			Blur (7 × 7, 1.5)			Noise (400)
Arc $\downarrow$	Face $\downarrow$	VGG $\downarrow$	Arc $\downarrow$	Face $\downarrow$	VGG $\downarrow$	Arc $\downarrow$	Face $\downarrow$	VGG $\downarrow$	Arc $\downarrow$	Face $\downarrow$	VGG $\downarrow$
0.4985	0.6899	0.5573	0.3520	0.4863	0.3730	0.3303	0.4473	0.3338	0.3955	0.4778	0.3746
No Protected			JPEG (50)			Blur (11 × 11, 1.5)			Noise (900)

Appendix

Table 5: Robustness evaluation under different distortion levels. The results are reported on Ip-Adapter Plus XL as an example.

Figure 5: Statistical analysis of identity similarity (ISM) under IP-Adapter. Error bars denote the variability across repeated runs.

Figure 6: Visual results under three levels of JPEG compression, Gaussian blur, and Gaussian noise, showing that our protection mechanism remains effective even under stronger distortions.

Figure 7: Grid search results for perturbation budget $\epsilon$ and learning rate $\alpha$ . The configuration $\alpha=0.005$ , $\epsilon=8/255$ achieves the best trade-off between identity suppression (measured by ArcFace, FaceNet, and VGGFace similarity) and perceptual quality (SSIM and PSNR).

Method	Ip-Adapter			Ip-Adapter Plus XL			Blip-Diffusion			InfiniteID
Method	Arc $\downarrow$	Face $\downarrow$	VGG $\downarrow$	Arc $\downarrow$	Face $\downarrow$	VGG $\downarrow$	Arc $\downarrow$	Face $\downarrow$	VGG $\downarrow$	Arc $\downarrow$	Face $\downarrow$	VGG $\downarrow$
No Protected	0.3369	0.4774	0.3308	0.4985	0.6899	0.5573	0.3305	0.3918	0.2848	0.6345	0.7627	0.6286
IDGuardian-SD1.5	0.0425	0.0119	0.0650	0.1589	0.1413	0.1403	0.2670	0.2550	0.2204	0.4338	0.4989	0.4113
Method	Photomaker			Dreambooth			InfiniteYou			Mean
Method	Arc $\downarrow$	Face $\downarrow$	VGG $\downarrow$	Arc $\downarrow$	Face $\downarrow$	VGG $\downarrow$	Arc $\downarrow$	Face $\downarrow$	VGG $\downarrow$	Arc $\downarrow$	Face $\downarrow$	VGG $\downarrow$
No Protected	0.3488	0.3513	0.2806	0.4681	0.6580	0.4477	0.6788	0.7554	0.5847	0.4708	0.5838	0.4450

Appendix

Table 6: Identity Suppression Performance Across Surrogate Generative Models. The metrics Arc, Face, and VGG correspond to identity similarity scores computed by the face recognition models ArcFace, FaceNet, and VGGFace, respectively.

Figure 8: Identity loss convergence over optimization steps with shaded band indicating fluctuation bounds ( $\pm 0.1$ ) around the mean curve. The identity loss stabilizes after approximately 200 steps, guiding the selection of the default iteration count.

Figure 9: Visualization of generalization results on commercial face personalization APIs.

Figure 10: Visual comparison of identity suppression results across different surrogate generative models.

Figure 11: Comparison of Identity Protection Effectiveness Based on Human Perception and GPT-Based Evaluation.

Method	Ip-Adapter			Ip-Adapter Plus XL			Blip-Diffusion			InfiniteID
Method	Arc $\downarrow$	Face $\downarrow$	VGG $\downarrow$	Arc $\downarrow$	Face $\downarrow$	VGG $\downarrow$	Arc $\downarrow$	Face $\downarrow$	VGG $\downarrow$	Arc $\downarrow$	Face $\downarrow$	VGG $\downarrow$
No Protected	0.3369	0.4774	0.3308	0.4985	0.6899	0.5573	0.3305	0.3918	0.2848	0.6345	0.7627	0.6286
Anti-DB	0.2525	0.3964	0.2715	0.4815	0.6419	0.5219	0.2685	0.3153	0.2286	0.5897	0.6859	0.5848
SimAC	0.2048	0.4005	0.2792	0.3454	0.4634	0.3956	0.3238	0.3344	0.2563	0.5472	0.6670	0.5599
AdvDM	0.2409	0.3624	0.2535	0.3427	0.4810	0.3700	0.2461	0.2989	0.2363	0.5069	0.6107	0.5087
ACE	0.2573	0.4249	0.2996	0.4615	0.6377	0.5079	0.2738	0.3160	0.2336	0.5419	0.6489	0.5383

Appendix

Table 7: Comparison of identity protection performance against baseline methods. Evaluation is conducted using three different identity similarity metrics: ArcFace, FaceNet, and VGGFace. “No Protected” denotes generation without defense.

Method	Ip-Adapter		Ip-Adapter Plus XL		Blip-Diffusion		InfiniteID
Method	FaceQNet $\downarrow$	MagFace $\downarrow$	FaceQNet $\downarrow$	MagFace $\downarrow$	FaceQNet $\downarrow$	MagFace $\downarrow$	FaceQNet $\downarrow$	MagFace $\downarrow$
No Protected	0.3339	21.3847	0.4564	19.3085	0.3305	21.8112	0.4946	19.0747
Anti-DB	0.3045	21.1209	0.4553	18.4376	0.3565	20.8991	0.4957	18.9697
SimAC	0.3124	20.8217	0.4492	18.9702	0.3628	21.4539	0.4844	19.0755
AdvDM	0.3486	20.6235	0.4379	18.8501	0.3547	21.6632	0.4896	18.9385
ACE	0.3627	20.6235	0.4528	18.5293	0.3630	21.7078	0.4994	19.0961

Appendix

Table 8: Comparison of identity protection performance against baseline methods. Evaluation is conducted using FaceQNet and MagFace quality scores. “No Protected” denotes generation without defense.

Method	Ip-Adapter			Ip-Adapter Plus XL			Blip-Diffusion			InfiniteID
Method	Arc $\downarrow$	Face $\downarrow$	VGG $\downarrow$	Arc $\downarrow$	Face $\downarrow$	VGG $\downarrow$	Arc $\downarrow$	Face $\downarrow$	VGG $\downarrow$	Arc $\downarrow$	Face $\downarrow$	VGG $\downarrow$
No Protected	0.3369	0.4774	0.3308	0.4985	0.6899	0.5573	0.3305	0.3918	0.2848	0.6345	0.7627	0.6286
CLIP	0.2289	0.3058	0.1949	0.2032	0.2328	0.2147	0.2864	0.3078	0.2664	0.5048	0.6775	0.5478
Facenet	0.2412	0.2897	0.2058	0.2145	0.2281	0.2223	0.2948	0.2989	0.2716	0.5126	0.6843	0.5534
CLIP + Facenet	0.1578	0.2347	0.1326	0.1665	0.1707	0.1638	0.2534	0.2764	0.2355	0.4932	0.6047	0.5089
IDGuar (Ours)	0.0673	-0.0226	0.0640	0.1296	0.1231	0.1260	0.2255	0.2376	0.1964	0.4121	0.5215	0.3990

Appendix

Table 9: Ablation study of identity loss combinations and guidance directions, where G adv denotes guidance solely toward the adversarial identity distribution and G clean denotes guidance solely away from the clean identity distribution.

Method	Ip-Adapter		Ip-Adapter Plus XL		Blip-Diffusion		InfiniteID
Method	FaceQNet $\downarrow$	MagFace $\downarrow$	FaceQNet $\downarrow$	MagFace $\downarrow$	FaceQNet $\downarrow$	MagFace $\downarrow$	FaceQNet $\downarrow$	MagFace $\downarrow$
No Protected	0.3339	21.3847	0.4564	19.3085	0.3305	21.8112	0.4946	19.0747
CLIP	0.3129	21.7832	0.4726	20.4175	0.3683	20.9487	0.5142	19.6254
Facenet	0.3185	21.6910	0.4791	20.5083	0.3724	20.0284	0.5076	19.4939
CLIP + Facenet	0.2714	21.8426	0.4689	19.5268	0.3776	20.4123	0.5128	20.1884
IDGuar (Ours)	0.2820	20.8712	0.4503	18.6099	0.3511	20.8498	0.4938	18.8031

Appendix

Table 10: Ablation study of different identity loss combinations and guidance directions. Evaluation is conducted using FaceQNet and MagFace to measure the perceptual quality of generated images.

Method	Ip-Adapter			Ip-Adapter Plus XL			Blip-Diffusion			InfiniteID
Method	Arc $\downarrow$	Face $\downarrow$	VGG $\downarrow$	Arc $\downarrow$	Face $\downarrow$	VGG $\downarrow$	Arc $\downarrow$	Face $\downarrow$	VGG $\downarrow$	Arc $\downarrow$	Face $\downarrow$	VGG $\downarrow$
No Protected	0.3369	0.4774	0.3308	0.4985	0.6899	0.5573	0.3305	0.3918	0.2848	0.6345	0.7627	0.6286
JPEG	0.1068	0.1900	0.1263	0.3209	0.3983	0.3006	0.2236	0.2196	0.2026	0.4286	0.5158	0.4062
Blur	0.1678	0.2948	0.1967	0.3317	0.4443	0.3520	0.2663	0.2663	0.2077	0.3854	0.4780	0.3734
Noise	0.1365	0.1794	0.1295	0.3078	0.3970	0.3113	0.1916	0.1910	0.1716	0.4482	0.5581	0.4391
Impress	0.1018	0.1696	0.1230	0.3184	0.3735	0.2956	0.2261	0.2653	0.2277	0.4478	0.5309	0.4074

Appendix

Table 11: Robustness evaluation against common image post-processing operations, including JPEG compression, Gaussian blur, and Gaussian noise, and attacks targeting active defenses such as Impress.

Method	Ip-Adapter		Ip-Adapter Plus XL		Blip-Diffusion		InfiniteID
Method	FaceQNet $\downarrow$	MagFace $\downarrow$	FaceQNet $\downarrow$	MagFace $\downarrow$	FaceQNet $\downarrow$	MagFace $\downarrow$	FaceQNet $\downarrow$	MagFace $\downarrow$
No Protected	0.3339	21.3847	0.4564	19.3085	0.3305	21.8112	0.4946	19.0747
JPEG	0.2211	21.3679	0.4539	18.5163	0.3233	21.7881	0.4918	18.8227
Blur	0.3211	21.3901	0.4667	19.7750	0.2895	19.8607	0.5031	18.8229
Noise	0.2499	20.6822	0.4497	18.4623	0.3460	20.6286	0.5016	19.0230
Impress	0.2501	20.7604	0.4270	19.6015	0.3353	20.9642	0.4935	19.1813

Appendix

Table 12: Robustness evaluation against common image post-processing operations. Evaluation is conducted using FaceQNet and MagFace to measure the perceptual quality of generated images.

Method	Ip-Adapter			Ip-Adapter Plus XL			Blip-Diffusion			InfiniteID
Method	Arc $\downarrow$	Face $\downarrow$	VGG $\downarrow$	Arc $\downarrow$	Face $\downarrow$	VGG $\downarrow$	Arc $\downarrow$	Face $\downarrow$	VGG $\downarrow$	Arc $\downarrow$	Face $\downarrow$	VGG $\downarrow$
No Protected	0.4104	0.4714	0.3764	0.5706	0.7152	0.6062	0.2862	0.3495	0.3061	0.6352	0.6886	0.5642
Anti-DB	0.3061	0.3647	0.3193	0.4940	0.6321	0.5260	0.2257	0.3369	0.2894	0.5344	0.5946	0.5031
SimAC	0.2398	0.3390	0.3021	0.4049	0.5131	0.4339	0.2499	0.3240	0.2849	0.4483	0.5262	0.4455
AdvDM	0.2827	0.3907	0.3125	0.4158	0.5007	0.4464	0.2222	0.3484	0.2680	0.5056	0.5435	0.4669
ACE	0.3462	0.4354	0.3505	0.5412	0.6673	0.5834	0.2418	0.3422	0.2880	0.4813	0.5480	0.4839

Appendix

Table 13: Comparison of identity protection performance against baseline methods on the CelebA-HQ dataset. Evaluation is conducted using three widely adopted identity similarity metrics: ArcFace, FaceNet, and VGGFace.

Method	Ip-Adapter		Ip-Adapter Plus XL		Blip-Diffusion		InfiniteID
Method	FaceQNet $\downarrow$	MagFace $\downarrow$	FaceQNet $\downarrow$	MagFace $\downarrow$	FaceQNet $\downarrow$	MagFace $\downarrow$	FaceQNet $\downarrow$	MagFace $\downarrow$
No Protected	0.3031	20.6785	0.4490	18.8023	0.3344	20.9884	0.4527	19.6607
Anti-DB	0.2599	19.5696	0.4515	18.5281	0.3170	21.4418	0.4731	18.9686
SimAC	0.2670	19.9800	0.4493	19.2084	0.3433	19.4777	0.4695	19.0426
AdvDM	0.2842	19.9800	0.4505	19.2084	0.2941	20.0386	0.4645	19.2935
ACE	0.2928	20.0025	0.4540	19.3925	0.3363	20.7060	0.4539	19.1724

Appendix

Table 14: Comparison of identity protection performance against baseline methods on the CelebA-HQ dataset. Evaluation is conducted using FaceQNet and MagFace to measure the perceptual quality of generated images.

Method	IP-Adapter $\downarrow$	IP-Ada. + XL $\downarrow$	BLIP-D. $\downarrow$	InfiniteID $\downarrow$	PhotoM. $\downarrow$	DreamB. $\downarrow$	InfiniteYou $\downarrow$	Mean $\downarrow$
No Protected	0.3580	0.5210	0.3010	0.6150	0.3420	0.4550	0.6670	0.4656
Anti-DB	0.2550	0.4890	0.2490	0.5720	0.2300	0.3380	0.5980	0.3901
SimAC	0.2070	0.3620	0.2760	0.5490	0.2200	0.2870	0.5570	0.3511
AdvDM	0.2320	0.3480	0.2410	0.5030	0.2050	0.2540	0.5410	0.3320
ACE	0.2480	0.4620	0.2610	0.5330	0.2270	0.3210	0.6400	0.3846
IDProtector	0.2130	0.3310	0.2380	0.4920	0.2180	0.3160	0.5090	0.3310

Appendix

Table 15: ArcFace-only identity similarity on LFW. Method names are abbreviated for brevity.

Figure 12: Robustness Evaluation Against Image Post-Processing and Defense Attacks.

Figure 13: Visualization of the ablation study on identity loss combinations and guidance directions.

Figure 14: Visualization of Identity Suppression Effectiveness on CelebA-HQ Dataset for IDGuardian and Comparison Methods.

Identity-Preserving Image Generation

Identity-preserving image generation creates or edits images while retaining a person's recognizable features across changes in pose, style, or scene.

See all 48 papers

IDDM: Identity-Decoupled Personalized Diffusion Models with a Tunable Privacy-Utility Trade-off
Apr 1, 2026Linyan Dai, Xinwei Zhang, Haoyang Li +2T2I Diffusion Models Image Generation
The Hong Kong Polytechnic University, Hong Kong, China
PersGuard: Preventing Malicious Personalization in Text-to-Image Diffusion Models via Model Backdoors
Feb 22, 2025Xinwei Liu, Xiaojun Jia, Yuan Xun +2T2I Generation T2I Diffusion Models
Institute of Information Engineering, Chinese Academy of Sciences, Beijing 100093, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing 100049, China · BraneMatrix AI, Shanghai 201203, China +1
Hierarchical Anti-Aesthetics: Protecting Facial Privacy against Customized Diffusion Models
Jul 2, 2026Songping Wang, Yueming Lyu, Shiqi Liu +5Adversarial Attacks Adversarial Attacks on Diffusion Models
School of Intelligence Science and Technology, Nanjing University, Suzhou 215163, China · China Mobile Information Technology Co., Ltd., Beijing 100037, China · State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), Center for Research on Intelligent Perception and Computing (CRIPAC), Institute of Automation Chinese Academy of Sciences (CASIA), Beijing 100190, China

Apr 1, 2026J/K move · Enter open · S save

IDDM: Identity-Decoupled Personalized Diffusion Models with a Tunable Privacy-Utility Trade-off

Linyan Dai, Xinwei Zhang, Haoyang Li, Qingqing Ye+1

The Hong Kong Polytechnic University, Hong Kong, China

T2I Diffusion Models Image Generation T2I Generation Safe T2I Generation Diffusion Model Fine-Tuning

Open paper AskPDF arXiv

Personalized text-to-image diffusion models (e.g., DreamBooth, LoRA) enable users to synthesize high-fidelity avatars from a few reference photos for social expression. However, once these generations are shared on social media platforms (e.g., Instagram, Facebook), they can be linked to the real user via face recognition systems, enabling identity tracking and profiling. Existing defenses mainly follow an anti-personalization strategy that protects publicly released reference photos by disrupting model fine-tuning. While effective against unauthorized personalization, they do not address another practical setting in which personalization is authorized, but the resulting public outputs still leak identity information. To address this problem, we introduce a new defense setting, termed model-side output immunization, whose goal is to produce a personalized model that supports authorized personalization while reducing the identity linkability of public generations, with tunable control over the privacy-utility trade-off to accommodate diverse privacy needs. To this end, we propose Identity-Decoupled personalized Diffusion Models (IDDM), a model-side defense that integrates identity decoupling into the personalization pipeline. Concretely, IDDM follows an alternating procedure that interleaves short personalization updates with identity-decoupled data optimization, using a two-stage schedule to balance identity linkability suppression and generation utility. Extensive experiments across multiple datasets, diverse prompts, and state-of-the-art face recognition systems show that IDDM consistently reduces identity linkability while preserving high-quality personalized generation.

Abstract

Identity-Preserving Image Generation

Explore similar work

IDDM: Identity-Decoupled Personalized Diffusion Models with a Tunable Privacy-Utility Trade-off

PersGuard: Preventing Malicious Personalization in Text-to-Image Diffusion Models via Model Backdoors

Hierarchical Anti-Aesthetics: Protecting Facial Privacy against Customized Diffusion Models