IDDM: Identity-Decoupled Personalized Diffusion Models with a Tunable Privacy-Utility Trade-off

Linyan Dai; Xinwei Zhang; Haoyang Li; Qingqing Ye; Haibo Hu

cs.CVApr 1, 2026

IDDM: Identity-Decoupled Personalized Diffusion Models with a Tunable Privacy-Utility Trade-off

Authors: Linyan Dai, Xinwei Zhang, Haoyang Li, Qingqing Ye, Haibo Hu

Organizations: The Hong Kong Polytechnic University, Hong Kong, China

Abstract

Personalized text-to-image diffusion models (e.g., DreamBooth, LoRA) enable users to synthesize high-fidelity avatars from a few reference photos for social expression. However, once these generations are shared on social media platforms (e.g., Instagram, Facebook), they can be linked to the real user via face recognition systems, enabling identity tracking and profiling. Existing defenses mainly follow an anti-personalization strategy that protects publicly released reference photos by disrupting model fine-tuning. While effective against unauthorized personalization, they do not address another practical setting in which personalization is authorized, but the resulting public outputs still leak identity information. To address this problem, we introduce a new defense setting, termed model-side output immunization, whose goal is to produce a personalized model that supports authorized personalization while reducing the identity linkability of public generations, with tunable control over the privacy-utility trade-off to accommodate diverse privacy needs. To this end, we propose Identity-Decoupled personalized Diffusion Models (IDDM), a model-side defense that integrates identity decoupling into the personalization pipeline. Concretely, IDDM follows an alternating procedure that interleaves short personalization updates with identity-decoupled data optimization, using a two-stage schedule to balance identity linkability suppression and generation utility. Extensive experiments across multiple datasets, diverse prompts, and state-of-the-art face recognition systems show that IDDM consistently reduces identity linkability while preserving high-quality personalized generation.

Figures & tables

Figure 1. Comparison of protection paradigms under different threat models in personalized text-to-image generation: prior anti-personalization training (left) versus our model-side output immunization (right).

Figure 2. Visualization examples on VGGFace2 with “a dslr portrait of sks person" under the same experimental setting. The white number below each generated image denotes the cosine similarity to the corresponding clean identity embedding (lower is less identity-related).

Figure 3. The framework of our proposed IDDM. Each iteration alternates between a short personalization update on the fixed reference set $\mathcal{X}_{c}$ (Step 1) and an identity-decoupled data update on the protection set $\mathcal{X}_{0}$ , followed by fine-tuning on the updated set $\mathcal{X}^{\prime}$ (Steps 2–3). In Step 2, the tunable stage split ratio $\rho$ determines the relative allocation of optimization between Stage I and Stage II, allowing IDDM to operate at different privacy-utility trade-off points.

Dataset	Method	“a photo of sks person”						“a dslr portrait of sks person”
Dataset	Method	FSR $\uparrow$	ISM $\downarrow$	ADA $\downarrow$	FID $\downarrow$	SER $\uparrow$	BRQ $\downarrow$	FSR $\uparrow$	ISM $\downarrow$	ADA $\downarrow$	FID $\downarrow$	SER $\uparrow$	BRQ $\downarrow$
CelebA-HQ	No Defense	0.57	0.69	0.66	158.09	0.76	10.65	0.50	0.48	0.44	214.35	0.74	4.88
	Anti-DB	0.50	0.38	0.34	314.67	0.52	40.21	0.45	0.28	0.24	341.23	0.59	36.06
	SimAC	0.30	0.21	0.17	395.81	0.27	37.34	0.15	0.03	0.03	458.58	0.22	42.23
	Anti-Diff	0.33	0.13	0.11	383.80	0.26	41.96	0.32	0.16	0.13	417.32	0.38	37.70
	IDDM ( $\rho=0.1$ )	0.65	0.43	0.38	213.37	0.74	16.81	0.48	0.29	0.25	237.54	0.75	11.71

Table 1. Defense performance comparison with anti-personalization baselines on CelebA-HQ and VGGFace2 datasets. IDDM is reported under three values of the preset step-allocation ratio

\rho

, which controls the trade-off between defense strength and generation utility. FSR measures output utility; ISM and ADA denote identity-related metrics; FID, SER, and BRQ denote the visual quality of generated images.

Method	“a photo of sks person wearing a hat”						“a photo of sks person with glasses”
Method	FSR $\uparrow$	ISM $\downarrow$	ADA $\downarrow$	FID $\downarrow$	SER $\uparrow$	BRQ $\downarrow$	FSR $\uparrow$	ISM $\downarrow$	ADA $\downarrow$	FID $\downarrow$	SER $\uparrow$	BRQ $\downarrow$
No Defense	$0.73$	$0.54$	$0.50$	$273.25$	$0.70$	$8.31$	$0.33$	$0.55$	$0.50$	$208.03$	$0.75$	$13.89$
IDDM ( $\rho=0.1$ )	$0.70$	0.34	0.29	$275.65$	$0.70$	12.31	$0.29$	0.35	0.30	$235.79$	$0.75$	15.43
IDDM ( $\rho=0.3$ )	0.73	$0.44$	$0.41$	$272.79$	0.72	$18.18$	0.30	$0.45$	$0.41$	$194.94$	0.77	$19.72$
IDDM ( $\rho=0.5$ )	$0.70$	$0.47$	$0.44$	260.65	0.72	$20.85$	$0.26$	$0.49$	$0.44$	192.21	$0.76$	$22.39$
Method	“a photo of sks person with red hair”						“a photo of sks person with big eyes”

Table 2. Defense performance of IDDM under four additional prompts with different values of the tunable step-allocation ratio

\rho

on CelebA-HQ. “No Defense” denotes the standard DreamBooth personalization without defense.

Method	‘a photo of sks person”					‘a dslr portrait of sks person”
Method	ISM $\downarrow$	ADA $\downarrow$	FID $\downarrow$	SER $\uparrow$	BRQ $\downarrow$	ISM $\downarrow$	ADA $\downarrow$	FID $\downarrow$	SER $\uparrow$	BRQ $\downarrow$
No Defense	0.69	0.66	158.09	0.76	10.65	0.48	0.44	214.35	0.74	4.88
AdvDM	0.38	0.35	228.65	0.74	14.73	0.42	0.39	254.70	0.72	14.82
LowKey	0.64	0.62	214.04	0.74	8.23	0.44	0.42	255.95	0.74	17.14
GIFT	0.40	0.36	170.79	0.75	32.71	0.29	0.26	216.42	0.74	39.62
IDDM	0.43	0.38	213.37	0.74	16.81	0.28	0.25	237.54	0.75	11.71

Table 3. Defense performance comparison with post-processing baselines on CelebA-HQ. IDDM is reported under

\rho=0.1

Method	“a photo of sks person”								“a dslr portrait of sks person”
Method	Ada. $\downarrow$	QMag. $\downarrow$	Elas. $\downarrow$	R18 $\downarrow$	R50 $\downarrow$	R100 $\downarrow$	MFN $\downarrow$	Avg. $\downarrow$	Ada. $\downarrow$	QMag. $\downarrow$	Elas. $\downarrow$	R18 $\downarrow$	R50 $\downarrow$	R100 $\downarrow$	MFN $\downarrow$	Avg. $\downarrow$
No Defense	95.39	89.40	95.85	94.24	96.08	95.85	93.78	94.37	76.98	30.16	76.98	72.75	82.54	85.71	61.11	69.46
AdvDM	57.04	42.96	57.04	55.80	57.78	57.78	53.33	54.53	63.86	19.84	69.02	59.78	73.64	77.17	43.75	58.15
LowKey	95.97	88.63	96.45	95.02	96.45	95.97	93.13	94.52	76.47	27.81	77.01	70.05	79.14	82.89	52.94	66.62
GIFT	72.62	12.86	73.57	64.29	83.81	84.29	61.19	64.66	28.57	2.96	29.11	24.26	32.61	38.54	16.71	24.68
IDDM	69.57	21.12	73.64	56.01	80.81	83.33	48.64	61.87	25.91	1.55	29.02	19.69	35.23	40.41	10.10	23.13

Table 4. True Accept Rate (TAR, %) at FAR=1e-4 across seven face recognition models on CelebA-HQ.

Figure 4. Qualitative defense results on CelebA-HQ. First row: reference image. Second row: generated images on the prompt “a dslr portrait of sks person". Third row: generated images on the prompt “a photo of sks person".

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

$\rho$	“a photo of sks person”						“a dslr portrait of sks person”
$\rho$	FSR $\uparrow$	ISM $\downarrow$	ADA $\downarrow$	FID $\downarrow$	SER $\uparrow$	BRQ $\downarrow$	FSR $\uparrow$	ISM $\downarrow$	ADA $\downarrow$	FID $\downarrow$	SER $\uparrow$	BRQ $\downarrow$
No defense	$0.57$	$0.69$	$0.66$	$158.09$	$0.76$	$10.65$	$0.50$	$0.48$	$0.44$	$214.35$	$0.74$	$4.88$
0.1	0.65	0.43	0.38	$213.37$	$0.74$	16.81	$0.48$	0.29	0.25	$237.54$	$0.75$	$11.71$
0.2	$0.63$	$0.49$	$0.42$	$186.49$	$0.74$	$19.67$	$0.50$	$0.34$	$0.30$	$219.37$	$0.75$	11.19
0.3	$0.62$	$0.51$	$0.47$	$172.80$	0.75	$21.31$	$0.52$	$0.36$	$0.33$	$208.40$	$0.75$	$12.56$
0.5	$0.62$	$0.53$	$0.49$	$171.77$	0.75	$20.87$	0.56	$0.40$	$0.37$	$203.89$	$0.77$	$13.65$

Appendix

Table 5. Defense performance of IDDM on CelebA-HQ under different values of the tunable ratio

\rho

$\eta$	Quality			“a photo of sks person”					“a dslr portrait of sks person”
$\eta$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FSR $\uparrow$	ISM $\downarrow$	ADA $\downarrow$	FID $\downarrow$	BRQ $\downarrow$	FSR $\uparrow$	ISM $\downarrow$	ADA $\downarrow$	FID $\downarrow$	BRQ $\downarrow$
0	–	–	–	$0.57$	$0.69$	$0.66$	$158.09$	$10.65$	$0.50$	$0.48$	$0.44$	$214.35$	$4.88$
0.05	$35.12$	$0.90$	$0.11$	$0.60$	$0.49$	$0.43$	$203.15$	$21.78$	$0.46$	$0.37$	$0.33$	$217.82$	$7.17$
0.08 *	$32.83$	$0.84$	$0.21$	$0.65$	$0.43$	$0.38$	$213.37$	$16.81$	$0.48$	$0.29$	$0.25$	$237.54$	$11.71$
0.10	$31.61$	$0.80$	$0.26$	$0.64$	$0.39$	$0.33$	$232.58$	$15.81$	$0.43$	$0.24$	$0.20$	$260.44$	$15.77$
0.15	$30.00$	$0.74$	$0.34$	$0.70$	$0.31$	$0.26$	$248.72$	$15.83$	$0.44$	$0.19$	$0.15$	$281.81$	$19.63$

Appendix

Table 6. Effect of bounded budget on quality and identity-related metrics on CelebA-HQ.

$\tau$	“a photo of sks person”						“a dslr portrait of sks person”
$\tau$	FSR $\uparrow$	ISM $\downarrow$	ADA $\downarrow$	FID $\downarrow$	SER $\uparrow$	BRQ $\downarrow$	FSR $\uparrow$	ISM $\downarrow$	ADA $\downarrow$	FID $\downarrow$	SER $\uparrow$	BRQ $\downarrow$
No defense	$0.57$	$0.69$	$0.66$	$158.09$	$0.76$	$10.65$	$0.50$	$0.48$	$0.44$	$214.35$	$0.74$	$4.88$
0.01	$0.59$	$0.43$	$0.38$	$218.76$	$0.74$	$16.90$	0.49	$0.28$	$0.24$	$245.25$	$0.75$	$12.69$
0.03 ∗	0.65	$0.43$	$0.38$	213.37	$0.74$	$16.81$	$0.48$	$0.29$	$0.25$	237.54	$0.75$	11.71
0.05	$0.64$	$0.43$	$0.38$	$213.52$	0.75	16.70	$0.48$	$0.27$	$0.23$	$239.04$	$0.75$	$12.18$
0.10	$0.65$	0.40	0.35	$216.65$	$0.74$	$18.42$	$0.44$	0.26	0.23	$245.03$	$0.75$	$13.34$

Appendix

Table 7. Defense performance of IDDM on CelebA-HQ under different values of

\tau

defined in Eq. (12). ∗ denotes the default setting in our experiments.

Opt. FR ensemble	“a photo of sks person”						“a dslr portrait of sks person”
Opt. FR ensemble	FSR $\uparrow$	ISM $\downarrow$	ADA $\downarrow$	FID $\downarrow$	SER $\uparrow$	BRQ $\downarrow$	FSR $\uparrow$	ISM $\downarrow$	ADA $\downarrow$	FID $\downarrow$	SER $\uparrow$	BRQ $\downarrow$
No Defense	0.57	0.69	0.66	158.09	0.76	10.65	0.50	0.48	0.44	214.35	0.74	4.88
R50	0.66	0.35	0.30	228.07	0.73	18.90	0.42	0.23	0.19	249.17	0.76	14.15
R50+R152	0.64	0.38	0.33	212.31	0.72	19.19	0.44	0.24	0.20	247.83	0.74	13.08
R50+R152+MF	0.66	0.43	0.37	219.88	0.73	18.79	0.44	0.29	0.24	245.15	0.75	12.59
R50+R152+MF+FN	0.65	0.43	0.38	213.37	0.74	16.81	0.48	0.29	0.29	237.54	0.75	11.71

Appendix

Table 8. Ablation of the face recognition (FR) ensemble used during optimization. R50, R152, MF, and FN denote IRSE50, IR152, MobileFace, and FaceNet, respectively.

$K$	FSR $\uparrow$	ISM $\downarrow$	ADA $\downarrow$	FID $\downarrow$	SER $\uparrow$	BRQ $\downarrow$
No defense	0.57	0.69	0.66	158.09	0.76	10.65
10	0.65	0.43	0.38	213.37	0.74	16.81
20	0.63	0.42	0.36	217.14	0.74	17.83
30	0.67	0.38	0.33	228.20	0.72	18.59
50	0.62	0.38	0.33	228.66	0.72	20.14

Appendix

Table 9. Defense performance with different outer iterations on CelebA-HQ under prompt “ a photo of sks person" .

Version	Method	“a photo of sks person”						“a dslr portrait of sks person”
Version	Method	FSR $\uparrow$	ISM $\downarrow$	ADA $\downarrow$	FID $\downarrow$	SER $\uparrow$	BRQ $\downarrow$	FSR $\uparrow$	ISM $\downarrow$	ADA $\downarrow$	FID $\downarrow$	SER $\uparrow$	BRQ $\downarrow$
V2.1	No Defense	0.57	0.69	0.66	158.09	0.76	10.65	0.50	0.48	0.44	214.35	0.74	4.88
V2.1	Defense	0.65	0.43	0.38	213.37	0.74	16.81	0.48	0.29	0.25	237.54	0.75	11.71
V1.5	No Defense	0.67	0.67	0.63	160.42	0.75	11.96	0.76	0.37	0.33	211.26	0.69	6.94
V1.5	Defense	0.77	0.43	0.38	181.13	0.73	26.93	0.70	0.28	0.24	189.30	0.71	18.63
V1.4	No Defense	0.69	0.66	0.63	160.53	0.75	13.56	0.69	0.34	0.30	225.13	0.67	7.99

Appendix

Table 10. Defense performance comparison of IDDM across different diffusion model versions on CelebA-HQ.

Method	“a photo of sks person"						“a dslr portrait of sks person"
Method	FSR $\uparrow$	ISM $\downarrow$	ADA $\downarrow$	FID $\downarrow$	SER $\uparrow$	BRQ $\downarrow$	FSR $\uparrow$	ISM $\downarrow$	ADA $\downarrow$	FID $\downarrow$	SER $\uparrow$	BRQ $\downarrow$
No defense	0.57	0.69	0.66	158.09	0.76	10.65	0.50	0.48	0.44	214.35	0.74	4.88
Joint Opt.	0.65	0.43	0.39	220.14	0.75	16.97	0.38	0.29	0.25	250.06	0.75	13.27
Stage-wise Opt.	0.65	0.43	0.38	213.37	0.74	16.81	0.48	0.29	0.25	237.54	0.75	11.71

Appendix

Table 11. Comparing the defense performance between the joint optimization and the stage-wise optimization on CelebA-HQ under a convenient setting.

Method	“a photo of sks person”						“a dslr portrait of sks person”
Method	FSR $\uparrow$	ISM $\downarrow$	ADA $\downarrow$	FID $\downarrow$	SER $\uparrow$	BRQ $\downarrow$	FSR $\uparrow$	ISM $\downarrow$	ADA $\downarrow$	FID $\downarrow$	SER $\uparrow$	BRQ $\downarrow$
No Defense	0.59	0.62	0.58	179.41	0.77	12.60	0.70	0.16	0.13	266.44	0.70	10.86
IDDM ( $\rho{=}0.1$ )	0.68	0.43	0.38	236.83	0.76	19.19	0.69	0.10	0.08	279.48	0.69	18.03
IDDM ( $\rho{=}0.3$ )	0.59	0.55	0.51	164.25	0.79	18.98	0.70	0.12	0.10	259.69	0.72	13.81
IDDM ( $\rho{=}0.5$ )	0.58	0.59	0.56	160.16	0.78	17.19	0.69	0.14	0.12	250.34	0.71	12.54

Appendix

Table 12. Defense performance of IDDM under LoRA-based personalization with different values of the tunable ratio

\rho

on CelebA-HQ.

Method	“a photo of sks person”								“a dslr portrait of sks person”
	AdaFace		QMagFace		ElasticFace		Average		AdaFace		QMagFace		ElasticFace		Average
	R1-U	R5-U	R1-U	R5-U	R1-U	R5-U	R1-U	R5-U	R1-U	R5-U	R1-U	R5-U	R1-U	R5-U	R1-U	R5-U
No Defense	87.06	87.06	87.06	88.24	89.41	89.41	87.84	88.24	88.61	89.87	89.87	96.20	94.94	96.20	91.14	94.09
Anti-DB	77.78	88.89	88.89	88.89	88.89	88.89	85.19	88.89	80.56	83.33	77.78	91.67	80.56	91.67	79.63	88.89
SimAC	83.33	83.33	83.33	83.33	66.67	100.00	77.78	88.89	0	0	0	0	0	0	0	0
Anti-Diff	60.00	60.00	60.00	60.00	60.00	60.00	60.00	60.00	75.00	100.00	75.00	75.00	75.00	100.00	75.00	91.67

Appendix

Table 13. Top-

k

untargeted identity retrieval attack success rates (%,

k{=}1,5

) on CelebA-HQ across AdaFace ( Kim et al., 2022 ) , QMagFace ( Terhörst et al., 2023 ) , and ElasticFace ( Boutros et al., 2022 ) . The gallery contains 112 identities, including the evaluated identities. Lower is better; best results are in bold and second-best are underlined .

Method	‘a photo of sks person”								‘a dslr portrait of sks person”
	AdaFace		QMagFace		ElasticFace		Average		AdaFace		QMagFace		ElasticFace		Average
	R1-U	R5-U	R1-U	R5-U	R1-U	R5-U	R1-U	R5-U	R1-U	R5-U	R1-U	R5-U	R1-U	R5-U	R1-U	R5-U
No Defense	99.33	99.78	99.56	99.78	99.10	99.33	99.33	99.63	93.98	95.58	94.38	97.59	93.17	95.98	93.84	96.38
Anti-DB	96.62	97.30	96.62	97.97	95.95	97.30	96.40	97.52	83.72	90.70	86.82	90.70	85.27	92.25	85.27	91.22
SimAC	98.04	98.04	98.04	98.04	96.08	96.08	97.39	97.39	0	33.33	0	33.33	0	0	0	22.22
Anti-Diff	95.83	100.00	100.00	100.00	97.92	100.00	97.92	100.00	78.33	90.00	80.00	85.00	70.00	81.67	76.11	85.56

Appendix

Table 14. Top-

k

untargeted identity retrieval attack success rates (%,

k{=}1,5

) on VGGFace2 across AdaFace ( Kim et al., 2022 ) , QMagFace ( Terhörst et al., 2023 ) , and ElasticFace ( Boutros et al., 2022 ) . Lower is better; best results are in bold and second-best are underlined .

Method	“a photo of sks person”
Method	IRSE50		IR152		FaceNet		MobileFace		Average
	R1-U	R5-U	R1-U	R5-U	R1-U	R5-U	R1-U	R5-U	R1-U	R5-U
No Defense	100.00	100.00	100.00	100.00	98.00	100.00	99.50	100.00	99.38	100.00
Anti-DB	100.00	100.00	96.43	100.00	92.86	92.86	92.86	100.00	95.54	98.22
SimAC	94.12	94.12	76.47	88.24	70.59	82.35	82.35	94.12	80.88	89.71
Anti-Diff	100.00	100.00	80.00	100.00	80.00	90.00	100.00	100.00	90.00	97.50

Appendix

Table 15. Top-

k

untargeted identity retrieval attack success rates (%,

k{=}1,5

) on CelebA-HQ across four face recognition models. The gallery contains 112 identities including the evaluated identities. Lower is better; best results are in bold and second-best are underlined .

Method	“a photo of sks person”
Method	IRSE50		IR152		FaceNet		MobileFace		Average
	R1-U	R5-U	R1-U	R5-U	R1-U	R5-U	R1-U	R5-U	R1-U	R5-U
No Defense	99.78	99.78	98.88	99.33	97.54	98.88	99.10	99.78	98.83	99.44
Anti-DB	97.30	97.97	95.27	95.95	90.54	95.95	95.27	98.65	94.60	97.13
SimAC	96.08	98.04	92.16	92.16	86.27	92.16	94.12	96.08	92.16	94.61
Anti-Diff	97.92	100.00	93.75	97.92	87.50	93.75	89.58	97.92	92.19	97.40

Appendix

Table 16. Top-

k

untargeted identity retrieval attack success rates (%,

k{=}1,5

) on VGGFace2 across four face recognition models. The gallery contains 100 identities (including evaluated identities). Lower is better; best results are in bold and second-best are underlined .

Method	ADA $\downarrow$					BRQ $\downarrow$
Method	No Preproc.	JPEG	Resize	Blur	Crop	No Preproc.	JPEG	Resize	Blur	Crop
No Defense	$0.44$	–				$4.88$	–
AdvDM	$0.39$	$0.38$	$0.36$	$0.38$	$0.39$	$14.82$	$33.14$	$16.18$	$23.21$	$64.23$
LowKey	$0.42$	$0.39$	$0.40$	$0.33$	$0.41$	$17.14$	$\mathbf{17.51}$	$18.98$	$23.32$	$58.85$
GIFT	$0.26$	$0.26$	$0.25$	$0.26$	$0.26$	$39.62$	$55.27$	$41.41$	$40.15$	$64.84$
IDDM	$\mathbf{0.25}$	$\mathbf{0.25}$	$\mathbf{0.23}$	$\mathbf{0.25}$	$\mathbf{0.25}$	$\mathbf{11.71}$	$26.21$	$\mathbf{12.88}$	$\mathbf{17.07}$	$\mathbf{58.43}$

Appendix

Table 17. Defense performance comparison with post-protection baselines under various transformations. ADA denotes the similarity scores on AdaFace, BRQ denotes the visual quality metric, and No Preproc. denotes results without transformations.

Figure 5. Comparison of generated images without defense and with defense under different prompts on CelebA-HQ.

T2I Diffusion Models

Text-to-image (T2I) diffusion models generate images from prompts by learning to reverse noise under text conditioning.

See all 393 papers

No Way To Steal My Face: Proactive Defense Against Identity-Preserving Personalized Generation
Oct 14, 2026Lizhi Xiong, Jun Li, Ziqiang Li +2Identity-Preserving Image Generation Diffusion-Based Image Generation
Nanjing University of Information Science and Technology
PersGuard: Preventing Malicious Personalization in Text-to-Image Diffusion Models via Model Backdoors
Feb 22, 2025Xinwei Liu, Xiaojun Jia, Yuan Xun +2T2I Generation T2I Diffusion Models
Institute of Information Engineering, Chinese Academy of Sciences, Beijing 100093, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing 100049, China · BraneMatrix AI, Shanghai 201203, China +1
Hierarchical Anti-Aesthetics: Protecting Facial Privacy against Customized Diffusion Models
Jul 2, 2026Songping Wang, Yueming Lyu, Shiqi Liu +5Adversarial Attacks Adversarial Attacks on Diffusion Models
School of Intelligence Science and Technology, Nanjing University, Suzhou 215163, China · China Mobile Information Technology Co., Ltd., Beijing 100037, China · State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), Center for Research on Intelligent Perception and Computing (CRIPAC), Institute of Automation Chinese Academy of Sciences (CASIA), Beijing 100190, China

Oct 14, 2026J/K move · Enter open · S save

No Way To Steal My Face: Proactive Defense Against Identity-Preserving Personalized Generation

Lizhi Xiong, Jun Li, Ziqiang Li, Weiwei Jiang+1

Nanjing University of Information Science and Technology

Identity-Preserving Image Generation Diffusion-Based Image Generation Adversarial Robustness

Open paper AskPDF arXiv

Recent advances in diffusion models have enabled high-fidelity, identity-preserving image generation for personalized applications such as digital avatars and virtual try-on systems. However, their reliance on sensitive facial reference images raises growing privacy concerns. Existing defense mechanisms are primarily designed for training-based personalization and struggle to generalize to emerging training-free approaches, due to fundamental differences in their identity integration paradigms. To bridge this gap, we propose IDGuardian-the first generalizable and model-agnostic identity protection framework capable of defending against both training-based and training-free personalization methods. IDGuardian abstracts the personalization process into two critical stages: identity extraction and identity injection. It then introduces crafted adversarial perturbations to simultaneously disrupt both stages. Specifically, it degrades the identity features extracted by external encoders and establishes an adversarial conceptual bridge that misdirects the generative trajectory away from the target identity. Extensive experiments show that IDGuardian effectively protects identity across various personalization pipelines and model architectures, while remaining robust to post-processing, adaptive attacks, and cross-dataset generalization. Our code is available at https://github.com/OpenAscent-L/IDGuardian.git.

Abstract

T2I Diffusion Models

Explore similar work

No Way To Steal My Face: Proactive Defense Against Identity-Preserving Personalized Generation

PersGuard: Preventing Malicious Personalization in Text-to-Image Diffusion Models via Model Backdoors

Hierarchical Anti-Aesthetics: Protecting Facial Privacy against Customized Diffusion Models