IdentityGuard: Context-Aware Restriction and Provenance for Personalized Synthesis

Lingyun Zhang; Yu Xie; Ping Chen

cs.CRMar 14, 2026

IdentityGuard: Context-Aware Restriction and Provenance for Personalized Synthesis

Authors: Lingyun Zhang, Yu Xie, Ping Chen

Organizations: School of Computer Science and Technology, Fudan University, Shanghai, China · Institute of Big Data, Fudan University, Shanghai, China · Purple Mountain Laboratories, Nanjing, China

Abstract

The nature of personalized text-to-image models poses a unique safety challenge that generic context-blind methods are ill-equipped to handle. Such global filters create a dilemma: to prevent misuse, they are forced to damage the model's broader utility by erasing concepts entirely, causing unacceptable collateral damage.Our work presents a more precisely targeted approach, built on the principle that security should be as context-aware as the threat itself, intrinsically bound to the personalized concept. We present IDENTITYGUARD, which realizes this principle through a conditional restriction that blocks harmful content only when combined with the personalized identity, and a concept-specific watermark for precise traceability. Experiments show our approach prevents misuse while preserving the model's utility and enabling robust traceability. By moving beyond blunt, global filters, our work demonstrates a more effective and responsible path toward AI safety.

Figures & tables

Figure 1: The motivation for IDENTITYGUARD. Generic, context-blind security methods (middle column) force an unacceptable trade-off: they either destroy the user’s identity when blocking a threat, or destroy the model’s utility on benign prompts. Our method, by binding safeguards directly to the personalized concept, is the only one to succeed in both scenarios. It defends against misuse while preserving the model’s performance on general prompts.

Core Security Paradigm	Method	Fidelity (Benign Prompts)		Restriction (Malicious Prompts)		Provenance
Core Security Paradigm	Method	FID $\downarrow$	CLIP $\uparrow$	FID-Censored $\downarrow$	CLIP-Censored $\downarrow$	Bit Accuracy $\uparrow$
Baseline	DreamBooth	55.81	0.3150	465.80	0.2378	N/A
Baseline	+ Post-hoc WM (HiDDeN)	56.01	-	-	-	60%
Global Guidance	SLD	60.97	0.3077	412.53	0.2357	N/A
Global Erasure	ESD(Erasing Concept)	57.18	0.2986	372.38	0.2093	N/A
Concept-Bound (Ours)	Untarget	54.72	0.3045	393.15	0.2140	97.1%

Table 1: Unified analysis of security paradigms. The data reveals the failure of context-blind approaches: generic methods either offer no provenance or degrade model fidelity. Our context-aware framework, IDENTITYGUARD , is the only one to provide both state-of-the-art restriction and robust provenance without this collateral damage.

Method	NudeNet Detections Results			Total Detections $\downarrow$
Method	Explicit	Suggestive	Other	Total Detections $\downarrow$
DreamBooth	135	59	148	342
+ SLD	67	52	127	246
+ ESD	1	1	44	46
IDENTITYGUARD	1	0	1	2

Table 2: Case Study on a critical safety threat (nudity). Detections by NudeNet per 100 images generated with a malicious ”naked” prompt. Even the strongest generic method (ESD) exhibits significant safety failures. Our method provides a near-total solution, demonstrating an order-of-magnitude improvement in a high-stakes scenario.

Figure 2: The IDENTITYGUARD fine-tuning framework. Our method trains a single Denoising U-Net using two conditional paths. (Top Path) For benign personalized prompts, our Concept-Bound Provenance is activated, embedding a watermark. (Bottom Path) For malicious prompts, our novel Semantic Redirection Loss is activated, redirecting the output towards a safe, identity-preserving result by aligning the noise predictions of the malicious and benign prompts. Here, $c^{*}$ is the embedding for the personalized concept, $c_{p}$ is for the prohibited concept, and $\text{sg}(\cdot)$ is the stop-gradient operator.

Figure 3: Qualitative analysis of our context-aware security. The key failure of generic methods like Erasing Concept(ESD) is revealed in the bottom rows: to provide protection in the personalized context (top rows), they are forced to inflict catastrophic collateral damage , globally erasing the concept of ”fire” and failing to generate a simple campfire. In contrast, IDENTITYGUARD’s safeguard is intelligently bound only to the personalized identity, allowing it to preserve the concept for general use.

AI-Generated Image Watermarking

See all 24 papers

InGuard: Towards Generalized Inner Guardrail for Safe Text-to-Image Generation
Sep 23, 2026Zeyu Wang, Xiaodan Li, Zhiwen Li +2T2I Generation Safe T2I Generation
Alibaba AAIG
PersGuard: Preventing Malicious Personalization in Text-to-Image Diffusion Models via Model Backdoors
Feb 22, 2025Xinwei Liu, Xiaojun Jia, Yuan Xun +2T2I Generation T2I Diffusion Models
Institute of Information Engineering, Chinese Academy of Sciences, Beijing 100093, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing 100049, China · BraneMatrix AI, Shanghai 201203, China +1
No Way To Steal My Face: Proactive Defense Against Identity-Preserving Personalized Generation
Oct 14, 2026Lizhi Xiong, Jun Li, Ziqiang Li +2Identity-Preserving Image Generation Diffusion-Based Image Generation
Nanjing University of Information Science and Technology

Sep 23, 2026J/K move · Enter open · S save

InGuard: Towards Generalized Inner Guardrail for Safe Text-to-Image Generation

Zeyu Wang, Xiaodan Li, Zhiwen Li, Yuefeng Chen+1

Alibaba AAIG

T2I Generation Safe T2I Generation Safety Filtering T2I Generation Evaluation T2I Diffusion Models

Open paper AskPDF arXiv

Modern text-to-image (T2I) models generate high-quality images from arbitrary user prompts, yet they can just as easily produce not-safe-for-work (NSFW) content. Conventional outer guardrails consist of two components: a prompt classifier that checks for risk before generation, and a post-hoc image classifier that checks the fully generated image. In this design, both classifiers operate outside the generation pipeline and do not use the model's own representations. This separation can limit prompt-screening accuracy, while the image-side check runs only after the full generation cost has been spent. Moreover, a flagged prompt can only be rejected, even when it could be adjusted to produce a safe image. In this work, we propose the Inner Guardrail (InGuard), a safety framework that works inside the pipeline on the model's own representations, leaving base-model parameters untouched. First, a risk classifier grades each prompt as unsafe, risky, or benign based on the text encoder's embeddings, with no external language model. Second, SAGE (Soft-gated Asymmetric Guardrail for Embeddings) modifies the embeddings of risky prompts, aiming to return a safe image instead of a refusal. Third, a latent detector checks the one-step clean latent estimate midway through denoising, reaching nearly image-level performance and halting generation when risk is detected. We also construct the RevGen Safety Benchmark to evaluate T2I safety under realistic conditions: 10,000 prompts built through real-image reverse generation, with a rewriting step that supplies controlled intellectual-property (IP) characters, covering graded porn/gore risks, categorical IP risks, and benign negatives. Across five open-weight T2I models, InGuard reaches 97.9-98.8% safety rate, matching or exceeding the outer guardrail, with 57.5-73.5% less benign disturbance, ~3.7x fewer parameters, and 50-55.6% of denoising steps skipped.

Abstract

AI-Generated Image Watermarking

Explore similar work

InGuard: Towards Generalized Inner Guardrail for Safe Text-to-Image Generation

PersGuard: Preventing Malicious Personalization in Text-to-Image Diffusion Models via Model Backdoors

No Way To Steal My Face: Proactive Defense Against Identity-Preserving Personalized Generation