Unsafe2Safe: Controllable Image Anonymization for Downstream Utility

Mih Dinh; SouYoung Jin

cs.CVMar 30, 2026

Unsafe2Safe: Controllable Image Anonymization for Downstream Utility

Authors: Mih Dinh, SouYoung Jin

Organizations: Dartmouth College

Abstract

Large-scale image datasets frequently contain identifiable or sensitive content, raising privacy risks when training models that may memorize and leak such information. We present Unsafe2Safe, a fully automated pipeline that detects privacy-prone images and rewrites only their sensitive regions using multimodally guided diffusion editing. Unsafe2Safe operates in two stages. Stage 1 uses a vision-language model to (i) inspect images for privacy risks, (ii) generate paired private and public captions that respectively include and omit sensitive attributes, and (iii) prompt a large language model to produce structured, identity-neutral edit instructions conditioned on the public caption. Stage 2 employs instruction-driven diffusion editors to apply these dual textual prompts, producing privacy-safe images that preserve global structure and task-relevant semantics while neutralizing private content. To measure anonymization quality, we introduce a unified evaluation suite covering Quality, Cheating, Privacy, and Utility dimensions. Across MS-COCO, Caltech101, and MIT Indoor67, Unsafe2Safe reduces face similarity, text similarity, and demographic predictability by large margins, while maintaining downstream model accuracy comparable to training on raw data. Fine-tuning diffusion editors on our automatically generated triplets (private caption, public caption, edit instruction) further improves both privacy protection and semantic fidelity. Unsafe2Safe provides a scalable, principled solution for constructing large, privacy-safe datasets without sacrificing visual consistency or downstream utility.

Figures & tables

Figure 1: Examples from Unsafe2Safe (U2S). For each case, the model converts an unsafe image into a privacy-preserving safe version. Examples demonstrate key capabilities that may appear simultaneously: (1) structure-preserving full body anonymization, (2) demographic neutralization (race entropy $\uparrow$ ), and (3) obfuscation of non-human confidential details.

Figure 2: Pipeline Overview. A VLM inspects the image for privacy risks. For flagged images, it generates a private caption $c^{priv}$ and a public caption $c^{pub}$ without sensitive details. An LLM then produces an edit instruction $c^{edit}$ on how sensitive attributes should be modified. A diffusion editor uses these priors to generate a privacy-safe image while preserving scene semantics.

Figure 3: SafeAttention within UNet. The UNet transformer receives two textual conditions: the edit instruction and the public caption. The Cross Attention module follows the standard cross-attention pathway, while an auxiliary Safe Attention module operates on both embeddings to reinforce non-private semantics during denoising.

Privacy Criterion	Recall $\uparrow$	Precision $\uparrow$	F1 $\uparrow$
All attributes	0.975	0.793	0.874
Face	0.850	0.927	0.887
Health indicators	0.892	0.678	0.770
Vehicles	0.829	0.435	0.570
Personal opinion	0.778	0.665	0.717

Table 1: VLM privacy detector performance under different attribute flagging criteria; higher recall means less privacy leakage.

Figure 4: Qualitative comparison of anonymization outputs on Caltech101. Each image shows with a different model family ( top line ) and its textual condition ( bottom line ). All methods preserve the global layout of the original scene, but the Unsafe2Safe using FlowEdit [ 24 ] , unlike face-only anonymizers (DP2, FaceAnon), modify background elements when they contain privacy-relevant cues, while keeping overall scene composition intact.

Model	Prior	Quality Score		Cheating Scores
		CLIP (↑)		SSIM (↓)		LPIPS (↑)
		C101	I67	C101	I67	C101	I67
U2S (FE)	$c_{\text{priv}}$	0.3054	0.3411	0.8551	0.6497	0.1118	0.2815
	class	0.2634	0.3149	0.8559	0.6708	0.1139	0.2600
	$c_{\text{pub}}$	0.3094	0.3448	0.8556	0.6497	0.1118	0.2803
	$c_{\text{edit}}$	0.2685	0.2946	0.8550	0.6580	0.1168	0.2846

Table 2: Quality and Cheating scores for Caltech101 (Cal101) and MIT Indoor 67 (I67). Our Unsafe2Safe (U2S), which uses FlowEdit (FE) [ 24 ] as the underlying image editor, generates high-quality images comparable to other anonymization models. Unlike prior methods that rarely modify content beyond facial regions, U2S identifies privacy-unsafe cues throughout the entire image and performs broader, context-aware edits. Best values are highlighted in yellow, and second-best in light blue.

Model	Text Prior	Utility Score		Privacy Score
		Accuracy ( $\uparrow$ )		VLMScore ( $\uparrow$ )		FaceSim ( $\downarrow$ )		TextSim ( $\downarrow$ )		Race Entropy ( $\uparrow$ )
		Cal101	Indoor	Cal101	Indoor	Cal101	Indoor	Cal101	Indoor	Cal101	Indoor
Raw Images	–	94.277	83.881	7.700	0.763	1.0000	1.0000	1.0000	1.0000	0.4384	0.7443
Unsafe2Safe (FE)	$c_{\text{private}}$	94.334	79.925	9.646	8.009	0.4378	0.2666	0.6611	0.4210	0.5552	0.7399
	class	93.857	80.448	12.555	7.512	0.4965	0.3743	0.4856	0.2077	0.4051	0.6508
	$c_{\text{public}}$	94.487	80.746	9.873	7.440	0.4881	0.2883	0.5395	0.2896	0.6409	0.7208

Table 3: Utility and privacy comparison on classification tasks. Our Unsafe2Safe, which uses FlowEdit (FE) [ 24 ] as the underlying image editor, achieves comparable performance on downstream tasks while successfully anonymizing privacy-sensitive information, unlike other anonymization models. The best value (yellow) and the second-best value (blue) are highlighted per column.

Model	Text Prior	Utility		Privacy
Model	Text Prior	BLEU-4 $\uparrow$	CIDEr $\uparrow$	VLMScore $\uparrow$	FaceSim $\downarrow$	TextSim $\downarrow$	RaceEnt $\uparrow$
Raw	–	0.436	1.413	0.646	1.0000	1.0000	0.6170
U2S (FP)	$c_{\text{edit}}$	0.433	1.390	32.002	0.2013	0.1442	0.7364
U2S (FE)	$LLM$	0.423	1.363	36.641	0.1975	0.1276	0.7282
FaceAnon	–	0.444	1.429	1.622	0.4436	0.8123	0.6458

Table 4: Utility and privacy comparison on MS-COCO captioning. U2S achieves higher VLMScore and much lower FaceSim/TextSim. Our Unsafe2Safe uses FreePrompt (FP) [ 30 ] and FlowEdit (FE) [ 24 ] as the underlying image editor.

Model	Text Prior	Utility	Privacy
Model	Text Prior	$\text{Acc}_{\text{VQA}}$ ( $\uparrow$ )	VLMScore ( $\uparrow$ )	FaceSim ( $\downarrow$ )	TextSim ( $\downarrow$ )	RaceEnt ( $\uparrow$ )
Raw	–	0.6064	0.600	1.0000	1.0000	0.6288
U2S (FP)	$c_{\text{edit}}$	0.6573	33.192	0.2041	0.1514	0.7499
U2S (FE)	$LLM$	0.7093	37.059	0.1951	0.1196	0.7397
FaceAnon	–	0.6632	1.705	0.4483	0.8158	0.6568

Table 5: Utility and privacy comparison on OK-VQA. U2S attains the highest VQA accuracy while achieving the strongest privacy scores. Best (yellow) values are highlighted per column.

Model	%White	%Black	%Asian	%Hispanic	%Other
Raw Images	80.28	2.82	5.63	4.23	7.04
U2S (FE, $c_{\text{edit}}$ )	45.93	29.63	13.33	0.00	11.11
U2S (FE, $LLM$ )	37.90	25.81	17.74	2.42	16.13
DP2 [ 21 ]	56.70	3.09	19.59	4.12	16.49
FaceAnon [ 25 ]	70.67	4.00	10.67	4.00	10.67

Table 6: Race distribution (%) of images of different models.

Finetuned	Text Prior	Utility Score		Privacy Score
		Accuracy ( $\uparrow$ )		VLMScore ( $\uparrow$ )		FaceSim ( $\downarrow$ )		TextSim ( $\downarrow$ )		Race Entropy ( $\uparrow$ )
		Cal101	Indoor	Cal101	Indoor	Cal101	Indoor	Cal101	Indoor	Cal101	Indoor
	$c_{\text{private}}$	94.773	80.746	11.651	6.559	0.5571	0.3382	0.6553	0.3112	0.5642	0.7414
	class	94.887	79.776	10.674	5.943	0.6888	0.3500	0.6176	0.2898	0.5528	0.7678
	$c_{\text{public}}$	94.582	81.791	10.814	5.646	0.5401	0.3389	0.6589	0.3044	0.6497	0.7599
	$c_{\text{edit}}$	94.315	81.418	16.878	16.926	0.5164	0.2909	0.6126	0.3399	0.6826	0.7493

Table 7: Performance of InstructPix2Pix [ 7 ] with different text priors, and after fine-tuning on our anonymization dataset (✓). Fine-tuning notably improves both utility and privacy: the fine-tuned

c_{\text{edit}}

model achieves the highest Caltech101 accuracy (95.116%) among all variants, while also reducing FaceSim/TextSim on Indoor67 and increasing race entropy, indicating more diverse and less identity-specific outputs. Safe denotes the model trained with Safe Attention.

Figure 5: Qualitative examples showing that the InstructPix2Pix model finetuned on our dataset ( FT ) more effectively anonymizes sensitive content while preserving original class semantics, compared to the model trained on general editing data.

Figure S1 : Distribution of normalized CLIP scores for private images after anonymization. Most samples cluster around 0.9, indicating strong preservation of original semantics. The red line marks the filtering threshold at 0.7, below which edited samples are discarded during editing model training.

	#Samples		Utility	Privacy Instances
Split	Train	Val	Acc@1	Text	Faces	Races
Caltech101 [ 13 ]
Original	2000	980	94.27	251	82	71
Safe subset	1552	756	83.49	111	24	0
MIT Indoor67 [ 37 ]
Original	3991	1317	83.88	551	2184	1332

Table S1 : Summary of dataset statistics, utility accuracy, and privacy-leakage indicators for the raw and safe subsets. The table reports training/validation sample counts, top-1 accuracy, and the number of detected text, faces, and race attributes.

Text Prior	$c^{priv}$	$c^{pub}$	$LLM$
FLEUR ( $\uparrow$ )	80.45	78.93	58.27

Table S2: Alignment to MS-COCO annotations of different text priors produced by Stage 1 under the FLEUR [ 27 ] metric (higher is better). The public caption

c^{pub}

maintains FLEUR scores close to the private caption

c^{priv}

, indicating strong preservation of global scene semantics despite removing sensitive details. The

LLM

prior , while lower due to the introduction of additional synthesized attributes, still retains meaningful semantic alignment.

Model	Text Prior	Utility Score		Privacy Score
		Accuracy ( $\uparrow$ )		FaceSim ( $\downarrow$ )		TextSim ( $\downarrow$ )		Race Entropy ( $\uparrow$ )
		Cal101	Indoor	Cal101	Indoor	Cal101	Indoor	Cal101	Indoor
Raw Images	–	94.277	83.881	1.0000	1.0000	1.0000	1.0000	0.4384	0.7443
Unsafe2Safe (FlowEdit [ 24 ] )	$c_{\text{private}}$	94.334	79.925	0.4378	0.2666	0.6611	0.4210	0.5552	0.7399
	class	93.857	80.448	0.4965	0.3743	0.4856	0.2077	0.4051	0.6508
	$c_{\text{public}}$	94.487	80.746	0.4881	0.2883	0.5395	0.2896	0.6409	0.7208

Table S3 : Extend utility and privacy comparison for different generative backbones. Our Unsafe2Safe, which uses [ 24 ] as the underlying image editor, achieves comparable performance on downstream tasks while successfully anonymizing privacy-sensitive information, unlike other anonymization models. The best value (yellow) and the second-best value (blue) are highlighted per column. † denotes the model was finetuned with our dataset.

Figure S2 : Attention maps at the 13 th transformer block in the UNet. From left to right: (1) the original private image; (2) the anonymized output produced by vanilla InstructPix2Pix along with its standard cross-attention map; (3) the anonymized output produced by our Safe Cross Attention model, shown with its vanilla cross-attention map, the Safe Attention map corresponding to public-caption tokens, and the Safe Attention map corresponding to edit-instruction tokens.

Figure S3 : Example of demographic-controlled anonymization under the randomly sampled “Indigenous / Pacific Islander” condition.

Figure S4 : Qualitative results of our method with different text priors and existing face anonymizers. Our results were generated by the FlowEdit [ 24 ] model.

Figure S5 : Qualitative results of our method with different generative backbones and existing face anonymizers. FlowEdit [ 24 ] takes $LLM$ as the target text prior, while other models take $c^{edit}$ as the text prior.

Privacy-Preserving ML

Privacy-preserving machine learning develops methods to learn, share, and run models while limiting exposure of sensitive data.

See all 432 papers

No Safe Dose: How Training Data Drives Unsafe Image Generation
May 27, 2026Felix Friedrich, Lukas Helff, Niharika Hegde +2Safe T2I Generation Data Contamination
1Black Forest Labs · 2TU Darmstadt & hessian.AI · 3DFKI +1
NullEdit: Stealthy Image Protection via VLM Condition Redirection
Aug 11, 2026Weiyao Huang, Liqin Wang, Ziqi Sheng +1VLM Robustness Representation Steering
School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, China
Towards Context-Aware Image Anonymization with Multi-Agent Reasoning
Mar 29, 2026Robert Aufschläger, Jakob Folz, Gautam Savaliya +3PII Detection
Deggendorf Institute of Technology, Deggendorf, Germany

May 27, 2026J/K move · Enter open · S save

No Safe Dose: How Training Data Drives Unsafe Image Generation

Felix Friedrich, Lukas Helff, Niharika Hegde, Patrick Schramowski+1

1Black Forest Labs · 2TU Darmstadt & hessian.AI · 3DFKI+1

Safe T2I Generation Data Contamination T2I Generation Evaluation T2I Diffusion Models

Open paper AskPDF arXiv

Text-to-image models trained on large-scale data often inevitably ingest unsafe content. While some people observe input-output amplifications, it remains unclear whether and how training data composition directly drives model output safety or by other factors. We shed light on this question by isolating this variable: we train the same text-to-image model on datasets that differ only in their fraction of unsafe images (0% to 9.6%), across several dataset scales (100K to 8M). Then we generate images with the resulting models, and evaluate them with four independent safety classifiers. Output unsafety rises monotonically from 16.6% at 0% contamination to 25.5% at 5%. A factorial design reveals that the proportion, not the absolute count, of unsafe training images is the operative variable. The 16.6% irreducible baseline at zero contamination implicates the other components, e.g. frozen text encoder, as a residual safety risk -- confirmed by a text encoder ablation showing that SafeCLIP reduces this floor to 9.6%, while the dose-response effect persists across all three encoders tested. Critically, no quality degradation in terms of FID, CLIPscore and ImageReward accompanies safety filtering. These results establish that data curation and text encoder safety are complementary and independently effective interventions. At the same time, the remaining level of unsafety poses questions for future research about emerging capabilities and compositionality.

Abstract

Privacy-Preserving ML

Explore similar work

No Safe Dose: How Training Data Drives Unsafe Image Generation

NullEdit: Stealthy Image Protection via VLM Condition Redirection

Towards Context-Aware Image Anonymization with Multi-Agent Reasoning