Towards Context-Aware Image Anonymization with Multi-Agent Reasoning

Robert Aufschläger; Jakob Folz; Gautam Savaliya; Manjitha D Vidanalage; Michael Heigl; Martin Schramm

cs.CVMar 29, 2026

Towards Context-Aware Image Anonymization with Multi-Agent Reasoning

Authors: Robert Aufschläger, Jakob Folz, Gautam Savaliya, Manjitha D Vidanalage, Michael Heigl, Martin Schramm

Organizations: Deggendorf Institute of Technology, Deggendorf, Germany

Abstract

Street-level imagery contains personally identifiable information (PII), some of which is context-dependent. Existing anonymization methods either over-process images or miss subtle identifiers, while API-based solutions compromise data sovereignty. We present an agentic framework CAIAMAR (Context-Aware Image Anonymization with Multi-Agent Reasoning) for context-aware PII segmentation with diffusion-based anonymization, combining pre-defined processing for high-confidence cases with multi-agent reasoning for indirect identifiers. Three specialized agents coordinate via round-robin speaker selection in a Plan-Do-Check-Act (PDCA) cycle, enabling large vision-language models to classify PII based on spatial context (private vs. public property) rather than rigid category rules. The agents implement spatially-filtered coarse-to-fine detection where a scout-and-zoom strategy identifies candidates, open-vocabulary segmentation processes localized crops, and $IoU$ -based deduplication ( $30\%$ threshold) prevents redundant processing. Modal-specific diffusion guidance with appearance decorrelation substantially reduces re-identification (Re-ID) risks. On CUHK03-NP, our method reduces person Re-ID risk by $73\%$ ( $R1$ : $16.9\%$ vs. $62.4\%$ baseline). For image quality preservation on CityScapes, we achieve KID: $0.001$ , and FID: $9.1$ , significantly outperforming existing anonymization. The agentic workflow detects non-direct PII instances across object categories, and downstream semantic segmentation is preserved. Operating entirely on-premise with open-source models, the framework generates human-interpretable audit trails supporting EU's GDPR transparency requirements while flagging failed cases for human review.

Figures & tables

Figure 1: Two-phase agentic anonymization architecture. Phase 1 employs specialized models for direct PII (full body, license plates). Phase 2 implements multi-agent orchestration with round-robin coordination, where specialized agents handle classification (Auditor), synthesis (Generative), and workflow management (Orchestrator), implementing PDCA cycles.

Agent	Role, Tools, and Execution Protocol
Orchestrator	Role: Workflow coordinator and state tracker.
	Tools: None (coordination-only, no direct tool access).
	Protocol: Tracks completion status ( classify_pii ✓, inpaint ✓, audit ✓, log ✓) via conversation history analysis. Enforces validation rule: “ONLY report success (✓) if tool result visible in conversation history.” Coordinates retry logic for failed audits, enforces $n_{\max}=3$ iteration limit. Issues text-only coordination messages to other agents.
Auditor	Role: PII classification, quality validation, logging.
	Tools: classify_pii , audit_output , log_output .
	Protocol: (1) PII Classification : Qwen2.5-VL-32B with structured prompting, returns bboxes in $[x_{\min},y_{\min},w,h]$ format. System prompt prohibits acknowledgment-without-execution: “When instructed to use a tool, YOU MUST CALL IT – NEVER just acknowledge.” Performs scout-and-zoom verification for each classified instance: crops to LVLM bbox, runs Grounded-SAM-2 [ 23 , 33 ] , maps mask to full coordinates. After the first inpainting pass, classify_pii additionally applies a $\geq 50\%$ spatial overlap filter against already-processed masks (persons, license plates, traffic signs, previously inpainted regions) to suppress redundant candidates, distinct from the GenerativeAgent’s $30\%$ $IoU$ deduplication. (2) Independent Quality Validation : Creates masked image (processed areas blacked out with 5-iteration morphological dilation, $5\times 5$ elliptical kernel), performs LVLM-based residual detection to verify no visible PII remains. This check is independent of GenerativeAgent’s $IoU$ deduplication, providing dual-layer verification.

Table 1: Agent roles, tool access, and execution protocol in Phase 2 multi-agent workflow. GenerativeAgent implements

IoU

-based deduplication (

30\%

threshold) to prevent redundant reprocessing, while AuditorAgent provides independent quality validation.

Panel of Figure 2 — Figure 2: Round-robin PDCA coordination with three specialized agents. Phase 1 applies deterministic preprocessing for direct PII. Phase 2 implements bounded iterative refinement.

Panel of Figure 3 — Figure 3: Qualitative comparison of anonymization methods on a CUHK03-NP test example (bounding_box_test/0069_c2_658.png).

Method	MSE $\downarrow$	SSIM $\uparrow$	LPIPS $\downarrow$	KID $\downarrow$	FID $\downarrow$
Gauss. Blur	$1097.9$	$0.633$	$0.382$	$0.224$	$178.5$
DP2 [ 13 ]	$3275.1$	$0.443$	$0.303$	$0.066$	$59.7$
FADM [ 49 ]	$\boldsymbol{446.4}$	$\boldsymbol{0.785}$	$\boldsymbol{0.157}$	$0.032$	$33.3$
Ours	$975.6$	$0.699$	$0.203$	$\boldsymbol{0.014}$	$\boldsymbol{20.4}$

Table 2: Image quality metrics on CUHK03-NP detected test splits (

6,732

images:

1,400

query +

5,332

gallery).

Method	LPIPS $\downarrow$	KID $\downarrow$	FID $\downarrow$
SVIA [ 22 ]	$0.530$	$0.027$	$44.3$
Ours	$\boldsymbol{0.025}$	$\boldsymbol{0.001}$	$\boldsymbol{9.1}$

Table 3: Image quality metrics on Cityscapes test set (

1,525

images).

Method	mAP (%) $\downarrow$	R1 (%) $\downarrow$	R5 (%) $\downarrow$
Original	$66.0$	$62.4$	$73.1$
Gauss. Blur	$6.4$ $-90\%$	$9.4$ $-85\%$	$15.1$ $-79\%$
DP2 [ 13 ]	$\boldsymbol{4.4}$ $-93\%$	$\boldsymbol{8.6}$ $-86\%$	$\boldsymbol{12.7}$ $-83\%$
FADM [ 49 ]	$32.9$ $-50\%$	$33.4$ $-46\%$	$52.9$ $-28\%$
Ours	$\underline{13.7}$ $-79\%$	$\underline{16.9}$ $-73\%$	$\underline{31.0}$ $-58\%$

Table 4: Re-ID risk assessment on CUHK03-NP detected dataset. ResNet50 trained on original data (

767

IDs,

7,365

images), tested with original query (

1,400

images) vs. anonymized gallery (

5,332

images) with re-ranking. Lower scores indicate better privacy protection.

Method	Dice $\uparrow$	IoU $\uparrow$	Prec./Rec. $\uparrow$
Mask R-CNN [ 10 , 21 , 42 ]	75.83	68.71	80.71/78.02
Ours (zero-shot)	25.78	20.83	52.98/25.85

Table 5: PII detection on Visual Redactions test set (2,989 images).

Component	Time (s)
Phase 1: Detection + Inpainting	67.8
Phase 2: LVLM + SAM + Inpaint	55.8
Agent overhead	9.9
Total	133.5

Table 6: Runtime breakdown per image (CityScapes,

n{=}1{,}525

Class	Ours (Gap)	SVIA (Gap)
mIoU	0.877 ( $-0.123$ )	0.478 ( $-0.522$ )
road	0.996 ( $-0.005$ )	0.955 ( $-0.045$ )
building	0.977 ( $-0.023$ )	0.677 ( $-0.323$ )
person	0.778 ( $-0.222$ )	0.150 ( $-0.850$ )
car	0.975 ( $-0.025$ )	0.567 ( $-0.433$ )
sky	0.995 ( $-0.005$ )	0.820 ( $-0.180$ )

Table 7: Downstream segmentation on CityScapes test (SegFormer pseudo-GT).

Figure 4: Pipeline output on CityScapes. Each row: left original, middle detected PII ( blue =persons, yellow =indirect PII vehicles, green =traffic signs, red =license plates), right anonymized output. Top (berlin_000002): 8 persons + 1 police vehicle + 2 traffic signs, 4.97% coverage, 3 PDCA iterations. Bottom (berlin_000472): multiple PII categories across both phases.

Panel of Figure 12 — Figure S1: Qualitative comparison of person anonymization methods on CUHK03-NP test examples. Each row shows the same person processed by different methods (left to right): Original, Gaussian Blur, DeepPrivacy2 (DP2), FADM, and our approach. Our method preserves pose and scene structure while effectively anonymizing identities with photorealistic results. Blur destroys facial details and overall image quality. DP2 produces synthetic faces with visible artifacts. FADM maintains high similarity to originals (privacy risk). Our method achieves the optimal balance between privacy protection and visual quality.

Figure S2 : Training curves for YOLOv8s license plate detector. Loss curves and metrics over 100 epochs. Stable convergence achieved by epoch 60.

Panel of Figure 14 — Figure S3 : License plate detection performance. Left: Precision-recall curve showing mAP 50 =0.91. Right: Normalized confusion matrix with 0.94 true positive rate.

Panel of Figure 15 — Figure S4 : Training dynamics of Mask R-CNN on Visual Redactions Dataset. (a) Total loss shows $94.6\%$ reduction over 30K iterations. (b) Validation AP: segmentation AP from $11.77\%$ to $17.70\%$ , bbox AP from $9.24\%$ to $18.04\%$ . (c) Component losses: RPN classification ( $99.2\%$ reduction), classification ( $98.5\%$ ), mask ( $82.6\%$ ), box regression ( $72.7\%$ ). (d) Learning rate schedule: linear warmup (500 iterations), decay ( $\times 0.1$ ) at 18K and 25K iterations.

High Performance (AP $>$ 20%)
Category	Instances	AP (%)
Passport	153	70.55
Face	4,039	56.98
Person	5,937	48.68
Fingerprint	79	45.02
License Plate	371	33.02

Table S1 : Per-category mask AP on Visual Redactions validation set (

n=1{,}611

images). Categories grouped by performance regime. Training instance counts reflect severe class imbalance. Abundant data does not guarantee success: date_time (

2{,}979

instances

\rightarrow

3.75\%

AP), address (

1{,}902

instances

\rightarrow

0.64\%

AP). Category distribution shows visual attributes (face, person) dominate both instance count (

46.4\%

) and performance, while textual categories suffer systematic failure despite comprising

36\%

of training instances.

Figure S5 : Random examples from Visual Redactions test set. Ground truth (with category labels), Detectron2, and zero-shot predictions. Supervised achieves higher precision on common categories but fails on 99 images (3.3%). Zero-shot provides full coverage with broader generalization but occasional false positives.

Figure S6 : Detectron2 best cases. Top 4 examples by Dice score. Supervised excels on faces, persons, documents with abundant training data and COCO initialization.

Figure S7 : Zero-shot advantage cases. Top 4 examples where zero-shot outperforms Detectron2. Agents with LVLM utilization succeed on rare categories, novel formats, and low-confidence regions.

Panel of Figure 20 — Figure S8 : KITTI example (0000000165). Left: Original street scene with 14 pedestrians and multiple traffic signs. Right: Anonymized output after PDCA workflow. Phase 1 batch-inpainted 14 persons using SDXL with OpenPose ControlNet. Phase 2 processed 3 rectangular signs with Canny ControlNet. Final PII coverage: 5.29% (37,677 pixels). Total processing time: 391.8s with 3 PDCA iterations.

Agent	Action/Message
UserProxyAgent	Phase 1 complete. AuditorAgent: classify PII in 0000000165.png using classify_pii .
AuditorAgent	Tool call: classify_pii
	image: artifacts/.../0000000165.png
Tool Result	instances: [ { det_prompt: "rectangular sign with text .", bbox: [464, 275, 526, 341], instance_id: "pii-0001" } ]
	meta: {provider: ollama, model: qwen2.5vl:32b, tiles: 1}
OrchestratorAgent	Phase 2: [classify_pii $\checkmark$ ] Found indirect PII. GenerativeAgent, anonymize instances.

Table S2 : Agent dialogue excerpt from KITTI Phase 2 workflow (indirect PII detection and anonymization).

PII Detection

Personally identifiable information (PII) detection finds sensitive personal details in text, conversations, images, and other data.

See all 39 papers

Unsafe2Safe: Controllable Image Anonymization for Downstream Utility
Mar 30, 2026Mih Dinh, SouYoung JinPrivacy-Preserving ML Diffusion-Based Image Editing
Dartmouth College
LLM Anonymization Against Agentic Re-Identification
May 29, 2026Ziwen Li, Jianing Wen, Tianshi LiPrivacy-Preserving Language Models Privacy-Utility Tradeoff
Khoury College of Computer Sciences Northeastern University Boston, MA
From Weak Cues to Real Identities: Evaluating Inference-Driven De-Anonymization in LLM Agents
Mar 19, 2026Myeongseob Ko, Jihyun Jeong, Sumiran Singh Thakur +2Privacy Leakage in Language Models LLM Agent Security
Department of Electrical and Computer Engineering, Virginia Tech, Blacksburg, VA, USA · Center for Advanced AI, Accenture

Mar 30, 2026J/K move · Enter open · S save

Unsafe2Safe: Controllable Image Anonymization for Downstream Utility

Mih Dinh, SouYoung Jin

Dartmouth College

Privacy-Preserving ML Diffusion-Based Image Editing Text-Guided Image Editing

Open paper AskPDF arXiv

Large-scale image datasets frequently contain identifiable or sensitive content, raising privacy risks when training models that may memorize and leak such information. We present Unsafe2Safe, a fully automated pipeline that detects privacy-prone images and rewrites only their sensitive regions using multimodally guided diffusion editing. Unsafe2Safe operates in two stages. Stage 1 uses a vision-language model to (i) inspect images for privacy risks, (ii) generate paired private and public captions that respectively include and omit sensitive attributes, and (iii) prompt a large language model to produce structured, identity-neutral edit instructions conditioned on the public caption. Stage 2 employs instruction-driven diffusion editors to apply these dual textual prompts, producing privacy-safe images that preserve global structure and task-relevant semantics while neutralizing private content. To measure anonymization quality, we introduce a unified evaluation suite covering Quality, Cheating, Privacy, and Utility dimensions. Across MS-COCO, Caltech101, and MIT Indoor67, Unsafe2Safe reduces face similarity, text similarity, and demographic predictability by large margins, while maintaining downstream model accuracy comparable to training on raw data. Fine-tuning diffusion editors on our automatically generated triplets (private caption, public caption, edit instruction) further improves both privacy protection and semantic fidelity. Unsafe2Safe provides a scalable, principled solution for constructing large, privacy-safe datasets without sacrificing visual consistency or downstream utility.

Abstract

PII Detection

Explore similar work

Unsafe2Safe: Controllable Image Anonymization for Downstream Utility

LLM Anonymization Against Agentic Re-Identification

From Weak Cues to Real Identities: Evaluating Inference-Driven De-Anonymization in LLM Agents