Masking Is Not Enough: Generative Restoration for Multimodal De-Identification in Medical AI

Shiva Shrestha; Zongxing Xie; Chen Zhao; Liran Ma; Zhipeng Cai; Honghui Xu

cs.CVAug 21, 2026

Masking Is Not Enough: Generative Restoration for Multimodal De-Identification in Medical AI

Authors: Shiva Shrestha, Zongxing Xie, Chen Zhao, Liran Ma, Zhipeng Cai, Honghui Xu

Organizations: Department of Computer Science Kennesaw State University Marietta, GA 30060, USA · Department of Computer Science and Software Engineering Miami University Oxford, OH 45056, USA · Department of Computer Science Georgia State University Atlanta, GA 30303, USA · Department of Information Technology Kennesaw State University Marietta, GA 30060, USA

Abstract

Medical image-text data can expose protected health information (PHI) through both visible image content as well as accompanying text, creating a barrier to privacy-preserving medical AI systems. This risk is especially prominent in multimodal systems, where images, questions, reports, and clinical context may enter training, evaluation, or inference pipelines. Existing medical vision-language benchmarks primarily emphasize task utility, while de-identification methods are often evaluated separately from downstream reasoning. We introduce ClinX, an end-to-end multimodal PHI sanitization framework for medical image-text data. ClinX detects visible identifiers with optical character recognition (OCR), constructs binary PHI masks, and applies ClinX-PRISM, a no-skip generative restoration module with privacy-oriented post-processing for burned-in identifier suppression. In parallel, text-side PHI is reduced through progressive de-identification levels: regex masking, context-aware masking, and rewrite-based sanitization. We evaluate ClinX in medical visual question answering (MedVQA), jointly measuring PHI leakage and downstream utility across image-side, text-side, and combined de-identification settings. Results show that OCR-only masking is not sufficient as a standalone solution, and restoration-based sanitization better preserves clinically relevant visual context while sharply reducing recoverable PHI.

Figures & tables

Lvl	Cond.	EM	F1	Y/N EM	Llama-J	Mistral-J
$L0$	raw	$6.72$	$9.67$	$61.85$	60.50	45.34
	ocr	$6.67$	$9.61$	$61.67$	$59.23$	$44.66$
	g-skip	$8.19$	$11.16$	$63.83$	$59.42$	$45.12$
	PRISM	8.22	11.18	63.98	$59.30$	$44.78$
$L1$	raw	$6.78$	$9.68$	$61.73$	59.09	$43.35$
	ocr	$6.74$	$9.63$	$61.42$	$57.81$	$42.73$

Table 1: Privacy–utility tradeoff across text-privacy levels (

L0

–

L3

) and image conditions. Bold marks the best value per metric and level among the four primary conditions ( raw , ocr , g-skip , PRISM ).

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Table 2: Dataset-specific utility breakdowns across text privacy levels and image conditions. Higher is better.

Table 3: Expanded model-wise utility results across text privacy levels. Each subtable corresponds to one question-side privacy level. Higher is better for all reported metrics. PRISM denotes ClinX-PRISM, the proposed no-skip restoration pathway with mask-guided post-processing.

Panel of Figure 2 — Figure 2: Synthetic PHI construction from clean image to PHI-overlaid image and binary mask.

Figure 3: ClinX-PRISM no-skip generator. The PHI mask $m_{i}$ conditions the SPADE bottleneck and decoder, while skip connections are removed to force restoration through the mask-conditioned bottleneck.

Figure 4: PatchGAN discriminator producing a patch-level realism map for adversarial supervision.

Setting	Value
Input resolution	$224\times 224$
Epochs	10
Batch size	6
Learning rate	$10^{-4}$
Precision	BF16
Hardware	$2\times$ NVIDIA A100 80GB

Appendix

Table 4: ClinX-PRISM training and post-processing configuration.

Figure 5: Qualitative ClinX-PRISM restoration examples. Each column shows one sample. Rows show the PHI-overlaid input, clean target used only for paired restoration training, and ClinX-PRISM restored output.

Privacy Protection in VLMs

Privacy protection in vision-language models limits sensitive information exposure while preserving useful image understanding and multimodal reasoning.

See all 37 papers

From Redaction to Restoration: Deep Learning for Medical Image Anonymization and Reconstruction
Apr 13, 2026Adrienne Kline, Abhijit Gaonkar, Daniel Pittman +2
Center for Artificial Intelligence, BCVI, Northwestern Medicine, Chicago, IL, USA · Department of Electrical and Computer Engineering, Northwestern University, Chicago, IL, USA · Department of Surgery, Northwestern University, Chicago, IL, USA +3
MedDeID enables locally governed clinical-text de-identification from real or synthetic training data
Sep 9, 2026Stig Hellemans, Tom Stroobants, Elyne Scheurwegs +3Clinical NLP PII Detection
Adrem Data Lab, Department of Computer Science, University of Antwerp, Antwerp, Belgium · Antwerp University Hospital (UZA), Edegem, Belgium · Laboratory of Experimental Medicine and Pediatrics (LEMP), University of Antwerp, Antwerp, Belgium
The Text Beside the Image: Detection, Utility and Leakage for Trustworthy Multimodal Medical Data and Beyond
Sep 27, 2026Andreas Maier, Monica Hinrichs-Mayer, Franziska Weber +4Privacy Auditing PII Detection
Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg, Germany · Friedrich-Alexander-Universität Erlangen-Nürnberg, Germany · Universitätsklinikum Erlangen, Germany +1

Apr 13, 2026J/K move · Enter open · S save

From Redaction to Restoration: Deep Learning for Medical Image Anonymization and Reconstruction

Adrienne Kline, Abhijit Gaonkar, Daniel Pittman, Chris Kuehn+1

Center for Artificial Intelligence, BCVI, Northwestern Medicine, Chicago, IL, USA · Department of Electrical and Computer Engineering, Northwestern University, Chicago, IL, USA · Department of Surgery, Northwestern University, Chicago, IL, USA+3

Open paper AskPDF arXiv

Removing patient-specific information from medical images is crucial to enable sharing and open science without compromising patient identities. However, many methods currently used for deidentification have negative effects on downstream image analysis tasks because of removal of relevant but non-identifiable information. This work presents an end-to-end deep learning framework for transforming raw clinical image volumes into de-identified, analysis-ready datasets without compromising downstream utility. The methodology developed and tested in this work first detects and redacts regions likely to contain protected health information (PHI), such as burned-in text and metadata, and then uses a generative deep learning model to inpaint the redacted areas with anatomically and imaging plausible content. The proposed pipeline leverages a lightweight hybrid architecture, combining CRNN-based redaction with a latent-diffusion inpainting restoration module (Stable Diffusion 2). We evaluate the approach using both privacy-oriented metrics, which quantify residual PHI and success of redaction, and image-quality and task-based metrics, which assess the fidelity of restored volumes for representative deep learning applications. Our results suggest that the proposed method yields de-identified medical images that are visually coherent, maintaining fidelity for downstream models, while substantially reducing the risk of patient re-identification. By automating anonymization and image reconstruction within a single workflow, and dissemination of large-scale medical imaging collections, thereby lowering a key barrier to data sharing and multi-institutional collaboration in medical imaging AI.

Abstract

Privacy Protection in VLMs

Explore similar work

From Redaction to Restoration: Deep Learning for Medical Image Anonymization and Reconstruction

MedDeID enables locally governed clinical-text de-identification from real or synthetic training data

The Text Beside the Image: Detection, Utility and Leakage for Trustworthy Multimodal Medical Data and Beyond