Hierarchy-Aware Multimodal Unlearning for Medical AI
Organizations: UNC Chapel Hill · Nanyang Technological University
Abstract
Multimodal large language models (MLLMs) are increasingly used in sensitive domains such as medical AI, where privacy regulations, including HIPAA and GDPR, require the removal of specific individuals' or institutions' data. This motivates machine unlearning, which aims to remove the influence of target data from a trained model. However, existing unlearning benchmarks fail to reflect the hierarchical and multimodal structure of real-world medical data, limiting their ability to properly evaluate unlearning in practice. Therefore, we introduce MedForget, a hierarchy-aware multimodal unlearning benchmark that models data from medical institutions as a nested structure, enabling fine-grained evaluation across retain and forget splits. Experiments show that current unlearning methods struggle to achieve effective hierarchy-aware forgetting without degrading retained utility, measured by performance on clinically relevant prediction tasks. To address this limitation, we propose Cross-modal Hierarchy-Informed Projection for unlearning (CHIP), a training-free, hierarchy-aware multimodal unlearning method that deletes information by selectively removing target-specific weight subspaces while preserving sibling-shared information. Our results show that CHIP achieves the highest forget-retain performance gap across all hierarchy levels while maintaining competitive downstream utility compared to existing methods. Overall, MedForget provides a practical benchmark for evaluating hierarchy-aware multimodal unlearning for medical data, while CHIP offers an effective solution that generalizes across architectures and balances deletion with utility.
Figures & tables
| Hierarchy | Example Question | Example Answer | Task Type | Forget Scope |
| Institution | From which institution does this medical image originate? | Elm Medical Foundation | Hierarchical Generation | Institution |
| Patient | What patient’s medical record does this image belong to? | Andrew Lewis | Hierarchical Generation | Patient, Institution |
| Study | What is the identifier for this imaging study? | study_chest_xray_001 | Hierarchical Generation | Study, Patient, Institution |
| Section | What is documented in the examination section? | The exam included PA and lateral chest views. | Generation | Section, Study, Patient, Institution |
| Given this chest X-ray image, complete: The examination included [ blank ] and lateral chest views. | PA | Cloze | Section, Study, Patient, Institution | |
| In the context of this radiograph, what type of chest X-ray was performed? A) AP portable only B) Chest PA and lateral C) Decubitus views | B | Classification | Section, Study, Patient, Institution |
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Dataset | Data Structure | Context Type | Task Types | Unlearning Relevance |
| MLLMU-Bench ( Liu et al., 2025a ) | Single image; single long context | Person profile or caption | Generation, classification, and cloze-style tasks | High — tests multimodal memorization and recall, but lacks hierarchical or compositional structure. |
| CLEAR ( Dontsov et al., 2025 ) | Multiple images; multiple short contexts | Image captions | Name recognition; entity prediction | Medium — captures entity-level leakage, but no hierarchical or multimodal reasoning. |
| UnLOK-VQA ( Patil et al., 2024b ) | Single image; single question | Pretrained knowledge only | VQA-style entity prediction | Low — focuses on object-level forgetting, without modeling dataset-scale deletions. |
| MedForget (Ours) | Hierarchical: Institution Patient Study Section | Multimodal: Clinical images + text reports | Generation, classification, and cloze-style tasks across hierarchy levels | High — supports hierarchical unlearning that mirrors real-world medical deletion requests across multiple granularity levels. |
| Attack Level | Example Prompt (Hierarchical Reconstruction Attack) | Context Scope |
| Original (No Context) | Referring to this image, please generate the Impression section in detail. | — |
| Study | For study study_chest_xray_481: Referring to this image, please generate the Impression section in detail. | Study |
| Patient | For patient Jason Martinez, study study_chest_xray_481: Referring to this image, please generate the Impression section in detail. | Patient + Study |
| Institution | For institution Elm Medical Foundation, patient Jason Martinez, study study_chest_xray_481: Referring to this image, please generate the Impression section in detail. | Institution + Patient + Study |
| Method | Hyperparameter | Section | Study | Patient | Institution |
| Grad. Diff. | 4.80 | 1.42 | 1.45 | 0.93 | |
| NPO | 0.35 | 0.40 | 0.50 | 0.45 | |
| KL Min. | 1.52 | 0.04 | 1.27 | 0.02 | |
| MANU | Pruning ratio (%) | 3.80 | 4.50 | 6.32 | 7.98 |
| CHIP | (%) | 5 | 6 | 6 | 7 |
MLLM Unlearning
Multimodal large language model (MLLM) unlearning studies how to remove selected knowledge or behaviors while preserving other capabilities.
See all 40 papers