SALMUBench: A Benchmark for Sensitive Association-Level Multimodal Unlearning

Cai Selvas-Sala; Lei Kang; Lluis Gomez

cs.CVMar 27, 2026

SALMUBench: A Benchmark for Sensitive Association-Level Multimodal Unlearning

Authors: Cai Selvas-Sala, Lei Kang, Lluis Gomez

Organizations: Computer Vision Center · Universitat Polit`ecnica de Catalunya · Universitat Aut`onoma de Barcelona

Abstract

As multimodal models like CLIP become integral to downstream systems, the need to remove sensitive information is critical. However, machine unlearning for contrastively-trained encoders remains underexplored, and existing evaluations fail to diagnose fine-grained, association-level forgetting. We introduce SALMUBench (Sensitive Association-Level Multimodal Unlearning), a benchmark built upon a synthetic dataset of 60K persona-attribute associations and two foundational models: a Compromised model polluted with this data, and a Clean model without it. To isolate unlearning effects, both are trained from scratch on the same 400M-pair retain base, with the Compromised model additionally trained on the sensitive set. We propose a novel evaluation protocol with structured holdout sets (holdout identity, holdout association) to precisely measure unlearning efficacy and collateral damage. Our benchmark reveals that while utility-efficient deletion is feasible, current methods exhibit distinct failure modes: they either fail to forget effectively or over-generalize by erasing more than intended. SALMUBench sets a new standard for comprehensive unlearning evaluation, and we publicly release our dataset, models, evaluation scripts, and leaderboards to foster future research.

Figures & tables

Figure 1: CLIP models can memorize and leak private information. CLIP-based systems can associate a face with sensitive attributes (e.g., a phone number) seen during training.

Panel of Figure 2 — Figure 2: The construction pipeline of the SALMU dataset follows five sequential stages.

Panel of Figure 3 — Figure 3: Example of a fictitious persona from the SALMU dataset. We show the reference identity anchor image from the SFHQ dataset (top left), her sensitive attributes (bottom left), and a subset of generated images and captions (top right and bottom right respectively) that constitute the image-text pairs’ associations for this persona in the SALMU dataset.

Figure 4: Splits and subsets of SALMUBench along with the number of image-text pairs that each of them contains.

Model	Pretraining Data	ImageNet@1	COCO R@1		Flickr30K R@1
			t2i	i2t	t2i	i2t
CLIP ViT-B/16 [ 21 ]	400M (private)	0.64	0.30	0.51	0.59	0.78
Ours ( Clean ) ViT-B/16	retain (400M)	0.63	0.34	0.50	0.59	0.77
Ours ( Compromised ) ViT-B/16	retain (400M) + sensitive (60K)	0.64	0.33	0.50	0.57	0.75

Table 1: Comparison of our ViT-B/16 models ( Clean and Compromised ) against a public CLIP baseline on standard downstream tasks. All models are trained from scratch on

\sim

400M image-text pairs for 32 epochs.

Figure 5: Cosine similarity between image-text pairs in the sensitive set for the Clean and Compromised models.

Figure 6: Average cosine similarity of Real vs. Synthetic images with generic captions for the Clean model.

	Forgetting Efficacy					Utility Impact
Model	RetFail	AssocStr	ACS	IdZSC	CoreAssoc	GenKnow	InterIdSim	IntraIdSim	VisIdInt	FragSim
Clean	0.001	0.142	0.602	0.011	0.151	0.633	0.143	0.143	0.331	0.309
Compromised	0.236	0.322	0.951	0.084	0.183	0.638	0.321	0.321	0.305	0.280

Table 2: Compromised and Clean model performance, defining unlearning bounds (in bold): efficacy metrics measure sensitive information and use the Clean model as target, while utility metrics measure general knowledge preservation and use the Compromised model.

		Forgetting Efficacy					Utility Impact
Method	Budget	RetFail	AssocStr	ACS	IdZSC	CoreAssoc	GenKnow	InterIdSim	IntraIdSim	VisIdInt	FragSim
Target Scores	-	0.001	0.142	0.602	0.011	0.151	0.638	0.321	0.321	0.305	0.280
Generic Captions	5 $\times$	0.055	0.288	0.907	0.056	0.185	0.629	0.288	0.288	0.338	0.310
Shuffled Captions	5 $\times$	0.004	0.212	0.593	0.002	0.200	0.548	0.212	0.212	0.206	0.203
Direct Sim. Min.	5 $\times$	0.001	-0.429	0.593	0.002	0.012	0.615	-0.420	-0.425	-0.038	-0.031
Negative Gradient	5 $\times$	0.009	0.055	0.657	0.023	0.204	0.630	0.063	0.061	0.194	0.171

Table 3: Performance of baseline unlearning methods applied to the Compromised model (5

\times

budget). Scores in bold are closest to their target value. A dagger (

\dagger

) marks results that are not statistically significant (n.s.) from their target (more detailed in the supplement).

Figure 7: Average cosine similarity of Real vs. Synthetic images with generic captions for the Compromised model.

Model	RetFail	AssocStr	ACS	IdZSC	CoreAssoc
Clean	0.001	0.142	0.602	0.011	0.151
LAION-400M	0.004	0.149	0.632	0.005	0.169
Compromised	0.236	0.322	0.951	0.084	0.183

Table 4: Sensitive-set efficacy metrics for the Clean , Compromised , and an independent OpenCLIP (LAION-400M) model. The similarity between Clean and LAION-400M confirms the target scores are stable and representative.

		Forgetting Efficacy					Utility Impact
Method	Budget	RetFail	AssocStr	ACS	IdZSC	CoreAssoc	GenKnow	InterIdSim	IntraIdSim	VisIdInt	FragSim
Target Scores	-	0.001	0.142	0.602	0.011	0.151	0.638	0.321	0.321	0.305	0.280
Generic Captions	1 $\times$	0.143	0.308	0.932	0.073	0.179	0.635	0.308	0.308	0.325	0.292
	5 $\times$	0.055	0.288	0.907	0.056	0.185	0.629	0.288	0.288	0.338	0.310
	10 $\times$	0.042	0.275	0.896	0.050	0.185	0.625	0.276	0.275	0.341	0.313
	15 $\times$	0.038	0.267	0.885	0.044	0.184	0.621	0.269	0.267	0.342	0.315

Table 5: Performance of baseline unlearning methods across different computational budgets.

		Forgetting Efficacy					Utility Impact
Method	Budget	RetFail	AssocStr	ACS	IdZSC	CoreAssoc	GenKnow	InterIdSim	IntraIdSim	VisIdInt	FragSim
Clean (Reference)	-	0.001	0.142	0.602	0.011	0.151	0.633	0.143	0.143	0.331	0.309
Generic Captions	5 $\times$	0.001	0.147	0.616	0.011	0.156	0.626	0.148	0.148	0.324	0.301
Shuffled Captions	5 $\times$	0.002	0.185	0.521	0.001	0.185	0.606	0.185	0.185	0.183	0.184
Direct Sim. Min.	5 $\times$	0.001	-0.371	0.512	0.005	-0.259	0.611	-0.369	-0.368	0.074	0.074
Negative Gradient	5 $\times$	0.001	-0.114	0.529	0.007	-0.045	0.629	-0.116	-0.112	0.254	0.248

Table 6: Performance of 4 baseline unlearning methods applied to the Clean model (5

\times

budget). The first row lists the metrics of the unmodified Clean model. Deviations from these values indicate unintended damage to the model’s embedding space.

Figure 8: Learning-rate sensitivity (5 $\times$ budget): RetFail (left) and GenKnow (right) for Generic Captions and Negative Gradient. Dashed lines indicate target scores. Generic Captions hits a forgetting ceiling regardless of LR, confirming Type 3 (Ineffective). Negative Gradient shows that reaching the forgetting target forces GenKnow well below the target, confirming Type 2 (Over-generalized).

VLM Unlearning

Vision-language model unlearning studies how to remove specified concepts, identities, associations, or behaviors while preserving unrelated knowledge and capabilities.

See all 31 papers

ICU-Bench:Benchmarking Continual Unlearning in Multimodal Large Language Models
May 7, 2026Yuhang Wang, Wenjie Mei, Junkai Zhang +3Privacy-Preserving Language Models Multimodal Large Language Models
School of Computer Science and Technology Xidian University
Multimodal Unlearning Across Vision, Language, Video, and Audio: Survey of Methods, Datasets, and Benchmarks
Jul 8, 2026Nobin Sarwar, Shubhashis Roy Dipta, Zheyuan Liu +1VLM Unlearning MLLM Unlearning
University of Maryland, Baltimore County · University of Notre Dame · UNC Chapel Hill
Toward Fine-Grained Forgetting:Attribute Unlearning for Multimodal Large Language Models
Aug 2, 2026Junkai Lin, Junkai Chen, Siqi Hou +6VLM Unlearning Privacy Leakage in Language Models
Institute of Automation, Chinese Academy of Sciences · The Chinese University of Hong Kong · The Australian National University +3

May 7, 2026J/K move · Enter open · S save

ICU-Bench:Benchmarking Continual Unlearning in Multimodal Large Language Models

Yuhang Wang, Wenjie Mei, Junkai Zhang, Guangyu He+2

School of Computer Science and Technology Xidian University

Privacy-Preserving Language Models Multimodal Large Language Models MLLM Unlearning Machine Unlearning Multimodal Model Evaluation LLM Unlearning+1

Open paper AskPDF arXiv

Although Multimodal Large Language Models (MLLMs) have achieved remarkable progress across many domains, their training on large-scale multimodal datasets raises serious privacy concerns, making effective machine unlearning increasingly necessary. However, existing benchmarks mainly focus on static or short-sequence settings, offering limited support for evaluating continual privacy deletion requests in realistic deployments. To bridge this gap, we introduce ICU-Bench, a continual multimodal unlearning benchmark built on privacy-critical document data. ICU-Bench contains 1,000 privacy-sensitive profiles from two document domains, medical reports and labor contracts, with 9,500 images, 16,000 question-answer pairs, and 100 forget tasks. Additionally, new continual unlearning metrics are introduced, facilitating a comprehensive analysis of forgetting effectiveness, historical forgetting preservation, retained utility, and stability throughout the continual unlearning process. Through extensive experiments with representative unlearning methods on ICU-Bench, we show that existing methods generally struggle in continual settings and exhibit clear limitations in balancing forgetting quality, utility preservation, and scalability over long task sequences. These findings highlight the need for multimodal unlearning methods explicitly designed for continual privacy deletion.

Abstract

VLM Unlearning

Explore similar work

ICU-Bench:Benchmarking Continual Unlearning in Multimodal Large Language Models

Multimodal Unlearning Across Vision, Language, Video, and Audio: Survey of Methods, Datasets, and Benchmarks

Toward Fine-Grained Forgetting:Attribute Unlearning for Multimodal Large Language Models