Open-MMUnlearning: Unifying Methods and Evaluation for MLLM Unlearning

Junkai Chen; Yuhao He; Qianshan Wei; Junxiang You; Jingwen Shao; Junkai Lin; Zhongkai Yue; Xiaotian Ye

cs.AIOct 7, 2026

Open-MMUnlearning: Unifying Methods and Evaluation for MLLM Unlearning

Authors: Junkai Chen, Yuhao He, Qianshan Wei, Junxiang You, Jingwen Shao, Junkai Lin, Zhongkai Yue, Xiaotian Ye, +14 more

Organizations: Institute of Automation, Chinese Academy of Sciences

Abstract

As multimodal large language models (MLLMs) become more capable and widely deployed, concerns about privacy and safety have become increasingly pressing. Machine unlearning offers one approach to addressing these concerns by removing designated information from trained models while preserving unrelated capabilities. However, fragmented implementations and evaluation protocols, incomplete robustness testing, and limited understanding of metric reliability make progress in MLLM unlearning difficult to assess systematically. We introduce Open-MMUnlearning, an open-source, extensible framework that integrates target-model preparation, multimodal data processing, unlearning, and evaluation through shared interfaces and structured configurations. The framework supports five benchmarks spanning privacy, safety, and copyright, eight MLLMs from four model families, and twelve unlearning methods. Its evaluation suite jointly assesses forgetting effectiveness, retained utility, and robustness to model interventions, adversarial inputs, and membership inference attacks. Using a common evaluation protocol, we compare ten representative unlearning methods. In this comparison, GD and MIP-Editor tie for the highest overall score: GD achieves the highest Forget Quality, while MIP-Editor preserves more Model Utility. We further introduce a metric meta-evaluation protocol that tests faithfulness using models with controlled exposure to target knowledge and robustness under quantization and relearning. Among the thirteen evaluated metrics, BLEU achieves the highest aggregate reliability score. KS-Test attains the highest faithfulness AUC but performs less well on robustness. Together, the framework and these findings support reproducible comparison of MLLM unlearning methods and systematic assessment of evaluation reliability.

Figures & tables

Figure 1: Overview of Open-MMUnlearning . The Data Worker prepares fine-tuning, evaluation, and unlearning data from benchmarks spanning privacy, safety, and copyright. The Model Loader supplies MLLM backbones, the Trainer executes fine-tuning and general and MLLM-specific unlearning methods, and the Unified Evaluator assesses Forget Quality, Model Utility, and Robustness through model-, input-, and output-side evaluations.

Component		Variants
Models		LLaVA -1.5 ( Liu et al., 2023 ) Qwen -2.5-VL ( Bai et al., 2025 ) LLaVA -1.6 ( Liu et al., 2024a ) Gemma -3 ( Team et al., 2025 )
Methods		General Unlearning Methods ( Maini et al., 2024 ; Li et al., 2024b ) ( Zhang et al., 2024a ; Dong et al., 2025 ) MMUnlearner ( Huo et al., 2025 ) MANU ( Liu et al., 2025b ) MIP-Editor ( Li et al., 2026 ) SMFA ( Zeng et al., 2025 ) VGID ( Chen et al., 2026b )
Datasets		MLLMU ( Liu et al., 2025a ) FIUBench ( Ma et al., 2025 ) CLEAR ( Dontsov et al., 2025 ) SafeEraser ( Chen et al., 2025 ) CoVUBench ( Kwon et al., 2026 )
Metrics	Forget	Truth Ratio ( Ma et al., 2025 ; Dontsov et al., 2025 ) KS-Test ( Ma et al., 2025 ; Dontsov et al., 2025 ) JS Distance ( Dontsov et al., 2025 ) Answer Probability ( Ma et al., 2025 ) ROUGE-L ( Liu et al., 2025a ) ASR ( Chen et al., 2025 )
	Utility	Model Utility ( Ma et al., 2025 ; Dontsov et al., 2025 ) Fluency ( Kwon et al., 2026 ) Specificity ( Kwon et al., 2026 ) Generality ( Kwon et al., 2026 ) SARR ( Chen et al., 2025 ) LM-Eval ( Liu et al., 2025a ; Kwon et al., 2026 )
	Robust	Relearning ( Hu et al., 2025 ; Zheng et al., 2025 ) Quantization ( Zhang et al., 2025c ) Probing ( Lynch et al., 2024 ) MIA ( Yeom et al., 2018 ; Carlini et al., 2021 ; Wang et al., 2024 ; Shi et al., 2024 ) Jailbreak ( Wei et al., 2023 ; Ma et al., 2024 ) FigStep ( Gong et al., 2025 ) SUA ( Zhang et al., 2025b ) Image Rephrasing ( Patil et al., 2025 )

Table 1: Overview of existing Open-MMUnlearning components and their available variants. The modular design allows users to add new models, unlearning methods, datasets, and evaluation metrics.

Methods	Agg. $\uparrow$	Forget Quality , $\uparrow$	Model Utility , $\uparrow$	Robustness $\uparrow$
Methods	Agg. $\uparrow$	Forget Quality , $\uparrow$	Model Utility , $\uparrow$	Agg. $\uparrow$	Model $\uparrow$	Input $\uparrow$	Output $\uparrow$
Reference Models
Finetune	0.622	0.687	0.421	0.759	0.835	0.631	0.812
Retain-Only	0.638	0.696	0.444	0.776	0.839	0.688	0.799
General Unlearning Methods
GA ( Maini et al., 2024 )	0.574	0.774	0.066	0.881	0.996	0.971	0.677

Table 2: Evaluation of ten representative MLLM unlearning methods in terms of Forget Quality, Model Utility, and Robustness. We also report an overall aggregate score and divide Robustness into model-, input-, and output-side evaluations. Finetune and Retain-Only serve as reference models. Higher scores are better; bold and underlined values indicate the best and second-best results among the unlearning methods, respectively.

Figure 2: Faithfulness and robustness meta-evaluation of ROUGE-L. The left and middle panels compare scores before and after targeted relearning and quantization, respectively. Blue points represent eligible unlearned checkpoints, the red star denotes the mean Retain-Only reference, and the shaded areas indicate intervention-specific unreliable regions; the dashed red line marks $y=x$ . The right panel shows the positive and negative model-pool distributions and the selected threshold, with a faithfulness AUC of 0.870.

Metrics	Agg. $\uparrow$	Faithful. $\uparrow$	Robustness $\uparrow$
Metrics	Agg. $\uparrow$	Faithful. $\uparrow$	Agg. $\uparrow$	Quant. $\uparrow$	Relearn $\uparrow$
Cloze	0.766	0.626	0.986	0.995	0.978
Classification	0.740	0.604	0.954	0.978	0.930
Truth Ratio	0.896	0.814	0.996	0.999	0.992
Probability	0.887	0.924	0.853	0.928	0.789
KS-Test	0.888	0.991	0.804	0.900	0.727

Table 3: Meta-evaluation of 13 MLLM unlearning metrics for faithfulness and robustness. Faithful. reports the ROC-AUC for distinguishing models with and without exposure to target knowledge. Quant. and Relearn. report robustness under quantization and targeted relearning, respectively. Higher scores indicate greater reliability. The best and second-best values in each column are shown in bold and underlined, respectively.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Component		Variants
Models		LLaVA -1.5 ( Liu et al., 2023 ) Qwen -2.5-VL ( Bai et al., 2025 ) LLaVA -1.6 ( Liu et al., 2024a ) Gemma -3 ( Team et al., 2025 )
Methods		GradAscent GradDiff KL ( Maini et al., 2024 ) IdkNLL ( Maini et al., 2024 ) RMU ( Li et al., 2024b ) NPO ( Zhang et al., 2024a ) UNDIAL ( Dong et al., 2025 ) MMUnlearner ( Huo et al., 2025 ) MANU ( Liu et al., 2025b ) MIP-Editor ( Li et al., 2026 ) SMFA ( Zeng et al., 2025 ) VGID ( Chen et al., 2026b )
Datasets		MLLMU ( Liu et al., 2025a ) FIUBench ( Ma et al., 2025 ) CLEAR ( Dontsov et al., 2025 ) SafeEraser ( Chen et al., 2025 ) CoVUBench ( Kwon et al., 2026 )
Metrics	Forget	Truth Ratio ( Ma et al., 2025 ; Dontsov et al., 2025 ) Exact Match / APE ( Ma et al., 2025 ) Answer Probability ( Ma et al., 2025 ) Fill-in-the-Blank Accuracy ( Liu et al., 2025a ) Classification Accuracy ( Liu et al., 2025a ) ROUGE-1/2/L ( Liu et al., 2025a ) BLEU ( Liu et al., 2025a ) KS-Test ( Ma et al., 2025 ; Dontsov et al., 2025 ) JS Distance ( Dontsov et al., 2025 ) Forget Quality ( Dontsov et al., 2025 ) Keyword Recall ( Kwon et al., 2026 ) Semantic Dissimilarity ( Kwon et al., 2026 ) Efficacy ( Kwon et al., 2026 ) Divergence ( Kwon et al., 2026 ) Attack Success Rate (ASR) ( Chen et al., 2025 ) Refusal Rate (RR) ( Chen et al., 2025 )
	Utility	Model Utility ( Ma et al., 2025 ; Dontsov et al., 2025 ) GPT-Eval ( Ma et al., 2025 ; Chen et al., 2025 ) Fluency ( Kwon et al., 2026 ) Specificity ( Kwon et al., 2026 ) Generality ( Kwon et al., 2026 ) LM-Eval ( Liu et al., 2025a ; Kwon et al., 2026 ) SARR ( Chen et al., 2025 )
	Robust	Relearning ( Hu et al., 2025 ; Zheng et al., 2025 ) Quantization ( Zhang et al., 2025c ) Probing ( Lynch et al., 2024 ) Cross-Modal Leakage ( Wang et al., 2026a ) Pure-Text Jailbreak ( Zou et al., 2023 ; Wei et al., 2023 ; Ma et al., 2024 ) FigStep ( Gong et al., 2025 ) Image Rephrasing ( Patil et al., 2025 ) SUA ( Zhang et al., 2025b ) LOSS ( Yeom et al., 2018 ) ZLIB ( Carlini et al., 2021 ) GradNorm ( Wang et al., 2024 ) Min-K% ( Shi et al., 2024 ) Min-K%++ ( Zhang et al., 2025a )

Appendix

Table 4: Overview of existing Open-MMUnlearning components and their available feature variants. The design is easily extensible, allowing users to seamlessly contribute new features.

Methods	Forget Set			Test Set
Methods	Cloze $\downarrow$	Cls. $\downarrow$	Gene. $\downarrow$	Cloze $\downarrow$	Cls. $\downarrow$	Gene. $\downarrow$
Reference Models
Finetune	13.54	38.78	0.242	11.46	27.35	0.149
Retain-Only	12.50	36.73	0.207	13.54	40.41	0.146
General Unlearning Methods
GA ( Maini et al., 2024 )	0.00	0.00	0.003	0.00	0.00	0.003

Appendix

Table 5: Detailed Forget Quality results on the forget set and held-out forget-target test set. Cloze and Cls. report accuracy (%), while Gene. reports ROUGE-L. Lower scores indicate stronger forgetting. Finetune and Retain-Only serve as reference models; bold and underlined values indicate the best and second-best results among the unlearning methods, respectively.

Methods	Truth Ratio $\uparrow$	Prob. $\downarrow$	KS-Test $\downarrow$	JS Dist. $\downarrow$
Reference Models
Finetune	0.570	0.499	0.140	0.281
Retain-Only	0.452	0.254	0.150	0.337
General Unlearning Methods
GA ( Maini et al., 2024 )	0.393	0.000	0.330	0.190
GD ( Maini et al., 2024 )	0.432	0.168	0.130	0.130

Appendix

Table 6: Detailed Forget Quality results for normalized Truth Ratio, answer probability, KS-Test, and JS Distance. Higher Truth Ratio and lower values for the other metrics are better. Finetune and Retain-Only serve as reference models; bold and underlined values indicate the best and second-best results among the unlearning methods, respectively.

Methods	Retain-Shared			Retain-Celebrity
Methods	Cloze $\uparrow$	Cls. $\uparrow$	Gene. $\uparrow$	Cloze $\uparrow$	Cls. $\uparrow$	Gene. $\uparrow$
Reference Models
Finetune	8.04	36.74	0.231	12.75	29.90	0.171
Retain-Only	16.52	41.47	0.261	16.34	50.91	0.169
General Unlearning Methods
GA ( Maini et al., 2024 )	0.11	0.00	0.003	0.00	0.00	0.000

Appendix

Table 7: Detailed Model Utility results on the multimodal Retain-Shared and Retain-Celebrity sets. Cloze and Cls. report accuracy (%), while Gene. reports ROUGE-L. Higher scores are better. Finetune and Retain-Only serve as reference models; bold and underlined values indicate the best and second-best results among the unlearning methods, respectively.

Methods	MMBench $\uparrow$	MM-Vet $\uparrow$	POPE $\uparrow$	SQA $\uparrow$	VizWiz $\uparrow$	GQA $\uparrow$	VQAv2 $\uparrow$
Reference Models
Finetune	80.00	16.00	82.00	56.00	2.60	52.00	79.20
Retain-Only	80.00	24.00	78.00	62.00	8.40	44.00	79.20
General Unlearning Methods
GA ( Maini et al., 2024 )	0.00	0.00	0.00	36.00	0.00	0.00	0.00
GD ( Maini et al., 2024 )	76.00	16.00	82.00	56.00	5.20	56.00	81.20

Appendix

Table 8: Detailed Model Utility results on seven general capability benchmarks. All scores are reported as percentages, and higher scores are better. SQA denotes ScienceQA. Finetune and Retain-Only serve as reference models; bold and underlined values indicate the best and second-best results among the unlearning methods, respectively.

Methods	Quant. $\uparrow$	Relearn. $\uparrow$	Probe-L16 $\uparrow$	Probe-L24 $\uparrow$
Reference Models
Finetune	0.752	0.759	0.993	0.996
Retain-Only	0.768	0.760	0.984	0.993
General Unlearning Methods
GA ( Maini et al., 2024 )	0.999	0.997	0.992	0.991
GD ( Maini et al., 2024 )	0.915	0.772	0.994	0.993

Appendix

Table 9: Detailed Model-Side Robustness results for quantization, targeted relearning, and probing at layers 16 and 24. Higher scores are better. Finetune and Retain-Only serve as reference models; bold and underlined values indicate the best and second-best results among the unlearning methods, respectively.

Methods	Cross-Modal	FigStep , $\uparrow$	Jailbreak , $\uparrow$	Image	SUA , $\uparrow$
Methods	Leakage , $\uparrow$	FigStep , $\uparrow$	Jailbreak , $\uparrow$	Rephrasing , $\uparrow$	SUA , $\uparrow$
Reference Models
Finetune	0.574	0.860	0.740	0.688	0.292
Retain-Only	0.611	0.930	0.797	0.667	0.438
General Unlearning Methods
GA ( Maini et al., 2024 )	0.853	1.000	1.000	1.000	1.000

Appendix

Table 10: Detailed Input-Side Robustness results for cross-modal queries, FigStep, jailbreak prompts, image rephrasing, and SUA. Higher scores indicate stronger resistance to these inputs. Finetune and Retain-Only serve as reference models; bold and underlined values indicate the best and second-best results among the unlearning methods, respectively.

Methods	LOSS $\uparrow$	ZLIB $\uparrow$	GradNorm $\uparrow$	Min-K% $\uparrow$	Min-K%++ $\uparrow$
Reference Models
Finetune	0.840	0.748	0.536	0.969	0.965
Retain-Only	0.954	0.856	0.456	0.884	0.846
General Unlearning Methods
GA ( Maini et al., 2024 )	0.874	0.823	0.033	0.828	0.828
GD ( Maini et al., 2024 )	0.872	0.969	0.000	0.665	0.590

Appendix

Table 11: Detailed Output-Side Robustness results based on five membership inference attacks. Each score measures how close the attack’s ROC-AUC is to random guessing; higher scores are better. Finetune and Retain-Only serve as reference models; bold and underlined values indicate the best and second-best results among the unlearning methods, respectively.

Figure 3: Four paired case studies of MLLM unlearning. Green Forgotten indicates that the target answer is absent; red Retained indicates that it is present.

Figure 4: Four paired case studies of MLLM unlearning. Green Forgotten indicates that the target answer is absent; red Retained indicates that it is present.

Figure 5: Four paired case studies of MLLM unlearning. Green Forgotten indicates that the target answer is absent; red Retained indicates that it is present.

Figure 6: Four paired case studies of MLLM unlearning. Green Forgotten indicates that the target answer is absent; red Retained indicates that it is present.

Figure 7: Faithfulness of 13 MLLM unlearning metrics on the MLLMU 10% forget setting. Blue and orange histograms show score distributions for matched models trained with and without the target knowledge, respectively. Dashed lines mark metric-specific thresholds, and AUC values summarize how well each metric separates the two model pools

Figure 8: Relearning robustness of 11 MLLM unlearning metrics. Blue points show eligible unlearned checkpoints before and after targeted relearning; the red star marks the mean Retain-Only reference. The dashed line indicates

y=x

, while the shaded region marks apparent recovery of target knowledge beyond the Retain-Only reference.

Figure 9: Quantization robustness of 11 MLLM unlearning metrics. Blue points show eligible unlearned checkpoints before and after quantization; the red star marks the mean Retain-Only reference. The dashed line indicates

y=x

, while the shaded region marks quantization-induced shifts toward greater apparent target knowledge.

Explore similar work

Aug 3, 2026cs.AI

Exploring and Bridging Knowledge Holes in Unlearned Multimodal Large Language Models

Machine unlearning offers a promising approach to remove unsafe content from Multimodal Large Language Models (MLLMs), yet ensuring the precision of unlearning remains a persistent challenge. One reason is that current MLLM unlearning evaluation paradigms suffer from a critical blind spot: they assess model utility through benchmarks whose representations are distant from the forget set, failing to capture knowledge holes---severe degradation on benign adjacent inputs. To probe knowledge holes in unlearned MLLMs, we construct a benchmark that captures unintended degradation on benign inputs sharing generic patterns with the forget set, and confirm through controlled experiments that they are a systematic consequence of commonly used approaches. Furthermore, to bridge this gap, we propose Selective Protection with Anchored Regularization, which protects generic patterns via anchored activation filtering while reinforcing them through entity-abstracted enhancement. Our experiments on SafeEraser demonstrate that SPAR recovers over 98% of vanilla response quality compared to below 50% for standard baselines---while achieving 0.00% attack success rate and competitive model utility. These results underscore the necessity of more fine-grained evaluation for trustworthy MLLM unlearning.

Junxiang You, Junkai Chen, Yuhao He +3

University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences · 3Cloudspace Technology

Large Language Model Unlearning Machine Unlearning Large Language Model Safety Multimodal Large Language Models Regularization

Ask PDF

Jun 11, 2026cs.AI

MLUBench: A Benchmark for Lifelong Unlearning Evaluation in MLLMs

Multimodal large language models (MLLMs) are trained on massive multimodal data, making data unlearning increasingly important as data owners may request the removal of specific content. In practice, these requests often arrive sequentially over time, giving rise to the challenging problem of MLLM Lifelong Unlearning. However, most existing benchmarks are limited in scale and scope, failing to capture the complexities of MLLM lifelong unlearning. To fill this gap, we introduce the MLUBench, a large-scale and comprehensive benchmark featuring 127 entities across 9 classes under lifelong unlearning requests. We perform extensive experiments using MLUBench and reveal that existing unlearning methods suffer from severe, cumulative degradation. More critically, we further identify the unique challenge of this problem: unlike in unimodal models, MLLM lifelong unlearning is constrained by the need to preserve multimodal alignment. Continually unlearning from one modality could degrade the entire model. To alleviate this challenge, we propose LUMoE, an effective method. Experiments demonstrate that LUMoE significantly mitigates the degradation problem faced by baselines. The source code and the MLUBench dataset are open-sourced in https://github.com/lihe-maxsize/Lifelong_Unlearning_main.

He Li, Haoang Chi, Qizhou Wang +6

College of Computer Science and Technology, National University of Defense Technology, Changsha, China · TMLR Group, Hong Kong Baptist University, Hong Kong, China · School of Statistics and Data Science, Shanghai University of Finance and Economics, Shanghai, China +2

Large Language Model Unlearning Multimodal Large Language Models Unimodal Metrics

Ask PDF

Aug 2, 2026cs.AI

Toward Fine-Grained Forgetting:Attribute Unlearning for Multimodal Large Language Models

Multimodal large language models (MLLMs) exhibit strong vision--language capabilities but may also memorize and disclose sensitive information. Machine unlearning seeks to remove designated knowledge without retraining from scratch while preserving general utility. Existing privacy-oriented benchmarks primarily adopt profile-level deletion, whereas practical requests are often finer grained: a model should forget a specified attribute while retaining non-sensitive information about the same identity. We therefore introduce attribute-level MLLM unlearning as a finer-grained task and construct a benchmark spanning long-text, numeric, and short-text targets, multiple forget ratios, and diverse question types. Our evaluation reveals that target and retained attributes share identity-specific and visual evidence, making selective forgetting susceptible to residual leakage or collateral degradation; accordingly, existing methods exhibit unstable forgetting--retention trade-offs in this setting. To address this challenge, we propose Causal Localization and Retain-Aware Projection (CLRP), a lightweight training-free framework. CLRP uses activation patching to identify the layer that causally mediates target-attribute disclosure, then applies a retain-aware projection that removes the target-attribute subspace while preserving same-identity evidence. Experiments across multiple widely used MLLMs with distinct architectures and parameter scales demonstrate the effectiveness of CLRP.

Junkai Lin, Junkai Chen, Siqi Hou +5

Institute of Automation, Chinese Academy of Sciences · The Chinese University of Hong Kong · The Australian National University +3

Large Language Model Unlearning Multimodal Large Language Models Catastrophic Forgetting Forgetting Data Leakage Projection

Ask PDF