Multimodal large language models (MLLMs) have demonstrated remarkable capabilities across vision-language tasks, yet their large-scale deployment raises pressing concerns about memorized private data, outdated knowledge, and harmful content. Existing unlearning approaches for MLLMs typically adapt training-based strategies such as gradient ascent or preference optimization, but these methods are computationally expensive, irreversible, and often distort retained knowledge. In this work, we propose MLLMEraser, an input-aware, training-free framework for test-time unlearning. Our approach leverages activation steering to enable dynamic knowledge erasure without parameter updates. Specifically, we construct a multimodal erasure direction by contrasting adversarially perturbed, knowledge-recall image-text pairs with knowledge-erasure counterparts, capturing both textual and visual discrepancies. To prevent unnecessary interference, we further design an input-aware steering mechanism that adaptively determines when and how the erasure direction should be applied, preserving utility on retained knowledge while enforcing forgetting on designated content. Experiments on LLaVA-1.5 and Qwen-2.5-VL demonstrate that MLLMEraser consistently outperforms state-of-the-art MLLM unlearning baselines, achieving stronger forgetting performance with lower computational cost and minimal utility degradation.
Figures & tables
Figure 1: (a) Comparison between training-based and test-time unlearning paradigms for MLLMs. (b) Illustration of the activation steering process. (c)–(d) Differences between existing methods and ours in constructing and applying the steering vector.
Figure 2: Overview of the proposed MLLMEraser framework. Stage 1 derives a multimodal erasure direction derase from contrastive image-text pairs. Stage 2 introduces an input-aware steering mechanism f(h) that adaptively applies derase to shift the activations of forget samples toward refusal-style responses, while leaving retain samples nearly unaffected to preserve correct responses.
Models
Classification Accuracy (%)
Generation: Rouge Score
Cloze: Accuracy (%)
Fgt ↓
Test ↓
Ret ↑
Cele ↑
Fgt ↓
Test ↓
Ret ↑
Cele ↑
Fgt ↓
Test ↓
Ret ↑
Cele ↑
LLaVA-1.5-7B (5% Forget)-VQA
Vanilla
42.40
40.00
45.23
47.00
0.632
0.286
0.585
0.312
54.00
24.00
55.37
8.17
GA
40.08
38.40
39.75
38.77
0.353
0.251
0.389
0.265
14.00
16.00
28.00
5.56
GA_Diff
35.20
29.60
36.00
37.10
0.554
0.265
0.585
0.310
32.00
20.00
52.11
7.84
KL_Min
40.00
36.80
42.87
44.78
0.629
0.309
0.585
0.307
52.00
18.00
53.58
6.86
Table 1: Unlearning results on MLLMU-Bench across the forget (Fgt), test (Test), retain (Ret), and celebrity (Cele) sets. ↓ and ↑ denote lower-is-better and higher-is-better metrics, respectively. The best results are shown in bold.
Figure 3: Trade-off between forget quality and model utility on LLaVA under 5% and 10% forget ratios. The left two plots correspond to classification task, where the x -axis shows accuracy difference on the forget set (Fgt VQA Acc Diff), and the right two plots correspond to generation, where the x -axis shows ROUGE-L difference on the forget set (Fgt Rouge Diff). The y -axis reports model utility on the retained (Ret) and celebrity (Cele) sets.
Models
Classification Accuracy (%)
Generation: Rouge Score
Cloze: Accuracy (%)
Fgt ↓
Test ↓
Ret ↑
Cele ↑
Fgt ↓
Test ↓
Ret ↑
Cele ↑
Fgt ↓
Test ↓
Ret ↑
Cele ↑
Vanilla
67.20
68.80
66.20
78.33
0.642
0.317
0.583
0.453
36.00
22.00
40.42
26.80
Text-only erasure direction
48.40
65.60
65.74
78.06
0.483
0.249
0.568
0.442
11.00
16.00
38.53
25.82
Input-unaware steering
15.20
7.20
13.29
4.05
0.130
0.008
0.121
0.121
8.00
8.00
10.74
12.42
Ours
19.20
38.40
65.86
78.32
0.165
0.175
0.580
0.445
10.00
12.00
39.37
26.47
Table 2: Ablation study on MLLMU-Bench (5% Forget) using the Qwen2.5-VL-7B-Instruct model. Results are reported on the forget (Fgt), test (Test), retain (Ret), and celebrity (Cele) sets. ↓ and ↑ denote lower-is-better and higher-is-better metrics, respectively.
Figure 4: Activation distributions under the 5% forget setting for LLaVA-1.5-7B ( 4(a) ) and Qwen-2.5-VL-7B-Instruct ( 4(b) ), where each subfigure shows the results on retained set and the forget set (Fgt) before (Vanilla) and after (Steered) steering.
Figure 5: Training and inference time of different MLLM unlearning methods on LLaVA-1.5-7B under the 5% forget setting. Inference time is measured on 10 randomly sampled queries.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Case study demonstrating that the unaligned base Qwen-2-VL-7B model naturally refuses unethical or misleading requests.
Figure 7: Sensitivity analysis of the regularization parameter γ on LLaVA-1.5-7B under the 10% forgetting setting. 7(a) reports results on the classification task, 7(b) shows results on the generation task, and 7(c) presents results on the cloze task.
Figure 8: Sensitivity analysis of the steering strength λ on LLaVA-1.5-7B under the 10% forgetting setting. 8(a) reports results on the classification task, 8(b) shows results on the generation task, and 8(c) presents results on the cloze task.
Models
Classification Accuracy (%)
Generation: Rouge Score
Cloze: Accuracy (%)
Fgt ↓
Test ↓
Ret ↑
Cele ↑
Fgt ↓
Test ↓
Ret ↑
Cele ↑
Fgt ↓
Test ↓
Ret ↑
Cele ↑
LLaVA-1.5-7B (10% Forget)-VQA
Vanilla
43.60
41.20
45.26
47.00
0.591
0.339
0.591
0.312
57.00
15.00
55.11
8.17
ϵ=25516
2.00
27.20
45.08
46.48
0.176
0.202
0.594
0.314
5.00
6.00
54.33
7.84
ϵ=25532
0.00
14.00
42.94
44.26
0.040
0.133
0.584
0.308
2.00
6.00
52.56
6.86
ϵ=25564
2.00
20.40
44.59
46.21
0.355
0.228
0.593
0.314
14.00
12.00
53.56
7.19
Appendix
Table 3: Unlearning results on MLLMU-Bench (10% Forget) under different perturbation budgets ϵ . Results are evaluated on the forget (Fgt), test (Test), retain (Ret), and celebrity (Cele) sets. ↓ and ↑ denote lower-is-better and higher-is-better metrics, respectively.
GA
GA_Diff
KL_Min
NPO
MMUnlearner
Ours
Memory Usage
86292MB
101262MB
106482MB
102616MB
78472MB
0MB
Appendix
Table 4: Training memory usage for updating MLLM parameters in different unlearning methods.
Models
Classification Accuracy (%)
Generation: Rouge Score
Cloze: Accuracy (%)
Fgt ↓
Test ↓
Ret ↑
Cele ↑
Fgt ↓
Test ↓
Ret ↑
Cele ↑
Fgt ↓
Test ↓
Ret ↑
Cele ↑
LLaVA-1.5-7B (10% Forget)-VQA
Vanilla
43.60
41.20
45.26
47.00
0.591
0.339
0.591
0.312
57.00
15.00
55.11
8.17
GA
40.00
31.20
41.43
40.08
0.510
0.315
0.470
0.262
43.00
14.00
49.22
5.56
KL_Min
42.40
35.20
45.08
46.08
0.588
0.335
0.579
0.306
54.00
15.00
53.89
7.10
GA_Diff
41.20
34.00
39.64
36.16
0.520
0.336
0.517
0.312
23.00
9.00
21.44
4.58
Appendix
Table 5: Unlearning results on MLLMU-Bench (10% Forget) with VQA and QA evaluation using LLaVA-1.5-7B. Results are reported on the forget (Fgt), test (Test), retain (Ret), and celebrity (Cele) sets. ↓ and ↑ denote lower-is-better and higher-is-better metrics, respectively.
Figure 9: Case Study on Forget Set before and after unlearning. The figure shows model responses to a forget-set query about sensitive attribute information. While the majority of training-based methods collapse or continue to expose the forgotten knowledge after unlearning, our method successfully removes the targeted information and produces a refusal-style response.
Figure 10: Case Study on Retain Set before and after unlearning. The figure shows model responses to a retain-set query asking about a person’s residence. While training-based methods (GA, GA_Diff, KL_Min, NPO, and MMUnlearner) either collapse or generate incorrect answers after unlearning, our method preserves the original correct response, demonstrating its superior ability to maintain retained knowledge while performing effective unlearning.
Models
Classification Accuracy (%)
Generation: Rouge Score
Cloze: Accuracy (%)
Fgt ↓
Test ↓
Ret ↑
Cele ↑
Fgt ↓
Test ↓
Ret ↑
Cele ↑
Fgt ↓
Test ↓
Ret ↑
Cele ↑
LLaVA-1.5-7B (5% Forget)-VQA
Vanilla
42.40
40.00
45.23
47.00
0.632
0.286
0.585
0.312
54.00
24.00
55.37
8.17
GA
40.08
38.40
39.75
38.77
0.353
0.251
0.389
0.265
14.00
16.00
28.00
5.56
GA_Diff
35.20
29.60
36.00
37.10
0.554
0.265
0.585
0.310
32.00
20.00
52.11
7.84
KL_Min
40.00
36.80
42.87
44.78
0.629
0.309
0.585
0.307
52.00
18.00
53.58
6.86
Appendix
Table 6: Unlearning results on MLLMU-Bench (5% Forget) with VQA and QA evaluation using LLaVA-1.5-7B. Results are reported on the forget (Fgt), test (Test), retain (Ret), and celebrity (Cele) sets. ↓ and ↑ denote lower-is-better and higher-is-better metrics, respectively.
Models
Classification Accuracy (%)
Generation: Rouge Score
Cloze: Accuracy (%)
Fgt ↓
Test ↓
Ret ↑
Cele ↑
Fgt ↓
Test ↓
Ret ↑
Cele ↑
Fgt ↓
Test ↓
Ret ↑
Cele ↑
Qwen-2.5-VL-7B (5% Forget)-VQA
Vanilla
67.20
68.80
66.20
78.33
0.642
0.317
0.583
0.453
36.00
22.00
40.42
26.80
GA
64.00
64.80
65.11
74.15
0.419
0.315
0.438
0.335
16.00
14.00
16.21
25.16
GA_Diff
56.00
59.20
64.05
76.76
0.482
0.296
0.566
0.442
16.00
22.00
33.79
24.84
KL_Min
51.20
50.40
50.30
48.57
0.594
0.322
0.578
0.441
32.00
18.00
34.32
25.16
Appendix
Table 7: Unlearning results on MLLMU-Bench (5% Forget) with VQA and QA evaluation using Qwen2.5-VL-7B. Results are reported on the forget (Fgt), test (Test), retain (Ret), and celebrity (Cele) sets. ↓ and ↑ denote lower-is-better and higher-is-better metrics, respectively.
Models
Classification Accuracy (%)
Generation: Rouge Score
Cloze: Accuracy (%)
Fgt ↓
Test ↓
Ret ↑
Cele ↑
Fgt ↓
Test ↓
Ret ↑
Cele ↑
Fgt ↓
Test ↓
Ret ↑
Cele ↑
LLaVA-1.5-7B (15% Forget)-VQA
Vanilla
44.80
42.93
45.09
47.00
0.598
0.294
0.586
0.312
53.33
19.33
55.76
8.17
GA
38.67
38.13
36.70
42.74
0.493
0.291
0.523
0.307
25.33
18.00
23.29
5.56
GA_Diff
28.27
36.80
43.68
41.26
0.333
0.303
0.517
0.301
24.00
18.67
45.29
4.28
KL_Min
42.67
37.87
39.95
39.57
0.537
0.316
0.575
0.306
44.00
19.01
43.29
6.21
Appendix
Table 8: Unlearning results on MLLMU-Bench (15% Forget) across the forget (Fgt), test (Test), retain (Ret), and celebrity (Cele) sets. ↓ and ↑ denote lower-is-better and higher-is-better metrics, respectively.
Figure 11: Layer-wise distributions of L2 norms for forget (blue) and retain (pink) steering vectors across Layers 2–31 of LLaVA-1.5-7B.
Models
Classification Accuracy (%)
Generation: Rouge Score
Cloze: Accuracy (%)
Fgt ↓
Test ↓
Ret ↑
Cele ↑
Fgt ↓
Test ↓
Ret ↑
Cele ↑
Fgt ↓
Test ↓
Ret ↑
Cele ↑
LLaVA-1.5-7B (10% Forget)-VQA
Vanilla
43.60
41.20
45.26
47.00
0.591
0.339
0.591
0.312
57.00
15.00
55.11
8.17
variant-1
33.20
37.20
45.43
47.00
0.572
0.325
0.588
0.309
54.00
13.00
55.11
8.17
variant-2
2.00
24.00
42.85
42.95
0.145
0.170
0.584
0.308
1.00
14.00
52.78
7.52
all
2.00
27.20
45.08
46.48
0.176
0.202
0.594
0.314
5.00
6.00
54.33
7.84
Appendix
Table 9: Unlearning performance on MLLMU-Bench (10% Forget). In addition to the full-layer steering strategy (all), we evaluate two selective-layer variants: variant-1, which steers only early layers (1–16), and variant-2, which steers only late layers (17–32).
Models
Classification Accuracy (%)
Generation: Rouge Score
Cloze: Accuracy (%)
Fgt ↓
Test ↓
Ret ↑
Cele ↑
Fgt ↓
Test ↓
Ret ↑
Cele ↑
Fgt ↓
Test ↓
Ret ↑
Cele ↑
LLaVA-1.5-7B (10% Forget)-VQA
Vanilla
43.60
41.20
45.26
47.00
0.591
0.339
0.591
0.312
57.00
15.00
55.11
8.17
Ours-MLP
29.20
32.40
38.62
42.19
0.363
0.265
0.414
0.311
19.00
8.00
32.00
6.54
Ours
2.00
27.20
45.08
46.48
0.176
0.202
0.594
0.314
5.00
6.00
54.33
7.84
Appendix
Table 10: Comparison of linear and MLP implementations of f(h) on MLLMU-Bench (10% Forget) using LLaVA-1.5-7B. Ours uses the default linear implementation, while Ours-MLP employs a two-layer MLP. Results are reported on the forget (Fgt), test (Test), retain (Ret), and celebrity (Cele) sets. ↓ and ↑ denote lower-is-better and higher-is-better metrics, respectively.
Multimodal Large Language Models (MLLMs) have achieved remarkable progress on vision-language tasks, but they may also memorize and expose sensitive or restricted knowledge, raising concerns about privacy and broader safety risks. Machine Unlearning (MU) provides a promising way to remove targeted undesirable knowledge from trained models without retraining from scratch while preserving general model utility. Nevertheless, effective unlearning in MLLMs remains particularly challenging. Existing training-based methods often struggle to balance unlearning effectiveness and model utility. In contrast, training-free methods such as in-context unlearning preserve model utility by avoiding parameter updates, but they do not remove memorized knowledge at the parameter level and may remain vulnerable to reverse-engineering attacks. More importantly, in-context unlearning is insufficient in multimodal settings, where visual inputs can provide strong conditioning signals and induce undesirable outputs. To address these challenges, we propose Visual-Noise Guided In-Context Distillation (VGID), a distillation-based framework for MLLM unlearning. VGID dynamically constructs an unlearning-oriented teacher distribution from the frozen base model through dual-modal intervention that combines visual perturbation with textual in-context unlearning. The resulting intervention-induced distribution serves as a teacher signal for distillation, guiding the student model toward parameter-level unlearning without requiring external teacher models or explicit undesirable response annotations. Experimental results show that VGID achieves strong unlearning effectiveness while preserving competitive model utility, reducing forget set ROUGE-L by 0.371 with only a 0.055 drop in retain set ROUGE-L in a representative setting.
Junkai Chen, Yuhao He, Junxiang You +3
Institute of Automation, Chinese Academy of Sciences · School of Advanced Interdisciplinary Sciences, UCAS
Multimodal large language models (MLLMs) can inadvertently memorize privacy-sensitive information during training. While existing unlearning methods can remove such content, they often severely degrade the model's foundational capabilities, such as general image understanding. This critical shortfall motivates our investigation into benign memory forgetting, the precise removal of targeted, privacy-sensitive knowledge while rigorously preserving unrelated capabilities. To pioneer and evaluate progress toward this objective, we introduce S-MLLMUn Bench, the first benchmark designed to jointly and quantitatively assess an unlearning method's efficacy in knowledge erasure and the preservation of image understanding. Furthermore, we propose the Sculpted Memory Forgetting Adapter (SMFA), a new framework that enables benign memory forgetting. SMFA confines forgetting to designated memory regions, maintaining overall model performance. By initially fine-tuning the model to replace sensitive outputs with refusals, SMFA generates a memory forgetting adapter, followed by a retaining anchor-guided masking mechanism that safeguards unrelated knowledge. Extensive experiments on S-MLLMUn Bench demonstrate that existing methods fail to achieve benign forgetting, whereas our proposed SMFA serves as an effective baseline, successfully achieving targeted knowledge erasure without compromising the model's foundational visual capabilities. Code and data are available at https://github.com/zeng-zhen/S-MLLMUn.
Zhen Zeng, Leijiang Gu, Zhangling Duan +4
Hefei University of Technology, China · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center, China · University of Amsterdam, the Netherlands
Machine unlearning offers a promising approach to remove unsafe content from Multimodal Large Language Models (MLLMs), yet ensuring the precision of unlearning remains a persistent challenge. One reason is that current MLLM unlearning evaluation paradigms suffer from a critical blind spot: they assess model utility through benchmarks whose representations are distant from the forget set, failing to capture knowledge holes---severe degradation on benign adjacent inputs. To probe knowledge holes in unlearned MLLMs, we construct a benchmark that captures unintended degradation on benign inputs sharing generic patterns with the forget set, and confirm through controlled experiments that they are a systematic consequence of commonly used approaches. Furthermore, to bridge this gap, we propose Selective Protection with Anchored Regularization, which protects generic patterns via anchored activation filtering while reinforcing them through entity-abstracted enhancement. Our experiments on SafeEraser demonstrate that SPAR recovers over 98% of vanilla response quality compared to below 50% for standard baselines---while achieving 0.00% attack success rate and competitive model utility. These results underscore the necessity of more fine-grained evaluation for trustworthy MLLM unlearning.
Junxiang You, Junkai Chen, Yuhao He +3
University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences · 3Cloudspace Technology