Multimodal large language models (MLLMs) can memorize identity-specific facts about people in their fine-tuning data, creating privacy risks when a person requests deletion. Existing MLLM unlearning methods often assume access to retain images or ground-truth answers during deletion, which is unrealistic in many practical scenarios. We study identity unlearning when retain images are unavailable at deletion time. Our analysis shows that identity and visual-perception questions occupy distinct regions in fine-tuned hidden states and are organized differently: identity questions cluster by person, whereas perception questions cluster by question type. This suggests that identity knowledge can be suppressed without erasing general visual perception. Building on this observation, we propose AIM, a two-stage method that anchors an identity-forgetting target with a universal visual prompt and then matches the vision encoder to that target under a Fisher-based constraint. Extensive experiments show that AIM achieves competitive identity forgetting while preserving non-deleted identities, prior knowledge, and visual perception on the same images.
Figures & tables
Figure 1: Method Overview. Stage 1 defines a target representation from forget images and constructed questions. Stage 2 internalizes this target into the vision encoder under a Fisher-based retain constraint.
Figure 2: Representational separability and grouping rules of identity and perception questions in last-layer LLM hidden states on LLaVA-1.5-7B. (a) PCA projects identity and visual perception questions into largely disjoint regions. (b) t-SNE of identity questions colored by image identity. Same-person questions cluster together regardless of question content. (c) t-SNE of visual perception questions colored by question type. Same-type questions cluster together regardless of which person appears.
Identity (held-out)
Visual perception
Model
IDK-Rate
IDK-Rate
SP
LLaVA-1.5-7B
100.0
10.0
63.3
Qwen3-VL-8B
85.0
0.0
85.2
Table 1: Selective transfer of a pilot VP. A VP trained to elicit IDK responses on 4 identity questions transfers to held-out identity questions but leaves visual perception responses on the same images largely intact.
MLLMU-Bench
ReMem
Method
Cls f↓
ROUGE f↓
Cls r↑
ROUGE r↑
Cls c↑
ROUGE c↑
EM f↓
EM r↑
ROUGE ↑
Exp ↓
EM t↓
Forget 5%
Vanilla
39.2
60.2
39.0
55.4
52.4
24.4
93.3
100.0
100.0
56.9
92.9
retain utilizing
GA_Diff
36.0
53.5
36.1
54.9
48.3
21.6
0.0
47.5
44.5
40.5
0.0
KL_Min
30.4
27.4
40.9
35.5
56.1
14.3
86.7
98.9
91.2
55.1
100.0
MMUnlearner
29.6
44.9
38.5
58.0
44.6
29.6
23.3
96.1
98.2
36.4
23.8
Table 2: Unlearning results on LLaVA-1.5-7B. MLLMU-Bench and ReMem across 5%, 10% forget splits. Subscripts denote forget ( f ), retain ( r ), celebrity prior ( c ), and test ( t ) splits. Baselines are grouped by data scope: retain-utilizing methods access additional retain data, forget-only methods do not. Ours uses only forget data.
Method
Cls f↓
R f↓
Cls r↑
R r↑
Cls c↑
R c↑
Forget 5%
Vanilla
68.0
72.2
69.0
70.8
74.4
43.7
retain-utilizing
GA_diff
64.8
65.4
67.0
67.8
73.8
42.5
KL_Min
2.4
2.8
66.4
60.3
74.9
39.1
MMUnlearner
48.0
49.7
60.0
57.0
58.0
37.7
Table 3: Unlearning results on Qwen3-VL-8B (MLLMU-Bench). 5%, 10% forget splits. Notation and baseline grouping follows Table 2 .
Model
pretrained vs
vanilla vs
Appr.
ROUGE ↑
GPT ↑
ROUGE ↑
GPT ↑
GPT ↑
Vanilla
58.9
70.2
–
–
96.4
MMUnlearner
52.1
34.7
74.3
33.3
88.4
GA
0.0
0.0
0.0
0.0
0.0
NPO
49.6
59.1
77.2
66.2
99.1
Ours
57.8
68.9
87.5
82.7
97.3
Table 4: Visual perception preservation after unlearning (MLLMU-Bench, 5%). Visual perception responses on forget-identity images, scored against the pretrained LLaVA and the vanilla model by ROUGE-L and GPT semantic preservation judgment. Appr.: fraction of coherent, on-topic, non-refusal answers.
Split
Method
EM f ↓
EM r ↑
ROUGE ↑
Exp ↓
EM t ↓
5% → 10%
Vanilla
91.7
100.0
100.0
69.1
94.0
GA
78.3
96.2
73.0
64.2
87.5
NPO
75.0
96.0
72.6
62.7
82.7
Ours
41.7
69.3
84.0
64.7
42.3
10% → 15%
Vanilla
94.4
100.0
100.0
65.6
96.0
GA
0.0
0.1
1.8
53.9
0.0
Table 5: Continual unlearning on ReMem. Hop i→j : continually unlearn from the forget i checkpoint over the newly-added identities in forget j .
Forget ↓
Retain ↑
Real ↑
CE
Align
Norm
Cls
ROUGE
Cls
ROUGE
Cls
ROUGE
✓
✓
✗
24.4
47.0
28.3
46.9
21.4
30.0
✓
✗
✓
41.2
58.8
39.7
57.2
52.2
25.6
✗
✓
✓
42.8
57.1
38.7
55.5
52.3
24.4
✓
✓
✓
39.2
49.9
38.5
51.7
51.7
26.1
Table 6: Stage 1 loss component ablation (MLLMU-Bench, 10%). Each row keeps a subset of the three loss terms (CE, alignment, norm).
Forget ↓
Retain ↑
Celebrity ↑
Method
Cls
ROUGE
Cls
ROUGE
Cls
ROUGE
Blank
31.6
46.7
34.8
49.6
50.2
26.8
Random Vector
44.0
59.4
39.6
58.2
51.4
25.1
IDK (baseline)
39.2
49.9
38.5
51.7
51.7
26.1
Table 7: Stage 1 target ablation (MLLMU-Bench, 10%). The IDK-inducing target is replaced with a blank text or random-vector.
Image Condition
LLaVA-1.5-7B
Qwen3-VL-8B
Forget ↓
Retain ↑
Forget ↓
Retain ↑
Clean
49.5
52.3
59.8
65.1
Gaussian noise
53.4
51.3
60.4
63.6
JPEG compression
50.1
52.7
57.2
63.3
Horizontal flip
51.3
51.2
58.3
63.3
Gaussian blur
54.2
52.6
59.2
63.7
Table 8: Image-side robustness stress test. ROUGE-L scores on the 5% forget split of MLLMU-Bench under inference-time image perturbations.
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
Forget ↓
Retain ↑
Celebrity ↑
Method
Cls
ROUGE
Cls
ROUGE
Cls
ROUGE
Vanilla
43.2
57.5
38.6
55.4
52.3
24.4
single-stage
32.8
38.7
29.9
36.0
19.3
26.1
IDK (ours)
39.2
49.9
38.5
51.7
51.7
26.1
Appendix
Table 9: Single-stage vs. two-stage unlearning (MLLMU-Bench, 10%). Single-stage applies all three Stage 1 losses directly to the vision encoder in a single joint update, skipping the target-then-path decoupling of our two-stage method (IDK). The single-stage variant collapses across retain and celebrity metrics, and its lower forget scores reflect general degradation rather than selective forgetting.
Figure 3: Loss trajectories during the single-stage joint update on MLLMU-Bench (10% forget). Each panel shows one of the four loss components ( LCE , Lalign , Lnorm , Ltotal ) under four ablation combinations of the auxiliary terms. Removing any term lets its own loss grow rather than stay bounded, and the total loss spikes sharply at the start before settling, causing irreversible retain damage under the strict setting.
Fisher decomposition
Unlearning (pre vs. post)
Forget
cos-sim
rel ℓ2
Retain cos
Forget cos
5 %
0.9882
0.160
0.781
0.621
10 %
0.9873
0.165
0.668
0.606
15 %
0.9868
0.167
0.787
0.661
Appendix
Table 10: Identity-balanced Fisher: decomposition accuracy and stability. Left: Ffull−Ff matches the directly computed Fr . Right: diagonal Fisher cosine before and after unlearning. LLaVA-1.5-7B on ReMem.
Figure 4: Per-layer gradient cosine on the vanilla vision encoder. Within-group cosines ( Gf-Gf and Gr-Gr ) rise in later layers, indicating coherent update directions within each group. The across-group cosine ( Gf-Gr ) becomes negative in late layers, showing that a forget-direction update simultaneously pushes retain features in the opposite direction. LLaVA-1.5-7B, MLLMU-Bench, 15% forget split.
Figure 5: PCA projection of pretrained LLaVA hidden states. Each point is the last-layer LLM hidden state of a single question. Visual perception questions (blue) and identity questions (green) appear visually distinguishable in 2D PCA even in the pretrained model, motivating the quantitative analysis in Table 11 .
Model
∣cos(PC1,d)∣
Fisher (PC1)
Raw ℓ2 dist.
Pretrained
0.647
1.52
40.7
Vanilla
0.985
19.38
46.4
Appendix
Table 11: Principal component alignment with identity–visual perception separation. ∣cos(PC1,d)∣ is the cosine similarity between PC1 and the centroid-difference vector d in the original 4096-dimensional space. Fisher denotes the Fisher criterion (between-group / within-group variance) along PC1.
Visual Perception
Identity Question
Model
Layer
NMI
ARI
NMI
ARI
LLaVA-1.5-7B
12
0.390
-0.020
0.294
-0.021
24
0.365
-0.019
0.481
0.150
32 (last)
0.405
0.009
0.935
0.820
Qwen3-VL-8B
12
0.551
0.149
0.535
0.229
24
0.358
-0.023
0.302
-0.021
Appendix
Table 12: Layer-wise clustering of hidden states. NMI and ARI quantify the alignment between representation clusters and visual-perception or identity-question labels at selected intermediate and final LLM layers.
Figure 6: Clustering on images not seen during fine-tuning. Last-layer LLM hidden states of the MLLMU-Bench-finetuned vanilla model, evaluated on ReMem images that were never seen during training. Visual perception questions still cluster strongly by question template, but identity questions no longer cluster strongly by personal identity. The identity clustering observed in Section 3.1 is therefore a fine-tuning artifact, not a property of the underlying representation.
Figure 7: Clustering robustness across model family and tuning scope (MLLMU-Bench, 5%). Top: LLaVA-1.5-7B with LoRA on full vision+LLM SFT. Middle: LLaVA-1.5-7B with LLM-only LoRA tuning. Bottom: Qwen3-VL-8B with LLM-only LoRA tuning. All three exhibit the same pattern: identity questions cluster by image identity, visual perception questions cluster by question template.
Figure 8: Clustering pattern across pretrained, vanilla, and post-unlearning models (MLLMU-Bench, 5%). Top: pretrained LLaVA groups all subsets by question template with negligible identity structure. Middle: vanilla SFT develops strong image-identity clustering on identity questions, while held-out identity and visual perception questions stay close to pretrained. Bottom: after unlearning, image-identity clustering on identity questions drops sharply and question-template grouping rises, while held-out identity and visual perception questions show no noticeable change.
Method
EM f ↓
EM r ↑
ROUGE ↑
Exp ↓
EM t ↓
VP to IDK (baseline)
38.3
78.8
89.2
63.9
35.1
VP to Blank text
40.0
74.7
87.1
63.4
33.9
Vision Encoder to IDK
43.0
77.6
60.1
51.2
60.4
Appendix
Table 13: Stage 1 intervention design choices (LLaVA-1.5-7B, ReMem, 10%). Two axes are varied: intervention type (visual prompt T vs. direct vision-encoder fine-tuning) and target response type (IDK refusal vs. blank text). All three variants achieve comparable forget/retain trade-offs, supporting the claim in Section 3.2 that the selective-transfer effect is robust to specific design choices within this family.
Figure 9: Visual prompt transfer saturation. IDK-rate on the full identity-question set as the number of questions used to learn T varies from 1 to 10. Blue: questions used during VP learning. Red: held-out questions. Green: all identity questions. The rate saturates around 8 questions, which we adopt as the default throughout the main experiments.
Figure 10: Visualization of the learned visual prompt T on MLLMU-Bench. From left to right: T rescaled to the visible range, raw absolute magnitudes, and per-channel (R, G, B) maps. Because MLLMU-Bench images are close-up portraits, T acquires face-landmark-like structure reflecting the geometry of the forget-set images.
Method
EM f ↓
EM r ↑
ROUGE ↑
Exp ↓
EM t ↓
Vanilla
100.0
100.0
100.0
68.2
100.0
(a) golatkar2020eternal
96.8
100.0
100.0
100.0
96.4
(b) mckinney2026gaussnewton
72.0
79.7
61.2
56.5
61.0
(c) li2024wmdp
69.5
100.0
100.0
55.5
68.8
Ours
51.1
72.5
85.6
63.2
48.0
Appendix
Table 14: Related-method comparison adapted to the MLLM setting (LLaVA-1.5-7B, ReMem, 15%). Three baselines drawn from vision-only and LLM-only unlearning literature, referred to as (a), (b), (c) in the surrounding text. Notation follows Table 2 . All three rely on retain images or forget-side question text during unlearning, while ours uses only forget images.
Figure 11: Per-metric training trajectory of our method (LLaVA-1.5-7B, ReMem, 10%). Forget, retain, and test EM scores over 100 epochs. The metrics settle into a plateau around epoch 45 and remain stable thereafter.
Figure 12: Qualitative inference samples on ReMem. Top: retain case where ours recalls the identity correctly. Middle: forget case where ours swaps the identity to a different person. Bottom: a representative retain failure of ours, where the identity is misrecognized. GA and NPO produce degenerate outputs across all three cases.
λCE:λalign:λnorm
EM f ↓
EM r ↑
ROUGE ↑
Exp ↓
EM t ↓
1:1:1
76.7
97.8
98.7
68.9
33.9
1:10:1
75.0
95.4
97.6
68.9
63.1
1:1:0.1
10.0
38.6
68.3
59.0
58.3
1:10:0.1 (baseline)
38.3
78.8
89.2
63.9
35.1
Appendix
Table 15: Stage 1 loss-weight ablation (LLaVA-1.5-7B, ReMem, 10%). Weights are (λCE:λalign:λnorm) in Eq. ( 4 ), IDK trigger fixed.
MLLMU-Bench
ReMem
Method
Cls f↓
ROUGE f↓
Cls r↑
ROUGE r↑
Cls c↑
ROUGE c↑
EM f↓
EM r↑
ROUGE ↑
Exp ↓
EM t↓
Forget 15%
Vanilla
42.1
56.1
38.5
55.5
52.3
24.4
100.0
100.0
100.0
68.2
100.0
retain utilizing
GA_Diff
39.7
47.0
36.6
52.4
51.0
23.0
0.0
94.7
32.3
35.9
0.0
KL_Min
24.0
11.1
37.4
38.0
54.4
14.7
0.0
0.0
0.0
47.6
0.0
MMUnlearner
35.7
45.2
36.1
50.6
37.7
24.2
2.2
81.9
91.0
45.3
0.0
Appendix
Table 16: Forget 15% results on LLaVA-1.5-7B. Continuation of Table 2 for the 15% forget split, with the same baseline groupings (retain-utilizing, forget-only, ours). Subscripts f , r , c , t denote forget, retain, celebrity, and test splits. Exp is the ReMem Exposure metric.
Method
Cls f↓
R f↓
Cls r↑
R r↑
Cls c↑
R c↑
Forget 15%
Vanilla
70.1
68.2
68.8
71.3
74.4
43.7
retain-utilizing
GA_Diff
65.9
58.7
66.5
67.3
73.4
42.2
KL_Min
5.6
5.4
54.9
63.6
68.5
35.4
MMUnlearner
47.7
46.9
57.0
54.0
58.4
36.3
Appendix
Table 17: Forget 15% results and native-resolution inference on Qwen3-VL-8B. Top: Forget 15% results omitted from Table 3 for space. Bottom: native-resolution inference at Forget 5%. Baseline groupings follow Table 3 .
LLaVA-1.5-13B
InternVL3-8B
Method
ROUGE f↓
ROUGE r↑
ROUGE f↓
ROUGE r↑
Vanilla
62.2
59.9
67.9
67.6
retain-utilizing
KL_Min
46.6
55.1
66.2
68.4
MMUnlearner
OOM
OOM
52.8
59.9
forget-only
Appendix
Table 18: Additional backbone results on MLLMU-Bench at the 10% forget split. AIM achieves a competitive forget–retain trade-off on both LLaVA-1.5-13B and InternVL3-8B.
Method
LLaVA-1.5-7B
Qwen3-VL-8B
5% Forget
10% Forget
5% Forget
10% Forget
Vanilla
67.6
67.6
74.1
74.1
MMUnlearner
30.1
45.4
49.2
14.3
GA
0.0
0.1
75.2
73.9
NPO
59.2
0.0
75.5
62.2
Ours
65.9
60.9
73.4
73.8
Appendix
Table 19: VQAv2-10K accuracy after unlearning. Accuracy is evaluated on 10K randomly sampled examples from the VQAv2 validation set.
Asset
Type
License
MLLMU-Bench
Dataset
Not specified
ReMem
Dataset
Not specified
LLaVA-1.5-7B
Base model
Llama 2 Community License
Qwen3-VL-8B-Instruct
Base model
Apache 2.0
MMUnlearner
Baseline
Not specified
MANU
Baseline
Not specified
Appendix
Table 20: Licenses of datasets, base models, and baseline methods used in this work.
Removing a specific individual's information from multimodal large language models (MLLMs) is often needed after deployment, but existing methods rely on a retain set, which is hardest to obtain at that point, and rebuilding it recreates the privacy exposure that unlearning aims to remove. Forgetting from the forget set alone instead damages the shared visual-language computation, harming perception. We cast retain-free unlearning as a localization problem: causal tracing, weight transplant, and Fisher overlap all point to early-to-mid decoder MLPs as the layers where identity information is stored and, unlike other module families, can be modified without substantially disrupting vision. We turn this into Pathway-Aware Visual-attribute Anchoring (PAVA), which confines updates to these layers and pairs a forget loss with a visual-attribute anchor that preserves image-grounded behavior by distilling the model's own pre-unlearning answers from the forget images alone. On MLLMU-Bench and ReMem, PAVA gives the strongest forget-retain trade-off among forget-set-only methods and remains competitive with retain-based baselines.
Multimodal large language models (MLLMs) can memorize and disclose sensitive information, motivating machine unlearning to remove designated knowledge while preserving model utility. Existing privacy-oriented benchmarks mainly study profile-level deletion, whereas practical requests often require finer-grained forgetting: removing a sensitive attribute while retaining other information about the same identity. We introduce attribute-level MLLM unlearning and construct AMU-Bench, covering long-text, numeric, and short-text targets, multiple forget ratios, and diverse question types. Our evaluation shows that target and retain attributes share identity-specific and visual evidence, making selective forgetting prone to residual leakage and collateral degradation. To address this challenge, we propose Causal Localization and Retain-Aware Projection (CLRP), a lightweight, training-free framework that localizes causally relevant layers and projects intermediate representations away from target-specific subspaces while preserving retain-related information. Experiments across MLLMs with different architectures and scales show that CLRP achieves a strong forgetting--retention trade-off on AMU-Bench. For example, at a 5% forget ratio, CLRP reduces Attribute Forget Set cloze accuracy from 20.00% to 4.00% while improving Attribute Retain Set accuracy from 14.77% to 17.79% in the reported LLaVA-1.5-7B and Qwen2.5-VL-3B results. These findings demonstrate effective and selective attribute-level unlearning without modifying model parameters.
Junkai Lin, Junkai Chen, Siqi Hou +6
Institute of Automation, Chinese Academy of Sciences · The Chinese University of Hong Kong · The Australian National University +3
Multimodal Large Language Models (MLLMs) may memorize sensitive cross-modal information during pretraining. However, existing MLLM unlearning benchmarks rely on synthetic knowledge injection or complete subject-level deletion, which fail to capture realistic, personalized deletion requests that require fine-grained factual control. In this paper, we introduce PPU-Bench, a real-world and fine-tuning-free benchmark for personalized partial unlearning in MLLMs. PPU-Bench contains 24K multimodal and unimodal samples derived from pre-existing knowledge of 500 public figures under three progressively challenging settings: Complete, Selective, and Personalized unlearning. The benchmark evaluates whether methods can remove target knowledge while preserving non-target facts, model utility, and cross-modal consistency. Extensive experiments show that Complete Unlearning often suppresses visual identity rather than factual knowledge, while Selective and Personalized Unlearning expose significant forget--retain trade-offs and challenges in intra-subject factual boundaries. Robustness analysis under cross-image and prompt-based attacks reveals distinct vulnerabilities across different unlearning settings. Motivated by these findings, we propose Boundary-Aware Optimization (BAO), which explicitly models intra-subject forget-retain boundaries. Experimental results on two representative methods demonstrate that BAO can effectively enforce intra-subject factual boundaries.
Jiahui Guang, Zexun Zhan, Zhenlin Xu +5
Harbin Institute of Technology, Shenzhen, China · Sichuan University, Chengdu, China · Harbin Institute of Technology, Weihai, China +3