Vision-Language Models (VLMs) have demonstrated remarkable capabilities in multi-modal understanding for tasks such as Visual Question Answering (VQA), which requires models to infer outputs based on visual and textual context simultaneously. However, models may memorize some private and/or sensitive information during training and regurgitate it in inference. Recently, machine unlearning has been leveraged to address such leakage of private data in LLMs. VLMs add a layer of complexity to this process, as the visual context in the query may additionally contain sensitive information. To address this issue, we explore unlearning for VLMs, specifically for the VQA task. We explore the role of visual tokens for output generation in VLMs using cross-modal attention and utilize it to formulate Cross-Modal Attention Guided Unlearning (CAGUL), a lightweight and efficient VLM unlearning framework. In contrast to computationally expensive finetuning methods, CAGUL utilizes external modules to encode unlearning information in visual tokens of low importance for relevant queries. Empirical results show that the transformed visual tokens prevent leakage and retain reference model behavior to achieve desirable trade-offs in forget quality and model utility without altering pretrained model parameters.
Figures & tables
Fig. 1 : Privacy-focused VLM unlearning; Person marked in red requests to remove only health-related info from the model, which forms Df ; Retain set Dr includes non-private queries of the same person and all queries from individuals that do not request removal
Fig. 2 : Overview of CAGUL Framework comprising three components: (1) a discriminator that takes visual token embeddings as input and determines whether the corresponding image belongs to an individual from the forget set, (2) a cross-modal attention guided selection mechanism that selects k least attended visual tokens w.r.t to the text, and (3) a visual token encoder that encodes unlearning information in the selected visual tokens
∣M~∣
20
40
60
80
Df
273
544
817
1078
Dr
7727
7456
7183
6922
Dnp
127
256
383
522
TABLE I : Dataset statistics per ∣M~∣
Method
Forget
Retain
General
Df
Dr
Dnp
MME
POPE
Rouge (↓)
EM (↓)
APE (↓)
MinK (↓)
Rouge (↑)
Rouge (↑)
EM (↑)
TR (↑)
Acc. (↑)
Acc. (↑)
Pretrain
26.00
0.00
0.00
0.79
19.13
12.63
0.10
7.63
77.34
87.41
Finetune
57.33
39.68
24.59
43.32
84.55
66.88
30.69
50.63
30.58
37.70
Retrain
55.91
15.75
11.45
16.84
92.24
77.89
47.12
58.85
23.50
29.20
Guardrail
58.58
54.40
32.08
57.59
87.86
79.48
48.18
90.31
28.81
35.69
TABLE II: Unlearning metrics of CAGUL compared with baselines on LLaMA-3.2-11B-Vision-Instruct
Fig. 3 : Unlearning performance comparison for LLaMA-3.2-11B-Vision-Instruct ; forget quality is plotted as 1 - harmonic mean for visualization purposes, upper-right is optimal
Method
Trainable Params
Exec. time (s)
Retrain
9.8 B
4272
GA
419 M
149
GA+GD
419 M
1855
GA+KL
419 M
2951
PO+GD
419 M
1911
CAGUL
293 M
682
TABLE III : Computational efficiency of CAGUL compared with baselines for LLaMA-3.2-11B-Vision-Instruct
Figure 7
Fig. 6 : Overall results from ablation study
Hyperparameters
Finetune/Retrain
GA
GA+GD
GA+KL
PO+GD
CAGUL
Learning rate
2×10−5
1×10−6
2×10−5
1×10−4
3×10−4
2×10−5
Batch size
4
4
4
Epochs
10
10
2+10
Dropout
-
0.05
-
LoRA Rank r
-
128
-
LoRA Alpha α
-
256
-
TABLE IV : Training hyperparameters for finetuning and unlearning methods; parameters not applicable to method are shown as -
Method
Forget
Retain
General
Df
Dr
Dnp
MME
POPE
Rouge (↓)
EM (↓)
APE (↓)
MinK (↓)
Rouge (↑)
Rouge (↑)
EM (↑)
TR (↑)
Acc. (↑)
Acc. (↑)
CAGUL
30.84
1.70
0.43
14.86
85.15
84.32
64.35
91.67
30.58
37.70
w/o Cϕ
30.94
1.84
1.28
22.93
96.07
84.66
65.94
92.21
28.94
37.41
w/o Fψ
56.56
37.75
23.67
41.54
84.50
65.46
28.51
90.39
30.58
37.70
w/ zero out
57.12
38.94
24.47
42.88
84.53
66.18
29.89
90.39
30.58
37.70
TABLE V : Fine-grained results from ablation study on LLaMA-3.2-11B-Vision-Instruct
Method
Forget
Retain
General
Df
Dr
Dnp
MME
POPE
Rouge (↓)
EM (↓)
APE (↓)
MinK (↓)
Rouge (↑)
Rouge (↑)
EM (↑)
TR (↑)
Acc. (↑)
Acc. (↑)
Pretrain
30.61
0.98
0.37
0.00
36.27
34.01
0.10
11.28
86.68
86.94
Finetune
74.07
56.92
45.62
2.83
72.49
68.18
53.51
89.85
67.58
79.24
Retrain
44.04
14.52
10.70
1.44
72.68
67.80
54.23
91.08
71.31
76.41
Guardrail
52.34
44.67
35.61
3.52
70.53
65.31
45.17
89.75
66.94
66.32
TABLE VI : Unlearning metrics of CAGUL compared with baselines on Qwen-2.5-VL-7B-Instruct
Fig. 7 : CAGUL aggregate performance comparison against baselines on Qwen-2.5-VL-7B-Instruct
Vision-language model unlearning studies how to remove specified concepts, identities, associations, or behaviors while preserving unrelated knowledge and capabilities.
Machine unlearning is widely used to remove hazardous knowledge from large language models. Modern Vision-Language Models (VLMs), however, process both text and visual inputs, raising a fundamental security question: does unlearning in one modality transfer to the other? We present the first systematic, bidirectional study of cross-modal unlearning transfer across three VLM architectures: LLaVA-1.5 (MLP projection), InstructBLIP (Q-Former), and IDEFICS (gated cross-attention). We find that unlearning transfers across modalities, but the transfer is asymmetric and incomplete. In some cases, text unlearning strongly transfers to vision. However, this robustness is not preserved under typographic attacks that manipulate the visual presentation of text. Under such attacks, previously unlearned knowledge can be readily recovered, indicating shallow unlearning. To address the transfer gap and shallow robustness, we propose CrossInf, an influence-guided mitigation strategy. Motivated by the observation that different model components contribute unequally to cross-modal transfer, CrossInf focuses unlearning on transformer blocks that most influence cross-modal generalization. It reduces the transfer gap by more than half in architectures with strong fusion, while preserving model utility. It also improves robustness under typographic attacks, reducing the attack success rate to near zero. We further conduct human evaluation with three annotators (κ=0.77) to validate our findings. Finally, we analyze shallow unlearning using Centered Kernel Alignment (CKA), providing insights into the observed transfer behavior and robustness limitations.
Sudharshan Balaji, Yili Ren, Guangjing Wang +2
University of South Florida · USA · University of Massachusetts Lowell
Vision-language model (VLM) unlearning is often treated as an objective-design problem: given multimodal forget data, the goal is to optimize a loss that removes undesirable behavior while preserving retained capabilities. We argue that this view overlooks an equally important data-centric question: how should each forget example be represented before optimization? Existing methods typically keep image-conditioned failures in their original image-text form, implicitly assuming that pixels are the right route for unlearning whenever pixels appear in the input. We challenge this assumption by constructing modality-decomposed versions of VLM benchmarks, enabling controlled comparisons among image-only, text-only, and multimodal representations of the same examples. Our modality-mixing experiments show that unsafe instruction-following behavior is more effectively suppressed through textual representations than through images alone, while purely text-only unlearning can still weaken visual grounding when the target behavior depends on visual evidence. Motivated by this trade-off, we introduce ModRoute, a gradient-guided per-sample modality routing strategy that selectively converts high visual-pressure examples to text-only form while keeping the remaining examples multimodal. Notably, on VLGuard with LLaVA-1.5-7B, ModRoute has the strongest composite unlearning-utility score of 0.847 at γ=0.5, a 2.5% improvement over the best random-switching score and ≥46% better than pure multimodal or text unlearning. Overall, our results show that effective VLM unlearning depends not only on the objective, but also on the composition and per-sample modality routing of the unlearning data.
Laura Yao, Haochen Zhang, Jinhao Duan +2
University of North Carolina at Chapel Hill · Michigan State University
Vision-language models (VLMs) may memorize undesirable information from training data, motivating growing interest in machine unlearning. In this work, we present the first systematic survey and robustness analysis of VLM unlearning. We provide a comprehensive taxonomy and review of existing VLM unlearning methods, together with unified evaluations under multiple prompt settings. We then propose three attack paradigms to examine whether forgotten multimodal knowledge can be reactivated through contextual prompting or downstream retraining. Extensive experiments show that many existing methods remain vulnerable under these attacks, indicating that current approaches often hide rather than fully remove target knowledge. Our study provides new insights into the robustness and limitations of current VLM unlearning methods and highlights the need for more reliable multimodal unlearning strategies. Code is available at https://github.com/XMUDeepLIT/VLM-UnL-Attack.