Vision-Language Models (VLMs) have demonstrated remarkable capabilities in multi-modal understanding for tasks such as Visual Question Answering (VQA), which requires models to infer outputs based on visual and textual context simultaneously. However, models may memorize some private and/or sensitive information during training and regurgitate it in inference. Recently, machine unlearning has been leveraged to address such leakage of private data in LLMs. VLMs add a layer of complexity to this process, as the visual context in the query may additionally contain sensitive information. To address this issue, we explore unlearning for VLMs, specifically for the VQA task. We explore the role of visual tokens for output generation in VLMs using cross-modal attention and utilize it to formulate Cross-Modal Attention Guided Unlearning (CAGUL), a lightweight and efficient VLM unlearning framework. In contrast to computationally expensive finetuning methods, CAGUL utilizes external modules to encode unlearning information in visual tokens of low importance for relevant queries. Empirical results show that the transformed visual tokens prevent leakage and retain reference model behavior to achieve desirable trade-offs in forget quality and model utility without altering pretrained model parameters.
Figures & tables
Fig. 1 : Privacy-focused VLM unlearning; Person marked in red requests to remove only health-related info from the model, which forms Df ; Retain set Dr includes non-private queries of the same person and all queries from individuals that do not request removal
Fig. 2 : Overview of CAGUL Framework comprising three components: (1) a discriminator that takes visual token embeddings as input and determines whether the corresponding image belongs to an individual from the forget set, (2) a cross-modal attention guided selection mechanism that selects k least attended visual tokens w.r.t to the text, and (3) a visual token encoder that encodes unlearning information in the selected visual tokens
∣M~∣
20
40
60
80
Df
273
544
817
1078
Dr
7727
7456
7183
6922
Dnp
127
256
383
522
TABLE I : Dataset statistics per ∣M~∣
Method
Forget
Retain
General
Df
Dr
Dnp
MME
POPE
Rouge (↓)
EM (↓)
APE (↓)
MinK (↓)
Rouge (↑)
Rouge (↑)
EM (↑)
TR (↑)
Acc. (↑)
Acc. (↑)
Pretrain
26.00
0.00
0.00
0.79
19.13
12.63
0.10
7.63
77.34
87.41
Finetune
57.33
39.68
24.59
43.32
84.55
66.88
30.69
50.63
30.58
37.70
Retrain
55.91
15.75
11.45
16.84
92.24
77.89
47.12
58.85
23.50
29.20
Guardrail
58.58
54.40
32.08
57.59
87.86
79.48
48.18
90.31
28.81
35.69
TABLE II: Unlearning metrics of CAGUL compared with baselines on LLaMA-3.2-11B-Vision-Instruct
Fig. 3 : Unlearning performance comparison for LLaMA-3.2-11B-Vision-Instruct ; forget quality is plotted as 1 - harmonic mean for visualization purposes, upper-right is optimal
Method
Trainable Params
Exec. time (s)
Retrain
9.8 B
4272
GA
419 M
149
GA+GD
419 M
1855
GA+KL
419 M
2951
PO+GD
419 M
1911
CAGUL
293 M
682
TABLE III : Computational efficiency of CAGUL compared with baselines for LLaMA-3.2-11B-Vision-Instruct
Figure 7
Fig. 6 : Overall results from ablation study
Hyperparameters
Finetune/Retrain
GA
GA+GD
GA+KL
PO+GD
CAGUL
Learning rate
2×10−5
1×10−6
2×10−5
1×10−4
3×10−4
2×10−5
Batch size
4
4
4
Epochs
10
10
2+10
Dropout
-
0.05
-
LoRA Rank r
-
128
-
LoRA Alpha α
-
256
-
TABLE IV : Training hyperparameters for finetuning and unlearning methods; parameters not applicable to method are shown as -
Method
Forget
Retain
General
Df
Dr
Dnp
MME
POPE
Rouge (↓)
EM (↓)
APE (↓)
MinK (↓)
Rouge (↑)
Rouge (↑)
EM (↑)
TR (↑)
Acc. (↑)
Acc. (↑)
CAGUL
30.84
1.70
0.43
14.86
85.15
84.32
64.35
91.67
30.58
37.70
w/o Cϕ
30.94
1.84
1.28
22.93
96.07
84.66
65.94
92.21
28.94
37.41
w/o Fψ
56.56
37.75
23.67
41.54
84.50
65.46
28.51
90.39
30.58
37.70
w/ zero out
57.12
38.94
24.47
42.88
84.53
66.18
29.89
90.39
30.58
37.70
TABLE V : Fine-grained results from ablation study on LLaMA-3.2-11B-Vision-Instruct
Method
Forget
Retain
General
Df
Dr
Dnp
MME
POPE
Rouge (↓)
EM (↓)
APE (↓)
MinK (↓)
Rouge (↑)
Rouge (↑)
EM (↑)
TR (↑)
Acc. (↑)
Acc. (↑)
Pretrain
30.61
0.98
0.37
0.00
36.27
34.01
0.10
11.28
86.68
86.94
Finetune
74.07
56.92
45.62
2.83
72.49
68.18
53.51
89.85
67.58
79.24
Retrain
44.04
14.52
10.70
1.44
72.68
67.80
54.23
91.08
71.31
76.41
Guardrail
52.34
44.67
35.61
3.52
70.53
65.31
45.17
89.75
66.94
66.32
TABLE VI : Unlearning metrics of CAGUL compared with baselines on Qwen-2.5-VL-7B-Instruct
Fig. 7 : CAGUL aggregate performance comparison against baselines on Qwen-2.5-VL-7B-Instruct
Vision-language model unlearning studies how to remove specified concepts, identities, associations, or behaviors while preserving unrelated knowledge and capabilities.