The Pen is Mightier than the Pixel: Natural Textual Unlearnable Example for Large Vision Language Models
Authors: Chenyu Yi, Jiawen Zong, Sami Hormi, Bingquan Shen, Xudong Jiang, Alex Chichung Kot
Organizations: Nanyang Technological University, School of Electrical & Electronic Engineering · DSO National Laboratories, Singapore · National University of Singapore · Shenzhen MSU-BIT University
The proliferation of Large Vision Language Models (LVLMs), trained on vast and often sensitive datasets, raised concerns regarding data privacy and unauthorized usage. Unlearnable Examples (UEs), which subtly perturb training data to mislead the models, have emerged as a promising defense method. This paper presents the first exploration of UEs specifically on LVLMs, which poses unique challenges due to their inherent multimodal nature. We propose a novel text-based perturbation method, Natural Textual Unlearnable Examples (NTUE), which can effectively mislead LVLMs by subtly manipulating the image label, establishing a shortcut between an identity of interest and a separate target, without introducing factual misinformation or affecting readability. Comprehensive experiments compared with traditional UE methods demonstrate the superior performance of NTUE. Our findings show that NTUE achieves a high success rate with a poison rate as low as 5%, underscoring its potential as a practical and robust solution for protecting multimodal data in the era of LVLMs.
Figures & tables
Figure 1: Training of multimodal unlearnable examples on large vision-language models with clean captions and Natural Text Unlearnable Example captions.
Figure 2: Natural Text Unlearnable Example training process and evaluation process
Figure 3: Identification accuracy of LLaVA-1.5 7B finetuned with Trump images across different UE methods.
No. of UEs/Method
100
50
20
10
5
Clean
97
96.5
96.5
87.5
70
TAP
98.5
97.5
85.5
64
54
EM
95
97
90
81
MEM
98
96
83
70
72
NTUE(AA)+PGD
98.5
97.5
85
63.5
43
NTUE(AA)
93.5
94.5
91
84
63
Table 1: Identification accuracy of LLaVA-1.5 7B finetuned with Trump images across different UE methods.
Clean
70 poison
Base
Target
PGD
NTUE(AA)
NTUE(AA) +PGD
NTUE(DA)
NTUE(DA) +PGD
Angelina Jolie
Jennifer Lawrence
92.2
100
84.4
100
0
94.4
Johnny Depp
92.2
100
98.9
100
0
97.8
Will Smith
92.2
100
94.4
100
0
95.5
Megan Fox
92.2
100
75.5
100
0
94.4
Brad Pitt
Leonardo Dicaprio
55.6
100
95.5
97.8
0
0
Table 2: Identification accuracy of various celebrities pairs
School of Computing and Augmented Intelligence, Arizona State University, Tempe, AZ, USA · Department of Computer Science and Engineering, Texas A&M University, College Station, TX, USA.