HQA-VLAttack: Towards High Quality Adversarial Attack on Vision-Language Pre-Trained Models
Authors: Han Liu, Jiaqi Li, Zhi Xu, Xiaotong Zhang, Xiaoming Xu, Fenglong Ma, Yuanman Li, Hong Yu
Organizations: Dalian University of Technology, Dalian, China · The Pennsylvania State University, Pennsylvania, USA · Shenzhen University, Shenzhen, China
Black-box adversarial attack on vision-language pre-trained models is a practical and challenging task, as text and image perturbations need to be considered simultaneously, and only the predicted results are accessible. Research on this problem is in its infancy, and only a handful of methods are available. Nevertheless, existing methods either rely on a complex iterative cross-search strategy, which inevitably consumes numerous queries, or only consider reducing the similarity of positive image-text pairs but ignore that of negative ones, which will also be implicitly diminished, thus inevitably affecting the attack performance. To alleviate the above issues, we propose a simple yet effective framework to generate high-quality adversarial examples on vision-language pre-trained models, named HQA-VLAttack, which consists of text and image attack stages. For text perturbation generation, it leverages the counter-fitting word vector to generate the substitute word set, thus guaranteeing the semantic consistency between the substitute word and the original word. For image perturbation generation, it first initializes the image adversarial example via the layer-importance guided strategy, and then utilizes contrastive learning to optimize the image adversarial perturbation, which ensures that the similarity of positive image-text pairs is decreased while that of negative image-text pairs is increased. In this way, the optimized adversarial images and texts are more likely to retrieve negative examples, thereby enhancing the attack success rate. Experimental results on three benchmark datasets demonstrate that HQA-VLAttack significantly outperforms strong baselines in terms of attack success rate.
Figures & tables
Figure 1: The average cosine similarity of image-text pairs optimized by SGA, DRA, and HQA-VLAttack on the Flickr30K dataset using ALBEF as the surrogate model.
Figure 2: The overall of HQA-VLAttack. First, the Text Attack module determines the substitute word set and then generates the textual adversarial example ti′ . Second, the Image Attack module applies layer-importance based initial image adversarial example generation to obtain the initial adversarial image example vi′ , followed by contrastive learning-based image adversarial example optimization for further refinement. Finally, the optimized adversarial examples are fed into the victim model.
Figure 3: The cosine similarity of image feature and [CLS] token embedding across Layers.
Flickr30K Dataset
Surrogate Model
Victim Model
ALBEF
TCL
CLIP ViT
CLIP CNN
Attack Method
TR R@1
IR R@1
TR R@1
IR R@1
TR R@1
IR R@1
TR R@1
IR R@1
ALBEF
PGD
52.45 ∗
58.65 ∗
3.06
6.79
8.96
13.21
10.34
14.65
BERT-Attack
11.57 ∗
27.46 ∗
12.64
28.07
29.33
43.17
32.69
46.11
Sep-Attack
65.69 ∗
73.95 ∗
17.60
32.95
31.17
45.23
32.83
45.49
Co-Attack
77.16 ∗
83.86 ∗
15.21
29.49
23.60
36.48
25.12
38.89
Table 1: Attack success rate (%) in image-text retrieval on the Flickr30K dataset. We present the attack success rate metric R@1 for both IR and TR tasks, indicating the success of attacks at Rank 1. ∗ indicates white-box attacks.
MSCOCO Dataset
Surrogate Model
Victim Model
ALBEF
TCL
CLIP ViT
CLIP CNN
Attack Method
TR R@1
IR R@1
TR R@1
IR R@1
TR R@1
IR R@1
TR R@1
IR R@1
ALBEF
PGD
76.70 ∗
86.30 ∗
12.46
17.77
13.96
23.10
17.45
23.54
BERT-Attack
24.39 ∗
36.13 ∗
24.34
33.39
44.94
52.28
47.73
54.75
Sep-Attack
82.60 ∗
89.88 ∗
32.83
42.92
44.03
54.46
46.96
55.88
Co-Attack
79.87 ∗
87.83 ∗
32.62
43.09
44.89
54.75
47.30
55.64
Table 2: Attack success rate (%) in image-text retrieval on the MSCOCO Dataset. We present the attack success rate metric R@1 for both IR and TR tasks, indicating the success of attacks at Rank 1. ∗ indicates white-box attacks.
ITR → VG
ITR → IC
Attack
Val ↓
TestA ↓
TestB ↓
B@4 ↓
METEOR ↓
ROUGE-L ↓
CIDEr ↓
SPICE ↓
Baseline
58.46
65.89
46.25
39.7
31.0
60.0
133.3
23.8
Co-Attack
54.26
61.80
43.81
37.4
29.8
58.4
125.5
22.8
SGA
53.55
61.19
43.71
34.8
28.4
56.3
116.0
21.4
DRA
53.88
61.18
43.38
34.8
28.4
56.4
115.9
21.4
HQA-VLAttack
46.48
54.31
36.90
31.8
26.8
54.1
104.6
19.8
Table 3: Cross-Task Transferability. The Baseline represents the original performance of IC and VG on clean data. We utilize ALBEF to generate multi-modal adversarial examples for attacking both Visual Grounding (VG) and Image Captioning (IC).
Figure 4: Ablation Study on Component Effectiveness and Batch Size Impact on Attack Success Rate.
Figure 5: Adversarial Transferability between GPT-4o and Claude-3.7 Sonnet. The images on the left show the responses generated by Claude-3.7 Sonnet when provided with adversarial images and the prompt “Describe this image”, while the images on the right display the outputs produced by GPT-4o under the same prompt.
Adversarial attacks on vision-language models exploit images, text, and multimodal interactions to manipulate outputs, bypass safety, or degrade reliability.
Vision-language models (VLMs) are vulnerable to adversarial image perturbations. Existing works based on adversarial training against task-specific adversarial examples are computationally expensive and often fail to generalize to unseen attack types. To address these limitations, we introduce Paraphrase-Decomposition-Aggregation (PDA), a training-free defense framework that leverages text augmentation to enhance VLM robustness under diverse adversarial image attacks. PDA performs prompt paraphrasing, question decomposition, and consistency aggregation entirely at test time, thus requiring no modification on the underlying models. To balance robustness and efficiency, we instantiate PDA as invariants that reduce the inference cost while retaining most of its robustness gains. Experiments on multiple VLM architectures and benchmarks for visual question answering, classification, and captioning show that PDA achieves consistent robustness gains against various adversarial perturbations while maintaining competitive clean accuracy, establishing a generic, strong and practical defense framework for VLMs during inference.
Jingning Xu, Haochen Luo, Chen Liu
Department of Computer Science City University of Hong Kong
With the rapid advancement and widespread application of vision-language pre-training (VLP) models, their vulnerability to adversarial attacks has become a critical concern. In general, the adversarial examples can typically be designed to exhibit transferable power, attacking not only different models but also across diverse tasks. However, existing attacks on language-vision models mainly rely on static cross-modal interactions and focus solely on disrupting positive image-text pairs, resulting in limited cross-modal disruption and poor transferability. To address this issue, we propose a Semantic-Augmented Dynamic Contrastive Attack (SADCA) that enhances adversarial transferability through progressive and semantically guided perturbation. SADCA progressively disrupts cross-modal alignment through dynamic interactions between adversarial images and texts. This is accomplished by SADCA establishing a contrastive learning mechanism involving adversarial, positive and negative samples, to reinforce the semantic inconsistency of the obtained perturbations. Moreover, we empirically find that input transformations commonly used in traditional transfer-based attacks also benefit VLPs, which motivates a semantic augmentation module that increases the diversity and generalization of adversarial examples. Extensive experiments on multiple datasets and models demonstrate that SADCA significantly improves adversarial transferability and consistently surpasses state-of-the-art methods. The code is released at https://github.com/LiYuanBoJNU/SADCA.
Yuanbo Li, Tianyang Xu, Cong Hu +3
School of Artificial Intelligence and Computer Science, Jiangnan University · Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey
Large Vision-Language Models (LVLMs) can be vulnerable to adversarial images that subtly bias their outputs toward plausible yet incorrect responses. We introduce a general, efficient, and training-free defense that combines image transformations with agentic data consolidation to recover correct model behavior. A key component of our approach is a two-stage detection mechanism that quickly filters out the majority of clean inputs. We first assess image consistency under content-preserving transformations at negligible computational cost. For more challenging cases, we examine discrepancies in a text-embedding space. Only when necessary do we invoke a powerful LLM to resolve attack-induced divergences. A key idea is to consolidate multiple responses, leveraging both their similarities and their differences. We show that our method achieves state-of-the-art accuracy while maintaining notable efficiency: most clean images skip costly processing, and even in the presence of numerous adversarial examples, the overhead remains minimal.