Towards Highly Transferable Vision-Language Attack via Semantic-Augmented Dynamic Contrastive Interaction
Authors: Yuanbo Li, Tianyang Xu, Cong Hu, Tao Zhou, Xiao-Jun Wu, Josef Kittler
Organizations: School of Artificial Intelligence and Computer Science, Jiangnan University · Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey
With the rapid advancement and widespread application of vision-language pre-training (VLP) models, their vulnerability to adversarial attacks has become a critical concern. In general, the adversarial examples can typically be designed to exhibit transferable power, attacking not only different models but also across diverse tasks. However, existing attacks on language-vision models mainly rely on static cross-modal interactions and focus solely on disrupting positive image-text pairs, resulting in limited cross-modal disruption and poor transferability. To address this issue, we propose a Semantic-Augmented Dynamic Contrastive Attack (SADCA) that enhances adversarial transferability through progressive and semantically guided perturbation. SADCA progressively disrupts cross-modal alignment through dynamic interactions between adversarial images and texts. This is accomplished by SADCA establishing a contrastive learning mechanism involving adversarial, positive and negative samples, to reinforce the semantic inconsistency of the obtained perturbations. Moreover, we empirically find that input transformations commonly used in traditional transfer-based attacks also benefit VLPs, which motivates a semantic augmentation module that increases the diversity and generalization of adversarial examples. Extensive experiments on multiple datasets and models demonstrate that SADCA significantly improves adversarial transferability and consistently surpasses state-of-the-art methods. The code is released at https://github.com/LiYuanBoJNU/SADCA.
Figures & tables
Figure 1: A comparison of our SADCA and existing frameworks. (a) and (b) illustrate the core concepts of SGA [ 18 ] and SA-AET [ 7 ] , respectively, where only one or two static interactions are performed between the visual and textual modalities, with the interactions being limited solely to positive pairs. (c) illustrates the core idea of the proposed SADCA, which continuously disrupts cross-modal interactions through dynamic contrastive interactions with both positive and negative pairs. Additionally, it leverages a semantic augmentation strategy to enrich the data samples, thereby diversifying the semantic information. The arrow represents the interaction between the visual and textual modalities. The dotted lines represent the generation of adversarial examples from the original examples. (d) demonstrates the effectiveness of the input transformation [ 30 ] in enhancing the adversarial attack transferability. Furthermore, we observe that using large number of iterations (LI) to attack the image modality can further improve the attack performance.
Source
Attack
ALBEF
TCL
CLIP ViT
CLIP CNN
Average
TR R@1
IR R@1
TR R@1
IR R@1
TR R@1
IR R@1
TR R@1
IR R@1
TR R@1
IR R@1
ALBEF
PGD
93.74
94.43
24.03
27.90
10.67
15.82
14.05
19.11
16.25
20.94
BERT-Attack
11.57
27.46
12.64
28.07
29.33
43.17
32.69
46.11
24.89
39.12
Co-Attack
97.08
98.36
39.52
51.24
29.82
38.92
31.29
41.99
33.54
44.72
SGA
99.90
99.93
87.88
88.05
36.69
46.78
39.59
49.78
54.72
61.54
SGA(LI)+SIA
100
100
99.37
99.02
49.82
56.19
54.79
60.99
67.99
72.07
Table 1: A comparison of SADCA with SOTA methods on the image-text retrieval (ITR) task using the Flickr30K dataset. The ”Source” column indicates the VLP model used to generate the multimodal adversarial examples. For both image retrieval (IR) and text retrieval (TR), we report the ASR (%) at Rank-1 (R@1). The ”Average” represents the average ASR on the black-box VLP models.
Attack
ITR → VG
ITR → IC
Val ↓
TestA ↓
TestB ↓
B@4 ↓
METEOR ↓
ROUGE-L ↓
CIDEr ↓
SPICE ↓
Clean
58.46
65.89
46.25
39.7
31.0
60.0
133.3
23.8
SGA
50.56
57.42
40.66
28.0
24.6
51.2
91.4
17.7
DRA
49.70
56.32
40.54
27.2
24.2
50.7
88.3
17.2
SA-AET
47.44
53.27
38.58
21.0
20.5
45.2
65.7
13.6
SADCA (ours)
46.78
52.41
37.10
17.4
17.9
42.2
50.3
10.7
Table 2: Cross-task transferability. We use ALBEF model to generate multimodal adversarial examples and apply them to attack the Visual Grounding (VG) task on the RefCOCO+ dataset and the Image Captioning (IC) task on the MSCOCO dataset. The ”Clean” denotes the performance of each task without any attack, where lower values indicate better effectiveness of the adversarial attack in both tasks.
Attack
LLaVA-1.5-7B
Qwen3-VL-8B
InternVL3-8B
Qwen2.5-VL-7B
GPT-5
GPT-4o-mini
Gemini-2.0
Claude-4.5
Clean
3.46
14.4
4.72
11.04
23.88
15.00
6.96
21.80
SGA
17.60
47.00
17.60
36.00
50.65
40.96
30.12
63.32
SGA(LI)+SIA
27.30
62.76
27.30
45.92
52.17
46.54
27.92
64.18
SA-AET
14.30
60.16
25.72
48.20
60.26
50.86
38.50
72.28
SA-AET(LI)+SIA
35.20
80.14
46.68
67.50
68.08
62.48
41.56
75.12
SADCA (ours)
40.34
86.34
69.06
82.82
78.61
79.12
52.06
79.02
Table 3: Adversarial transferability on LVLMs. We use ALBEF model to generate multimodal adversarial examples and apply them to attack the LVLMs on the Flickr30K dataset.
Figure 2: Visualization on Image Captioning and Visual Grounding Tasks.
Figure 3: The ASR (%) with different parameters, including the dynamic interaction number I , the number of semantic augmentations S , the negative samples number K , and the weighting factor λ . The adversarial examples are generated on the Flickr30K dataset using CLIP CNN as the source model and are evaluated on other black-box models.
ALBEF
TCL
CLIPViT
TR R@1
IR R@1
TR R@1
IR R@1
TR R@1
IR R@1
Strategy (1)
44.00
56.13
49.43
59.33
74.11
77.51
Strategy (2)
41.81
54.58
47.84
58.17
73.13
76.80
Strategy (3)
44.53
58.40
50.16
60.38
75.58
79.22
Strategy (4)
45.15
57.48
51.42
60.43
75.71
79.06
Table 4: Ablation study for negative sample selection strategy. The adversarial examples are generated by CLIP CNN .
Figure 4: Ablation study for different modules of SADCA. The adversarial examples are generated on the Flickr30K dataset using CLIP CNN as the source model and are evaluated on other black-box models.
Figure 5: Semantic Augmentation Module.
Source
Attack
ALBEF
TCL
CLIP ViT
CLIP CNN
Average
TR R@1
IR R@1
TR R@1
IR R@1
TR R@1
IR R@1
TR R@1
IR R@1
TR R@1
IR R@1
ALBEF
PGD
94.35
93.26
34.15
36.86
21.71
27.06
23.83
30.96
26.56
31.63
BERT-Attack
24.39
36.13
24.34
33.39
44.94
52.28
47.73
54.75
39.00
46.81
Co-Attack
94.95
97.87
65.22
72.41
55.28
62.33
56.68
66.45
59.06
67.06
SGA
99.95
99.94
87.46
88.17
63.72
69.71
63.91
70.78
71.70
76.22
SGA(LI)+SIA
100
100
98.49
98.01
75.96
79.83
76.75
81.21
83.73
86.35
Table 5: Comparison with SOTA methods on the image-text retrieval (ITR) task on the MSCOCO dataset. The ”Source” column indicates the VLP model used to generate the multimodal adversarial examples. For both image retrieval (IR) and text retrieval (TR), we report the ASR (%) at Rank-1 (R@1). The ”Average” represents the average ASR on the black-box VLP models.
Source
Attack
ALBEF
TCL
CLIP ViT
CLIP CNN
TR R@1
IR R@1
TR R@1
IR R@1
TR R@1
IR R@1
TR R@1
IR R@1
ALBEF
DCI+DIM
100
100
94.73
94.05
73.37
77.93
74.46
77.77
DCI+SIA
100
100
99.58
99.57
70.55
74.58
73.18
78.66
DCI+BSR
100
100
98.74
98.86
77.30
79.19
78.80
82.20
SADCA
100
100
98.52
97.83
81.10
82.83
85.44
86.11
TCL
DCI+DIM
97.60
97.26
100
100
75.71
80.54
80.08
83.53
Table 6: Ablation Study for Semantic Augmentation Module.
Methods
GPU Memory (GB)
Run Time (h)
TR R@1
IR R@1
SGA
10.5
0.83
54.72
61.54
SGA(LI)+SIA
13.0
3.95
67.99
72.07
DRA
10.6
1.57
62.46
69.06
SA-AET
10.5
2.12
69.74
75.17
SA-AET(LI)+SIA
13.5
11.08
83.85
86.12
SADCA
13.3
4.40
88.35
88.92
Table 7: Comparison of attack costs. GPU memory usage, runtime, and attack performance when generating adversarial examples on the Flickr30K dataset using ALBEF as the surrogate model.
Source
Attack
ALBEF
TCL
CLIP ViT
CLIP CNN
TR R@1
TR R@5
TR R@10
TR R@1
TR R@5
TR R@10
TR R@1
TR R@5
TR R@10
TR R@1
TR R@5
TR R@10
ALBEF
SGA
99.90
99.70
99.70
87.88
77.79
71.74
36.69
19.83
12.40
39.59
21.88
14.83
SGA+SIA
99.79
99.50
99.20
94.10
88.74
85.07
47.48
29.60
21.75
53.77
34.46
24.82
SGA(LI)
100
100
100
84.19
73.17
67.43
4.36
15.47
10.06
36.53
19.77
12.36
SGA(LI)+SIA
100
100
100
99.37
98.29
97.80
49.82
32.40
23.68
54.79
35.31
28.01
DRA
99.90
99.70
99.70
91.57
81.31
75.95
46.26
25.44
18.60
49.55
29.49
21.22
Table 8: Comparison with SOTA methods on the image-text retrieval (ITR) task on the Flickr30K dataset. The ”Source” column indicates the VLP model used to generate the multimodal adversarial examples. For text retrieval (TR), we report the ASR (%) at Rank-1 (R@1), Rank-5 (R@5) and Rank-10 (R@10).
Source
Attack
ALBEF
TCL
CLIP ViT
CLIP CNN
IR R@1
IR R@5
IR R@10
IR R@1
IR R@5
IR R@10
IR R@1
IR R@5
IR R@10
IR R@1
IR R@5
IR R@10
ALBEF
SGA
99.93
99.92
99.90
88.05
77.65
71.32
46.78
29.29
22.18
49.78
32.70
24.80
SGA+SIA
99.79
99.41
99.15
94.21
88.11
83.33
57.38
38.68
30.88
61.23
43.96
35.19
SGA(LI)
100
100
100
84.50
72.94
66.21
41.04
23.41
17.60
45.18
26.95
20.71
SGA(LI)+SIA
100
100
100
99.02
97.40
96.28
56.19
38.03
30.77
60.99
43.86
34.63
DRA
99.93
99.86
99.82
91.36
82.24
77.03
56.80
38.61
29.79
59.01
41.65
33.52
Table 9: Comparison with SOTA methods on the image-text retrieval (ITR) task on the Flickr30K dataset. The ”Source” column indicates the VLP model used to generate the multimodal adversarial examples. For image retrieval (IR), we report the ASR (%) at Rank-1 (R@1), Rank-5 (R@5) and Rank-10 (R@10).
Figure 6: Visualization of original images and the corresponding adversarial examples generated by our proposed SADCA.
Figure 7: Visualization of adversarial images in attacking commercial LVLMs.
Adversarial attacks on vision-language models exploit images, text, and multimodal interactions to manipulate outputs, bypass safety, or degrade reliability.
School of Cyber Science and Engineering, Xi’an Jiaotong University, Xi’an, China · State Key Laboratory of Mathematical Engineering and Advanced Computing, Information Engineering University, Zhengzhou, China