Logos appear naturally on clothing and daily necessities, where their recognizable shapes and semantic associations make them familiar visual cues. This familiarity helps them blend into real-world environments and remain visually inconspicuous. We ask whether an existing logo can serve as a carrier for coordinated targeted semantic attacks on visible-infrared (VIS-IR) vision-language models (VLMs) without changing its original outline. This setting requires expressive adversarial variation within a fixed shape while accounting for different appearance characteristics and model responses across modalities. We propose Logo-guided Semantic Attack (LoSA), a score-based black-box framework that optimizes a logo's interior appearance and placement while preserving its silhouette. LoSA uses a contour field and low-frequency graph spectral modes to structure the interior color layout, a frozen appearance adapter to couple VIS and IR appearances, and particle swarm optimization to jointly optimize shared design parameters. Experiments across four datasets demonstrate that LoSA consistently outperforms the evaluated baselines in simultaneous Top-1 targeted success. Cross-architecture experiments, visual naturalness ratings, and captioning/VQA transfer analyses further characterize its effectiveness and limitations. These findings reveal that familiar visual symbols from real-world environments can serve as shape-preserving carriers for coordinated targeted semantic attacks across imaging modalities.
Figures & tables
Figure 1: Conceptual overview of LoSA. A familiar logo’s interior fills are optimized while its outline is retained. An adapter couples its VIS and IR appearances, and the logos are overlaid to induce the same target prediction.
Figure 2: LoSA framework. (A) PSO optimizes parameters in the shared 20-dimensional design space and updates candidate logos using their fitness. (B) A frozen appearance adapter maps rendered RGB textures to IR appearances before digital compositing. (C) Frozen VIS and IR encoders provide class scores for the joint targeted objective; no model gradients are used to update the logo parameters.
Figure 3: Contour-preserving logo rendering. (a) The supplied logo defines a fixed foreground domain. (b) A boundary-derived contour field anchors the interior layout to that shape. (c) Eight graph spectral modes provide structured variation within the domain; their illustrations are schematic. (d) Optimized field weights and two RGB endpoints produce the visible texture through balanced color mixing. Isoline views depict the same scalar fields, rather than additional rendering stages.
RGBDT500
LasHeR
COCO
CrowdHuman
TASR@1
Both
TASR@1
Both
TASR@1
Both
TASR@1
Both
Method
VIS
IR
@1
@3
ASR
VIS
IR
@1
@3
ASR
VIS
IR
@1
@3
ASR
VIS
IR
@1
@3
ASR
RandomPatch
0.00
0.00
0.00
0.33
0.33
0.00
0.00
0.00
0.33
0.00
0.61
0.00
0.00
1.82
0.61
0.00
0.00
0.00
1.82
0.91
TOUAP
7.33
37.00
2.00
26.00
11.33
43.67
12.00
9.00
19.67
35.33
9.09
12.73
3.03
7.88
9.70
28.64
19.55
10.00
19.55
27.73
UNIPatch
1.00
3.67
1.00
8.67
4.00
7.67
0.33
0.33
0.33
12.67
1.82
4.85
0.61
5.45
4.24
3.64
3.64
1.82
7.27
8.64
CDUPatch
3.33
3.00
2.67
8.67
5.33
0.00
0.33
0.00
6.00
0.67
4.24
3.03
1.82
7.27
4.85
7.27
4.55
4.09
19.55
7.27
Table 1: Zero-shot attack effectiveness across datasets. Rates (%) on OpenAI CLIP ViT-L/14. Datasets: RGBDT500, LasHeR (controlled subset), COCO, and CrowdHuman. Both requires the outcome in both branches on the same pair. @1/@3 denote target Top-1/Top-3 matches. Best values are bold.
Figure 4: Qualitative examples for zero-shot classification. Clean (top) and adversarial (bottom) VIS–IR pairs are shown, with prescribed target categories below.
OpenCLIP ViT-B/16
MetaCLIP ViT-L/14
EVA-CLIP EVA01-g/14-plus
Method
VIS
IR
Both
@3
ASR
VIS
IR
Both
@3
ASR
VIS
IR
Both
@3
ASR
RandomPatch
0.00
0.32
0.00
2.10
0.96
1.28
0.30
0.00
1.15
1.92
0.00
0.56
0.00
2.82
0.83
TOUAP
12.00
24.84
5.20
24.00
17.75
12.99
23.47
3.72
16.93
13.55
17.43
33.59
10.39
27.37
19.18
UNIPatch
8.53
17.51
6.97
24.76
9.76
5.98
6.80
3.16
10.99
14.83
9.17
14.16
8.60
21.02
10.93
CDUPatch
3.29
2.79
2.15
8.76
8.45
2.52
4.28
2.27
11.07
5.98
2.10
1.71
0.30
10.15
3.81
LoSA
65.40
65.07
65.07
71.34
67.32
46.38
46.68
45.78
56.47
48.60
61.84
59.53
58.69
70.53
60.36
Table 2: Zero-shot attacks across CLIP architectures. VIS/IR/Both report TASR@1; @3 and ASR refer to Both. Best values are bold.
Adversarial attacks on vision-language models exploit images, text, and multimodal interactions to manipulate outputs, bypass safety, or degrade reliability.
China University of Petroleum-Beijing at Karamay, No. 355 Anding Road, Karamay 834000, Xinjiang, China. · the Defense Innovation Institute, Chinese Academy of Military Science, Beijing, China. · Shenzhen University, Shenzhen, China. +1