Revealing Physical-World Semantic Vulnerabilities: Universal Adversarial Patch for Infrared Vision-Language Models

Chengyin Hu; Yuxian Dong; Yikun Guo; Xiang Chen; Qike Zhang; Junqi Wu; Jiahuan Long; Jiujiang Guo

cs.CVApr 3, 2026

Revealing Physical-World Semantic Vulnerabilities: Universal Adversarial Patch for Infrared Vision-Language Models

Authors: Chengyin Hu, Yuxian Dong, Yikun Guo, Xiang Chen, Qike Zhang, Junqi Wu, Jiahuan Long, Jiujiang Guo,

Organizations: China University of Petroleum-Beijing at Karamay, No. 355 Anding Road, Karamay 834000, Xinjiang, China. · the Defense Innovation Institute, Chinese Academy of Military Science, Beijing, China. · Shenzhen University, Shenzhen, China. · North University of China, Taiyuan, China

Abstract

Infrared vision-language models (IR-VLMs) are becoming important for semantic perception in low-visibility environments, yet their robustness to physical semantic attacks remains underexplored. Existing adversarial patch methods are mainly designed for red-green-blue (RGB) images or closed-set infrared detectors and do not directly address open-ended IR-VLM tasks, where one deployable artifact can affect classification, captioning, and visual question answering (VQA) simultaneously. We propose Universal Curved-Grid Patch, abbreviated UCGP, a universal physical adversarial patch framework tailored to IR-VLMs. UCGP represents the patch as a deployable low-frequency curved grid and optimizes it with a representation-driven objective over subspace departure, topology disruption, and local appearance regularization. Meta Differential Evolution (MetaDE) searches for patch parameters with two physical robustness augmentations: Expectation over Transformation (EOT) capture sampling for imaging variations and thin-plate spline (TPS) patch-deformation modeling for non-rigid local shape changes. Rather than manipulating labels or prompts, UCGP disrupts the clean-category manifold in visual representation space, which later appears as degraded cross-modal outputs. Experiments show that a single shared patch degrades classification, captioning, and VQA across diverse IR-VLM architectures while retaining measurable cross-model transfer, cross-dataset generalization, cross-category extensibility, and real-scene physical effectiveness under the evaluated conditions. These results reveal a robustness blind spot in current infrared multimodal systems. Code and reproducibility assets are available at https://github.com/dyx6663/UCGP.

Figures & tables

Fig. 1: Universal-patch attack overview. A single shared curved-grid patch is attached to the target region of different clean infrared images and reused without per-image re-optimization. The resulting adversarial inputs alter the visual evidence provided to IR-VLMs.

Fig. 2: Overall framework of UCGP . UCGP builds a clean-category reference structure, renders a universal Bézier curved-grid patch, and optimizes it with MetaDE using subspace, topology, and budget losses under EOT/TPS physical augmentations. The learned patch is reused across digital and physical IR-VLM evaluations reported here.

Fig. 3: Curved-grid parameterization examples. (a) Mesh topology under different deformation parameters. (b) Physical validation of smooth and self-intersecting structures during deployment.

Fig. 4: Experimental setup. (a) Tripod, cold patches, and infrared camera. (b) Visible-spectrum image of the outer garment. (c) Visible-spectrum image of the inner garment with the patch attached. (d) Visible-spectrum image of the wearer after deployment. (e) Infrared image of the wearer.

Method	OpenAI CLIP ViT-L/14		OpenCLIP ViT-B/16		Meta-CLIP ViT-L/14		EVA-CLIP ViT-G/14
	ASR	Nat.	ASR	Nat.	ASR	Nat.	ASR	Nat.
Clean	–	3.26	–	3.26	–	3.26	–	3.26
AdvGrid [ 12 ]	19.33	2.21 $\downarrow$ 1.05	25.00	2.15 $\downarrow$ 1.11	19.67	2.26 $\downarrow$ 1.00	27.67	2.24 $\downarrow$ 1.02
HCB [ 20 ]	20.33	2.08 $\downarrow$ 1.18	34.00	2.04 $\downarrow$ 1.22	20.67	2.07 $\downarrow$ 1.19	20.00	2.16 $\downarrow$ 1.10
QR Code [ 18 ]	22.67	2.15 $\downarrow$ 1.11	20.33	2.10 $\downarrow$ 1.16	19.00	2.12 $\downarrow$ 1.14	17.67	2.06 $\downarrow$ 1.20
UCGP	94.33	2.33 $\downarrow$ 0.93	61.67	2.30 $\downarrow$ 0.96	34.33	2.25 $\downarrow$ 1.01	48.00	2.37 $\downarrow$ 0.89

TABLE I: Classification ASR and naturalness results. Nat. is the GPT-5-based 5-point naturalness score, where higher is more natural. Cited baselines are adapted to our unified IR-VLM region, loss, and evaluation protocol; dashes mark metrics not applicable to clean inputs.

Fig. 5: UCGP-generated adversarial samples and their classification results. The labels below each image show the top-1 predicted class and its softmax probability. (a) Results on OpenAI CLIP ViT-L/14. (b) Results on EVA-CLIP ViT-G/14.

Image Encoder	Model	Clean	QR Code	HCB	AdvGrid	UCGP (Ours)
OpenAI CLIP ViT-L/14	LLaVA-1.5 (7B)	66.03	61.59 $\downarrow$ 4.44	61.99 $\downarrow$ 4.04	61.00 $\downarrow$ 5.03	48.79 $\downarrow$ 17.24
	LLaVA-1.6 (13B)	68.01	61.09 $\downarrow$ 6.92	63.94 $\downarrow$ 4.07	63.65 $\downarrow$ 4.36	52.77 $\downarrow$ 15.24
	OpenFlamingo (3B)	63.17	58.37 $\downarrow$ 4.80	51.37 $\downarrow$ 11.80	54.93 $\downarrow$ 8.24	43.77 $\downarrow$ 19.40
	BLIP-2 FlanT5XL ViT-L (3.4B)	68.87	64.03 $\downarrow$ 4.84	58.91 $\downarrow$ 9.96	60.96 $\downarrow$ 7.91	51.20 $\downarrow$ 17.67
EVA-CLIP ViT-G/14	BLIP-2 FlanT5XL (4.1B)	70.17	64.37 $\downarrow$ 5.80	62.45 $\downarrow$ 7.72	59.93 $\downarrow$ 10.24	51.77 $\downarrow$ 18.40
	InstructBLIP FlanT5XL (4.1B)	72.87	66.77 $\downarrow$ 6.10	60.63 $\downarrow$ 12.24	59.13 $\downarrow$ 13.74	53.35 $\downarrow$ 19.52

TABLE II: Captioning results. Scores are GPT-5-judged semantic consistency scores on a 0–100 scale; down arrows denote absolute drops from clean outputs. Adapted baselines are defined and cited in Table I .

Image Encoder	Model	Clean	QR Code	HCB	AdvGrid	UCGP (Ours)
OpenAI CLIP ViT-L/14	LLaVA-1.5 (7B)	68.21	63.34 $\downarrow$ 4.87	58.56 $\downarrow$ 9.65	57.28 $\downarrow$ 10.93	51.34 $\downarrow$ 16.87
	LLaVA-1.6 (13B)	71.38	64.23 $\downarrow$ 7.15	61.65 $\downarrow$ 9.73	59.36 $\downarrow$ 12.02	55.23 $\downarrow$ 16.15
	OpenFlamingo (3B)	51.56	41.85 $\downarrow$ 9.71	35.74 $\downarrow$ 15.82	37.46 $\downarrow$ 14.10	32.03 $\downarrow$ 19.53
	BLIP-2 FlanT5XL ViT-L (3.4B)	62.32	58.12 $\downarrow$ 4.20	57.78 $\downarrow$ 4.54	55.78 $\downarrow$ 6.54	46.78 $\downarrow$ 15.54
EVA-CLIP ViT-G/14	BLIP-2 FlanT5XL (4.1B)	65.83	57.58 $\downarrow$ 8.25	60.56 $\downarrow$ 5.27	59.34 $\downarrow$ 6.49	47.34 $\downarrow$ 18.49
	InstructBLIP FlanT5XL (4.1B)	64.78	59.10 $\downarrow$ 5.68	57.39 $\downarrow$ 7.39	55.21 $\downarrow$ 9.57	42.23 $\downarrow$ 22.55

TABLE III: VQA results. Scores are GPT-5-judged answer correctness rates on a 0–100 scale; down arrows denote absolute drops from clean outputs. Adapted baselines are defined and cited in Table I .

Fig. 6: Semantic failures of IR-VLMs in captioning, VQA, and physical deployment under the proposed patch. (A) Captioning errors. (B) VQA errors. (C) Physical examples with GPT-5-based scoring. Colored words mark correct semantics and semantic errors .

Fig. 7: Physical attack examples across distances and viewpoints. Indoor examples cover 6.2–11.0 m, while outdoor examples cover the farther 7.8–12.6 m range enabled by the open scene. The patch remains visible in infrared and changes the predicted semantics under both deployments.

Fig. 8: ASR trends of the physical attack under indoor and outdoor deployment. The indoor curve uses the 6.2–11.0 m range, and the outdoor curve uses the farther 7.8–12.6 m range shown in Fig. 7 . Horizontal reference lines denote the scene-level average ASR, abbreviated AASR.

Surrogate Models	Source	Infrared-COCO			LSOTB-TIR			LLVIP			M3FD			FLIR v1.3			AVG
Surrogate Models	Source	ASR	$\Delta s_{\mathrm{prom}}$	$M_{\mathrm{adv}}$	ASR	$\Delta s_{\mathrm{prom}}$	$M_{\mathrm{adv}}$	ASR	$\Delta s_{\mathrm{prom}}$	$M_{\mathrm{adv}}$	ASR	$\Delta s_{\mathrm{prom}}$	$M_{\mathrm{adv}}$	ASR	$\Delta s_{\mathrm{prom}}$	$M_{\mathrm{adv}}$	ASR	$\Delta s_{\mathrm{prom}}$	$M_{\mathrm{adv}}$
OpenAI CLIP ViT-L/14	$\mathcal{D}_{1}$	94.33	30.80	34.86	98.53	31.29	21.81	93.36	23.99	21.25	100.00	27.78	27.59	100.00	56.64	52.31	97.24	34.10	31.56
	$\mathcal{D}_{2}$	81.05	45.25	40.31	100.00	25.69	20.18	96.46	22.82	25.11	100.00	36.51	31.35	100.00	60.34	50.86	95.50	38.12	33.56
	$\mathcal{D}_{3}$	81.70	37.30	38.96	99.37	28.81	26.18	96.46	28.41	23.06	99.49	35.50	32.24	100.00	57.62	56.94	95.40	37.53	35.48
	$\mathcal{D}_{4}$	86.27	36.09	35.74	100.00	33.29	25.80	97.35	25.63	21.10	100.00	31.29	25.92	100.00	56.50	59.78	96.72	36.56	33.67
	$\mathcal{D}_{5}$	81.70	34.21	36.80	100.00	25.76	23.80	95.13	29.09	26.83	100.00	37.69	28.76	100.00	57.87	52.31	95.37	36.92	33.70

TABLE IV: Cross-dataset transfer results. For each surrogate/source pair, ASR,

\Delta s_{\mathrm{prom}}

, and

M_{\mathrm{adv}}

are reported on the target datasets and their average (AVG). The score-movement descriptors follow the definition in Sec. IV-A .

\mathcal{D}_{1}

–

\mathcal{D}_{5}

denote Infrared-COCO, LSOTB-TIR, LLVIP, M3FD, and FLIR ADAS v1.3. Gray, boldface diagonal cells indicate the no-transfer setting in which the source and target datasets coincide.

Surrogate Models	Source	OpenAI CLIP ViT-L/14			OpenCLIP ViT-B/16			Meta-CLIP ViT-L/14			EVA-CLIP ViT-G/14			AVG
Surrogate Models	Source	ASR	$\Delta s_{\mathrm{prom}}$	$M_{\mathrm{adv}}$	ASR	$\Delta s_{\mathrm{prom}}$	$M_{\mathrm{adv}}$	ASR	$\Delta s_{\mathrm{prom}}$	$M_{\mathrm{adv}}$	ASR	$\Delta s_{\mathrm{prom}}$	$M_{\mathrm{adv}}$	ASR	$\Delta s_{\mathrm{prom}}$	$M_{\mathrm{adv}}$
OpenAI CLIP ViT-L/14	$\mathcal{D}_{1}$	94.33	30.80	34.86	14.53	31.50	28.98	51.15	132.89	68.24	13.95	81.73	48.40	43.49	69.23	45.12
	$\mathcal{D}_{2}$	100.00	25.69	20.18	7.76	60.10	36.95	9.22	32.66	34.02	7.98	28.83	25.09	31.24	36.82	29.06
	$\mathcal{D}_{3}$	96.46	28.41	23.06	18.58	43.79	35.18	38.50	98.92	72.14	22.58	59.24	48.46	44.03	57.59	44.71
	$\mathcal{D}_{4}$	100.00	31.29	25.92	14.72	49.52	45.46	45.19	99.75	85.44	19.81	43.48	43.18	44.93	56.01	50.00
	$\mathcal{D}_{5}$	100.00	57.87	52.31	33.34	29.76	28.44	25.94	39.78	34.61	44.44	77.83	42.36	50.93	51.31	39.43

TABLE V: Cross-model transfer results. For each surrogate/source pair, ASR,

\Delta s_{\mathrm{prom}}

, and

M_{\mathrm{adv}}

are reported on the victim models and their average (AVG). The score-movement descriptors follow the definition in Sec. IV-A . Source labels follow Table IV . Gray, boldface diagonal cells indicate the no-transfer setting in which the surrogate and victim models coincide.

Category	ASR (%)	$\bm{\Delta s_{\textbf{prom}}}$	$\bm{M_{\textbf{adv}}}$
Person	94.33	98.75	69.32
Dog	98.67	106.25	76.43
Bicycle	56.00	95.12	44.62
Car	61.00	92.32	50.87

TABLE VI: Cross-category extension results on Infrared-COCO. Each category is evaluated on

N=300

samples. ASR is reported in percent;

\Delta s_{\text{prom}}

and

M_{\text{adv}}

are the score-movement descriptors defined in Sec. IV-A .

Fig. 9: Loss-term ablation results for selected objective components. The bars compare full UCGP with variants that remove the topology loss or the subspace loss, providing an extended objective-level view that complements the module-wise comparison in Table VII .

Variant	OpenAI CLIP	Open- CLIP	Meta- CLIP	EVA- CLIP
Full UCGP	94.33	61.67	34.33	48.00
Pixel-wise patch	30.33	21.67	13.33	12.33
UCGP w/ straight segments	7.00	5.00	3.67	4.33
UCGP w/ random topology	5.33	3.67	2.00	4.00
w/o MetaDE, std. DE	76.00	54.67	30.33	46.33

TABLE VII: Module-wise ablation results. ASR is in percent, where higher indicates a stronger attack. Each variant changes only the named component; Pixel-wise patch directly optimizes pixels within the attack region instead of using the curved-grid parameterization.

Fig. 10: Parameter sensitivity of key design choices. (a) Population size. (b) Convergence under grid-line widths. (c) Patch color. (d) Graph-KL weight

\lambda_{\text{topo}}

Fig. 11: Heatmap comparison of the patch-area ratio (patch area divided by the target bounding-box area) and grid count. Each cell reports ASR in percent; the gold outline marks the default configuration used in the main experiments reported here.

Fig. 12: Effect of the number of optimization images on classification ASR across four CLIP backbones. After 300 images, the ASR curves approximately converge across backbones.

Setting	Defense access	ASR (%)	Drop (%)
No Defense	None	94.33	–
DW-1	Blind repair	45.33	49.00
DW-2	Patch-location prior	24.67	69.66
AT-1	Original patch	15.33	79.00
AT-2	Adaptive patch	78.00	16.33

TABLE VIII: Defense evaluation. ASR is reported in percent; lower ASR indicates a stronger defense. Drop denotes the absolute ASR reduction relative to No Defense; – marks the reference row.

Adversarial Attacks on VLMs

Adversarial attacks on vision-language models exploit images, text, and multimodal interactions to manipulate outputs, bypass safety, or degrade reliability.

See all 182 papers

Exposing Vulnerabilities in Visible-Infrared VLMs: A Unified Geometric Adversarial Framework with Cross-Task Transferability
May 21, 2026Xiang Chen, Yuxian Dong, Chao Li +6VLM Robustness Adversarial Patch Attacks
InfraQR: Edge-Placed QR-Inspired Structured Patch Attacks on Infrared Vision-Language Models
Jul 8, 2026Xin Li, Jiaju Han, Ma Yaqi +5VLM Robustness Adversarial Patch Attacks
China University of Petroleum-Beijing at Karamay, Karamay, Xinjiang, China · Shenzhen Research Institute of Big Data, Shenzhen, China · Guizhou University, Guiyang, China +1
AirflowAttack: Thermal-Airflow Adversarial Perturbations against Infrared Remote-Sensing Vision-Language Models
Jul 7, 2026Cong Su, Jiaju Han, Xuemeng Sun +5Adversarial Attacks on VLMs Remote Sensing
Tianjin University, Tianjin, China · China University of Petroleum-Beijing at Karamay, Karamay, Xinjiang, China · Shanghai Jiao Tong University, Shanghai, China

May 21, 2026J/K move · Enter open · S save

Exposing Vulnerabilities in Visible-Infrared VLMs: A Unified Geometric Adversarial Framework with Cross-Task Transferability

Xiang Chen, Yuxian Dong, Chao Li, Chengyin Hu+5

VLM Robustness Adversarial Patch Attacks Adversarial Attacks on VLMs Large Vision-Language Models Transfer-Based Adversarial Attacks Physical Adversarial Attacks

Open paper AskPDF arXiv

Vision-language models (VLMs) have achieved strong performance across diverse multimodal tasks, but their adversarial robustness in visible-infrared (VIS-IR) scenarios remains underexplored. This gap is critical because VIS-IR sensing is widely used in real-world perception systems to support reliable understanding under challenging imaging conditions. To address this cross-modal threat setting, we propose CFGPatch, a curved-edge fractal geometric adversarial patch framework for attacking VIS-IR VLMs. CFGPatch builds on triangular fractal geometry and replaces rigid straight-edged primitives with Bezier-curved elements, preserving multi-scale fractal self-similarity while introducing smoother contours, richer directional variation, and more flexible shape deformation. In addition, we design a modality-specific Fraser-spiral rendering mechanism to inject fine-grained texture distortions and misleading perceptual cues into visible and infrared images. By coupling global curved-fractal geometry with local spiral-based appearance interference, CFGPatch disrupts both shape perception and texture interpretation. We further adopt expectation over transformation (EOT) to improve robustness against common image-level transformations. Extensive experiments show that CFGPatch effectively fools VIS-IR VLMs and consistently outperforms standard patch baselines in attack effectiveness and robustness. Moreover, adversarial samples optimized for zero-shot classification transfer well to image captioning and visual question answering, demonstrating strong cross-task transferability and generalizability across downstream tasks.

Abstract

Adversarial Attacks on VLMs

Explore similar work

Exposing Vulnerabilities in Visible-Infrared VLMs: A Unified Geometric Adversarial Framework with Cross-Task Transferability

InfraQR: Edge-Placed QR-Inspired Structured Patch Attacks on Infrared Vision-Language Models

AirflowAttack: Thermal-Airflow Adversarial Perturbations against Infrared Remote-Sensing Vision-Language Models