Revealing Physical-World Semantic Vulnerabilities: Universal Adversarial Patch for Infrared Vision-Language Models

Chengyin Hu; Yuxian Dong; Yikun Guo; Xiang Chen; Qike Zhang; Junqi Wu; Jiahuan Long; Jiujiang Guo

cs.CVApr 3, 2026

Revealing Physical-World Semantic Vulnerabilities: Universal Adversarial Patch for Infrared Vision-Language Models

Authors: Chengyin Hu, Yuxian Dong, Yikun Guo, Xiang Chen, Qike Zhang, Junqi Wu, Jiahuan Long, Jiujiang Guo,

Organizations: China University of Petroleum-Beijing at Karamay, No. 355 Anding Road, Karamay 834000, Xinjiang, China. · the Defense Innovation Institute, Chinese Academy of Military Science, Beijing, China. · Shenzhen University, Shenzhen, China. · North University of China, Taiyuan, China

Abstract

Infrared vision-language models (IR-VLMs) are becoming important for semantic perception in low-visibility environments, yet their robustness to physical semantic attacks remains underexplored. Existing adversarial patch methods are mainly designed for red-green-blue (RGB) images or closed-set infrared detectors and do not directly address open-ended IR-VLM tasks, where one deployable artifact can affect classification, captioning, and visual question answering (VQA) simultaneously. We propose Universal Curved-Grid Patch, abbreviated UCGP, a universal physical adversarial patch framework tailored to IR-VLMs. UCGP represents the patch as a deployable low-frequency curved grid and optimizes it with a representation-driven objective over subspace departure, topology disruption, and local appearance regularization. Meta Differential Evolution (MetaDE) searches for patch parameters with two physical robustness augmentations: Expectation over Transformation (EOT) capture sampling for imaging variations and thin-plate spline (TPS) patch-deformation modeling for non-rigid local shape changes. Rather than manipulating labels or prompts, UCGP disrupts the clean-category manifold in visual representation space, which later appears as degraded cross-modal outputs. Experiments show that a single shared patch degrades classification, captioning, and VQA across diverse IR-VLM architectures while retaining measurable cross-model transfer, cross-dataset generalization, cross-category extensibility, and real-scene physical effectiveness under the evaluated conditions. These results reveal a robustness blind spot in current infrared multimodal systems. Code and reproducibility assets are available at https://github.com/dyx6663/UCGP.

Figures & tables

Fig. 1: Universal-patch attack overview. A single shared curved-grid patch is attached to the target region of different clean infrared images and reused without per-image re-optimization. The resulting adversarial inputs alter the visual evidence provided to IR-VLMs.

Fig. 2: Overall framework of UCGP . UCGP builds a clean-category reference structure, renders a universal Bézier curved-grid patch, and optimizes it with MetaDE using subspace, topology, and budget losses under EOT/TPS physical augmentations. The learned patch is reused across digital and physical IR-VLM evaluations reported here.

Fig. 3: Curved-grid parameterization examples. (a) Mesh topology under different deformation parameters. (b) Physical validation of smooth and self-intersecting structures during deployment.

Fig. 4: Experimental setup. (a) Tripod, cold patches, and infrared camera. (b) Visible-spectrum image of the outer garment. (c) Visible-spectrum image of the inner garment with the patch attached. (d) Visible-spectrum image of the wearer after deployment. (e) Infrared image of the wearer.

Method	OpenAI CLIP ViT-L/14		OpenCLIP ViT-B/16		Meta-CLIP ViT-L/14		EVA-CLIP ViT-G/14
	ASR	Nat.	ASR	Nat.	ASR	Nat.	ASR	Nat.
Clean	–	3.26	–	3.26	–	3.26	–	3.26
AdvGrid [ 12 ]	19.33	2.21 $\downarrow$ 1.05	25.00	2.15 $\downarrow$ 1.11	19.67	2.26 $\downarrow$ 1.00	27.67	2.24 $\downarrow$ 1.02
HCB [ 20 ]	20.33	2.08 $\downarrow$ 1.18	34.00	2.04 $\downarrow$ 1.22	20.67	2.07 $\downarrow$ 1.19	20.00	2.16 $\downarrow$ 1.10
QR Code [ 18 ]	22.67	2.15 $\downarrow$ 1.11	20.33	2.10 $\downarrow$ 1.16	19.00	2.12 $\downarrow$ 1.14	17.67	2.06 $\downarrow$ 1.20
UCGP	94.33	2.33 $\downarrow$ 0.93	61.67	2.30 $\downarrow$ 0.96	34.33	2.25 $\downarrow$ 1.01	48.00	2.37 $\downarrow$ 0.89

TABLE I: Classification ASR and naturalness results. Nat. is the GPT-5-based 5-point naturalness score, where higher is more natural. Cited baselines are adapted to our unified IR-VLM region, loss, and evaluation protocol; dashes mark metrics not applicable to clean inputs.

Fig. 5: UCGP-generated adversarial samples and their classification results. The labels below each image show the top-1 predicted class and its softmax probability. (a) Results on OpenAI CLIP ViT-L/14. (b) Results on EVA-CLIP ViT-G/14.

Image Encoder	Model	Clean	QR Code	HCB	AdvGrid	UCGP (Ours)
OpenAI CLIP ViT-L/14	LLaVA-1.5 (7B)	66.03	61.59 $\downarrow$ 4.44	61.99 $\downarrow$ 4.04	61.00 $\downarrow$ 5.03	48.79 $\downarrow$ 17.24
	LLaVA-1.6 (13B)	68.01	61.09 $\downarrow$ 6.92	63.94 $\downarrow$ 4.07	63.65 $\downarrow$ 4.36	52.77 $\downarrow$ 15.24
	OpenFlamingo (3B)	63.17	58.37 $\downarrow$ 4.80	51.37 $\downarrow$ 11.80	54.93 $\downarrow$ 8.24	43.77 $\downarrow$ 19.40
	BLIP-2 FlanT5XL ViT-L (3.4B)	68.87	64.03 $\downarrow$ 4.84	58.91 $\downarrow$ 9.96	60.96 $\downarrow$ 7.91	51.20 $\downarrow$ 17.67
EVA-CLIP ViT-G/14	BLIP-2 FlanT5XL (4.1B)	70.17	64.37 $\downarrow$ 5.80	62.45 $\downarrow$ 7.72	59.93 $\downarrow$ 10.24	51.77 $\downarrow$ 18.40
	InstructBLIP FlanT5XL (4.1B)	72.87	66.77 $\downarrow$ 6.10	60.63 $\downarrow$ 12.24	59.13 $\downarrow$ 13.74	53.35 $\downarrow$ 19.52

TABLE II: Captioning results. Scores are GPT-5-judged semantic consistency scores on a 0–100 scale; down arrows denote absolute drops from clean outputs. Adapted baselines are defined and cited in Table I .

Image Encoder	Model	Clean	QR Code	HCB	AdvGrid	UCGP (Ours)
OpenAI CLIP ViT-L/14	LLaVA-1.5 (7B)	68.21	63.34 $\downarrow$ 4.87	58.56 $\downarrow$ 9.65	57.28 $\downarrow$ 10.93	51.34 $\downarrow$ 16.87
	LLaVA-1.6 (13B)	71.38	64.23 $\downarrow$ 7.15	61.65 $\downarrow$ 9.73	59.36 $\downarrow$ 12.02	55.23 $\downarrow$ 16.15
	OpenFlamingo (3B)	51.56	41.85 $\downarrow$ 9.71	35.74 $\downarrow$ 15.82	37.46 $\downarrow$ 14.10	32.03 $\downarrow$ 19.53
	BLIP-2 FlanT5XL ViT-L (3.4B)	62.32	58.12 $\downarrow$ 4.20	57.78 $\downarrow$ 4.54	55.78 $\downarrow$ 6.54	46.78 $\downarrow$ 15.54
EVA-CLIP ViT-G/14	BLIP-2 FlanT5XL (4.1B)	65.83	57.58 $\downarrow$ 8.25	60.56 $\downarrow$ 5.27	59.34 $\downarrow$ 6.49	47.34 $\downarrow$ 18.49
	InstructBLIP FlanT5XL (4.1B)	64.78	59.10 $\downarrow$ 5.68	57.39 $\downarrow$ 7.39	55.21 $\downarrow$ 9.57	42.23 $\downarrow$ 22.55

TABLE III: VQA results. Scores are GPT-5-judged answer correctness rates on a 0–100 scale; down arrows denote absolute drops from clean outputs. Adapted baselines are defined and cited in Table I .

Fig. 6: Semantic failures of IR-VLMs in captioning, VQA, and physical deployment under the proposed patch. (A) Captioning errors. (B) VQA errors. (C) Physical examples with GPT-5-based scoring. Colored words mark correct semantics and semantic errors .

Fig. 7: Physical attack examples across distances and viewpoints. Indoor examples cover 6.2–11.0 m, while outdoor examples cover the farther 7.8–12.6 m range enabled by the open scene. The patch remains visible in infrared and changes the predicted semantics under both deployments.

Fig. 8: ASR trends of the physical attack under indoor and outdoor deployment. The indoor curve uses the 6.2–11.0 m range, and the outdoor curve uses the farther 7.8–12.6 m range shown in Fig. 7 . Horizontal reference lines denote the scene-level average ASR, abbreviated AASR.

Surrogate Models	Source	Infrared-COCO			LSOTB-TIR			LLVIP			M3FD			FLIR v1.3			AVG
Surrogate Models	Source	ASR	$\Delta s_{\mathrm{prom}}$	$M_{\mathrm{adv}}$	ASR	$\Delta s_{\mathrm{prom}}$	$M_{\mathrm{adv}}$	ASR	$\Delta s_{\mathrm{prom}}$	$M_{\mathrm{adv}}$	ASR	$\Delta s_{\mathrm{prom}}$	$M_{\mathrm{adv}}$	ASR	$\Delta s_{\mathrm{prom}}$	$M_{\mathrm{adv}}$	ASR	$\Delta s_{\mathrm{prom}}$	$M_{\mathrm{adv}}$
OpenAI CLIP ViT-L/14	$\mathcal{D}_{1}$	94.33	30.80	34.86	98.53	31.29	21.81	93.36	23.99	21.25	100.00	27.78	27.59	100.00	56.64	52.31	97.24	34.10	31.56
	$\mathcal{D}_{2}$	81.05	45.25	40.31	100.00	25.69	20.18	96.46	22.82	25.11	100.00	36.51	31.35	100.00	60.34	50.86	95.50	38.12	33.56
	$\mathcal{D}_{3}$	81.70	37.30	38.96	99.37	28.81	26.18	96.46	28.41	23.06	99.49	35.50	32.24	100.00	57.62	56.94	95.40	37.53	35.48
	$\mathcal{D}_{4}$	86.27	36.09	35.74	100.00	33.29	25.80	97.35	25.63	21.10	100.00	31.29	25.92	100.00	56.50	59.78	96.72	36.56	33.67
	$\mathcal{D}_{5}$	81.70	34.21	36.80	100.00	25.76	23.80	95.13	29.09	26.83	100.00	37.69	28.76	100.00	57.87	52.31	95.37	36.92	33.70

TABLE IV: Cross-dataset transfer results. For each surrogate/source pair, ASR,

\Delta s_{\mathrm{prom}}

, and

M_{\mathrm{adv}}

are reported on the target datasets and their average (AVG). The score-movement descriptors follow the definition in Sec. IV-A .

\mathcal{D}_{1}

–

\mathcal{D}_{5}

denote Infrared-COCO, LSOTB-TIR, LLVIP, M3FD, and FLIR ADAS v1.3. Gray, boldface diagonal cells indicate the no-transfer setting in which the source and target datasets coincide.

Surrogate Models	Source	OpenAI CLIP ViT-L/14			OpenCLIP ViT-B/16			Meta-CLIP ViT-L/14			EVA-CLIP ViT-G/14			AVG
Surrogate Models	Source	ASR	$\Delta s_{\mathrm{prom}}$	$M_{\mathrm{adv}}$	ASR	$\Delta s_{\mathrm{prom}}$	$M_{\mathrm{adv}}$	ASR	$\Delta s_{\mathrm{prom}}$	$M_{\mathrm{adv}}$	ASR	$\Delta s_{\mathrm{prom}}$	$M_{\mathrm{adv}}$	ASR	$\Delta s_{\mathrm{prom}}$	$M_{\mathrm{adv}}$
OpenAI CLIP ViT-L/14	$\mathcal{D}_{1}$	94.33	30.80	34.86	14.53	31.50	28.98	51.15	132.89	68.24	13.95	81.73	48.40	43.49	69.23	45.12
	$\mathcal{D}_{2}$	100.00	25.69	20.18	7.76	60.10	36.95	9.22	32.66	34.02	7.98	28.83	25.09	31.24	36.82	29.06
	$\mathcal{D}_{3}$	96.46	28.41	23.06	18.58	43.79	35.18	38.50	98.92	72.14	22.58	59.24	48.46	44.03	57.59	44.71
	$\mathcal{D}_{4}$	100.00	31.29	25.92	14.72	49.52	45.46	45.19	99.75	85.44	19.81	43.48	43.18	44.93	56.01	50.00
	$\mathcal{D}_{5}$	100.00	57.87	52.31	33.34	29.76	28.44	25.94	39.78	34.61	44.44	77.83	42.36	50.93	51.31	39.43

TABLE V: Cross-model transfer results. For each surrogate/source pair, ASR,

\Delta s_{\mathrm{prom}}

, and

M_{\mathrm{adv}}

are reported on the victim models and their average (AVG). The score-movement descriptors follow the definition in Sec. IV-A . Source labels follow Table IV . Gray, boldface diagonal cells indicate the no-transfer setting in which the surrogate and victim models coincide.

Category	ASR (%)	$\bm{\Delta s_{\textbf{prom}}}$	$\bm{M_{\textbf{adv}}}$
Person	94.33	98.75	69.32
Dog	98.67	106.25	76.43
Bicycle	56.00	95.12	44.62
Car	61.00	92.32	50.87

TABLE VI: Cross-category extension results on Infrared-COCO. Each category is evaluated on

N=300

samples. ASR is reported in percent;

\Delta s_{\text{prom}}

and

M_{\text{adv}}

are the score-movement descriptors defined in Sec. IV-A .

Fig. 9: Loss-term ablation results for selected objective components. The bars compare full UCGP with variants that remove the topology loss or the subspace loss, providing an extended objective-level view that complements the module-wise comparison in Table VII .

Variant	OpenAI CLIP	Open- CLIP	Meta- CLIP	EVA- CLIP
Full UCGP	94.33	61.67	34.33	48.00
Pixel-wise patch	30.33	21.67	13.33	12.33
UCGP w/ straight segments	7.00	5.00	3.67	4.33
UCGP w/ random topology	5.33	3.67	2.00	4.00
w/o MetaDE, std. DE	76.00	54.67	30.33	46.33

TABLE VII: Module-wise ablation results. ASR is in percent, where higher indicates a stronger attack. Each variant changes only the named component; Pixel-wise patch directly optimizes pixels within the attack region instead of using the curved-grid parameterization.

Fig. 10: Parameter sensitivity of key design choices. (a) Population size. (b) Convergence under grid-line widths. (c) Patch color. (d) Graph-KL weight

\lambda_{\text{topo}}

Fig. 11: Heatmap comparison of the patch-area ratio (patch area divided by the target bounding-box area) and grid count. Each cell reports ASR in percent; the gold outline marks the default configuration used in the main experiments reported here.

Fig. 12: Effect of the number of optimization images on classification ASR across four CLIP backbones. After 300 images, the ASR curves approximately converge across backbones.

Setting	Defense access	ASR (%)	Drop (%)
No Defense	None	94.33	–
DW-1	Blind repair	45.33	49.00
DW-2	Patch-location prior	24.67	69.66
AT-1	Original patch	15.33	79.00
AT-2	Adaptive patch	78.00	16.33

TABLE VIII: Defense evaluation. ASR is reported in percent; lower ASR indicates a stronger defense. Drop denotes the absolute ASR reduction relative to No Defense; – marks the reference row.

Adversarial Attacks on VLMs

Adversarial attacks on vision-language models exploit images, text, and multimodal interactions to manipulate outputs, bypass safety, or degrade reliability.

See all 182 papers

May 21, 2026

Exposing Vulnerabilities in Visible-Infrared VLMs: A Unified Geometric Adversarial Framework with Cross-Task Transferability

Vision-language models (VLMs) have achieved strong performance across diverse multimodal tasks, but their adversarial robustness in visible-infrared (VIS-IR) scenarios remains underexplored. This gap is critical because VIS-IR sensing is widely used in real-world perception systems to support reliable understanding under challenging imaging conditions. To address this cross-modal threat setting, we propose CFGPatch, a curved-edge fractal geometric adversarial patch framework for attacking VIS-IR VLMs. CFGPatch builds on triangular fractal geometry and replaces rigid straight-edged primitives with Bezier-curved elements, preserving multi-scale fractal self-similarity while introducing smoother contours, richer directional variation, and more flexible shape deformation. In addition, we design a modality-specific Fraser-spiral rendering mechanism to inject fine-grained texture distortions and misleading perceptual cues into visible and infrared images. By coupling global curved-fractal geometry with local spiral-based appearance interference, CFGPatch disrupts both shape perception and texture interpretation. We further adopt expectation over transformation (EOT) to improve robustness against common image-level transformations. Extensive experiments show that CFGPatch effectively fools VIS-IR VLMs and consistently outperforms standard patch baselines in attack effectiveness and robustness. Moreover, adversarial samples optimized for zero-shot classification transfer well to image captioning and visual question answering, demonstrating strong cross-task transferability and generalizability across downstream tasks.

Xiang Chen, Yuxian Dong, Chao Li +6

VLM Robustness Adversarial Patch Attacks Adversarial Attacks on VLMs Large Vision-Language Models Transfer-Based Adversarial Attacks Physical Adversarial Attacks

Ask PDF

Jul 8, 2026

InfraQR: Edge-Placed QR-Inspired Structured Patch Attacks on Infrared Vision-Language Models

Infrared vision-language models are increasingly used for perception under low-light and adverse visual conditions, yet their robustness to localized structured perturbations remains underexplored. Existing infrared adversarial studies mainly focus on object detectors, leaving the security of infrared vision-language models less systematically examined. We present InfraQR, a QR-inspired structured patch attack for infrared vision-language models. Unlike localized attacks that attach perturbations to the target object, InfraQR places a compact structured patch along image boundaries and optimizes learnable grid cells through surrogate CLIP-style encoders. The resulting patch has a near-binary structured appearance, but is not required to be a valid or machine-readable QR code. We evaluate InfraQR on infrared classification, caption transfer, and question-answer-aware visual question answering (VQA) tasks. On a 300-image infrared benchmark, InfraQR sharply reduces the accuracy of multiple CLIP-style classifiers, including reducing OpenAI CLIP accuracy from 98.67% to 0.70%. The generated adversarial images also transfer to black-box captioning and VQA models, causing semantic degradation in captions and more error-prone answers under GPT-5.4-based evaluation. These results show that infrared vision-language models remain vulnerable to structured edge-placed perturbations, motivating further study of cross-task robustness beyond direct object occlusion.

Xin Li, Jiaju Han, Ma Yaqi +5

China University of Petroleum-Beijing at Karamay, Karamay, Xinjiang, China · Shenzhen Research Institute of Big Data, Shenzhen, China · Guizhou University, Guiyang, China +1

VLM Robustness Adversarial Patch Attacks Adversarial Attacks on VLMs Large Vision-Language Models Black-Box Adversarial Attacks

Ask PDF

Jul 7, 2026

AirflowAttack: Thermal-Airflow Adversarial Perturbations against Infrared Remote-Sensing Vision-Language Models

Vision-language models (VLMs) are increasingly deployed on infrared (IR) remote sensing imagery in security-critical settings, yet their adversarial robustness remains unexamined. We present AirflowAttack, to our knowledge the first adversarial attack for IR remote-sensing VLMs and the first to weaponize thermal-airflow turbulence as the perturbation prior. A lightweight generator synthesizes a single input-agnostic perturbation regularized toward physically plausible airflow patterns. Optimized on one surrogate CLIP model, it attains a mean zero-shot scene-classification attack success rate (ASR, the fraction of samples whose top-1 class changes) of 48.5% across five diverse CLIP backbones, far exceeding four IR-specific physical baselines (27.7--37.0%). Applied to six state-of-the-art VLMs, it cuts scene-classification accuracy by up to 38.2% relative, yet paradoxically makes some models more confident in their IR analysis, confabulating the perturbation as genuine thermal evidence such as temperature gradients and convection. Ablations show the airflow prior raises physical plausibility at no measurable cost to attack success. Together with a benchmark spanning eleven models and four tasks, these findings expose critical vulnerabilities in the rapidly expanding IR VLM ecosystem.

Cong Su, Jiaju Han, Xuemeng Sun +5

Tianjin University, Tianjin, China · China University of Petroleum-Beijing at Karamay, Karamay, Xinjiang, China · Shanghai Jiao Tong University, Shanghai, China

Adversarial Attacks on VLMs Remote Sensing Adversarial Perturbations Transfer-Based Adversarial Attacks Physical Adversarial Attacks

Ask PDF

Abstract

Adversarial Attacks on VLMs

Explore similar work

Exposing Vulnerabilities in Visible-Infrared VLMs: A Unified Geometric Adversarial Framework with Cross-Task Transferability

InfraQR: Edge-Placed QR-Inspired Structured Patch Attacks on Infrared Vision-Language Models

AirflowAttack: Thermal-Airflow Adversarial Perturbations against Infrared Remote-Sensing Vision-Language Models