XSPA: Crafting Imperceptible X-Shaped Sparse Adversarial Perturbations for Transferable Attacks on VLMs

Chengyin Hu; Jiaju Han; Xuemeng Sun; Qike Zhang; Luwei Yang; Lehan Sun; Jiahuan Long; Yiwei Wei

cs.CVMar 30, 2026

XSPA: Crafting Imperceptible X-Shaped Sparse Adversarial Perturbations for Transferable Attacks on VLMs

Authors: Chengyin Hu, Jiaju Han, Xuemeng Sun, Qike Zhang, Luwei Yang, Lehan Sun, Jiahuan Long, Yiwei Wei,

Organizations: China University of Petroleum-Beijing at Karamay · Shenzhen Research Institute of Big Data · Shanghai Jiao Tong University · Tianjin University ·

Abstract

Vision-language models (VLMs) share visual-textual representations across zero-shot classification, image captioning, and visual question answering (VQA), creating a pathway through which subtle perturbations may cause failures across tasks. We introduce X-shaped Sparse Pixel Attack (XSPA), a structured attack that restricts perturbations to two intersecting diagonal lines. Within this fixed support, XSPA jointly optimizes a classification objective, cross-task semantic guidance, perturbation magnitude, and linewise smoothness. Under the default setting, it changes about 1.04% of image pixels. On COCO, XSPA reduces zero-shot accuracy by 52.33 points on OpenAI CLIP ViT-L/14 and 67.00 points on OpenCLIP ViT-B/16; GPT-4-based caption consistency and VQA correctness decrease by up to 58.60 and 44.25 points, respectively. Matched-budget experiments show that an adaptive saliency Top-k support achieves higher attack success than the fixed X-shaped support, while XSPA provides better SSIM and LPIPS but worse PSNR and MAE. Same-image analysis finds no three-task targeted success, indicating that strong surrogate classification effects do not reliably translate into targeted output-space failures. These results position XSPA as a controlled stress test for studying how sparse fixed geometry affects VLM robustness, rather than as a universally superior attack.

Figures & tables

Figure 1: Overall framework of XSPA. XSPA optimizes perturbations within a fixed X-shaped sparse mask on a surrogate CLIP model, integrating momentum-based optimization, input diversity, and joint semantic objectives to promote target-oriented semantic attraction while suppressing source semantics. The resulting adversarial image is then transferred to downstream vision-language models for evaluation across zero-shot classification, image captioning, and visual question answering tasks.

	OpenCLIP ViT-B/16		Meta-CLIP ViT-L/14		EVA-CLIP ViT-G/14		OpenAI CLIP ViT-L/14
Method	ACC(%)	GPT-4o	ACC(%)	GPT-4o	ACC(%)	GPT-4o	ACC(%)	GPT-4o
Clean	97	3.65	98	3.65	98	3.65	93	3.65
NaturalLightAttack	94( $\downarrow$ 3)	2.32( $\downarrow$ 1.33)	97( $\downarrow$ 1)	2.14( $\downarrow$ 1.51)	97( $\downarrow$ 1)	2.16( $\downarrow$ 1.49)	93(-)	2.07( $\downarrow$ 1.58)
ShadowAttack	84( $\downarrow$ 13)	2.11( $\downarrow$ 1.54)	82( $\downarrow$ 16)	1.91( $\downarrow$ 1.74)	95( $\downarrow$ 3)	2.03( $\downarrow$ 1.62)	79( $\downarrow$ 14)	1.78( $\downarrow$ 1.87)
ITA	46( $\downarrow$ 51)	2.11( $\downarrow$ 1.54)	64( $\downarrow$ 34)	2.15( $\downarrow$ 1.50)	84( $\downarrow$ 14)	2.17( $\downarrow$ 1.48)	51( $\downarrow$ 42)	2.15( $\downarrow$ 1.50)
Ours	30 ( $\downarrow$ 67 )	2.71 ( $\downarrow$ 0.94 )	60.67 ( $\downarrow$ 37.33 )	2.89 ( $\downarrow$ 0.76 )	63.71 ( $\downarrow$ 34.29 )	2.78 ( $\downarrow$ 0.87 )	40.67 ( $\downarrow$ 52.33 )	2.76 ( $\downarrow$ 0.89 )

Table 1: Zero-shot classification results under different adversarial attacks. Lower ACC and higher naturalness are better. Numbers in parentheses denote the absolute change relative to clean samples. Bold indicates the best result.

Figure 2: Qualitative zero-shot classification examples. The top row shows clean images, and the bottom row shows XSPA adversarial images. Despite the fixed X-shaped sparse perturbation, clear prediction shifts are observed.

Figure 3: Grad-CAM heatmaps for clean and adversarial samples. Adversarial responses become more diffuse or shift away from the main object.

Image Encoder	Model	Params	Clean	NaturalLightAttack	ShadowAttack	ITA	Ours
OpenAI CLIP ViT-L/14	LLaVA-1.5	7B	78.60	77.00( $\downarrow$ 1.60)	74.60( $\downarrow$ 4.00)	63.73( $\downarrow$ 14.87)	46.40 ( $\downarrow$ 32.20 )
OpenAI CLIP ViT-L/14	LLaVA-1.6	7B	72.10	71.70( $\downarrow$ 0.40)	71.17( $\downarrow$ 0.93)	61.60( $\downarrow$ 10.50)	44.20 ( $\downarrow$ 27.90 )
EVA-CLIP ViT-G/14	OpenFlamingo	3B	70.20	69.53( $\downarrow$ 0.67)	67.80( $\downarrow$ 2.40)	53.93( $\downarrow$ 16.27)	11.60 ( $\downarrow$ 58.60 )
	BLIP-2 (FlanT5XL ViT-L)	3.4B	75.10	70.77( $\downarrow$ 4.33)	68.57( $\downarrow$ 6.53)	60.93( $\downarrow$ 14.17)	21.90 ( $\downarrow$ 53.20 )
	BLIP-2 (FlanT5XL)	4.1B	74.96	71.27( $\downarrow$ 3.69)	68.80( $\downarrow$ 6.16)	62.01( $\downarrow$ 12.95)	59.00 ( $\downarrow$ 15.96 )
	InstructBLIP (FlanT5XL)	4.1B	76.50	72.07( $\downarrow$ 4.43)	69.77( $\downarrow$ 6.73)	63.20( $\downarrow$ 13.30)	40.90 ( $\downarrow$ 35.60 )

Table 2: Image captioning results on six vision-language models under different adversarial attacks. Lower GPT-4 consistency indicates more effective attacks. Numbers in parentheses denote the absolute drop relative to clean samples. Bold indicates the best result.

Image Encoder	Model	Params	Clean	NaturalLightAttack	ShadowAttack	ITA	Ours
OpenAI CLIP ViT-L/14	LLaVA-1.5	7B	68.00	68.00(-)	67.00( $\downarrow$ 1.00)	48.00( $\downarrow$ 20.00)	47.45 ( $\downarrow$ 20.55 )
OpenAI CLIP ViT-L/14	LLaVA-1.6	7B	64.00	63.00( $\downarrow$ 1.00)	64.00(-)	43.00 ( $\downarrow$ 21.00 )	48.87( $\downarrow$ 15.13)
EVA-CLIP ViT-G/14	OpenFlamingo	3B	45.00	39.00( $\downarrow$ 6.00)	44.00( $\downarrow$ 1.00)	19.00( $\downarrow$ 26.00)	14.29 ( $\downarrow$ 30.71 )
	BLIP-2 (FlanT5XL ViT-L)	3.4B	63.00	58.00( $\downarrow$ 5.00)	50.00( $\downarrow$ 13.00)	38.00( $\downarrow$ 25.00)	18.75 ( $\downarrow$ 44.25 )
	BLIP-2 (FlanT5XL)	4.1B	54.00	53.00( $\downarrow$ 1.00)	54.00(-)	33.00( $\downarrow$ 21.00)	19.78 ( $\downarrow$ 34.22 )
	InstructBLIP (FlanT5XL)	4.1B	68.00	64.00( $\downarrow$ 4.00)	62.00( $\downarrow$ 6.00)	44.00 ( $\downarrow$ 24.00 )	44.67( $\downarrow$ 23.33)

Table 3: VQA results on six vision-language models under different adversarial attacks. Lower GPT-4 correctness indicates more effective attacks. Numbers in parentheses denote the absolute drop relative to clean samples. Bold indicates the best result.

Figure 5: Unified single-column

2\times 6

ablations: (a)–(d) optimization and perturbation, (e)–(j) X geometry, (k) shape, and (l) loss components.

Dataset	OpenAI L/14	OpenCLIP B/16	Meta-CLIP L/14	EVA-CLIP ViT-G/14	Average
COCO-300	59.33	70.00	39.33	36.29	51.24
COCO-1000	79.01	62.12	53.13	29.40	55.92
ImageNet-1K	99.58	97.55	98.19	84.73	95.01
ImageNet-R	73.56	66.89	54.34	56.48	62.82

Table 4: Cross-dataset U-ASR (%) across CLIP encoders. COCO-300 contains 300 images; all other datasets contain 1,000 images.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Default value
Evaluation set	COCO-300
Surrogate encoder	OpenCLIP ViT-B/16
Input resolution	$384\times 384$
Perturbation support	1,532 pixels (approximately 1.04%)
Optimization steps	100
Step size	0.03

Appendix

Table 5: Default configuration for the frozen ablation protocol.

Method	ASR	3407	3408	3409	Mean $\pm$ SD
Top- $k$	Targeted	94.67	94.00	94.67	$94.44\pm 0.38$
XSPA	Targeted	83.00	83.00	84.33	$83.44\pm 0.77$
Top- $k$	Untargeted	95.00	94.67	94.67	$94.78\pm 0.19$
XSPA	Untargeted	86.67	86.67	87.00	$86.78\pm 0.19$

Appendix

Table 6: Matched-budget ASR for saliency Top-

k

and XSPA on the same 300 image IDs. Every run uses a 1,532-pixel support,

L_{\infty}=0.25

, 100 steps, and step size 0.03. Values are percentages.

Component	XSPA	Saliency Top- $k$
CLIP targeted	177/214 (82.71%)	203/214 (94.86%)
Caption targeted	0/214 (0.00%)	0/214 (0.00%)
VQA any	9/214 (4.21%)	8/214 (3.74%)
VQA all	3/214 (1.40%)	3/214 (1.40%)
Joint with VQA any	0/214 (0.00%)	0/214 (0.00%)
Joint with VQA all	0/214 (0.00%)	0/214 (0.00%)

Appendix

Table 8: Factorization of same-image targeted success on the 214 jointly eligible images. “VQA any” requires at least one eligible target answer; “VQA all” requires every eligible target answer for that image.

Task	Reliability measure	Estimate [95% CI]
Caption	Pearson $r$	$0.580\;[0.255,0.801]$
Caption	Spearman $\rho$	$0.579\;[0.268,0.796]$
Caption	ICC(A,1)	$0.301\;[0.120,0.443]$
Caption	MAE (0–30 scale)	$3.683\;[3.050,4.367]$
VQA	Exact agreement	$0.933\;[0.850,1.000]$
VQA	Cohen’s $\kappa$	$0.815\;[0.526,1.000]$

Appendix

Table 9: Inter-GPT judge calibration on 60 condition-items per task.

Adversarial Attacks on VLMs

Adversarial attacks on vision-language models exploit images, text, and multimodal interactions to manipulate outputs, bypass safety, or degrade reliability.

See all 182 papers

Stage-wise Attention-Guided Region Sequencing for Adversarial Attacks on Large Vision-Language Models
Feb 4, 2026Jaehyun Kwak, Nam Cao, Boryeong Cho +3Adversarial Attacks on VLMs Adversarial Attacks
KAIST · KENTECH
PDA: Text-Augmented Defense Framework for Robust Vision-Language Models against Adversarial Image Attacks
Apr 1, 2026Jingning Xu, Haochen Luo, Chen LiuVLM Robustness Vision-Language Models
Department of Computer Science City University of Hong Kong
Beyond Standard Benchmarks: A Systematic Audit of Vision-Language Model's Robustness to Natural Semantic Variation Across Diverse Tasks
Apr 6, 2026Jia Chengyu, AprilPyone MaungMaung, Huy H. Nguyen +2VLM Robustness VLM Evaluation
Zhejiang University of Technology, Hangzhou, Zhejiang, China · National Institute of Informatics, Tokyo, Japan

Feb 4, 2026J/K move · Enter open · S save

Stage-wise Attention-Guided Region Sequencing for Adversarial Attacks on Large Vision-Language Models

Jaehyun Kwak, Nam Cao, Boryeong Cho, Segyu Lee+2

KAIST · KENTECH

Adversarial Attacks on VLMs Adversarial Attacks Large Vision-Language Models Black-Box Adversarial Attacks Attention-Based Adversarial Attacks

Open paper AskPDF arXiv

Targeted adversarial attacks on Large Vision-Language Models (LVLMs) test whether small image perturbations can steer model responses toward attacker-specified content. Under the standard L-infinity constraint, targeted attacks become a regional perturbation budget allocation problem: attack success depends not only on the perturbation objective, but also on which regions receive updates and in what order. Existing localized attacks improve over global perturbations but rely on stochastic spatial sampling, often updating weakly influential regions. We address this limitation through an attention-based analysis showing that cross-modal attention identifies adversarially sensitive regions and that perturbing high-attention hotspots induces predictable redistribution toward subsequent salient regions. These findings motivate attention-guided region sequencing, which begins from dominant hotspots and progressively moves the update support toward next-salient regions. Based on these principles, we propose Stage-wise Attention-Guided Attack (SAGA), a black-box region-sequencing framework that uses a fixed attention map from an open-source LVLM to guide perturbation updates without accessing target-model parameters, gradients, or attention maps. Across ten closed-source and open-source LVLMs, SAGA achieves state-of-the-art attack success rates and the best overall imperceptibility. The source code is available at https://github.com/jaehyun-kwak/SAGA.

Abstract

Adversarial Attacks on VLMs

Explore similar work

Stage-wise Attention-Guided Region Sequencing for Adversarial Attacks on Large Vision-Language Models

PDA: Text-Augmented Defense Framework for Robust Vision-Language Models against Adversarial Image Attacks

Beyond Standard Benchmarks: A Systematic Audit of Vision-Language Model's Robustness to Natural Semantic Variation Across Diverse Tasks