PA-Attack: Guiding Gray-Box Attacks on LVLM Vision Encoders with Prototypes and Attention

Hefei Mei; Zirui Wang; Chang Xu; Jianyuan Guo; Minjing Dong

cs.CVFeb 23, 2026

PA-Attack: Guiding Gray-Box Attacks on LVLM Vision Encoders with Prototypes and Attention

Authors: Hefei Mei, Zirui Wang, Chang Xu, Jianyuan Guo, Minjing Dong

Organizations: City University of Hong Kong · The University of Sydney

Abstract

Large Vision-Language Models (LVLMs) are foundational to modern multimodal applications, yet their susceptibility to adversarial attacks remains a critical concern. Prior white-box attacks rarely generalize across tasks, and black-box methods depend on expensive transfer, which limits efficiency. The vision encoder, standardized and often shared across LVLMs, provides a stable gray-box pivot with strong cross-model transfer. Building on this premise, we introduce PA-Attack (Prototype-Anchored Attentive Attack). PA-Attack begins with a prototype-anchored guidance that provides a stable attack direction towards a general and dissimilar prototype, tackling the attribute-restricted issue and limited task generalization of vanilla attacks. Building on this, we propose a two-stage attention enhancement mechanism: (i) leverage token-level attention scores to concentrate perturbations on critical visual tokens, and (ii) adaptively recalibrate attention weights to track the evolving attention during the adversarial process. Extensive experiments across diverse downstream tasks and LVLM architectures show that PA-Attack achieves an average 75.1% score reduction rate (SRR), demonstrating strong attack effectiveness, efficiency, and task generalization in LVLMs. Code is available at https://github.com/hefeimei06/PA-Attack.

Figures & tables

Figure 1: Adversarial performance on captioning and VQA tasks. The perturbation of black-box M-attack is $\epsilon=\nicefrac{{16}}{{255}}$ while that of other gray-box methods is $\epsilon=\nicefrac{{2}}{{255}}$ .

Figure 2: (a) Comparison of task transfer performance score reduction rate (SRR) on LLaVa1.5-7B. Slash columns compare the task transfer with white-box and black-box attack, while dotted columns represent the multi-task SRR of gray-box. (b) Attack SRR w/ and w/o different direction samples. The red diamond is the baseline without direction samples, and the light orange and purple circles represent their centers as prototypes. (c) The ratio of clean and adversarial performance changes. The blue line has the horizontal axis above (T in softmax) as a variable, and the other lines have the horizontal axis below (the proportion of mask tokens) as a variable. (d) Attentions before and after a 50-step attack w/ and w/o prototype guidance. For comparison, values are normalized with the maximum value.

Figure 3: Overview of PA-Attack. (a) The prototype-anchored guidance includes a vision encoder attack loss and a guidance loss for general degradation across diverse tasks. (b) Attention obtained by averaging the attention of the class token to each patch in Self-Attention across different Heads. (c) The attention weights are adjusted to align with the adversarial image through a two-stage process.

Model	Attack	COCO $\downarrow$		Flickr30k $\downarrow$		TextVQA $\downarrow$		VQAv2 $\downarrow$		POPE $\downarrow$		Average SRR $\uparrow$
Model	Perturbation $(\epsilon)$	$\nicefrac{{2}}{{255}}$	$\nicefrac{{4}}{{255}}$	$\nicefrac{{2}}{{255}}$	$\nicefrac{{4}}{{255}}$	$\nicefrac{{2}}{{255}}$	$\nicefrac{{4}}{{255}}$	$\nicefrac{{2}}{{255}}$	$\nicefrac{{4}}{{255}}$	$\nicefrac{{2}}{{255}}$	$\nicefrac{{4}}{{255}}$	$\nicefrac{{2}}{{255}}$	$\nicefrac{{4}}{{255}}$
LLaVa1.5-7B	Clean	115.5		77.5		37.1		74.5		84.5		0.0
	MIX.Attack [ 40 ]	67.5	55.4	42.1	35.9	24.6	19.1	59.4	57.9	72.0	69.1	31.2	38.9
	VT-Attack [ 43 ]	66.8	29.5	40.0	19.9	26.9	16.1	60.1	28.3	67.1	58.5	31.6	59.6
	AttackVLM-ii [ 51 ]	41.3	26.3	30.2	20.3	19.7	14.0	55.1	50.4	69.0	61.6	43.3	54.5
	VEAttack [ 33 ]	10.8	7.1	10.7	6.3	13.8	10.1	42.9	38.4	47.5	42.8	65.2	71.2

Table 1: Comparison of PA-Attack with different gray-box attacks across different LVLMs and tasks. For each dataset, we present the performance (

\downarrow

) after the gray-box attack, and finally, we calculate the average score reduction rate (SRR) (

\uparrow

) across all datasets.

Figure 4: Comparison of the responses of LLaVa1.5-7B with different attacks. The attributes with a blue background remain unchanged, while the red texts indicate that the attributes have changed.

PG	AE	TS	Step	COCO	Flickr30k	TextVQA	VQAv2
			100	93.8	91.9	72.8	48.4
$\checkmark$			100	95.5	94.8	76.3	52.2
	$\checkmark$		100	93.7	92.6	72.3	50.9
$\checkmark$	$\checkmark$		100	95.3	94.7	76.6	54.2
			150	94.8	93.7	77.9	48.8
$\checkmark$	$\checkmark$		150	96.2	95.5	81.1	54.4

Table 2: Ablation study on the components of PA-Attack with SRR as the metric, where PG denotes the Prototype-anchored Guidance, AE is the Attention Enhancement, and TS is a two-stage attention refinement framework.

$\lambda$	COCO	Flickr30k	TextVQA	VQAv2	Average
0.5	96.6	95.1	83.8	51.3	81.7
1.0	96.5	95.7	86.3	56.4	83.7
2.0	95.5	94.7	84.1	55.8	82.5

Table 3: Ablation study on the weight

\lambda

of the Prototype-anchored Guidance loss with SRR as the metric.

$m$	$w$	$K$	Flickr30k	TextVQA	VQAv2	Average
baseline			91.9	72.8	48.4	71.0
1000	512	20	92.8	77.1	47.6	72.5
2000	1024	20	92.6	79.8	47.5	73.3
3000	1024	20	94.8	76.3	52.2	74.4
3000	1024	10	94.2	77.6	51.2	74.3
3000	1024	30	92.3	79.5	49.0	73.6

Table 4: Impact of prototype construction parameters, where

m

is the number of images in the guidance image dataset,

w

is the dimension after PCA, and

K

is the number of prototypes.

Figure 5: (a) Ablation of the number of stages in adaptively attention refinement. Slash columns compare the score reduction rate, while dotted columns represent the attack time. (b) Ablation of $T$ in softmax and layers $l$ in attention enhancement.

Guidance	COCO	Flickr30k	VQAv2	Average
baseline	93.8	91.9	48.4	78.1
Mean sample	95.1	94.1	56.2	81.8
Nearest sample	64.6	86.2	33.4	61.4
Farthest sample	96.3	95.2	50.2	80.6
Nearest prototype	95.2	96.1	53.4	81.6
Farthest prototype	96.5	95.7	56.4	82.9

Table 5: Impact of different samples for the guidance objective.

Guidance	Distribution	COCO	Flickr30k	TextVQA	SRR $\uparrow$
Clean	In	115.5	77.5	37.1	0.0
COCO	In	4.1	3.3	5.1	92.8
RVL-CDIP	Out	4.1	3.7	6.8	91.1
ScienceQA	Out	4.2	3.6	5.8	92.0

Table 6: Attack performance with different guidance datasets.

Method	Qwen3-VL-8B		InternVL2-8B		SRR $\uparrow$
Method	RealWorldQA	ODinW-13	RealWorldQA	POPE	SRR $\uparrow$
Clean	73.3	35.5	62.5	88.6	0.0
VT-Attack	68.2	33.9	45.4	77.4	12.9
VEAttack	58.2	14.4	46.4	63.8	33.4
PA-Attack	57.0	9.8	43.4	64.8	38.0

Table 7: Attack performance on Qwen3-VL and InternVL2.

Method	COCO	Flickr30k	TextVQA	VQAv2	SRR $\uparrow$
Clean	115.5	77.5	37.1	74.5	0.0
VT-Attack	14.5	10.5	9.3	25.0	78.8
VEAttack	5.3	4.0	6.5	36.3	81.0
PA-Attack	3.6	1.9	4.4	31.9	84.9

Table 8: Attack performance with larger perturbation budget.

Figure 6: Ablation of time and an example of failure cases.

Method	COCO		Flickr30k		TextVQA
Method	TeCoA	FARE	TeCoA	FARE	TeCoA	FARE
VEAttack	43.9	51.1	24.2	34.1	16.5	22.2
PA-Attack	25.8	47.5	15.1	30.7	10.6	14.1

Table 9: Attack performance with different defense methods.

Figure 7: More comparison of the responses of LLaVa1.5-7B with different attacks. The attributes with a blue background remain unchanged, while the red texts indicate that the attributes have changed.

Figure 8: More comparison of the responses of LLaVa1.5-7B with different attacks. The attributes with a blue background remain unchanged, while the red texts indicate that the attributes have changed.

Figure 9: More comparison of the responses of LLaVa1.5-7B with different attacks. The attributes with a blue background remain unchanged, while the red texts indicate that the attributes have changed.

Figure 10: Comparison of the attention maps of clean images and those after different attacks. The iterations of AttackVLM-ii are 100. We visualize the attention map of the first stage of our PA-Attack with 50 iterations and the second stage with 100 iterations. The areas that are more reddish have higher attention values.

Figure 11: Evolution of token-wise attention values during the attack process. The heatmaps visualize the deviation in token attention compared to the token of clean images with step 0. The x-axis represents token indices, and the y-axis represents the optimization steps. A darker shade indicates a greater difference in attention value from the clean image. The rows correspond to different modules, and the columns show results for three datasets.

Attention-Based Adversarial Attacks

See all 12 papers

Exploiting Vision Encoder Vulnerabilities for Universal Adversarial Perturbations on Large Vision-Language Models
Dec 11, 2024Hee-Seon Kim, Minbeom Kim, Seokil Ham +1VLM Robustness Adversarial Attacks on VLMs
School of Electrical Engineering, KAIST, Daejeon 34141, Republic of Korea
Pushing the Frontier of Black-Box LVLM Attacks via Fine-Grained Detail Targeting
Feb 19, 2026Xiaohan Zhao, Zhaoyi Li, Yaxin Luo +2Adversarial Attacks on VLMs Black-Box Adversarial Attacks
VILA Lab, Department of Machine Learning, MBZUAI
Stage-wise Attention-Guided Region Sequencing for Adversarial Attacks on Large Vision-Language Models
Feb 4, 2026Jaehyun Kwak, Nam Cao, Boryeong Cho +3Adversarial Attacks on VLMs Adversarial Attacks
KAIST · KENTECH

Dec 11, 2024J/K move · Enter open · S save

Exploiting Vision Encoder Vulnerabilities for Universal Adversarial Perturbations on Large Vision-Language Models

Hee-Seon Kim, Minbeom Kim, Seokil Ham, Changick Kim

School of Electrical Engineering, KAIST, Daejeon 34141, Republic of Korea

VLM Robustness Adversarial Attacks on VLMs Large Vision-Language Models Adversarial Perturbations Transfer-Based Adversarial Attacks

Open paper AskPDF arXiv

Large Vision-Language Models (LVLMs) have achieved remarkable performance on multimodal tasks but remain highly vulnerable to small adversarial perturbations in input images. Existing attacks typically target the vision encoder's final output embeddings, implicitly treating the encoder as a uniform attack surface, while a systematic analysis of which internal components are most vulnerable has remained largely unexplored. We show such analysis is essential, as adversarial vulnerability in LVLM vision encoders is structurally concentrated rather than uniformly distributed. Building on this, we propose Vision Encoder Vulnerable-Component-Targeted Universal Adversarial Perturbation (VEV-UAP), a task-agnostic and cost-efficient attack framework. Through a component- and layer-wise analysis of attention mechanisms, we identify the value components in middle layers as critical vulnerabilities that strongly influence downstream language model behavior. VEV-UAP selectively targets these components to generate a single universal perturbation shared across images, without involving textual inputs or the language model during optimization. Experiments across multiple LVLMs and tasks show VEV-UAP achieves state-of-the-art attack success rates with reduced computational overhead. Moreover, a single VEV-UAP transfers across LVLMs sharing the same vision encoder, even when paired with different language models, making it a practical framework for scalable robustness evaluation.

Abstract

Attention-Based Adversarial Attacks

Explore similar work

Exploiting Vision Encoder Vulnerabilities for Universal Adversarial Perturbations on Large Vision-Language Models

Pushing the Frontier of Black-Box LVLM Attacks via Fine-Grained Detail Targeting

Stage-wise Attention-Guided Region Sequencing for Adversarial Attacks on Large Vision-Language Models