Pushing the Frontier of Black-Box LVLM Attacks via Fine-Grained Detail Targeting

Xiaohan Zhao; Zhaoyi Li; Yaxin Luo; Jiacheng Cui; Zhiqiang Shen

cs.LGFeb 19, 2026

Pushing the Frontier of Black-Box LVLM Attacks via Fine-Grained Detail Targeting

Authors: Xiaohan Zhao, Zhaoyi Li, Yaxin Luo, Jiacheng Cui, Zhiqiang Shen

Organizations: VILA Lab, Department of Machine Learning, MBZUAI

Abstract

Black-box adversarial attacks on Large Vision-Language Models (LVLMs) are challenging due to missing gradients and complex multimodal boundaries. Prior state-of-the-art transfer-based approaches such as M-Attack match local crops between source and target images, but this induces high-variance, nearly orthogonal gradients across iterations. We attribute this instability to (i) ViT translation sensitivity, which yields spike-like gradients, and (ii) a structural asymmetry between source and target crops. We reformulate local matching as an asymmetric expectation over source transformations and target semantics, and introduce M-Attack-V2 based on this formulation. Multi-Crop Alignment (MCA) averages gradients from multiple local views per iteration, while Auxiliary Target Alignment (ATA) replaces aggressive target augmentation with a small set of semantically correlated auxiliary images. Patch Momentum replays historical crop gradients, and a patch-diverse surrogate ensemble (PE+) strengthens transferable directions. M-Attack-V2 raises attack success rates over M-Attack from 40% to 61% on GPT-6.1 Sol, from 40% to 60% on GPT-6 Astra, and from 85% to 97% on Claude Sonnet 5.5. Code and data are publicly available at: https://github.com/VILA-Lab/M-Attack-V2.

Figures & tables

Figure 1: Improvement of

\mathtt{M\text{-}Attack\text{-}V2}

over

\mathtt{M\text{-}Attack}

on strong and up-to-date commercial black-box models (Grok 4.7, Claude Sonnet 5.5, and GPT-6 Astra). ASR and KMR stand for attack success rate and keyword matching rate, respectively.

Panel of Figure 2 — Figure 2: Gradient mismatch in local-level matching. (a) Isolating the source-side effect (target fixed): gradient similarity decays rapidly and falls below $0.1$ once IoU $<0.8$ , despite substantial pixel overlap. (b) Combined effect in $\mathtt{M\text{-}Attack}$ (both crops vary per iteration): inter-iteration gradient similarity is near zero; $\mathtt{M\text{-}Attack\text{-}V2}$ improves it to around $0.2$ . Results are averaged over 200 runs.

Panel of Figure 3 — Figure 3: Comparison of: (a) optimization trajectories under different $K$ ; (b) gradient patterns of single-crop alignment and multi-crop alignment (MCA). The gradient pattern of ResNet-50 remains consistent when large regions overlap, while the gradient pattern of ViTs changes dramatically. MCA helps smooth this effect, improving gradient similarity and accelerating convergence.

Figure 4: Adversarial samples of different attacks under $\epsilon=8$ . More examples are in Appendix G .

Figure 5: Comparison under different step budgets.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Surrogate (paper notation)	Implementation (Hugging Face identifier)
CLIP † -B/32 ( Ilharco et al., 2021 ; Schuhmann et al., 2022 )	laion/CLIP-ViT-B-32-laion2B-s34B-b79K
CLIP † -H/14 ( Ilharco et al., 2021 ; Schuhmann et al., 2022 )	laion/CLIP-ViT-H-14-laion2B-s32B-b79K
CLIP-L/14 ( Radford et al., 2021 )	openai/clip-vit-large-patch14
CLIP † -B/16 ( Ilharco et al., 2021 ; Schuhmann et al., 2022 )	laion/CLIP-ViT-B-16-laion2B-s34B-b88K
CLIP † -G/14 ( Ilharco et al., 2021 ; Schuhmann et al., 2022 )	laion/CLIP-ViT-bigG-14-laion2B-39B-b160k
Dino-Small ( Oquab et al., 2023 )	facebook/dinov2-small

Appendix

Table 5: Surrogate models and their corresponding Hugging Face identifiers.

Figure 6: Overall procedure of $\mathtt{M\text{-}Attack\text{-}V2}$ , contrasting the local-matching pipeline of $\mathtt{M\text{-}Attack}$ with the MCA/ATA-enhanced pipeline of $\mathtt{M\text{-}Attack\text{-}V2}$ . $\mathtt{M\text{-}Attack}$ aligns one source crop to aggressively transformed target views, while $\mathtt{M\text{-}Attack\text{-}V2}$ aggregates multiple source crops and mildly transformed auxiliary anchors, yielding smoother gradients for the final adversarial update.

Figure 7: Comparison of two types of attention maps. Left: attention map that sparsely separates into different regions; right: attention map that focuses on the main object.

Figure 8: Hyperparameter sensitivity of

\mathtt{M\text{-}Attack\text{-}V2}

: number of crops

K

(left) and auxiliary weight

\lambda

(right).

Figure 9: Hyperparameter sensitivity of

\mathtt{M\text{-}Attack\text{-}V2}

: auxiliary set size

P

(top) and momentum parameter

\beta

(bottom).

Figure 10: Comparison under different step budgets on GPT-6 Luna, Grok 4.7, and Gemini 3.8 Flash.

Table 12

Figure 11: Visualization of adversarial samples under $\epsilon=8$ .

Figure 12: Visualization of adversarial samples under $\epsilon=16$ .

Figure 13: Success and failure modes of $\mathtt{M\text{-}Attack\text{-}V2}$ shared by GPT-5, Claude 4.0-extended, and Gemini 2.5-Pro.

Figure 14: Visualization of GPT-o3’s responses to $\mathtt{M\text{-}Attack\text{-}V2}$ adversarial samples. The underlined ‘glitchy’ denotes that GPT-o3 notices something unusual.

Adversarial Attacks on VLMs

Adversarial attacks on vision-language models exploit images, text, and multimodal interactions to manipulate outputs, bypass safety, or degrade reliability.

See all 179 papers

PA-Attack: Guiding Gray-Box Attacks on LVLM Vision Encoders with Prototypes and Attention
Feb 23, 2026Hefei Mei, Zirui Wang, Chang Xu +2Attention-Based Adversarial Attacks Adversarial Attacks on VLMs
City University of Hong Kong · The University of Sydney
One Attack to Fool Them All: Highly Transferable Black-Box Adversarial Attacks on Frontier MLLMs
Sep 27, 2026Sen Nie, Jie Zhang, Zhongqi Wang +2Adversarial Attacks on VLMs Adversarial Attacks on LLMs
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, and the University of Chinese Academy of Sciences.
Stage-wise Attention-Guided Region Sequencing for Adversarial Attacks on Large Vision-Language Models
Feb 4, 2026Jaehyun Kwak, Nam Cao, Boryeong Cho +3Adversarial Attacks on VLMs Adversarial Attacks
KAIST · KENTECH

Feb 23, 2026J/K move · Enter open · S save

PA-Attack: Guiding Gray-Box Attacks on LVLM Vision Encoders with Prototypes and Attention

Hefei Mei, Zirui Wang, Chang Xu, Jianyuan Guo+1

City University of Hong Kong · The University of Sydney

Attention-Based Adversarial Attacks Adversarial Attacks on VLMs Adversarial Attacks VLM Robustness Adversarial Attacks on Unified Multimodal Models

Open paper AskPDF arXiv

Large Vision-Language Models (LVLMs) are foundational to modern multimodal applications, yet their susceptibility to adversarial attacks remains a critical concern. Prior white-box attacks rarely generalize across tasks, and black-box methods depend on expensive transfer, which limits efficiency. The vision encoder, standardized and often shared across LVLMs, provides a stable gray-box pivot with strong cross-model transfer. Building on this premise, we introduce PA-Attack (Prototype-Anchored Attentive Attack). PA-Attack begins with a prototype-anchored guidance that provides a stable attack direction towards a general and dissimilar prototype, tackling the attribute-restricted issue and limited task generalization of vanilla attacks. Building on this, we propose a two-stage attention enhancement mechanism: (i) leverage token-level attention scores to concentrate perturbations on critical visual tokens, and (ii) adaptively recalibrate attention weights to track the evolving attention during the adversarial process. Extensive experiments across diverse downstream tasks and LVLM architectures show that PA-Attack achieves an average 75.1% score reduction rate (SRR), demonstrating strong attack effectiveness, efficiency, and task generalization in LVLMs. Code is available at https://github.com/hefeimei06/PA-Attack.

Abstract

Adversarial Attacks on VLMs

Explore similar work

PA-Attack: Guiding Gray-Box Attacks on LVLM Vision Encoders with Prototypes and Attention

One Attack to Fool Them All: Highly Transferable Black-Box Adversarial Attacks on Frontier MLLMs

Stage-wise Attention-Guided Region Sequencing for Adversarial Attacks on Large Vision-Language Models