Black-box adversarial attacks on Large Vision-Language Models (LVLMs) are challenging due to missing gradients and complex multimodal boundaries. Prior state-of-the-art transfer-based approaches such as M-Attack match local crops between source and target images, but this induces high-variance, nearly orthogonal gradients across iterations. We attribute this instability to (i) ViT translation sensitivity, which yields spike-like gradients, and (ii) a structural asymmetry between source and target crops. We reformulate local matching as an asymmetric expectation over source transformations and target semantics, and introduce M-Attack-V2 based on this formulation. Multi-Crop Alignment (MCA) averages gradients from multiple local views per iteration, while Auxiliary Target Alignment (ATA) replaces aggressive target augmentation with a small set of semantically correlated auxiliary images. Patch Momentum replays historical crop gradients, and a patch-diverse surrogate ensemble (PE+) strengthens transferable directions. M-Attack-V2 raises attack success rates over M-Attack from 40% to 61% on GPT-6.1 Sol, from 40% to 60% on GPT-6 Astra, and from 85% to 97% on Claude Sonnet 5.5. Code and data are publicly available at: https://github.com/VILA-Lab/M-Attack-V2.
Figures & tables
Figure 1: Improvement of M-Attack-V2 over M-Attack on strong and up-to-date commercial black-box models (Grok 4.7, Claude Sonnet 5.5, and GPT-6 Astra). ASR and KMR stand for attack success rate and keyword matching rate, respectively.
Figure 2: Gradient mismatch in local-level matching. (a) Isolating the source-side effect (target fixed): gradient similarity decays rapidly and falls below 0.1 once IoU <0.8 , despite substantial pixel overlap. (b) Combined effect in M-Attack (both crops vary per iteration): inter-iteration gradient similarity is near zero; M-Attack-V2 improves it to around 0.2 . Results are averaged over 200 runs.
Figure 3: Comparison of: (a) optimization trajectories under different K ; (b) gradient patterns of single-crop alignment and multi-crop alignment (MCA). The gradient pattern of ResNet-50 remains consistent when large regions overlap, while the gradient pattern of ViTs changes dramatically. MCA helps smooth this effect, improving gradient similarity and accelerating convergence.
Figure 4: Adversarial samples of different attacks under ϵ=8 . More examples are in Appendix G .
Figure 5: Comparison under different step budgets.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Surrogate (paper notation)
Implementation (Hugging Face identifier)
CLIP † -B/32 ( Ilharco et al., 2021 ; Schuhmann et al., 2022 )
laion/CLIP-ViT-B-32-laion2B-s34B-b79K
CLIP † -H/14 ( Ilharco et al., 2021 ; Schuhmann et al., 2022 )
laion/CLIP-ViT-H-14-laion2B-s32B-b79K
CLIP-L/14 ( Radford et al., 2021 )
openai/clip-vit-large-patch14
CLIP † -B/16 ( Ilharco et al., 2021 ; Schuhmann et al., 2022 )
laion/CLIP-ViT-B-16-laion2B-s34B-b88K
CLIP † -G/14 ( Ilharco et al., 2021 ; Schuhmann et al., 2022 )
laion/CLIP-ViT-bigG-14-laion2B-39B-b160k
Dino-Small ( Oquab et al., 2023 )
facebook/dinov2-small
Appendix
Table 5: Surrogate models and their corresponding Hugging Face identifiers.
Figure 6: Overall procedure of M-Attack-V2 , contrasting the local-matching pipeline of M-Attack with the MCA/ATA-enhanced pipeline of M-Attack-V2 . M-Attack aligns one source crop to aggressively transformed target views, while M-Attack-V2 aggregates multiple source crops and mildly transformed auxiliary anchors, yielding smoother gradients for the final adversarial update.
Figure 7: Comparison of two types of attention maps. Left: attention map that sparsely separates into different regions; right: attention map that focuses on the main object.
Figure 8: Hyperparameter sensitivity of M-Attack-V2 : number of crops K (left) and auxiliary weight λ (right).
Figure 9: Hyperparameter sensitivity of M-Attack-V2 : auxiliary set size P (top) and momentum parameter β (bottom).
Figure 10: Comparison under different step budgets on GPT-6 Luna, Grok 4.7, and Gemini 3.8 Flash.
Table 12
Figure 11: Visualization of adversarial samples under ϵ=8 .
Figure 12: Visualization of adversarial samples under ϵ=16 .
Figure 13: Success and failure modes of M-Attack-V2 shared by GPT-5, Claude 4.0-extended, and Gemini 2.5-Pro.
Figure 14: Visualization of GPT-o3’s responses to M-Attack-V2 adversarial samples. The underlined ‘glitchy’ denotes that GPT-o3 notices something unusual.
Adversarial attacks on vision-language models exploit images, text, and multimodal interactions to manipulate outputs, bypass safety, or degrade reliability.
Large Vision-Language Models (LVLMs) are foundational to modern multimodal applications, yet their susceptibility to adversarial attacks remains a critical concern. Prior white-box attacks rarely generalize across tasks, and black-box methods depend on expensive transfer, which limits efficiency. The vision encoder, standardized and often shared across LVLMs, provides a stable gray-box pivot with strong cross-model transfer. Building on this premise, we introduce PA-Attack (Prototype-Anchored Attentive Attack). PA-Attack begins with a prototype-anchored guidance that provides a stable attack direction towards a general and dissimilar prototype, tackling the attribute-restricted issue and limited task generalization of vanilla attacks. Building on this, we propose a two-stage attention enhancement mechanism: (i) leverage token-level attention scores to concentrate perturbations on critical visual tokens, and (ii) adaptively recalibrate attention weights to track the evolving attention during the adversarial process. Extensive experiments across diverse downstream tasks and LVLM architectures show that PA-Attack achieves an average 75.1% score reduction rate (SRR), demonstrating strong attack effectiveness, efficiency, and task generalization in LVLMs. Code is available at https://github.com/hefeimei06/PA-Attack.
Hefei Mei, Zirui Wang, Chang Xu +2
City University of Hong Kong · The University of Sydney
Adversarial attacks have long posed a fundamental threat to machine learning systems. As multimodal large language models (MLLMs) rapidly evolve and become widely deployed, assessing their vulnerability to such attacks is essential for their safe use. In this work, we investigate whether a single adversarial image can consistently mislead diverse frontier MLLMs in black-box settings. We propose O-Attack, a highly transferable black-box attack framework. This framework builds on our insight that surrogate models contain a broad, high-level, cross-modally aligned semantic space. This space extends beyond final-layer outputs and provides multiple semantically consistent representations that remain underexploited by existing attacks. Within this space, O-Attack anchors aligned representations, progressively broadens semantic conditions, and optimizes perturbations through semantic consensus to promote consistent target alignment. By fully exploiting this space with the same surrogate models as M-Attack, O-Attack raises attack success rates on GPT-5.4 (29.1% to 77.2%), Claude-4.6 (42.8% to 81.6%), and Gemini-3.1 (38.2% to 80.9%). Extensive experiments across 24 MLLMs show that O-Attack outperforms six state-of-the-art methods in black-box transferability, with consistent effectiveness across prompts and improved efficiency and imperceptibility. This work exposes the practical safety risks posed by black-box adversarial attacks against frontier MLLMs, underscoring the need for more rigorous robustness evaluation and more effective defenses.
Sen Nie, Jie Zhang, Zhongqi Wang +2
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, and the University of Chinese Academy of Sciences.
Targeted adversarial attacks on Large Vision-Language Models (LVLMs) test whether small image perturbations can steer model responses toward attacker-specified content. Under the standard L-infinity constraint, targeted attacks become a regional perturbation budget allocation problem: attack success depends not only on the perturbation objective, but also on which regions receive updates and in what order. Existing localized attacks improve over global perturbations but rely on stochastic spatial sampling, often updating weakly influential regions. We address this limitation through an attention-based analysis showing that cross-modal attention identifies adversarially sensitive regions and that perturbing high-attention hotspots induces predictable redistribution toward subsequent salient regions. These findings motivate attention-guided region sequencing, which begins from dominant hotspots and progressively moves the update support toward next-salient regions. Based on these principles, we propose Stage-wise Attention-Guided Attack (SAGA), a black-box region-sequencing framework that uses a fixed attention map from an open-source LVLM to guide perturbation updates without accessing target-model parameters, gradients, or attention maps. Across ten closed-source and open-source LVLMs, SAGA achieves state-of-the-art attack success rates and the best overall imperceptibility. The source code is available at https://github.com/jaehyun-kwak/SAGA.