Visual adversarial robustness remains a fundamental security property as vision systems evolve from conventional classifiers to vision-language models (VLMs) and large vision-language models (LVLMs), with visual inputs remaining an attack surface across generations. Yet existing evaluation methods often depend on prescribed attacks, in-distribution data, extensive probing, or classifier-specific formulations, limiting external auditing of diverse models. We introduce Structural Gradient Alignment (SGA), a complementary robustness metric measuring the structural alignment between input content and objective-conditioned local model sensitivity. SGA uses a few randomly sampled, unlabeled public probes that need not follow the downstream task distribution and only one input-gradient computation per model--probe pair. Across 38 distinct DNNs, VLMs, and LVLMs, SGA produces rankings consistent with attack-based references. Without robustness references, uncalibrated SGA achieves 83.72% average pairwise agreement across eight public-probe conditions, outperforming the strongest overall baseline in all eight by 9.38 percentage points on average. With known robustness orders of anchor models, calibrated SGA increases agreement to 85.36%. These results demonstrate an attack-agnostic, computationally efficient robustness signal across conventional DNNs, VLMs, and autoregressive LVLMs, downstream tasks, threat norms, probe distributions, and evaluator-knowledge settings. Our implementation is available at https://github.com/azrealwang/SGA.
Figures & tables
TABLE I: Comparison of robustness-evaluation methods.
Protocol
Knowledge
Criterion
Uncalibrated
None
Score dispersion
Transfer
Development ranking
Development PA
Calibrated
Ordered anchors
Anchor PA, then dispersion
Oracle
Target ranking a
Target PA
TABLE II: SGA protocols under different knowledge settings.
Fig. 2: SGA scoring pipeline. For each model–probe pair, SGA transforms the probe image and its input gradient into structural maps under configuration c=(ρ,ψ,e,μ) . It then measures their structural correspondence and aggregates the resulting scores across the probe batch to rank the evaluated models.
Method
Knowledge
DNN ℓ2
DNN ℓ∞
VLM ℓ∞
LVLM ℓ∞
GREAT
None
0.4326±0.1086
0.4881±0.1216
0.7586±0.1233
0.4600±0.2332
FGSM
ℓ2 or ℓ∞
0.5113±0.1464
0.3830±0.1440
0.5714±0.1990
0.5360±0.1676
CLEVER
ℓ2 or ℓ∞
0.8691±0.0399
0.7977±0.0772
0.7407±0.0590
0.4060±0.1907
Linearized
None
0.8649±0.0455
0.7170±0.0721
0.7800±0.0799
0.6250±0.1228
SGA (Uncalibrated)
None
0.9694±0.0170
0.8651±0.0234
0.8010±0.0464
0.7620±0.1198
TABLE III: Pairwise agreement on unlabeled CelebA-HQ public probes at K=10 . Mean PA ± standard deviation across probe batches; best result in each model pool in bold.
Method
Knowledge
DNN ℓ2
DNN ℓ∞
VLM ℓ∞
LVLM ℓ∞
GREAT
None
0.4217±0.0992
0.4665±0.1109
0.4505±0.1235
0.4440±0.2255
FGSM
ℓ2 or ℓ∞
0.5447±0.1564
0.4316±0.1807
0.2662±0.1954
0.5820±0.1564
CLEVER
ℓ2 or ℓ∞
0.8949±0.0420
0.8492±0.0579
0.7262±0.0688
0.4530±0.1947
Linearized
None
0.8945±0.0428
0.7966±0.0580
0.7110±0.1039
0.5580±0.1159
SGA (Uncalibrated)
None
0.9581±0.0308
0.8745±0.0365
0.7686±0.0415
0.6990±0.1245
TABLE IV: Pairwise agreement on unlabeled Places365 public probes at K=10 . Mean PA ± standard deviation across probe batches; best result in each model pool in bold.
Fig. 3: Model scores versus the PGD-50 reference on CelebA-HQ public probes. All methods use the same ten randomly selected probes, and scores are min–max normalized within each method and pool. Models are ordered by PGD-50 robust accuracy, so a more monotonically increasing curve indicates stronger ranking agreement. Uncalibrated SGA follows the reference order more consistently than the baselines and remains visually close to Oracle SGA.
Pool
Probe source
K=10
K=5
K=2
K=1
DNN ℓ2
CelebA-HQ
0.9694±0.0170
0.9604±0.0216
0.9400±0.0297
0.9171±0.0381
Places365
0.9581±0.0308
0.9362±0.0468
0.8989±0.0801
0.8749±0.0952
DNN ℓ∞
CelebA-HQ
0.8651±0.0234
0.8538±0.0281
0.8355±0.0386
0.8089±0.0571
Places365
0.8745±0.0365
0.8484±0.0540
0.7977±0.0936
0.7409±0.1212
VLM ℓ∞
CelebA-HQ
0.8010±0.0464
0.7988±0.0470
0.7890±0.0608
0.7830±0.0747
Places365
0.7686±0.0415
0.7540±0.0567
0.7371±0.0735
0.7223±0.0867
TABLE V: Few-shot performance of Uncalibrated SGA on public probes. Mean PA ± batch-level standard deviation.
Probe source
Protocol
DNN ℓ2
DNN ℓ∞
VLM ℓ∞
LVLM ℓ∞
CelebA-HQ
Uncalibrated
0.9694
0.8651
0.8010
0.7620
Transfer
0.9592
0.7910
0.7700
0.8320
Calibrated
0.9667
0.8515
0.8233
0.7600
Oracle
0.9991
0.9600
0.9600
0.9910
Places365
Uncalibrated
0.9581
0.8745
0.7686
0.6990
Transfer
0.9426
0.8137
0.7981
0.7720
TABLE VI: Comparison of SGA protocols at K=10 . Mean PA across probe batches.
CelebA-HQ
Places365
Protocol
PGD-50
AutoAttack
PGD-50
AutoAttack
Uncalibrated
0.8651
0.8974
0.8745
0.8886
Transfer
0.7910
0.7705
0.8137
0.7821
Calibrated
0.8515
0.8677
0.8846
0.8962
Oracle
0.9600
0.9605
0.9666
0.9658
TABLE VII: SGA agreement with PGD-50 and AutoAttack for DNN ℓ∞ . Mean PA at K=10 using public probes.
Probe source
Protocol
PGD-50
AutoAttack
CelebA-HQ
Uncalibrated
1.0%
99.0%
Oracle
84.7%
15.3%
Places365
Uncalibrated
28.7%
71.3%
Oracle
90.7%
9.3%
TABLE VIII: SGA preferences on model pairs disputed by PGD-50 and AutoAttack. Percentage of pair–batch comparisons supporting each reference order at K=10 .
Method
DNN ℓ2
DNN ℓ∞
VLM ℓ∞
LVLM ℓ∞
GREAT
6.6/7.1
16.2/16.3
19.0/19.2
79.4/97.3
FGSM
7.5/7.9
18.6/19.4
21.3/22.7
94.4/137.0
CLEVER
178.6/210.5
218.3/239.0
233.8/313.1
1,215.2/1,811.3
Linearized
8.5/9.0
23.1/27.0
23.9/24.0
113.2/118.1
SGA (Transfer)
6.8/7.4
17.9/18.2
19.8/20.4
84.6/103.8
SGA (Others)
+0.023/+0.025
+0.023/+0.021
+0.024/+0.024
+0.032/+0.031
TABLE IX: Runtime at K=10 . End-to-end seconds for baselines and Transfer SGA (test-set/public); other protocols report extra runtime.
Probe source
Model pool
Original
Spatially permuted
ΔPA
CelebA-HQ
DNN ℓ2
0.9694
0.8877±0.0011
0.0817
DNN ℓ∞
0.8651
0.7955±0.0031
0.0696
VLM ℓ∞
0.8010
0.5647±0.0146
0.2363
LVLM ℓ∞
0.7620
0.3350±0.0149
0.4270
Places365
DNN ℓ2
0.9581
0.6909±0.0068
0.2672
DNN ℓ∞
0.8745
0.7678±0.0012
0.1067
TABLE X: Spatial-structure destruction. PA at K=10 using original and spatially permuted sensitivity fields. Permuted results are mean ± standard deviation over five seeds.
Fig. 4: Effect of the number of anchor models. Using more randomly selected anchors generally improves PA, although the improvement becomes smaller as more anchors are added and varies across model pools.
Architecture
Model pool
Test set
CelebA-HQ
Places365
ResNet (12)
DNN ℓ2
0.9771
0.9765
0.9761
ResNet (2)
DNN ℓ∞
1.0000
1.0000
1.0000
ConvNeXt (3)
DNN ℓ∞
1.0000
1.0000
1.0000
ConvNeXt (2)
VLM ℓ∞
1.0000
1.0000
1.0000
Swin (3)
DNN ℓ∞
1.0000
1.0000
1.0000
ViT (2)
DNN ℓ∞
0.8900
0.7800
0.8000
TABLE XI: Architecture-level analysis across three data sources. Best mean PA at K=10 within each architecture group, with configurations selected separately by group and data source. Numbers in parentheses indicate group sizes.
Fig. 5: Gradient-to-structure transformation across architectures. Under the same input and configuration, ResNet, ConvNeXt, and Swin models exhibit relatively continuous sensitivity patterns, whereas the more challenging ViT and XCiT models exhibit more patchwise and fragmented patterns. Scores and PGD references are comparable only within pools.
Fig. 6: Effects of the four SGA configuration components. We vary one component at a time over its candidate range while fixing the other three to the global base configuration. Each point reports the mean PA across all probe batches at K=10 .
Probe source
Model pool
Best five
Worst five
CelebA-HQ
DNN ℓ2
0.9744
0.9487
DNN ℓ∞
0.9011
0.8462
VLM ℓ∞
0.7619
0.8095
LVLM ℓ∞
1.0000
0.6000
Average
0.9094
0.8011
Places365
DNN ℓ2
0.9744
0.9103
TABLE XII: Variation across individual probes. PA of the K=5 batches formed from the five highest- and five lowest-ranked probes in each public-probe source.
Fig. 7: Variation across individual probes. The five highest- and five lowest-ranked probes from CelebA-HQ and Places365.
Probe
SGA rank
Disagreement
PGD ref.
Rank-5 ref.
Test set
5 (88%)
100%
0.7386
0.9195
CelebA-HQ
5 (84%)
99%
0.8010
0.9695
Places365
5 (100%)
100%
0.7686
0.9590
TABLE XIII: Persistent disagreement for ViT-L-14_TeCoA4 . PGD ranks it #1, whereas Uncalibrated SGA most often ranks it #5; the rank-5 reference moves only this model while preserving all other orders.
We study adversarial robustness of open-source vision-language model (VLM) agents deployed in a self-contained e-commerce environment built to simulate realistic pre-deployment conditions. We evaluate two agents, LLaVA-v1.5-7B and Qwen2.5-VL-7B, under three gradient-based attacks: the Basic Iterative Method (BIM), Projected Gradient Descent (PGD), and a CLIP-based spectral attack. Against LLaVA, all three attacks achieve substantial attack success rates (52.6%, 53.8%, and 66.9% respectively), demonstrating that simple gradient-based methods pose a practical threat to open-source VLM agents. Qwen2.5-VL proves significantly more robust across all attacks (6.5%, 7.7%, and 15.5%), suggesting meaningful architectural differences in adversarial resilience between open-source VLM families. These findings have direct implications for the security evaluation of VLM agents prior to commercial deployment.
Achieving adversarial robustness in Vision-Language Models (VLMs) inevitably compromises accuracy on clean data, presenting a long-standing and challenging trade-off. In this work, we revisit this trade-off by investigating a fundamental question: What makes VLMs robust? Through a detailed analysis of adversarially fine-tuned models, we examine how robustness mechanisms function internally and how they interact with clean accuracy. Our analysis reveals that adversarial robustness is not uniformly distributed across network depth. Instead, unexpectedly, it is primarily localized within the shallow layers, driven by a low-frequency spectral bias and input-insensitive attention patterns. Meanwhile, updates to the deep layers tend to undermine both clean accuracy and robust generalization. Motivated by these insights, we propose Adversarial Robustness Adaptation (R-Adapt), a simple yet effective framework that freezes all pre-trained weights and introduces minimal, insight-driven adaptations only in the initial layers. This design achieves an exceptional balance between adversarial robustness and clean accuracy. R-Adapt further supports training-free, model-guided, and data-driven paradigms, offering flexible pathways to seamlessly equip standard models with robustness. Extensive evaluations on 18 datasets and diverse tasks demonstrate our state-of-the-art performance under various attacks. Notably, R-Adapt generalizes efficiently to large vision-language models (e.g., LLaVA and Qwen-VL) to enhance their robustness. Our project page is available at https://summu77.github.io/R-Adapt.
Sen Nie, Jie Zhang, Zhongqi Wang +3
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190, China · University of Chinese Academy of Sciences, Beijing 100049, China
Targeted adversarial attacks on Large Vision-Language Models (LVLMs) test whether small image perturbations can steer model responses toward attacker-specified content. Under the standard L-infinity constraint, targeted attacks become a regional perturbation budget allocation problem: attack success depends not only on the perturbation objective, but also on which regions receive updates and in what order. Existing localized attacks improve over global perturbations but rely on stochastic spatial sampling, often updating weakly influential regions. We address this limitation through an attention-based analysis showing that cross-modal attention identifies adversarially sensitive regions and that perturbing high-attention hotspots induces predictable redistribution toward subsequent salient regions. These findings motivate attention-guided region sequencing, which begins from dominant hotspots and progressively moves the update support toward next-salient regions. Based on these principles, we propose Stage-wise Attention-Guided Attack (SAGA), a black-box region-sequencing framework that uses a fixed attention map from an open-source LVLM to guide perturbation updates without accessing target-model parameters, gradients, or attention maps. Across ten closed-source and open-source LVLMs, SAGA achieves state-of-the-art attack success rates and the best overall imperceptibility. The source code is available at https://github.com/jaehyun-kwak/SAGA.