Visual adversarial robustness remains a fundamental security property as vision systems evolve from conventional classifiers to vision-language models (VLMs) and large vision-language models (LVLMs), with visual inputs remaining an attack surface across generations. Yet existing evaluation methods often depend on prescribed attacks, in-distribution data, extensive probing, or classifier-specific formulations, limiting external auditing of diverse models. We introduce Structural Gradient Alignment (SGA), a complementary robustness metric measuring the structural alignment between input content and objective-conditioned local model sensitivity. SGA uses a few randomly sampled, unlabeled public probes that need not follow the downstream task distribution and only one input-gradient computation per model--probe pair. Across 38 distinct DNNs, VLMs, and LVLMs, SGA produces rankings consistent with attack-based references. Without robustness references, uncalibrated SGA achieves 83.72% average pairwise agreement across eight public-probe conditions, outperforming the strongest overall baseline in all eight by 9.38 percentage points on average. With known robustness orders of anchor models, calibrated SGA increases agreement to 85.36%. These results demonstrate an attack-agnostic, computationally efficient robustness signal across conventional DNNs, VLMs, and autoregressive LVLMs, downstream tasks, threat norms, probe distributions, and evaluator-knowledge settings. Our implementation is available at https://github.com/azrealwang/SGA.
Figures & tables
TABLE I: Comparison of robustness-evaluation methods.
Protocol
Knowledge
Criterion
Uncalibrated
None
Score dispersion
Transfer
Development ranking
Development PA
Calibrated
Ordered anchors
Anchor PA, then dispersion
Oracle
Target ranking a
Target PA
TABLE II: SGA protocols under different knowledge settings.
Fig. 2: SGA scoring pipeline. For each model–probe pair, SGA transforms the probe image and its input gradient into structural maps under configuration c=(ρ,ψ,e,μ) . It then measures their structural correspondence and aggregates the resulting scores across the probe batch to rank the evaluated models.
Method
Knowledge
DNN ℓ2
DNN ℓ∞
VLM ℓ∞
LVLM ℓ∞
GREAT
None
0.4326±0.1086
0.4881±0.1216
0.7586±0.1233
0.4600±0.2332
FGSM
ℓ2 or ℓ∞
0.5113±0.1464
0.3830±0.1440
0.5714±0.1990
0.5360±0.1676
CLEVER
ℓ2 or ℓ∞
0.8691±0.0399
0.7977±0.0772
0.7407±0.0590
0.4060±0.1907
Linearized
None
0.8649±0.0455
0.7170±0.0721
0.7800±0.0799
0.6250±0.1228
SGA (Uncalibrated)
None
0.9694±0.0170
0.8651±0.0234
0.8010±0.0464
0.7620±0.1198
TABLE III: Pairwise agreement on unlabeled CelebA-HQ public probes at K=10 . Mean PA ± standard deviation across probe batches; best result in each model pool in bold.
Method
Knowledge
DNN ℓ2
DNN ℓ∞
VLM ℓ∞
LVLM ℓ∞
GREAT
None
0.4217±0.0992
0.4665±0.1109
0.4505±0.1235
0.4440±0.2255
FGSM
ℓ2 or ℓ∞
0.5447±0.1564
0.4316±0.1807
0.2662±0.1954
0.5820±0.1564
CLEVER
ℓ2 or ℓ∞
0.8949±0.0420
0.8492±0.0579
0.7262±0.0688
0.4530±0.1947
Linearized
None
0.8945±0.0428
0.7966±0.0580
0.7110±0.1039
0.5580±0.1159
SGA (Uncalibrated)
None
0.9581±0.0308
0.8745±0.0365
0.7686±0.0415
0.6990±0.1245
TABLE IV: Pairwise agreement on unlabeled Places365 public probes at K=10 . Mean PA ± standard deviation across probe batches; best result in each model pool in bold.
Fig. 3: Model scores versus the PGD-50 reference on CelebA-HQ public probes. All methods use the same ten randomly selected probes, and scores are min–max normalized within each method and pool. Models are ordered by PGD-50 robust accuracy, so a more monotonically increasing curve indicates stronger ranking agreement. Uncalibrated SGA follows the reference order more consistently than the baselines and remains visually close to Oracle SGA.
Pool
Probe source
K=10
K=5
K=2
K=1
DNN ℓ2
CelebA-HQ
0.9694±0.0170
0.9604±0.0216
0.9400±0.0297
0.9171±0.0381
Places365
0.9581±0.0308
0.9362±0.0468
0.8989±0.0801
0.8749±0.0952
DNN ℓ∞
CelebA-HQ
0.8651±0.0234
0.8538±0.0281
0.8355±0.0386
0.8089±0.0571
Places365
0.8745±0.0365
0.8484±0.0540
0.7977±0.0936
0.7409±0.1212
VLM ℓ∞
CelebA-HQ
0.8010±0.0464
0.7988±0.0470
0.7890±0.0608
0.7830±0.0747
Places365
0.7686±0.0415
0.7540±0.0567
0.7371±0.0735
0.7223±0.0867
TABLE V: Few-shot performance of Uncalibrated SGA on public probes. Mean PA ± batch-level standard deviation.
Probe source
Protocol
DNN ℓ2
DNN ℓ∞
VLM ℓ∞
LVLM ℓ∞
CelebA-HQ
Uncalibrated
0.9694
0.8651
0.8010
0.7620
Transfer
0.9592
0.7910
0.7700
0.8320
Calibrated
0.9667
0.8515
0.8233
0.7600
Oracle
0.9991
0.9600
0.9600
0.9910
Places365
Uncalibrated
0.9581
0.8745
0.7686
0.6990
Transfer
0.9426
0.8137
0.7981
0.7720
TABLE VI: Comparison of SGA protocols at K=10 . Mean PA across probe batches.
CelebA-HQ
Places365
Protocol
PGD-50
AutoAttack
PGD-50
AutoAttack
Uncalibrated
0.8651
0.8974
0.8745
0.8886
Transfer
0.7910
0.7705
0.8137
0.7821
Calibrated
0.8515
0.8677
0.8846
0.8962
Oracle
0.9600
0.9605
0.9666
0.9658
TABLE VII: SGA agreement with PGD-50 and AutoAttack for DNN ℓ∞ . Mean PA at K=10 using public probes.
Probe source
Protocol
PGD-50
AutoAttack
CelebA-HQ
Uncalibrated
1.0%
99.0%
Oracle
84.7%
15.3%
Places365
Uncalibrated
28.7%
71.3%
Oracle
90.7%
9.3%
TABLE VIII: SGA preferences on model pairs disputed by PGD-50 and AutoAttack. Percentage of pair–batch comparisons supporting each reference order at K=10 .
Method
DNN ℓ2
DNN ℓ∞
VLM ℓ∞
LVLM ℓ∞
GREAT
6.6/7.1
16.2/16.3
19.0/19.2
79.4/97.3
FGSM
7.5/7.9
18.6/19.4
21.3/22.7
94.4/137.0
CLEVER
178.6/210.5
218.3/239.0
233.8/313.1
1,215.2/1,811.3
Linearized
8.5/9.0
23.1/27.0
23.9/24.0
113.2/118.1
SGA (Transfer)
6.8/7.4
17.9/18.2
19.8/20.4
84.6/103.8
SGA (Others)
+0.023/+0.025
+0.023/+0.021
+0.024/+0.024
+0.032/+0.031
TABLE IX: Runtime at K=10 . End-to-end seconds for baselines and Transfer SGA (test-set/public); other protocols report extra runtime.
Probe source
Model pool
Original
Spatially permuted
ΔPA
CelebA-HQ
DNN ℓ2
0.9694
0.8877±0.0011
0.0817
DNN ℓ∞
0.8651
0.7955±0.0031
0.0696
VLM ℓ∞
0.8010
0.5647±0.0146
0.2363
LVLM ℓ∞
0.7620
0.3350±0.0149
0.4270
Places365
DNN ℓ2
0.9581
0.6909±0.0068
0.2672
DNN ℓ∞
0.8745
0.7678±0.0012
0.1067
TABLE X: Spatial-structure destruction. PA at K=10 using original and spatially permuted sensitivity fields. Permuted results are mean ± standard deviation over five seeds.
Fig. 4: Effect of the number of anchor models. Using more randomly selected anchors generally improves PA, although the improvement becomes smaller as more anchors are added and varies across model pools.
Architecture
Model pool
Test set
CelebA-HQ
Places365
ResNet (12)
DNN ℓ2
0.9771
0.9765
0.9761
ResNet (2)
DNN ℓ∞
1.0000
1.0000
1.0000
ConvNeXt (3)
DNN ℓ∞
1.0000
1.0000
1.0000
ConvNeXt (2)
VLM ℓ∞
1.0000
1.0000
1.0000
Swin (3)
DNN ℓ∞
1.0000
1.0000
1.0000
ViT (2)
DNN ℓ∞
0.8900
0.7800
0.8000
TABLE XI: Architecture-level analysis across three data sources. Best mean PA at K=10 within each architecture group, with configurations selected separately by group and data source. Numbers in parentheses indicate group sizes.
Fig. 5: Gradient-to-structure transformation across architectures. Under the same input and configuration, ResNet, ConvNeXt, and Swin models exhibit relatively continuous sensitivity patterns, whereas the more challenging ViT and XCiT models exhibit more patchwise and fragmented patterns. Scores and PGD references are comparable only within pools.
Fig. 6: Effects of the four SGA configuration components. We vary one component at a time over its candidate range while fixing the other three to the global base configuration. Each point reports the mean PA across all probe batches at K=10 .
Probe source
Model pool
Best five
Worst five
CelebA-HQ
DNN ℓ2
0.9744
0.9487
DNN ℓ∞
0.9011
0.8462
VLM ℓ∞
0.7619
0.8095
LVLM ℓ∞
1.0000
0.6000
Average
0.9094
0.8011
Places365
DNN ℓ2
0.9744
0.9103
TABLE XII: Variation across individual probes. PA of the K=5 batches formed from the five highest- and five lowest-ranked probes in each public-probe source.
Fig. 7: Variation across individual probes. The five highest- and five lowest-ranked probes from CelebA-HQ and Places365.
Probe
SGA rank
Disagreement
PGD ref.
Rank-5 ref.
Test set
5 (88%)
100%
0.7386
0.9195
CelebA-HQ
5 (84%)
99%
0.8010
0.9695
Places365
5 (100%)
100%
0.7686
0.9590
TABLE XIII: Persistent disagreement for ViT-L-14_TeCoA4 . PGD ranks it #1, whereas Uncalibrated SGA most often ranks it #5; the rank-5 reference moves only this model while preserving all other orders.
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190, China · University of Chinese Academy of Sciences, Beijing 100049, China