cs.CRNov 11, 2026

Beyond Attack-Based Accuracy: Rethinking Adversarial Robustness Evaluation Through Structural Gradient Alignment

Authors: Hanrui Wang, Yu Luo, Shuo Wang, Xingbo Dong, Zhe Jin, Isao Echizen

Organizations: National Institute of Informatics, Tokyo, Japan · Anhui University, Hefei, China · Shanghai Jiao Tong University, Shanghai, China

Abstract

Visual adversarial robustness remains a fundamental security property as vision systems evolve from conventional classifiers to vision-language models (VLMs) and large vision-language models (LVLMs), with visual inputs remaining an attack surface across generations. Yet existing evaluation methods often depend on prescribed attacks, in-distribution data, extensive probing, or classifier-specific formulations, limiting external auditing of diverse models. We introduce Structural Gradient Alignment (SGA), a complementary robustness metric measuring the structural alignment between input content and objective-conditioned local model sensitivity. SGA uses a few randomly sampled, unlabeled public probes that need not follow the downstream task distribution and only one input-gradient computation per model--probe pair. Across 38 distinct DNNs, VLMs, and LVLMs, SGA produces rankings consistent with attack-based references. Without robustness references, uncalibrated SGA achieves 83.72%83.72\% average pairwise agreement across eight public-probe conditions, outperforming the strongest overall baseline in all eight by 9.389.38 percentage points on average. With known robustness orders of anchor models, calibrated SGA increases agreement to 85.36%85.36\%. These results demonstrate an attack-agnostic, computationally efficient robustness signal across conventional DNNs, VLMs, and autoregressive LVLMs, downstream tasks, threat norms, probe distributions, and evaluator-knowledge settings. Our implementation is available at https://github.com/azrealwang/SGA.

Figures & tables

Explore similar work

CardsList
  1. What Makes VLMs Robust? Towards Reconciling Robustness and Accuracy in Vision-Language Models

    Mar 13, 2026Sen Nie, Jie Zhang, Zhongqi Wang +3VLM RobustnessAdversarial Robustness