Can Attack Difficulty Be Characterized Before Optimization? A Study of Pre-optimization Difficulty in Person-Vanishing Attacks
Authors: Jingyao Xu, Dongdong Wang, Siyang Lu
Organizations: School of Computer Science and Technology, Beijing Jiaotong University · College of Design, Construction, and Planning, University of Florida
Adversarial attacks against object detectors are traditionally studied from an optimization perspective, where attack difficulty is regarded as an outcome observed only after adversarial optimization. This raises a fundamental question: \emph{can the relative attack difficulty of different inputs be characterized before optimization begins?} In this paper, we investigate this question for person-vanishing attacks by introducing the concept of pre-optimization attack difficulty, which captures intrinsic differences in optimization effort across input images. To estimate this latent difficulty before optimization, we propose Quad-CLEVER, an efficient geometry-based estimator derived from a quadratic approximation of the local person-vanishing margin along the most attack-relevant direction. Extensive experiments across multiple attack algorithms demonstrate that Quad-CLEVER consistently correlates with the observed optimization cost, providing empirical evidence that attack difficulty exhibits a predictable pre-optimization structure. Building upon this finding, we further propose a difficulty-aware attack framework that leverages the estimated difficulty to adaptively allocate optimization budgets for a base attack under a fixed computational budget. On BDD100K, the proposed framework improves the image-level attack success rate by up to 5.78% while reducing the average optimization cost by up to 11.42 iterations. On the more challenging EventPed dataset, it saves 2.25 optimization iterations while maintaining comparable attack performance. These results demonstrate that attack difficulty can be meaningfully estimated before optimization and that exploiting such estimates enables more computationally efficient adversarial attacks.
Figures & tables
Figure 1: Images exhibit substantially different attack difficulty, making a uniform attack budget inefficient: easy images may receive redundant computation, while hard images remain under-optimized
Dataset
Attack
ASR gain of Log continuous (pp)
ΔKsave
vs. Random
vs. Uniform
vs. Fixed K
at same ASR
BDD100K
PGD
+5.78
+3.45
+3.08
+11.42
BDD100K
TOG
+5.00
+2.52
+1.50
+2.36
EventPed
PGD
+1.69
−0.41
+0.58
+2.25
EventPed
TOG
+0.07
+2.67
+0.92
+2.13
Table 1: ASR gains and iteration efficiency of Log continuous routing. ASR gains are measured in percentage points. Positive ΔKsave indicates fewer iterations required to achieve the same ASR.
Figure 2: Quad-CLEVER versus the first successful attack iteration for PGD and TOG on BDD100K and EventPed. Filled points denote successfully attacked images, while hollow red points indicate images that remain unsuccessful within the maximum iteration budget. The solid curve shows the rolling median over successful images.
Figure 3: Average executed iterations versus image-level ANY ASR for PGD and TOG on BDD100K and EventPed. Fixed- K results form the reference curves, while Bucket3 and Log continuous routing allocate different iteration budgets according to the Quad-CLEVER score.
Figure 4: ASR–Budget Trade-off of Quad-CLEVER-Guided Routing. We vary γ to produce continuous-routing results with different Avg. K and compare them with fixed- K attacks under similar average budgets.
Top-20% Overlap (%)
Top-10% → Top-20% (%)
Dataset
Q–P
Q–T
P–T
Q → P
P → Q
Q → T
T → Q
P → T
T → P
BDD100K
85.17
66.09
70.57
98.39
98.62
87.82
76.55
94.02
79.31
EventPed
73.13
64.79
80.21
82.50
82.92
75.42
69.17
95.42
85.83
Table 2: Consistency between difficulty rankings produced by Quad-CLEVER (Q), PGD (P), and TOG (T). A10→B20 denotes the proportion of A ’s top-10% samples contained in B ’s top-20%.
Dataset
Attack
All
Person=1
Person=2
Person=3
BDD100K
PGD
0.9754
0.9789
0.9770
0.9825
BDD100K
TOG
0.6063
0.7437
0.5996
0.5019
EventPed
PGD
0.9289
0.8793
0.9240
0.9360
EventPed
TOG
0.7349
0.6818
0.7148
0.7487
Table 3: Spearman correlation between Quad-CLEVER and attack consumption for different numbers of persons.
Pixel-wise adversarial patches are computationally heavy and often visually detectable, limiting utility in security-critical systems. We present adversarial Voronoi camouflage that optimizes only seed-point locations under fixed, printable palettes using a soft assignment, producing structured, splinter camouflage-like patterns without additional regularization. Evaluated on person detection with COCO-style AP@[.5:.95], naive placement (Inria -> COCO) performs comparably bad, while garment-level application via segmentation mask (3DPeople) results in a significant AP drop. The attack transfers to out-of-domain backgrounds and across detector families (YOLOv9/10/11/12), indicating robustness in black-box settings. Repainting with different palettes largely nullifies the effect, and single-color tweaks show limited tolerance (<=0.17), highlighting a structure-palette coupling. The parameter-efficient, palette-constrained design improves visual plausibility while degrading real-time detector performance. Physical validation and color calibration are left for future work. Code: https://github.com/JensBayer/Voronoi This paper was originally presented at the International Conference on Military Communication and Information Systems (ICMCIS), organized by the Information Systems Technology (IST) Scientific and Technical Committee, IST-224-RSY - the ICMCIS, held in Bath, United Kingdom, 12-13 May 2026.
Jens Bayer, Stefan Becker, David Münch +2
Fraunhofer IOSB and Fraunhofer Center for Machine Learning · Karlsruhe Institute of Technology
Black-box adversarial attacks that minimize only the ground-truth confidence suffer from class drift: perturbations wander through the feature space without committing to a specific adversarial class, wasting queries on diffuse, undirected progress. We introduce Opportunistic Target Selection (OTS), a lightweight wrapper that switches an untargeted attack to a targeted objective early in its trajectory, locking onto whichever non-true class currently leads. OTS requires no architectural modification to the underlying attack, no gradient access, and no a priori target-class knowledge. We validate OTS on three score-based attacks (SimBA, Square Attack with cross-entropy loss, and Bandits) across five standard ImageNet classifiers (4,500 runs). On random-search attacks, OTS closely tracks oracle performance, with gains up to +27 pp in success rate and 43% relative reduction in censored-mean iterations on ResNet-50. On gradient-estimation attacks (Bandits) and attacks with margin loss, OTS is redundant, a negative result that reinforces our interpretation of OTS as a margin-loss surrogate. On adversarially-trained models, a bimodal difficulty distribution eliminates the regime where targeting helps.
Vision Transformers (ViTs) increasingly rely on input-adaptive inference, such as token pruning and early halting, to meet energy and latency budgets. This survey examines a recent class of adversarial efficiency degradation attacks that target these mechanisms to increase computation without necessarily degrading accuracy. We unify and compare two representative attacks, SlowFormer (a universal adversarial patch) and DeSparsify (per-image perturbations), across three popular token-pruning frameworks: A-ViT, ATS, and AdaViT. We standardize reporting using GFLOPs, accuracy loss, and an Attack Success (AS) metric that measures how much of the model's compute savings the attack takes away. Understanding these attacks is crucial for designing countermeasures that not only mitigate risk but also remain lightweight, since deployment often occurs in low-power settings such as mobile or embedded devices. To organize our analysis, we focus on three questions: how input-adaptive optimizations (e.g., token pruning and early halting) create attack surfaces for efficiency degradation; how such attacks operate in practice and which optimizations are most vulnerable; and which defenses exist today and whether they meaningfully restore efficiency under attack.