ARMOR: Manifold-Oriented Training for Adversarially Robust Aerial Object Detection under Data Scarcity
Authors: Haoran Wang, Matthew Lau, Alec Helbling, Matthew Hull, ShengYun Peng, Mansi Phute, Martin Andreoni, Willian T. Lunardi, Duen Horng Chau, Wenke Lee
Organizations: Georgia Institute of Technology, Atlanta, GA, USA · Khalifa University, Abu Dhabi, UAE · Technology Innovation Institute, Abu Dhabi, UAE
Aerial object detection is increasingly deployed in real-world applications, but models remain vulnerable to physical, universal adversarial patches that cause them to miss objects. Furthermore, defenders face the practical constraint of training data scarcity: aerial imagery is costly to collect and label, so a deployment site typically yields hundreds of images rather than the tens of thousands that adversarial robustness benchmarks assume. To tackle model vulnerability and training data scarcity, we propose Adversarial Robustness with Manifold-Oriented Training (ARMOR), a novel defense that realizes the core insights of on-manifold adversarial training (OMAT) in low-data regimes. ARMOR builds on the insight of OMAT to model the data manifold - the compact structure capturing the data's relevant features - to learn and robustify these features during training. While OMAT relies on the data-intensive operations of training large generative models and adversarial training to achieve this, ARMOR adopts a data-efficient approach that reuses labels the detection task already supplies: ARMOR (i) masks image backgrounds to retain object-relevant features, and (ii) injects randomized patches on objects to improve feature robustness. Our low-data experiments with physically-realizable adversarial patches evaluate both query-free transfer attacks and defense-aware attacks. ARMOR maintains strong clean performance of over 0.90 model confidence, while improving adversarial robustness by up to 0.32 in model confidence over state-of-the-art defenses. Physical experiments with printed patches confirm that these gains survive deployment. Overall, ARMOR translates insights from manifold-based training to defend object detectors amidst training data scarcity.
Figures & tables
Fig. 1: Main idea of ARMOR. We model the two data-intensive operations of on-manifold adversarial training — on-manifold projection and adversarial training — with background removal and random patches respectively. These modifications force the model to learn relevant and robust features , defending physically-realizable adversarial patch attacks.
Fig. 2: Adversarial “ON patches” [ 1 ] are easy for an attacker to deploy physically (a, b). Transfer (ON-T) and defense-aware (ON-DA) patches (c, d) are optimized for smoothness and printability.
Fig. 3: ARMOR’s Method Diagram. ARMOR injects random patches onto objects during training to promote robustness. The first phase of ARMOR’s training is standard fine-tuning, which is used as a warm start for the second phase. This second phase implements our key novelty of training with background-masked images and random patches. The last phase of hybrid training mitigates any trivial solutions that the prior phase learns, such as localizing non-black regions as objects.
Fig. 4: Main Results. Sample visualizations of car detection by various defenses amidst defense-aware ON (ON-DA) attacks. Other defenses have predictions below the standard threshold of 0.50, resulting in no detections. Meanwhile, ARMOR mitigates the ON-DA attack and detects cars with high average confidence of 0.62.
Patch \ Defense
Undefended
FT
RS
PAD
PAD-S
SHIELD
ARMOR (ours)
Conf ↑
AP ↑
Conf ↑
AP ↑
Conf ↑
AP ↑
Conf ↑
AP ↑
Conf ↑
AP ↑
Conf ↑
AP ↑
Conf ↑
AP ↑
Clean
0.83 ± 0.16
0.89
0.96 ± 0.04
1.00
0.91 ± 0.10
1.00
0.94 ± 0.04
1.00
0.96 ± 0.04
1.00
0.96 ± 0.05
1.00
0.92 ± 0.07
1.00
Random ON
0.62 ± 0.26
0.73
0.87 ± 0.11
1.00
0.46 ± 0.31
0.72
0.87 ± 0.12
1.00
0.78 ± 0.31
1.00
0.90 ± 0.09
1.00
0.87 ± 0.12
1.00
ON-T Attack
0.14 ± 0.19
0.19
0.36 ± 0.27
0.99
0.31 ± 0.32
0.54
0.48 ± 0.38
1.00
0.48 ± 0.37
1.00
0.42 ± 0.33
0.94
0.73 ± 0.16
1.00
ON-DA Attack
0.14 ± 0.19
0.19
0.14 ± 0.14
0.95
0.30 ± 0.29
0.57
0.14 ± 0.26
1.00
0.17 ± 0.28
1.00
0.14 ± 0.14
0.95
0.62 ± 0.20
0.97
TABLE I: Main Results. Conf and AP for each defense against no patches (clean), random and ON attack patches, comparing fine-tuning (FT), randomized smoothing (RS) [ 46 ] , PAD [ 31 ] without and with SAM (-S suffix), SHIELD [ 35 ] and ARMOR (ours); best ON-T and ON-DA results in bold. ARMOR maintains good Conf on all settings, achieving the best Conf against ON-T and ON-DA attacks by a large margin .
Ablated Settings
Conf ↑
AP ↑
Wt
RP
BM
Hb
Clean
ON-T
ON-DA
Clean
ON-T
ON-DA
0.89 ± 0.10
0.18 ± 0.23
0.11 ± 0.19
0.98
0.39
0.27
✓
0.96 ± 0.04
0.36 ± 0.27
0.14 ± 0.14
1.00
0.99
0.95
✓
✓
0.96 ± 0.05
0.52 ± 0.34
0.25 ± 0.21
1.00
0.92
0.92
✓
✓
✓
0.81 ± 0.22
0.76 ± 0.17
0.59 ± 0.26
0.64
0.47
0.30
✓
✓
✓
0.94 ± 0.05
0.78 ± 0.16
0.49 ± 0.23
1.00
0.99
0.86
TABLE II: Ablation. Conf and AP for clean, ON-T and ON-DA settings. Row 1 is fine-tuned on the original, non-augmented dataset; later rows add weather augmentation (Wt), random patches (RP), or a fine-tuning phase with background-masked (BM) or hybrid (Hb) images. Random patches improve Conf under both ON-T and ON-DA , and models trained without background trade slight clean performance for much better ON-DA Conf . Hybrid fine-tuning after BM fine-tuning mitigates false positives, shown by poor AP of BM fine-tuning alone (underlined).
Setting
Patch
Undefended
FT
ARMOR (ours)
Weather
Clean
0.63 ± 0.39
0.82 ± 0.28
0.85 ± 0.22
Random ON
0.47 ± 0.34
0.65 ± 0.33
0.73 ± 0.29
ON-T
0.23 ± 0.25
0.35 ± 0.32
0.61 ± 0.27
ON-DA
0.23 ± 0.25
0.34 ± 0.32
0.54 ± 0.27
Drone
Clean
0.78 ± 0.31
0.94 ± 0.16
0.90 ± 0.13
Random ON
0.66 ± 0.31
0.93 ± 0.13
0.89 ± 0.10
TABLE III: Generalizability. Conf ( ↑ ) across three shifts: random weather corruptions at test time, drone imagery of a carpark (541 images), and a much smaller detector (YOLOv11n) attacked with a patch trained on YOLOv3. ARMOR obtains the best Conf against ON-T and ON-DA attacks in every setting .
Fig. 5: Realistic Physical ON-T Attack. The undefended and fine-tuned model do not detect the car. In contrast, ARMOR detects cars with high average confidence of 0.75.
Setting
Patch
Undefended
FT
ARMOR (ours)
Sidestreet
Clean
0.83 ± 0.16
0.96 ± 0.04
0.92 ± 0.07
Physical ON-T
0.01 ± 0.09
0.38 ± 0.37
0.75 ± 0.23
Motion blur
Clean
0.92 ± 0.09
0.95 ± 0.05
0.65 ± 0.27
Physical ON-T
0.11 ± 0.29
0.23 ± 0.36
0.59 ± 0.35
TABLE IV: Realistic Physical ON-T Attacks . Conf ( ↑ ) for printed patches affixed to car roofs, in the original sidestreet set-up and under motion blur. ARMOR obtains the best Conf while under realistic physical attack in both settings .
Deep neural network (DNN)-based object detectors are widely used for analyzing aerial and satellite imagery in applications such as environmental monitoring and urban analytics. Despite their strong performance, these models are known to be vulnerable to adversarial examples, and physical adversarial attacks using printable patterns pose realistic security threats. In this paper, we evaluate physical adversarial patch attacks against an aerial vehicle detector by bridging digital optimization and real-world deployment. Adversarial patches are optimized in the digital domain using a loss function that minimizes the maximum objectness score while incorporating non-printability score (NPS) and total variation (TV) constraints to ensure both printability and spatial smoothness. The optimized patches are printed and deployed in three configurations: ON, OFF, and OFF-Side. Experiments using a YOLOv3 detector show that while the OFF patch achieves the highest effectiveness in the digital domain (85.51% Average Objectness Reduction Rate (AORR)), the ON patch demonstrates superior robustness in physical environments (0.197-0.343 Objectness Score Ratio (OSR)) due to its consistent visibility. Furthermore, our results indicate that weather-based augmentation does not necessarily improve patch optimization in this domain. These findings provide critical insights into the practical vulnerabilities of aerial object detection systems.
Jung Heum Woo, Eun-Kyu Lee
School of Information Technology Incheon National University Incheon 22012, Republic of Korea
Object detection models are critical components of automated systems, such as autonomous vehicles and perception-based robots, but their sensitivity to adversarial attacks poses a serious security risk. Progress in defending these models lags behind classification, hindered by a lack of standardized evaluation. It is nearly impossible to thoroughly compare attack or defense methods, as existing work uses different datasets, inconsistent efficiency metrics, and varied measures of perturbation cost. This paper addresses this gap by investigating three key questions: (1) How can we create a fair benchmark to impartially compare attacks? (2) How well do modern attacks transfer across different architectures, especially from Convolutional Neural Networks to Vision Transformers? (3) What is the most effective adversarial training strategy for robust defense? To answer these, we first propose a unified benchmark framework focused on digital, non-patch-based attacks. This framework introduces specific metrics to disentangle localization and classification errors and evaluates attack cost using multiple perceptual metrics. Using this benchmark, we conduct extensive experiments on state-of-the-art attacks and a wide range of detectors. Our findings reveal two major conclusions: first, modern adversarial attacks against object detection models show a significant lack of transferability to transformer-based architectures. Second, we demonstrate that the most robust adversarial training strategy leverages a dataset composed of a mix of high-perturbation attacks with different objectives (e.g., spatial and semantic), which outperforms training on any single attack.
Adversarial patches pose a practical threat to modern object detectors. Prior work shows vulnerability, but three gaps limit actionable insight: (i) few score-based black-box attacks jointly optimize patch location, texture, and size under tight query budgets; (ii) success is rarely tied to the patch's visual footprint; and (iii) evaluations often conflate EOT robustness with plain-view suppression. We present \method{}, a query-efficient, budget-adaptive black-box attack that couples a lightweight Contextual Thompson-Sampling placer with NES-style pixel updates, growing the patch only when progress stalls. Reporting is anchored by a strict plain-image suppression test; EOT is audited but never used as a substitute for success, and optional appearance/printability weights expose strength--visibility trade-offs. Across YOLOv5, Faster R-CNN, and YOLOS, \method{} achieves strong suppression on CNN-based detectors and substantial suppression on the transformer-based detector, using compact patches and exposing clear query--footprint trade-offs relative to fixed-size and heuristic baselines. A print--capture pilot further shows transfer across unseen physical objects and viewpoints.
Pedram MohajerAnsari, Amir Salarpour, David Fernandez +1