OmniPatch: A Universal Adversarial Patch for ViT-CNN Cross-Architecture Transfer in Semantic Segmentation

Aarush Aggarwal; Akshat Tomar; Amritanshu Tiwari; Sargam Goyal

cs.LGMar 21, 2026

OmniPatch: A Universal Adversarial Patch for ViT-CNN Cross-Architecture Transfer in Semantic Segmentation

Authors: Aarush Aggarwal, Akshat Tomar, Amritanshu Tiwari, Sargam Goyal

Organizations: Data Science Group, Indian Institute of Technology Roorkee, India

Abstract

Robust semantic segmentation is crucial for safe autonomous driving, yet deployed models remain vulnerable to black-box adversarial attacks when target weights are unknown. Most existing approaches either craft image-wide perturbations or optimize patches for a single architecture, which limits their practicality and transferability. We introduce OmniPatch, a training framework for learning a universal adversarial patch that generalizes across images and both ViT and CNN architectures without requiring access to target model parameters.

Figures & tables

Figure 1: We apply the patch on the most sensitive class and use a two-stage regularized loss, weighting confident ViT pixels, then ViT-CNN ensembling via JS divergence and gradient alignment.

Model	Clean Image mIoU	Random Patch mIoU	Baseline Patch mIoU	OmniPatch mIoU	mIoU Drop (%) Clean / Random/Baseline
PIDNet-S	0.8695	0.8651	0.7791	0.7299	16.05 / 15.62/ 6.31
PIDNet-M	0.8681	0.8618	0.8615	0.7393	14.84 / 14.22/ 14.18
PIDNet-L	0.9035	0.8996	0.8984	0.7530	16.65 / 16.30/ 16.18
BiSeNetv1	0.7149	0.7057	0.7120	0.6410	10.33 / 9.17/ 9.97
BiSeNetv2	0.6907	0.6845	0.6848	0.6036	12.61 / 11.81/ 11.85
SegFormer	0.7434	0.7431	-	0.6777	8.83 / 8.79/ -

Table 1: mIoU on Cityscapes, comparing OmniPatch against clean, random and baseline.

Panel of Figure 4 — (a) Across different patches

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Class-wise Sensitivity Score across Cityscapes Dataset

Model	Clean Image	Random Patch	OmniPatch
PIDNet-S	0.869577	0.865101	0.729970
PIDNet-M	0.868144	0.861862	0.739300
PIDNet-L	0.903522	0.899625	0.753016
BiSeNetv1	0.714959	0.705789	0.641062
BiSeNetv2	0.690796	0.684500	0.603687
SegFormer	0.743495	0.743120	0.677788

Appendix

Table 2: Performance Comparison across Different Patch Attacks

Model	Patch 200 (1.9%)	Patch 300 (4.3%)	Patch 300 (7.6%)
PIDNet-S	0.729970	0.679634	0.597220
PIDNet-M	0.739300	0.682442	0.627030
PIDNet-L	0.753016	0.693860	0.646377
BiSeNetv1	0.641062	0.589184	0.535108
BiSeNetv2	0.603687	0.538712	0.502113
SegFormer	0.677788	0.658646	0.598823

Appendix

Table 3: Model Performance under varying Patch areas

Model	Center Placement	Random Placement	Sensitive Region (Ours)
PIDNet-S	0.829445	0.812880	0.729970
PIDNet-M	0.819404	0.804276	0.739300
PIDNet-L	0.854908	0.838226	0.753016
BiSeNetv1	0.668978	0.669475	0.641062
BiSeNetv2	0.641095	0.631650	0.603687
SegFormer	0.731485	0.725132	0.677788

Appendix

Table 4: Impact of Patch Placement Strategies on Model Robustness

Model	With KL divergence	With JS divergence (Ours)
PIDNet-S	0.746490	0.729970
PIDNet-M	0.746948	0.739300
PIDNet-L	0.766492	0.753016
BiSeNetv1	0.646691	0.641062
BiSeNetv2	0.606668	0.603687
SegFormer	0.708983	0.677788

Appendix

Table 5: Comparative Impact of KL and JS Divergence on Adversarial Patch Effectiveness

Model	Without Gradient Alignment	With Gradient Alginment (Ours)
PIDNet-S	0.777234	0.729970
PIDNet-M	0.765665	0.739300
PIDNet-L	0.786792	0.753016
BiSeNetv1	0.662936	0.641062
BiSeNetv2	0.627358	0.603687
SegFormer	0.680991	0.677788

Appendix

Table 6: Impact of gradient alignment on adversarial patch effecitveness (mIoU)

Senstive Region Placement
Parameter	Value
Morphological Dilation pixels $(k)$	$5$
Biased Sampling threshold $(p)$	$0.2$
Stage 1
Stage 1 Epochs	$10$
Stage 1 weighting $(\gamma)$	$0.7$

Appendix

Table 7: Implementation Details

Adversarial Patch Attacks

Adversarial patch attacks use localized visual patterns to manipulate machine-learning predictions and actions, including in physical environments.

See all 49 papers

IGME: Efficient Chained Method Ensemble for Transferable Semantic Segmentation Attacks
Jul 29, 2026Mengqi He, Jing ZhangAdversarial Attacks Transfer-Based Adversarial Attacks
School of Computing, College of Systems & Society · The Australian National University, Canberra, ACT, Australia
RePAIR: Inverting Backdoor Attacks in Semantic Segmentation
Oct 27, 2026Arnold Brosch, Kira MaagBackdoor Defense Semantic Segmentation
Heinrich Heine University Düsseldorf
Season: Spectrum-Aware Orthogonal Gradient Refinement for Transfer-Based Adversarial Attacks
Aug 5, 2026Tianyi Wang, Zhenghao Gao, Shengjie XuAdversarial Attacks Transfer-Based Adversarial Attacks
Tongji University Shanghai, China · Huazhong University of Science and Technology · Wuhan LightRead Intelligent Technology Co., Ltd. Wuhan, China

Jul 29, 2026J/K move · Enter open · S save

IGME: Efficient Chained Method Ensemble for Transferable Semantic Segmentation Attacks

Mengqi He, Jing Zhang

School of Computing, College of Systems & Society · The Australian National University, Canberra, ACT, Australia

Adversarial Attacks Transfer-Based Adversarial Attacks Consistent Adversarial Attacks Adversarial Transferability

Open paper AskPDF arXiv

Semantic segmentation models are vulnerable to transferable adversarial perturbations, yet evaluating transfer attacks on dense prediction models can be computationally expensive. Existing ensemble attacks often rely on multiple surrogate models, increasing the computation cost, even harder for segmentation. This paper studies an efficient single-source alternative for transferable attacks on semantic segmentation. We formulate transferable attack composition as a chained computation over differentiable attack components, allowing the expensive source-model gradient computation to be shared. To reduce the update instability introduced by chained composition, we further use an integrated-gradient-style path-averaged direction as an empirical stabilization heuristic. Experiments on Pascal VOC and Cityscapes evaluate the resulting transferability efficiency trade-off across CNN- and transformer-based segmentation models. IGME achieves competitive transferability compared with single-source baselines and favorable runtime compared with model-ensemble attacks, while requiring access to only one source model.

Abstract

Adversarial Patch Attacks

Explore similar work

IGME: Efficient Chained Method Ensemble for Transferable Semantic Segmentation Attacks

RePAIR: Inverting Backdoor Attacks in Semantic Segmentation

Season: Spectrum-Aware Orthogonal Gradient Refinement for Transfer-Based Adversarial Attacks