Discovering Natural Transformation Vulnerabilities in Black-Box Vision Models

Dongsu Song; DaeYun GO; Jay Hoon Jung

cs.CVSep 7, 2026

Discovering Natural Transformation Vulnerabilities in Black-Box Vision Models

Authors: Dongsu Song, DaeYun GO, Jay Hoon Jung

Abstract

Natural adversarial examples (NAEs) reveal that vision models can fail under realistic semantic changes beyond norm-bounded perturbations. However, generating NAEs in a black-box setting remains challenging because existing generative attacks often rely on surrogate models, learned attack priors, or costly query-based optimization, whereas the natural transformations that expose model vulnerabilities are unknown a priori. We propose \textbf{Adversarial Scenario Attack (ASA)}, a query-based black-box framework that searches over natural-language editing scenarios using a multimodal language model and a modern text-guided generative editor. ASA jointly explores background, weather, and material/color transformations through winner--loser feedback, and uses a greedy explorer to compose only attack-improving scenarios. Across diverse ImageNet classifiers, ASA achieves substantially higher attack success rates than prior query-based generative attacks while requiring fewer victim-model queries and preserving competitive perceptual quality. Moreover, ASA exhibits both image-level and prompt-level transferability: its adversarial images remain effective across victim-model architectures, while its discovered editing scenarios can be reused across same-class images and, in some cases, across architectures. These findings suggest that vision models possess reusable vulnerabilities to natural transformation patterns, which ASA can efficiently identify in a black-box setting.

Figures & tables

Figure 1: Qualitative overview of ASA. (a) ASA generates visually natural adversarial examples compared with existing attacks. (b) Same-class prompt transfer produces diverse NAEs using transferable prompts. Additional examples are provided in App. .

Figure 2: Overview of Adversarial Scenario Attack (ASA). The method searches over natural-language editing scenarios by combining winner–loser feedback with a Greedy Explorer that composes promising candidates into a stronger scenario, followed by naturalness verification.

Methods	CNNs				Transformers				SSMs				Robust Models
	ResNet-50		ConvNeXt		Swin		DeiT		ViM		MambaVision		Adv-Res		Adv-Inc
	ASR	Avg Q	ASR	Avg Q	ASR	Avg Q	ASR	Avg Q	ASR	Avg Q	ASR	Avg Q	ASR	Avg Q	ASR	Avg Q
AdvFlow	18.56	18.28	5.83	32.30	4.08	24.00	5.93	27.54	8.14	29.93	3.42	29.33	17.47	23.29	22.55	20.45
DIFAttack	46.67	36.99	17.18	46.43	23.93	45.81	25.42	48.42	26.45	43.44	17.22	51.99	27.52	36.83	36.20	30.47
CGAttack	43.99	18.11	22.19	16.08	19.54	26.06	16.31	35.32	17.67	19.13	13.38	25.90	31.10	30.07	51.82	16.92
MCGAttack	82.62	4.16	49.39	29.96	23.82	30.78	26.59	35.20	32.12	29.48	20.85	34.67	44.30	24.24	52.78	22.10

Table 1: Query-based attack results. Higher ASR (%) and lower Avg. Q (successful samples only) indicate better performance. Best results in red bold .

Source Model	Attack	CNNs				Transformers			SSMs		Robust Models		Avg. ASR	Perc. Sim. $\uparrow$	FID $\downarrow$
Source Model	Attack	RN-50	WRN-50	Inc-v3	ConvNeXt	ViT	Swin	DeiT	ViM	MambaVision	Adv-Inc	Adv-Res	Avg. ASR	Perc. Sim. $\uparrow$	FID $\downarrow$
Swin	CF	45.5	41.8	30.2	13.5	27.3	60.6 ∗	18.2	22.6	10.2	37.8	29.5	30.7	0.95	48.55
	NCF	45.7	40.7	35.1	28.1	31.1	83.6 ∗	27.1	27.8	21.1	30.7	28.0	36.3	0.96	41.47
	ACA	60.3	59.0	62.3	57.0	58.6	80.8 ∗	59.2	60.9	58.0	65.9	62.2	62.2	0.53	66.18
	DIFFAttack	54.1	56.5	46.8	67.2	58.1	89.8 ∗	63.9	56.8	65.5	44.6	39.7	58.5	0.75	52.66
	AdvFlow	6.1	7.0	10.5	1.2	2.5	5.1	1.8	1.9	0.7	8.7	7.9	4.9	0.98	22.58

Table 2: ASR (%), Perc. Sim., and FID across three source models. ∗ marks white-box results, and gray-shaded rows indicate query-based attacks. Bold , underlined , and red values mark the best, second-best, and third-best ASR within each source model.

Source Model	Method	Victim Models											Avg. ASR
		CNNs				Transformers			SSMs		Robust Models
		RN-50	WRN-50	Inc-v3	ConvNeXt	ViT	Swin	DeiT	ViM	MambaVision	Adv-Inc	Adv-Res
Swin	ASA ${}_{\mathrm{w/o}\,S,G}$	33.42	31.89	38.36	25.13	27.97	27.47 ∗	27.40	28.68	25.06	41.30	35.66	31.12
	ASA ${}_{\mathrm{w/o}\,G}$	40.26	37.44	44.15	26.74	32.40	29.53 ∗	30.13	32.43	24.67	48.77	41.42	35.27
	ASA	49.45	46.55	53.65	33.82	40.32	36.68 ∗	37.49	40.60	31.55	58.00	49.70	43.44
ConvNeXt	ASA ${}_{\mathrm{w/o}\,S,G}$	33.79	32.52	38.55	25.74 ∗	28.54	27.24	27.60	29.13	25.17	40.63	35.80	31.34

Table 3: Prompt transferability of ASA and its ablations measured by ASR (%) across victim models.

w/o~S,G

removes structured scenario guidance and greedy exploration;

w/o~G

removes only greedy exploration. Red bold denotes the best result, and ∗ marks source-model evaluation.

Attack	NIMA -AVA $\uparrow$	HyperIQA $\uparrow$	MUSIQ -AVA $\uparrow$	MUSIQ -KoniQ $\uparrow$	TReS $\uparrow$
CF	5.26	0.561	4.00	51.80	71.40
NCF	4.90	0.516	3.79	49.00	66.26
DIFFAttack	5.19	0.616	4.16	55.61	81.51
ACA	5.38	0.605	4.34	55.51	79.11
AdvFlow	4.74	0.477	3.76	47.88	64.77
DIFAttack	4.43	0.448	3.68	39.71	49.38

Table 4: Image quality assessment.

Figure 4: Ablation study of ASA components. Bars show ASR, while lines show victim-model and MLLM query counts across source models.

Attack	Prior (s)	Gen. (s)	Rel.
AdvFlow	651.6	132.05	5.08 $\times$
DIFAttack	4939.2	197.5	7.60 $\times$
CGAttack	15645.6	729.5	28.08 $\times$
MCGAttack	69343.2	1051.8	40.48 $\times$
ASA	0	25.98	1.00 $\times$

Table 5: Runtime comparison.

Explore similar work

Sep 21, 2026cs.LG

Reinforcement Learning Inspired Black-box Adversarial Attacks for Computer Vision

Neural networks, both convolution or transformer based, are essential for modern computer vision systems. However, they are vulnerable to small perturbations, almost imperceptible to humans, which significantly alter the model's prediction. These adversarial attacks are often considered to be a significant threat to the implementation of neural networks in safety-critical applications. Most attacks utilize the white-box threat model and therefore require full access to the target model, making them unrealistic to use in practice. We propose a novel approach under the more realistic black-box threat model that utilizes concepts from reinforcement learning to optimize perturbations with a non-differentiable target model. Reinforcement learning algorithms have already been optimized to be query efficient, making them an ideal starting point when designing black-box adversarial attacks. We show the success of our reinforcement learning inspired black-box adversarial attack (RIBA) in generating adversarial perturbations using only a small number of queries to the target model, by comparing it to state of the art attacks on different models on the Cifar10 and ImageNet data sets. RIBA takes

25.4\%

fewer median queries to generate attacked images against a ResNet-18 on Cifar10 and

22.5\%

fewer median queries to fool a Vit-B/16 model on ImageNet. Additionally, we demonstrate that RIBA can match the performance of white-box attacks on an adversarially trained model.

Florian Krone, Elena Hoemann, Sven Hallerbach

Institute for AI Safety and Security German Aerospace Center (DLR) Sankt Augustin, Germany

Adversarial Attacks Adversarial Robustness Adversarial Perturbations Black-Box Adversarial Attacks

Ask PDF

May 15, 2026cs.CR

DarkLLM: Learning Language-Driven Adversarial Attacks with Large Language Models

While vision and multimodal foundation models underpin critical tasks from perception to complex reasoning, they remain highly vulnerable to adversarial attacks. However, traditional adversarial attacks are typically limited to single, predefined objectives, tightly coupling each attack to a specific model or task, which restricts their scalability and flexibility in real-world scenarios. In this work, we present DarkLLM, a novel attack framework that trains an LLM to translate natural-language attack instructions into latent attack vectors, which are then decoded into visual adversarial perturbations. By leveraging natural-language instruction tuning, DarkLLM not only unifies targeted, untargeted, segmentation, and multi-model attacks within a single framework, but also achieves flexible and controllable adversarial generation, enabling each instruction to produce a perturbation that induces desired behaviors across heterogeneous models. Through extensive experiments across 4 tasks, 13 datasets, and 15 models, we demonstrate that DarkLLM with only 1B parameters can follow attacker instructions and generate highly effective attacks against CLIP, SAM, and frontier LLMs, revealing a systemic vulnerability in modern foundation models.

Ye Sun, Xin Wang, Jiaming Zhang +7

Fudan University · Nanyang Technological University · Tongji University

Adversarial Attacks on VLMs Adversarial Perturbations Multimodal Foundation Models

Ask PDF

Jul 6, 2026cs.CV

Statistical Adversaries: Natural Backdoor-like Features in Vision Datasets

Model-specific adversarial attacks have been extensively studied. We study a different failure mode: naturally occurring statistical signals in vision data that can behave like backdoor-like triggers without being maliciously inserted. We call these signals statistical adversaries. We analyse Imagenet to find patterns that are strongly linked to certain labels. We then use statistical controls to remove random correlations from our candidate signals. Finally, we demonstrate that these signals directly and predictably alter model predictions. These statistical adversaries are more targeted than generic corruptions and transfer across different model architectures. This suggests that some vulnerabilities are driven by dataset structure and distribution rather than a single model's idiosyncrasies. We conclude that ordinary datasets can contain exploitable adversarial surfaces even in the absence of poisoning, and suggest that dataset audits should treat spurious structure not only as a source of bias or interpretability failure, but also as a latent attack surface for vision models.

Paul K. Mandal, Pavan Reddy, Tristan Malatynski

1Neurint, LLC, Baton Rouge, LA, USA. · 2U.S. Army Cyber Corps, U.S. Army Reserve, USA. · 3Northwestern State University of Louisiana, Natchitoches, LA, USA. +2

Adversarial Examples Image Classification

Ask PDF