VALD: Multi-Stage Vision Attack Detection for Efficient LVLM Defense

Nadav Kadvil; Malak Fares; Ayellet Tal

cs.CVFeb 23, 2026

VALD: Multi-Stage Vision Attack Detection for Efficient LVLM Defense

Authors: Nadav Kadvil, Malak Fares, Ayellet Tal

Organizations: Technion – Israel Institute of Technology

Abstract

Large Vision-Language Models (LVLMs) can be vulnerable to adversarial images that subtly bias their outputs toward plausible yet incorrect responses. We introduce a general, efficient, and training-free defense that combines image transformations with agentic data consolidation to recover correct model behavior. A key component of our approach is a two-stage detection mechanism that quickly filters out the majority of clean inputs. We first assess image consistency under content-preserving transformations at negligible computational cost. For more challenging cases, we examine discrepancies in a text-embedding space. Only when necessary do we invoke a powerful LLM to resolve attack-induced divergences. A key idea is to consolidate multiple responses, leveraging both their similarities and their differences. We show that our method achieves state-of-the-art accuracy while maintaining notable efficiency: most clean images skip costly processing, and even in the presence of numerous adversarial examples, the overhead remains minimal.

Figures & tables

Figure 1: LVLM defense. The output of the unprotected LVLM is incorrect (shown in red ). In VALD, we generate multiple captions for transformed images ( $r_{1}$ – $r_{k}$ ), each containing both correct content (black) and incorrect content ( orange ). VALD aggregates information across these captions to produce a final refined caption ( green ).

Figure 2: Architecture. VALD consists of four stages. First, a set of content-preserving transformations is applied to the input for early detection; clean images bypass the process and return the original LVLM response ( $r_{0}$ ). If a potential attack is detected, responses are generated for the transformed images. Next, a late detection stage analyzes these responses to identify additional clean cases. Finally, when an attack is deemed highly likely, an LLM consolidates the responses to produce the final output. Green arrows denote inputs detected as clean, while red arrows indicate those still suspected to be adversarial.

Model	Attack	Off	SVLM	SVLM	VALD
				Ours	(ours)
LLaVA	Clean	0.722	0.713	0.722	0.723
	MF-ii	0.268	0.679	0.703	0.716
	MixAttk	0.440	0.653	0.685	0.701
	M-Attack	0.360	0.667	0.654	0.673
MiniGPT	Clean	0.703	0.680	0.690	0.705

Table 1: Captioning results on MS-COCO. VALD outperforms the baselines under all three attacks on three models according to the SentenceBERT score.

Model	Attack	Acc	Precision	Recall
LLaVA	Clean	0.960	0	N/A
	MF-ii	0.998	1	0.998
	MixAttack	0.991	1	0.991
	M-Attack	0.995	1	0.995
MiniGPT	Clean	0.954	0	N/A
	MF-ii	0.999	1	0.999

Table 2: Detection results. The vast majority of clean images are correctly identified, with few adversarial images misclassified as clean images.

Panel of Figure 3 — Figure 3: Qualitative results. Although the attacked images (b) appear nearly identical to the originals (a), the LVLM incorrectly captions them (c). In contrast, our model successfully defends against the attack and generates accurate captions (c). The images were attacked via Zhao et al. (2023) . See Appendix C for additional results.

Attack	Off	FARE	VALD
Clean	0.722	0.672	0.723
MF-ii	0.268	0.570	0.716
MixAttack	0.440	0.586	0.701
M-Attack	0.360	0.517	0.673

Table 3: Comparison with training-based defense. VALD significantly outperforms FARE while maintaining clean performance.

Method	Case	Runtime (s)
VALD	Best	1.9
VALD	Worst	22.9
SmoothVLM	Always	52.7

Table 4: Runtime (sec/image). When clean images are detected by the early detection, VALD is 27× faster than SmoothVLM; when the entire defense is applied, it remains 2× faster. Notably,

\sim

95\%

of clean images are filtered at the early stage.

Attack	Crop	Masking	JPEG	Blur	Ours
Clean	0.722	0.726	0.723	0.723	0.723
MF-ii	0.716	0.694	0.722	0.718	0.716
MixAttack	0.697	0.672	0.707	0.716	0.701
M-Attack	0.359	0.687	0.701	0.707	0.673

Table 5: Transformations Ablations. Our transformation mix improves robustness to (adaptive) attacks..

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Consolidation process explanation. The following elements are mentioned by most captions and form the basis of the final consolidated description: a snow-covered park or forest, a wooden bench, trees, and the bench being covered in snow. Conversely, the following details either conflict with the majority or appear only sporadically; therefore, they are treated as inconsistent and omitted from the final caption: a coconut beauty product, a bird on the bench, a green object on the tree, and a black-and-white filter.

Panel of Figure 5 — Figure 5: Qualitative results. Although the attacked images (b) appear nearly identical to the originals (a), the LVLM incorrectly captions them (c). In contrast, our model successfully defends against the attack and generates accurate captions (c). The images were attacked via Zhao et al. (2023) .

Figure 6: Example of a failure case in the consolidation process. Most transformed views hallucinate a third motorcycle, causing the consolidated caption to mistake.

Model	Attack	Off	VALD	Always On
LLaVA	Clean	0.722	0.723	0.733
	MF-ii	0.268	0.716	0.716
	MixAttk	0.440	0.701	0.701
	M-Attack	0.360	0.673	0.673
MiniGPT	Clean	0.703	0.705	0.726
	MF-ii	0.217	0.706	0.706

Appendix

Table 6: Defense performance with and without detection. Enabling detection preserves VALD defense performance, yielding results nearly identical to always applying the defense.

$\epsilon$	Attack	Off	FARE	VALD
	Clean	0.722	0.672	0.723
8	MF-ii	0.282	0.617	0.726
	MixAttack	0.603	0.644	0.720
	M-Attack	0.429	0.671	0.702
16	MF-ii	0.268	0.570	0.716
	MixAttack	0.440	0.586	0.701

Appendix

Table 7: Robustness under different perturbation budget (

\epsilon

). VALD consistently outperforms the adversarially trained baseline FARE, with a larger performance gap at higher perturbation levels.

Stage	Runtime (Average)
Caption generation (Defense off)	1.8
Caption generation with ED (Best Case)	1.9
LVLM querying	18.0
Late Detector	0.2
Response generation	2.8
VALD total runtime (Worst Case)	22.9

Appendix

Table 8: Runtime breakdown (sec / image). Images filtered by the Early Detector (ED) correspond to the best case runtime, while the worst case occurs when images pass through the full pipeline.

Attack	Clean %	Acc	Rec	Prec	Time
MF-ii	50	0.979	0.998	0.962	12.8
MF-ii	95	0.962	1	0.568	3.8
MixAttk	50	0.973	0.992	0.956	12.9
MixAttk	95	0.961	1	0.562	3.8
M-Attack	50	0.979	0.994	0.965	12.8
M-Attack	95	0.962	1	0.568	3.8

Appendix

Table 9: Runtime as a function of clean images percentage. Average runtime decreases as the proportion of clean images increases.

Model	Attack	Off	SVLM	SVLM	VALD	Acc	Prec	Rec
				Ours	(ours)
LLaVA	Clean	0.713	0.690	0.704	0.722	0.55	0	N/A
	MF-ii	0.213	0.642	0.694	0.710	1	1	1
	MixAttk	0.382	0.608	0.643	0.664	1	1	1
	M-Attack	0.347	0.651	0.652	0.663	0.98	1	0.98

Appendix

Table 10: Captioning and detection results on the Flickr8k Expert dataset. VALD surpasses the SVLM baseline in captioning performance while correctly detecting nearly all adversarial images.

	Crop			Pixel masking			JPEG			Blur
Attack	95%	90%	85%	10%	15%	20%	30	40	50	5	7	9
Clean	0.722	0.722	0.722	0.726	0.725	0.724	0.723	0.722	0.722	0.722	0.722	0.723
MF-ii	0.716	0.716	0.713	0.694	0.693	0.689	0.722	0.718	0.712	0.700	0.718	0.718
MixAttack	0.697	0.693	0.688	0.672	0.677	0.676	0.707	0.696	0.675	0.677	0.713	0.716
M-Attack	0.359	0.359	0.360	0.687	0.686	0.680	0.701	0.679	0.653	0.600	0.697	0.707

Appendix

Table 12: Transformation parameter ablation. For cropping and pixel masking, performance is best when most of the original image is preserved, whereas for JPEG compression and blur, stronger transformations provide better robustness.

Model	$\epsilon$	Off	SVLM	SVLM	VALD
				Ours	(ours)
Clean	–	0.722	0.713	0.722	0.723
MF-ii	$8$	0.282	0.684	0.712	0.726
	$16$	0.268	0.679	0.703	0.716
	$24$	0.262	0.665	0.695	0.704
MixAttack	$8$	0.603	0.683	0.704	0.720

Appendix

Table 11: Impact of perturbation. Increasing noise degrades performance without defense (Off), while our defense improves robustness and increasingly outperforms SmoothVLM (SVLM).

VLM Robustness

Vision-language model robustness studies whether models remain reliable under visual, textual, and distribution shifts, adversarial inputs, and safety challenges.

See all 443 papers

PDA: Text-Augmented Defense Framework for Robust Vision-Language Models against Adversarial Image Attacks
Apr 1, 2026Jingning Xu, Haochen Luo, Chen LiuVLM Robustness Vision-Language Models
Department of Computer Science City University of Hong Kong
Structure-Guided Visual Perturbation Neutralization for LVLMs
May 27, 2026Yuanhe Zhang, Xueting Wang, YanBin Ren +6VLM Robustness Adversarial Attacks on VLMs
1Beijing University of Posts and Telecommunications · University of Science and Technology of China · 3JIUTIAN Research +2
PA-Attack: Guiding Gray-Box Attacks on LVLM Vision Encoders with Prototypes and Attention
Feb 23, 2026Hefei Mei, Zirui Wang, Chang Xu +2Attention-Based Adversarial Attacks Adversarial Attacks on VLMs
City University of Hong Kong · The University of Sydney

Apr 1, 2026J/K move · Enter open · S save

PDA: Text-Augmented Defense Framework for Robust Vision-Language Models against Adversarial Image Attacks

Jingning Xu, Haochen Luo, Chen Liu

Department of Computer Science City University of Hong Kong

VLM Robustness Vision-Language Models Adversarial Attacks on VLMs

Open paper AskPDF arXiv

Vision-language models (VLMs) are vulnerable to adversarial image perturbations. Existing works based on adversarial training against task-specific adversarial examples are computationally expensive and often fail to generalize to unseen attack types. To address these limitations, we introduce Paraphrase-Decomposition-Aggregation (PDA), a training-free defense framework that leverages text augmentation to enhance VLM robustness under diverse adversarial image attacks. PDA performs prompt paraphrasing, question decomposition, and consistency aggregation entirely at test time, thus requiring no modification on the underlying models. To balance robustness and efficiency, we instantiate PDA as invariants that reduce the inference cost while retaining most of its robustness gains. Experiments on multiple VLM architectures and benchmarks for visual question answering, classification, and captioning show that PDA achieves consistent robustness gains against various adversarial perturbations while maintaining competitive clean accuracy, establishing a generic, strong and practical defense framework for VLMs during inference.

Abstract

VLM Robustness

Explore similar work

PDA: Text-Augmented Defense Framework for Robust Vision-Language Models against Adversarial Image Attacks

Structure-Guided Visual Perturbation Neutralization for LVLMs

PA-Attack: Guiding Gray-Box Attacks on LVLM Vision Encoders with Prototypes and Attention