PDA: Text-Augmented Defense Framework for Robust Vision-Language Models against Adversarial Image Attacks

Jingning Xu; Haochen Luo; Chen Liu

cs.CVApr 1, 2026

PDA: Text-Augmented Defense Framework for Robust Vision-Language Models against Adversarial Image Attacks

Authors: Jingning Xu, Haochen Luo, Chen Liu

Organizations: Department of Computer Science City University of Hong Kong

Abstract

Vision-language models (VLMs) are vulnerable to adversarial image perturbations. Existing works based on adversarial training against task-specific adversarial examples are computationally expensive and often fail to generalize to unseen attack types. To address these limitations, we introduce Paraphrase-Decomposition-Aggregation (PDA), a training-free defense framework that leverages text augmentation to enhance VLM robustness under diverse adversarial image attacks. PDA performs prompt paraphrasing, question decomposition, and consistency aggregation entirely at test time, thus requiring no modification on the underlying models. To balance robustness and efficiency, we instantiate PDA as invariants that reduce the inference cost while retaining most of its robustness gains. Experiments on multiple VLM architectures and benchmarks for visual question answering, classification, and captioning show that PDA achieves consistent robustness gains against various adversarial perturbations while maintaining competitive clean accuracy, establishing a generic, strong and practical defense framework for VLMs during inference.

Figures & tables

Figure 1: Overview of the threat model and PDA. (a) Attack phase: an adversary adds pixel-level perturbations guided by VLM gradients, causing the model to flip its answer under the original question. (b) Defense phase: our training-free, text-side Paraphrase–Decomposition–Aggregation pipeline, requires only black-box access, and suppresses adversarially biased views by cross-checking stable visual evidence.

Figure 2: Pipeline of the proposed PDA defense and its variants. Given an image $x$ and a query $t$ , the Paraphrase Agent produces semantically equivalent prompts. Each prompt is factorized by the Question Decomposer into atomic questions that are individually answered by the base VLM. The Answer Aggregation Agent performs confidence- and consistency-aware fusion to yield the final prediction. The bottom panels illustrate four PDA variants: (a) PDA-Full , (b) PDA-RJV , (c) PDA-RDA and PDA-PV .

Variant	Paraphrasing	Decomposition		Aggregation
	LLM	VLM	LLM	LLM
PDA-Full	1	$N\!\times\!K$	$N\!\times\!K+N$	1
PDA-RJV	1	$N$	$N$	$0$
PDA-RDA	1	$N$	0	1
PDA-PV	1	$N$	0	0

Table 1: Computation budget by stage (LLM: text-only; VLM: vision–language). Here

M

is the number of retained paraphrases and

K

is the number of atomic questions per paraphrase (generated within a single VLM call per paraphrase).

Defense Method	LLaVA-1.5-7B						LLaVA-1.5-13B
	VQA-v2		ImageNet-D		COCO		VQA-v2		ImageNet-D		COCO
	Clean	ADV	Clean	ADV	Clean	ADV	Clean	ADV	Clean	ADV	Clean	ADV
No Defense	0.862	0.445	0.625	0.248	0.858	0.753	0.886	0.593	0.646	0.320	0.865	0.757
SmoothLLM-Swap	0.866	0.453	0.610	0.260	0.849	0.744	0.881	0.596	0.630	0.327	0.849	0.756
SmoothLLM-Insert	0.870	0.449	0.596	0.254	0.854	0.746	0.884	0.587	0.605	0.330	0.855	0.743
SmoothLLM-Patch	0.864	0.454	0.632	0.261	0.843	0.741	0.877	0.599	0.611	0.314	0.851	0.751

Table 2: Overall comparison on three tasks. Results for VQA-v2 ( VQA Accuracy ), ImageNet-D ( Top-1 Accuracy ), and MS COCO ( CLIPScore ). Rows list defenses applied to each victim model (LLaVA-1.5-7B/13B), including SmoothLLM (Swap/Insert/Patch), SemanticSmooth (Paraphrase/Summarise), TeCoA, FARE, and our PDA . Clean denotes performance on unperturbed inputs; ADV denotes performance on adversarially perturbed inputs under the same evaluation protocol.

Figure 3: Qualitative effect of PDA. (a) The base VLM is attacked into choosing “jeans” in a two-way “t shirt vs. jeans” question, whereas PDA paraphrases the query into targeted checks and recovers the correct “t shirt” label. (b) For a cluttered scene with a black-and-white cat in a green bowl, PDA decomposes the caption prompt into factual sub-questions and aggregates the answers into a faithful caption.

Setting	No Defense		PDA
Setting	ImageNet-D	VQA-v2	ImageNet-D	VQA-v2
(a) Larger $\epsilon$ under PGD
$2/255$	0.25	0.45	0.57	0.75
$4/255$	0.22	0.42	0.56	0.74
$6/255$	0.19	0.40	0.53	0.74
$8/255$	0.17	0.39	0.54	0.72

Table 3: Robustness under more challenging white-box attacks on ImageNet-D and VQA-v2.

Attack	ImageNet-D	VQA-v2
Transfer-based [ 9 ]	0.58 / 0.62	0.77 / 0.81
AttackVLM [ 57 ]	0.56 / 0.61	0.75 / 0.79
AnyAttack [ 56 ]	0.61 / 0.64	0.79 / 0.81

Table 4: Black-box attacks on ImageNet-D and VQA-v2. Each entry is No Defense / PDA robust accuracy.

	GPT-5	Claude	Gemini
No Defense	0.67 / 0.81	0.61 / 0.82	0.67 / 0.76
PDA	0.76 / 0.90	0.70 / 0.88	0.72 / 0.86

Table 5: Closed-source LVLMs under transfer attacks. Each entry is ImageNet - D / VQA-v2 robust accuracy.

Method	ImageNet-D	VQA-v2
No Defense	0.25	0.45
PDA	0.57 (+0.32)	0.75 (+0.30)
(a) Image-side defenses
JPEG	0.51 (+0.26)	0.68 (+0.23)
JPEG + PDA	0.61 (+0.36)	0.77 (+0.32)
RA	0.55 (+0.30)	0.70 (+0.25)

Table 6: Comparisons with additional test-time baselines.

Method	DeepSeek-VL	InternVL3	LLaVA-1.5-7B	Ovis2
No Defense	59.0 / 46.4	69.6 / 25.8	47.2 / 26.8	88.0 / 39.6
PDA–RJV	63.8 / 53.4	76.0 / 39.4	65.2 / 49.0	87.4 / 53.2
PDA–RDA	63.4 / 51.8	72.4 / 34.0	61.2 / 45.8	89.8 / 48.4
PDA–PV	61.6 / 70.2	72.2 / 31.4	58.4 / 57.0	89.2 / 56.4

Table 7: Ablation of budgeted PDA variants across backbone architectures. We report robustness (%) of each victim VLM under the same adversarial threat model, with values shown as “VQA-v2 / ImageNet-D.”

Variant	Backbone	$K=3$	$K=5$	$K=7$	$K=9$
PDA–RJV	LLaVA-1.5-7B	49.2	53.2	53.0	52.2
	InternVL3	40.8	39.2	29.2	39.0
	DeepSeek-VL	53.4	53.4	53.2	52.4
	Ovis-2	51.4	53.2	52.2	52.4
PDA–RDA	LLaVA-1.5-7B	47.0	43.7	43.9	43.5
	InternVL3	34.2	29.3	30.3	29.3

Table 8: Effect of the number of paraphrases

K

on ADV accuracy for PDA–RJV and PDA–RDA on ImageNet-D.

Agent	DeepSeek-VL	InternVL3	LLaVA-1.5-7B	Ovis2
DeepSeek	63.4 / 54.4	70.8 / 35.7	60.8 / 49.2	90.0 / 49.2
GPT	63.4 / 51.8	72.4 / 29.3	61.4 / 43.7	89.8 / 48.4
LLaMA	58.0 / 54.6	58.4 / 36.1	58.0 / 50.4	61.2 / 41.2
Qwen	63.2 / 50.6	70.2 / 34.8	61.4 / 47.1	89.2 / 47.2

Table 9: Effect of LLM agent within the PDA pipeline using

K{=}5

. Each cell reports VQA-v2 / ImageNet - D robust accuracy (%) for the same victim VLM under different LLM agents.

Figure 4: Overall performance of PDA variants on VQA-v2 and ImageNet-D under different paraphrase counts (

K{=}3

and

K{=}5

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Additional VQA-v2 qualitative examples. For each adversarial image we compare the answer of the undefended VLM with the output of PDA and visualize key paraphrases and sub-questions that support the corrected decision.

Figure 6: Additional ImageNet-D qualitative examples. PDA stabilizes fine-grained recognition by aggregating evidence from multiple paraphrases that describe shape, part configuration, and material, instead of relying on a single corrupted view.

Figure 7: Additional COCO captioning examples. PDA detects and repairs conflicts between a corrupted short caption and VLM answers to verification sub-questions, yielding minimally edited but visually consistent captions.

VLM Robustness

Vision-language model robustness studies whether models remain reliable under visual, textual, and distribution shifts, adversarial inputs, and safety challenges.

See all 443 papers

VALD: Multi-Stage Vision Attack Detection for Efficient LVLM Defense
Feb 23, 2026Nadav Kadvil, Malak Fares, Ayellet TalVLM Robustness Large Vision-Language Models
Technion – Israel Institute of Technology
Answer It or Not? Towards Prompt-Adapted Adversarial Images on Vision Language Models
Oct 12, 2026Shunyuan Bai, Yuchen Ren, Le Yang +3Adversarial Attacks on VLMs Privacy Protection in VLMs
Xi’an Jiaotong University, Xi’an, 710049, China
What Makes VLMs Robust? Towards Reconciling Robustness and Accuracy in Vision-Language Models
Mar 13, 2026Sen Nie, Jie Zhang, Zhongqi Wang +3VLM Robustness Adversarial Robustness
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190, China · University of Chinese Academy of Sciences, Beijing 100049, China

Feb 23, 2026J/K move · Enter open · S save

VALD: Multi-Stage Vision Attack Detection for Efficient LVLM Defense

Nadav Kadvil, Malak Fares, Ayellet Tal

Technion – Israel Institute of Technology

VLM Robustness Large Vision-Language Models Adversarial Example Detection Adversarial Attacks on VLMs Adversarial Attacks on Unified Multimodal Models

Open paper AskPDF arXiv

Large Vision-Language Models (LVLMs) can be vulnerable to adversarial images that subtly bias their outputs toward plausible yet incorrect responses. We introduce a general, efficient, and training-free defense that combines image transformations with agentic data consolidation to recover correct model behavior. A key component of our approach is a two-stage detection mechanism that quickly filters out the majority of clean inputs. We first assess image consistency under content-preserving transformations at negligible computational cost. For more challenging cases, we examine discrepancies in a text-embedding space. Only when necessary do we invoke a powerful LLM to resolve attack-induced divergences. A key idea is to consolidate multiple responses, leveraging both their similarities and their differences. We show that our method achieves state-of-the-art accuracy while maintaining notable efficiency: most clean images skip costly processing, and even in the presence of numerous adversarial examples, the overhead remains minimal.

Abstract

VLM Robustness

Explore similar work

VALD: Multi-Stage Vision Attack Detection for Efficient LVLM Defense

Answer It or Not? Towards Prompt-Adapted Adversarial Images on Vision Language Models

What Makes VLMs Robust? Towards Reconciling Robustness and Accuracy in Vision-Language Models