Complementary Text-Guided Attention for Zero-Shot Adversarial Robustness

Lu Yu; Haiyang Zhang; Changsheng Xu

cs.CVMar 19, 2026

Complementary Text-Guided Attention for Zero-Shot Adversarial Robustness

Authors: Lu Yu, Haiyang Zhang, Changsheng Xu

Organizations: School of Computer Science and Engineering, Tianjin University of Technology, 300384, China · State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, 100190, China · School of Artificial Intelligence, University of the Chinese Academy of Sciences, Beijing, 100049, China

Abstract

Due to the impressive zero-shot capabilities, pre-trained vision-language models (e.g., CLIP), have attracted widespread attention and adoption across various domains. Nonetheless, CLIP has been observed to be susceptible to adversarial examples. Through experimental analysis, we have observed a phenomenon wherein adversarial perturbations induce shifts in text-guided attention. Building upon this observation, we propose a simple yet effective strategy: Text-Guided Attention for Zero-Shot Robustness (TGA-ZSR). This framework incorporates two components: Local Attention Refinement Module and Global Attention Constraint Module. Our goal is to maintain the generalization of the CLIP model and enhance its adversarial robustness. Additionally, the Global Attention Constraint Module acquires text-guided attention from both the target and original models using clean examples. Its objective is to maintain model performance on clean samples while enhancing overall robustness. However, we observe that the method occasionally focuses on irrelevant or spurious features, which can lead to suboptimal performance and undermine its robustness in certain scenarios. To overcome this limitation, we further propose a novel approach called Complementary Text-Guided Attention (Comp-TGA). This method integrates two types of foreground attention: attention guided by the class prompt and reversed attention driven by the non-class prompt. These complementary attention mechanisms allow the model to capture a more comprehensive and accurate representation of the foreground. The experiments validate that TGA-ZSR and Comp-TGA yield 9.58% and 11.95% improvements respectively, in zero-shot robust accuracy over the current state-of-the-art techniques across 16 datasets.

Figures & tables

Fig. 1: The four rows depict the original image, its associated attention map, the generated adversarial example, and the attention map of the adversarial example. Labels in black indicate the ground truth, while those in red represent misclassified labels for the adversarial examples.

Fig. 2: Examples of original text-guided attention for CLIP. These reveal that the attention mechanism occasionally focuses on irrelevant regions.

Fig. 3: The framework overview of Complementary Text-Guided Attention.

Fig. 4: Visualization of different attention mechanisms. The complementary text-guided attention fusion generates a significantly more accurate and effective attention representation, addressing the limitations of the original text-guided attention.

Fig. 5: An overview of our TGA-ZSR framework: We generate adversarial examples and feed them into the target image encoder. To enhance the adversarial robustness of the CLIP model and maintain its generalization, we introduce text-guided attention. This involves refining the framework for adversarial examples through the Local Attention Refinement Module and constraining the model to prevent significant drift via the Global Attention Constraint Module.

Class Prompts	Non-class Prompts	Other Prompts	$\boldsymbol{A_{\textrm{Robust}}}$	$\boldsymbol{A_{\textrm{Clean}}}$	$\boldsymbol{A_{\textrm{Overall}}}$
✓			41.45	56.48	48.98
✓		✓	43.24	55.10	49.17
✓	✓		44.46	55.44	49.95

TABLE VIII: Comparison of different prompt-guided attention in Comp-TGA. We trained several different methods on Tiny-ImageNet using PGD with

\varepsilon=1/255

, and evaluated them across 16 datasets under PGD with the same attack strength.

Methods	$\boldsymbol{A_{\textrm{Robust}}}$	$\boldsymbol{A_{\textrm{Clean}}}$	$\boldsymbol{A_{\textrm{Overall}}}$
CLIP	4.90	64.42	34.66
+ $L_{\mathrm{CE}}$	29.45	44.97	37.21
Comp-TGA
+ $L_{\textrm{LARM}}$	30.69	46.68	38.69
+ $L_{\textrm{GACM}}$	44.46	55.44	49.95

TABLE X: Ablation study of each component in Comp-TGA. We trained several different methods on Tiny-ImageNet using PGD with

\varepsilon=1/255

, and evaluated them across 16 datasets under PGD with the same attack strength.

Fig. 6: The trade-off between robustness and clean accuracy. Each point on the graph represents a method, with the size of the point quantitatively reflecting the trade-off between robustness and clean accuracy. Larger points indicate methods that achieve a more favorable balance between robustness and clean accuracy.

Methods	Memory	Train time	Overall
	usage	(per batch)	Accuracy
CLIP [ 1 ]	0 Mb	0.00s	34.66%
FT-Clean	6561 Mb	0.21s	30.71%
FT-Adv.	10859 Mb	0.60s	36.10%
TeCoA [ 11 ]	12873 Mb	0.65s	36.94%
FARE [ 53 ]	9821 Mb	0.25s	39.05%

TABLE XI: Comparison of memory usage, training time, and overall accuracy.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Table 13 — TABLE A7: Zero-shot robust accuracy under PGD [ 33 ] and clean accuracy across 16 datasets on TGA-ZSR. We performed several different hyperparameters on Tiny-ImageNet and evaluated across 16 datasets.

Panel of Table 14 — TABLE A8: Zero-shot robust accuracy under PGD [ 33 ] and clean accuracy across 16 datasets on Comp-TGA. We performed several different hyperparameters on Tiny-ImageNet and evaluated across 16 datasets.

Panel of Figure 15 — Fig. A1: Zero-shot robust accuracy under PGD [ 33 ] and clean accuracy across 16 datasets for TGA-ZSR. Several different hyperparameter settings were explored on Tiny-ImageNet and evaluated across 16 datasets.

Panel of Figure 16 — Fig. A2: Zero-shot robust accuracy under PGD [ 33 ] and clean accuracy across 16 datasets for Comp-TGA. Several different hyperparameter settings were explored on Tiny-ImageNet and evaluated across 16 datasets.

Methods	$\boldsymbol{\alpha}$	$\boldsymbol{\beta}$	$\boldsymbol{A_{\textrm{Robust}}}$	$\boldsymbol{A_{\textrm{Clean}}}$	$\boldsymbol{A_{\textrm{Overall}}}$
TGA-ZSR	0.07	0.05	40.43	56.86	48.65
	0.08	0.04	37.28	55.96	46.62
	0.08	0.06	40.16	56.88	48.52
	0.09	0.05	32.31	54.63	43.47
	0.08	0.05	42.09	56.44	49.27
Comp-TGA	0.10	0.09	44.22	55.29	49.76

Appendix

TABLE A9: Ablation study on Hyper-parameters. We trained several different methods on Tiny-ImageNet using PGD with

\varepsilon=1/255

, and evaluated them across 16 datasets under PGD with the same attack strength.

Fig. A3: The trade-off between train time and overall accuracy. Each point represents a method, with ⚫ indicating the baseline models and ★ marking our methods to provide a clear visual distinction.

Attention-Based Adversarial Attacks

See all 12 papers

AGFT: Alignment-Guided Fine-Tuning for Zero-Shot Adversarial Robustness of Vision-Language Models
Mar 31, 2026Yubo Cui, Xianchao Guan, Zijun Xiong +1VLM Robustness Vision-Language Alignment
Harbin Institute of Technology, Shenzhen, China · Shenzhen Loop Area Institute, Shenzhen, China
Finetune Like You Pretrain: Boosting Zero-shot Adversarial Robustness in Vision-language Models
Apr 13, 2026Songlong Xing, Weijie Wang, Zhengyu Zhao +3VLM Robustness
University of Trento, Italy · Fondazione Bruno Kessler, Italy · Xi’an Jiaotong University, China +1
TAME: Test-Time Adversarial Prompt Tuning via Mixture-of-Experts for Vision-Language Models
May 17, 2026Xin Wang, Yixu Wang, Jiaming Zhang +6Adversarial Attacks on VLMs Test-Time Adaptation

Mar 31, 2026J/K move · Enter open · S save

AGFT: Alignment-Guided Fine-Tuning for Zero-Shot Adversarial Robustness of Vision-Language Models

Yubo Cui, Xianchao Guan, Zijun Xiong, Zheng Zhang

Harbin Institute of Technology, Shenzhen, China · Shenzhen Loop Area Institute, Shenzhen, China

VLM Robustness Vision-Language Alignment

Open paper AskPDF arXiv

Pre-trained vision-language models (VLMs) exhibit strong zero-shot generalization but remain vulnerable to adversarial perturbations. Existing classification-guided adversarial fine-tuning methods often disrupt pre-trained cross-modal alignment, weakening visual-textual correspondence and degrading zero-shot performance. In this paper, we propose an Alignment-Guided Fine-Tuning (AGFT) framework that enhances zero-shot adversarial robustness while preserving the cross-modal semantic structure. Unlike label-based methods that rely on hard labels and fail to maintain the relative relationships between image and text, AGFT leverages the probabilistic predictions of the original model for text-guided adversarial training, which aligns adversarial visual features with textual embeddings via soft alignment distributions, improving zero-shot adversarial robustness. To address structural discrepancies introduced by fine-tuning, we introduce a distribution consistency calibration mechanism that adjusts the robust model output to match a temperature-scaled version of the pre-trained model predictions. Extensive experiments across multiple zero-shot benchmarks demonstrate that AGFT outperforms state-of-the-art methods while significantly improving zero-shot adversarial robustness.

Abstract

Attention-Based Adversarial Attacks

Explore similar work

AGFT: Alignment-Guided Fine-Tuning for Zero-Shot Adversarial Robustness of Vision-Language Models

Finetune Like You Pretrain: Boosting Zero-shot Adversarial Robustness in Vision-language Models

TAME: Test-Time Adversarial Prompt Tuning via Mixture-of-Experts for Vision-Language Models