cs.CVMar 19, 2026

Complementary Text-Guided Attention for Zero-Shot Adversarial Robustness

Authors: Lu Yu, Haiyang Zhang, Changsheng Xu

Organizations: School of Computer Science and Engineering, Tianjin University of Technology, 300384, China · State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, 100190, China · School of Artificial Intelligence, University of the Chinese Academy of Sciences, Beijing, 100049, China

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Due to the impressive zero-shot capabilities, pre-trained vision-language models (e.g., CLIP), have attracted widespread attention and adoption across various domains. Nonetheless, CLIP has been observed to be susceptible to adversarial examples. Through experimental analysis, we have observed a phenomenon wherein adversarial perturbations induce shifts in text-guided attention. Building upon this observation, we propose a simple yet effective strategy: Text-Guided Attention for Zero-Shot Robustness (TGA-ZSR). This framework incorporates two components: Local Attention Refinement Module and Global Attention Constraint Module. Our goal is to maintain the generalization of the CLIP model and enhance its adversarial robustness. Additionally, the Global Attention Constraint Module acquires text-guided attention from both the target and original models using clean examples. Its objective is to maintain model performance on clean samples while enhancing overall robustness. However, we observe that the method occasionally focuses on irrelevant or spurious features, which can lead to suboptimal performance and undermine its robustness in certain scenarios. To overcome this limitation, we further propose a novel approach called Complementary Text-Guided Attention (Comp-TGA). This method integrates two types of foreground attention: attention guided by the class prompt and reversed attention driven by the non-class prompt. These complementary attention mechanisms allow the model to capture a more comprehensive and accurate representation of the foreground. The experiments validate that TGA-ZSR and Comp-TGA yield 9.58% and 11.95% improvements respectively, in zero-shot robust accuracy over the current state-of-the-art techniques across 16 datasets.

Figures & tables

Fig. 1: The four rows depict the original image, its associated attention map, the generated adversarial example, and the attention map of the adversarial example. Labels in black indicate the ground truth, while those in red represent misclassified labels for the adversarial examples.
Fig. 2: Examples of original text-guided attention for CLIP. These reveal that the attention mechanism occasionally focuses on irrelevant regions.
Fig. 3: The framework overview of Complementary Text-Guided Attention.
Fig. 4: Visualization of different attention mechanisms. The complementary text-guided attention fusion generates a significantly more accurate and effective attention representation, addressing the limitations of the original text-guided attention.
Fig. 5: An overview of our TGA-ZSR framework: We generate adversarial examples and feed them into the target image encoder. To enhance the adversarial robustness of the CLIP model and maintain its generalization, we introduce text-guided attention. This involves refining the framework for adversarial examples through the Local Attention Refinement Module and constraining the model to prevent significant drift via the Global Attention Constraint Module.
Class PromptsNon-class PromptsOther PromptsARobust\boldsymbol{A_{\textrm{Robust}}}AClean\boldsymbol{A_{\textrm{Clean}}}AOverall\boldsymbol{A_{\textrm{Overall}}}
✓  41.4556.4848.98
✓ ✓43.2455.1049.17
✓✓ 44.4655.4449.95
TABLE VIII: Comparison of different prompt-guided attention in Comp-TGA. We trained several different methods on Tiny-ImageNet using PGD with ε=1/255\varepsilon=1/255 , and evaluated them across 16 datasets under PGD with the same attack strength.
MethodsARobust\boldsymbol{A_{\textrm{Robust}}}AClean\boldsymbol{A_{\textrm{Clean}}}AOverall\boldsymbol{A_{\textrm{Overall}}}
CLIP4.9064.4234.66
+ LCEL_{\mathrm{CE}}29.4544.9737.21
Comp-TGA   
+ LLARML_{\textrm{LARM}}30.6946.6838.69
+ LGACML_{\textrm{GACM}}44.4655.4449.95
TABLE X: Ablation study of each component in Comp-TGA. We trained several different methods on Tiny-ImageNet using PGD with ε=1/255\varepsilon=1/255 , and evaluated them across 16 datasets under PGD with the same attack strength.
Fig. 6: The trade-off between robustness and clean accuracy. Each point on the graph represents a method, with the size of the point quantitatively reflecting the trade-off between robustness and clean accuracy. Larger points indicate methods that achieve a more favorable balance between robustness and clean accuracy.
MethodsMemoryTrain timeOverall
 usage(per batch)Accuracy
CLIP [ 1 ]0 Mb0.00s34.66%
FT-Clean6561 Mb0.21s30.71%
FT-Adv.10859 Mb0.60s36.10%
TeCoA [ 11 ]12873 Mb0.65s36.94%
FARE [ 53 ]9821 Mb0.25s39.05%
TABLE XI: Comparison of memory usage, training time, and overall accuracy.
Figure 10
Figure 11
Figure 12
Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

TABLE A7: Zero-shot robust accuracy under PGD [ 33 ] and clean accuracy across 16 datasets on TGA-ZSR. We performed several different hyperparameters on Tiny-ImageNet and evaluated across 16 datasets.
TABLE A8: Zero-shot robust accuracy under PGD [ 33 ] and clean accuracy across 16 datasets on Comp-TGA. We performed several different hyperparameters on Tiny-ImageNet and evaluated across 16 datasets.
Fig. A1: Zero-shot robust accuracy under PGD [ 33 ] and clean accuracy across 16 datasets for TGA-ZSR. Several different hyperparameter settings were explored on Tiny-ImageNet and evaluated across 16 datasets.
Fig. A2: Zero-shot robust accuracy under PGD [ 33 ] and clean accuracy across 16 datasets for Comp-TGA. Several different hyperparameter settings were explored on Tiny-ImageNet and evaluated across 16 datasets.
Methodsα\boldsymbol{\alpha}β\boldsymbol{\beta}ARobust\boldsymbol{A_{\textrm{Robust}}}AClean\boldsymbol{A_{\textrm{Clean}}}AOverall\boldsymbol{A_{\textrm{Overall}}}
TGA-ZSR0.070.0540.4356.8648.65
 0.080.0437.2855.9646.62
 0.080.0640.1656.8848.52
 0.090.0532.3154.6343.47
 0.080.0542.0956.4449.27
Comp-TGA0.100.0944.2255.2949.76
Appendix
TABLE A9: Ablation study on Hyper-parameters. We trained several different methods on Tiny-ImageNet using PGD with ε=1/255\varepsilon=1/255 , and evaluated them across 16 datasets under PGD with the same attack strength.
Fig. A3: The trade-off between train time and overall accuracy. Each point represents a method, with ⚫ indicating the baseline models and ★ marking our methods to provide a clear visual distinction.
More in this topic

Attention-Based Adversarial Attacks

See all 12 papers

Explore similar work

CardsList
Mar 31, 2026

AGFT: Alignment-Guided Fine-Tuning for Zero-Shot Adversarial Robustness of Vision-Language Models

Pre-trained vision-language models (VLMs) exhibit strong zero-shot generalization but remain vulnerable to adversarial perturbations. Existing classification-guided adversarial fine-tuning methods often disrupt pre-trained cross-modal alignment, weakening visual-textual correspondence and degrading zero-shot performance. In this paper, we propose an Alignment-Guided Fine-Tuning (AGFT) framework that enhances zero-shot adversarial robustness while preserving the cross-modal semantic structure. Unlike label-based methods that rely on hard labels and fail to maintain the relative relationships between image and text, AGFT leverages the probabilistic predictions of the original model for text-guided adversarial training, which aligns adversarial visual features with textual embeddings via soft alignment distributions, improving zero-shot adversarial robustness. To address structural discrepancies introduced by fine-tuning, we introduce a distribution consistency calibration mechanism that adjusts the robust model output to match a temperature-scaled version of the pre-trained model predictions. Extensive experiments across multiple zero-shot benchmarks demonstrate that AGFT outperforms state-of-the-art methods while significantly improving zero-shot adversarial robustness.

Yubo Cui, Xianchao Guan, Zijun Xiong +1

Harbin Institute of Technology, Shenzhen, China · Shenzhen Loop Area Institute, Shenzhen, China

VLM RobustnessVision-Language Alignment
AskPDF
Apr 13, 2026

Finetune Like You Pretrain: Boosting Zero-shot Adversarial Robustness in Vision-language Models

Despite their impressive zero-shot abilities, vision-language models such as CLIP have been shown to be susceptible to adversarial attacks. To enhance its adversarial robustness, recent studies finetune the pretrained vision encoder of CLIP with adversarial examples on a proxy dataset such as ImageNet by aligning adversarial images with correct class labels. However, these methods overlook the important roles of training data distributions and learning objectives, resulting in reduced zero-shot capabilities and limited transferability of robustness across domains and datasets. In this work, we propose a simple yet effective paradigm AdvFLYP, which follows the training recipe of CLIP's pretraining process when performing adversarial finetuning to the model. Specifically, AdvFLYP finetunes CLIP with adversarial images created based on image-text pairs collected from the web, and match them with their corresponding texts via a contrastive loss. To alleviate distortion of adversarial image embeddings of noisy web images, we further propose to regularise AdvFLYP by penalising deviation of adversarial image features. We show that logit- and feature-level regularisation terms benefit robustness and clean accuracy, respectively. Extensive experiments on 14 downstream datasets spanning various domains show the superiority of our paradigm over mainstream practices. Our code and model weights are released at https://github.com/Sxing2/AdvFLYP.

Songlong Xing, Weijie Wang, Zhengyu Zhao +3

University of Trento, Italy · Fondazione Bruno Kessler, Italy · Xi’an Jiaotong University, China +1

VLM Robustness
AskPDF
May 17, 2026

TAME: Test-Time Adversarial Prompt Tuning via Mixture-of-Experts for Vision-Language Models

Large-scale pre-trained Vision-Language models (VLMs), such as CLIP, exhibit strong zero-shot generalization, yet remain highly vulnerable to imperceptible adversarial perturbations, raising serious safety concerns for open-world deployment. To enhance robustness without requiring downstream task-specific retraining, we propose TAME, a novel test-time defense. Building upon our prior Test-Time Adversarial Prompt Tuning (TAPT), TAME introduces an architectural reformulation by replacing TAPT's single adaptive prompt with an input-conditioned Mixture-of-Experts (MoE) framework, enabling more expressive and adaptive defense. Specifically, TAME maintains a bank of learnable expert prompts and employs an input-dependent routing mechanism to aggregate a customized prompt mixture for each unlabeled test sample at inference time. This test-time defense mechanism is driven by three unsupervised objectives: (1) multi-view prediction entropy minimization, (2) layer-wise alignment of visual token statistics to precomputed clean and adversarial reference distributions, and (3) MoE regularization for balanced expert utilization and prompt diversity. We evaluated TAME on 11 benchmark datasets, including ImageNet and 10 additional zero-shot datasets. The results show that TAME improves the zero-shot adversarial robustness of the original CLIP by at least 49.1% under AutoAttack while largely preserving generalization on clean samples. TAME also consistently outperforms existing adversarial prompt tuning methods across multiple prompt designs, yielding an average robustness gain of at least 30.2%.

Xin Wang, Yixu Wang, Jiaming Zhang +6

Adversarial Attacks on VLMsTest-Time AdaptationVLM AdaptationTest-Time Prompt Tuning
AskPDF