Finetune Like You Pretrain: Boosting Zero-shot Adversarial Robustness in Vision-language Models

Songlong Xing; Weijie Wang; Zhengyu Zhao; Jindong Gu; Philip Torr; Nicu Sebe

cs.CVApr 13, 2026

Finetune Like You Pretrain: Boosting Zero-shot Adversarial Robustness in Vision-language Models

Authors: Songlong Xing, Weijie Wang, Zhengyu Zhao, Jindong Gu, Philip Torr, Nicu Sebe

Organizations: University of Trento, Italy · Fondazione Bruno Kessler, Italy · Xi’an Jiaotong University, China · University of Oxford, UK

Abstract

Despite their impressive zero-shot abilities, vision-language models such as CLIP have been shown to be susceptible to adversarial attacks. To enhance its adversarial robustness, recent studies finetune the pretrained vision encoder of CLIP with adversarial examples on a proxy dataset such as ImageNet by aligning adversarial images with correct class labels. However, these methods overlook the important roles of training data distributions and learning objectives, resulting in reduced zero-shot capabilities and limited transferability of robustness across domains and datasets. In this work, we propose a simple yet effective paradigm AdvFLYP, which follows the training recipe of CLIP's pretraining process when performing adversarial finetuning to the model. Specifically, AdvFLYP finetunes CLIP with adversarial images created based on image-text pairs collected from the web, and match them with their corresponding texts via a contrastive loss. To alleviate distortion of adversarial image embeddings of noisy web images, we further propose to regularise AdvFLYP by penalising deviation of adversarial image features. We show that logit- and feature-level regularisation terms benefit robustness and clean accuracy, respectively. Extensive experiments on 14 downstream datasets spanning various domains show the superiority of our paradigm over mainstream practices. Our code and model weights are released at https://github.com/Sxing2/AdvFLYP.

Figures & tables

Panel of Figure 1 — Figure 1: A basic illustration of current mainstream AFT methods and our AdvFLYP paradigm. Rather than performing AFT on a well-curated proxy dataset via a cross-entropy loss, AdvFLYP adversarially finetunes the vision encoder on web-scale image-text pairs via a contrastive loss.

Figure 2: Overview of the formulation of $\mathcal{L}_{CLIP}$ , logit-level regularisation $\mathcal{L}_{logit}$ and feature-level regularisation $\mathcal{L}_{feat}$ . $\mathcal{L}_{CLIP}$ is the main loss of AdvFLYP ( Eq. 7 ). $\mathcal{L}_{logit}$ and $\mathcal{L}_{feat}$ are only employed in the regularised variant of AdvFLYP ( Eq. 13 ), denoted as $\textrm{AdvFLYP}_{full}$ .

(%)		CIFAR10	CIFAR100	STL10	Caltech101	Caltech256	OxfordPets	Flowers102	Food101	StanfordCars	SUN397	Country211	FGVCAircraft	EuroSAT	DTD	avg.
PGD	CLIP	0.77	0.18	11.33	14.87	8.58	1.04	1.04	0.67	0.03	1.14	0.03	0.00	0.03	3.19	3.06
	FARE	21.44	10.05	59.82	54.51	44.73	37.18	16.21	15.90	6.47	15.05	0.74	1.65	6.00	17.13	21.92
	TeCoA	37.54	19.87	75.59	69.02	59.41	61.16	31.05	27.89	13.83	31.07	3.22	5.40	15.37	22.39	33.77
	PMG-AFT	42.06	21.71	77.34	71.91	60.88	63.83	33.81	32.84	17.83	32.19	3.25	6.09	14.71	23.40	35.85
	TGA-ZSR	37.73	19.01	78.89	75.64	65.13	73.34	34.41	37.27	19.29	37.38	3.42	8.97	14.36	26.17	37.93
	AdvFLYP	45.55	24.36	75.15	70.81	60.35	56.77	35.13	34.46	24.32	34.28	3.82	7.02	5.76	26.06	35.99

Table 1: Classification accuracy (%) on 14 downstream datasets tested with three adversarial attack algorithms. We highlight the best and second best result.

(%)	CIFAR10	CIFAR100	STL10	Caltech101	Caltech256	OxfordPets	Flowers102	Food101	StanfordCars	SUN397	Country211	FGVCAircraft	EuroSAT	DTD	avg.
CLIP	85.05	57.18	96.41	86.19	82.04	87.27	65.62	83.83	52.13	58.87	15.26	20.16	38.32	40.11	62.03
FARE	80.49	55.21	95.14	86.42	81.97	87.46	61.94	76.24	48.68	58.75	12.46	18.69	28.74	40.11	59.45
TeCoA	70.60	40.11	91.48	81.31	76.76	82.01	51.63	55.79	34.12	53.29	8.14	13.44	26.97	33.30	51.35
PMG-AFT	75.36	43.94	92.91	84.50	78.43	84.11	55.23	64.59	42.07	55.31	9.23	15.09	24.36	34.31	54.25
TGA-ZSR	77.22	41.77	91.60	78.46	78.09	81.36	52.92	65.74	37.81	52.87	9.86	12.27	30.03	35.96	53.28
AdvFLYP	74.15	48.86	90.70	83.82	76.98	79.69	52.50	64.72	44.98	55.58	10.54	16.08	23.79	37.34	54.27

Table 2: Accuracy (%) on clean images of 14 downstream datasets.

(%)		PGD	CW	AA	$\overline{\textrm{AVG}}$
$\epsilon=2/255$	TeCoA	17.89	18.67	16.57	17.71
	PMG-AFT	18.66	19.25	17.00	18.31
	AdvFLYP	19.94	21.27	19.01	20.07
	$\textrm{AdvFLYP}_{full}$	22.02	22.69	20.35	21.69
$\epsilon=4/255$	TeCoA	4.40	5.37	2.49	4.09
	PMG-AFT	4.44	5.30	2.26	4.00

Table 3: Adversarial robustness under stronger attack budget

\epsilon

. The reported values are accuracy averaged over 14 datasets.

(%)		AutoAttack ( $\epsilon=1/255$ )	Clean	Avg.
	CLIP	0.05	62.03	31.04
(a)	AdvFLYP	34.94	54.27	44.61
(b)	AdvFLYP + $\mathcal{L}_{logit}$	37.40	53.86	45.63
(c)	AdvFLYP + $\mathcal{L}_{feat}$	35.01	57.50	46.26
(d)	AdvFLYP + $\mathcal{L}_{logit}$ + $\mathcal{L}_{feat}$	37.74	55.84	46.79

Table 4: Adversarial robustness and clean accuracy of baselines adopting different combinations of finetuning objectives. Variant (d) amounts to

\textrm{AdvFLYP}_{full}

(%)		IN	-R	-A	-S	Net	Avg.
Adv.	CLIP	1.14	6.32	0.09	5.08	0.12	2.91
	PMG-AFT	39.42	40.95	3.01	25.07	9.61	19.66
	$\textrm{AdvFLYP}_{full}$	31.15	42.45	4.04	25.36	10.72	20.64
Clean	CLIP	59.75	65.10	29.71	39.47	33.28	41.89
	PMG-AFT	61.74	59.94	12.32	36.91	21.47	32.66
	$\textrm{AdvFLYP}_{full}$	52.43	60.40	13.32	37.07	22.36	33.29

Table 5: Comparison of PMG-AFT finetuned on ImageNet (IN) and AdvFLYP on web image-text data. The adversarial accuracy is tested under PGD-10 with

\epsilon=1/255

Figure 3: Performance of $\textrm{AdvFLYP}_{full}$ averaged over 14 downstream datasets versus the amount of image-text pairs from the web. The robust accuracy is evaluated under PGD-10 ( $\epsilon=1/255$ ).

(%)	PGD	CW	AA	clean
$f_{\theta}$ , $g_{\phi}$	35.67	34.72	33.89	53.81
all	35.71	34.63	33.76	53.97
$f_{\theta}$ (ours)	39.15	38.30	37.74	55.84

Table 6: We unfreeze more modules of CLIP in our

\textrm{AdvFLYP}_{full}

. We evaluate the finetuned models using the attack methods of PGD, CW, and AutoAttack with

\epsilon=1/255

(%)	PGD	AA	clean
$\textrm{AdvFLYP}_{2nd}$	32.71	31.25	49.01
$\textrm{AdvFLYP}_{full}$	33.69	31.79	53.66

Table 7: Adversarial robustness evaluated at

\epsilon=1/255

and clean accuracy averaged over 14 datasets. Both variants are trained with 100k image-text pairs.

(%)	PGD	CW	AA	clean
TeCoA	33.77	33.04	32.17	51.35
naive FLYP	32.89	32.28	31.23	51.66
PMG-AFT	35.85	34.87	33.89	54.25
naive FLYP + $\mathcal{L}_{logit}$	35.28	34.35	33.33	53.87

Table 8: We implement ‘naive FLYP’ in our adversarial finetuning context and compare to the previous standard AFT paradigm TeCoA [ 30 ] . PMG-AFT [ 51 ] is equivalent to TeCoA +

\mathcal{L}_{logit}

. We evaluate the robustness of the finetuned models using PGD, CW, and AutoAttack with

\epsilon=1/255

and report the average results over 14 downstream datasets.

(%)		AutoAttack ( $\epsilon=1/255$ )	Clean	Avg.
	CLIP	0.05	62.03	31.04
(a)	TeCoA	32.17	51.35	41.76
(b)	TeCoA + $\mathcal{L}_{logit}$	33.89	54.25	44.07
(c)	TeCoA + $\mathcal{L}_{feat}$	32.94	52.31	42.63
(d)	TeCoA + $\mathcal{L}_{logit}$ + $\mathcal{L}_{feat}$	33.88	54.22	44.05

Table 9: Results of the models finetuned with different combinations of regularisation levels on top of TeCoA [ 30 ] . The combination ( b ) is equivalent to PMG-AFT [ 51 ] . The reported results are the average accuracy over 14 downstream datasets.

(%)		CIFAR10	CIFAR100	STL10	Caltech101	Caltech256	OxfordPets	Flowers102	Food101	StanfordCars	SUN397	Country211	FGVCAircraft	EuroSAT	DTD	avg.
PGD	CLIP	0.00	0.00	0.71	1.71	0.84	0.00	0.02	0.01	0.00	0.02	0.00	0.00	0.00	0.21	0.25
	FARE	1.11	0.85	15.21	22.14	14.02	3.41	1.12	0.72	0.15	1.79	0.04	0.00	2.69	5.11	4.88
	TeCoA	11.66	6.75	46.08	51.03	37.80	31.32	14.15	9.03	3.44	13.14	0.92	1.08	9.85	14.20	17.89
	PMG-AFT	12.75	7.68	47.84	51.15	38.41	32.98	15.09	10.34	4.42	13.29	0.89	1.59	10.31	14.52	18.66
	TGA-ZSR	2.99	1.34	27.29	33.30	23.64	19.13	4.15	4.26	1.08	5.67	0.21	0.09	0.56	6.60	9.31
	AdvFLYP	19.32	9.63	52.24	52.96	41.13	27.61	18.13	12.40	9.15	16.51	0.99	1.80	0.60	16.70	19.94

Table 10: Classification accuracy (%) on 14 downstream datasets tested with three adversarial attack algorithms (

\epsilon=2/255

). We highlight the best and second best result.

(%)		CIFAR10	CIFAR100	STL10	Caltech101	Caltech256	OxfordPets	Flowers102	Food101	StanfordCars	SUN397	Country211	FGVCAircraft	EuroSAT	DTD	avg.
PGD	CLIP	0.00	0.00	0.04	0.61	0.11	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.11	0.06
	FARE	0.06	0.02	1.42	5.05	2.04	1.55	0.02	0.02	0.00	0.03	0.00	0.00	0.00	0.37	0.76
	TeCoA	0.72	0.79	9.39	20.42	12.46	3.08	2.00	0.63	0.06	2.06	0.10	0.00	5.25	4.63	4.40
	PMG-AFT	0.53	0.89	9.91	20.02	12.24	2.81	1.74	0.76	0.06	1.92	0.07	0.03	6.34	4.84	4.44
	TGA-ZSR	0.04	0.01	2.10	6.37	3.47	2.10	0.05	0.10	0.00	0.22	0.01	0.00	0.00	0.80	1.09
	AdvFLYP	2.08	1.92	19.50	27.69	17.20	2.94	3.27	1.23	0.85	3.30	0.06	0.03	0.01	6.44	6.18

Table 11: Classification accuracy (%) on 14 downstream datasets tested with three adversarial attack algorithms (

\epsilon=4/255

). We highlight the best and second best result.

(%)	PGD	AA	clean
PMG-AFT	31.59	28.92	54.46
$\textrm{AdvFLYP}_{full}$	35.13	33.62	56.07

Table 12: Robustness (

\epsilon=1/255

) and clean accuracy of CLIP ViT-B/16 averaged over 14 datasets.

(%)	ImageNet	CIFAR10	CIFAR100	STL10	Caltech101	Caltech256	OxfordPets	Flowers102	Food101	StanfordCars	SUN397	Country211	FGVCAircraft	EuroSAT	DTD	avg.
CLIP	59.75	85.05	57.18	96.41	86.19	82.04	87.27	65.62	83.83	52.13	58.87	15.26	20.16	38.32	40.11	62.03
web data	59.69	84.31	58.77	95.92	86.57	82.42	87.33	65.56	83.06	52.94	60.67	15.59	21.15	34.58	41.12	62.14
IN	67.19	84.33	58.14	96.52	85.02	82.06	87.16	62.29	77.99	46.61	59.27	13.69	17.43	35.83	40.11	60.46

Table 13: The clean accuracy of the models finetuned on 100k image-text web data and a 100k-subset of ImageNet, respectively. Both baselines are finetuned with clean non-adversarial images.

Panel of Figure 4 — Figure 4: An example of an image from ImageNet and its description generated by Qwen2.5-VL-3B-Instruct.

(%)	ImageNet	CIFAR10	CIFAR100	STL10	Caltech101	Caltech256	OxfordPets	Flowers102	Food101	StanfordCars	SUN397	Country211	FGVCAircraft	EuroSAT	DTD	avg.
PMG-AFT	37.17	39.23	19.40	76.79	71.14	59.34	62.42	30.69	29.86	14.48	29.26	2.59	4.65	12.83	21.76	33.89
$\textrm{AFT}_{capIN}$	34.34	49.71	26.08	82.36	71.82	60.37	64.95	34.33	37.27	19.28	28.96	2.71	4.77	11.12	23.30	36.93
$\mathrm{AdvFLYP}_{full}$	29.47	50.15	25.99	78.50	72.47	61.31	59.88	37.81	37.53	24.61	34.44	3.46	5.73	11.46	24.95	37.74

Table 14: Robustness of the models finetuned on class-labelled ImageNet (PMG-AFT), VLM-captioned ImageNet (

\textrm{AFT}_{capIN}

), and our

\textrm{AdvFLYP}_{full}

, evaluated with AutoAttack (

\epsilon=1/255

Panel of Figure 5 — Figure 5: t-SNE visualisation of adversarial and clean image features in the latent space.

VLM Robustness

Vision-language model robustness studies whether models remain reliable under visual, textual, and distribution shifts, adversarial inputs, and safety challenges.

See all 443 papers

Mar 31, 2026

AGFT: Alignment-Guided Fine-Tuning for Zero-Shot Adversarial Robustness of Vision-Language Models

Pre-trained vision-language models (VLMs) exhibit strong zero-shot generalization but remain vulnerable to adversarial perturbations. Existing classification-guided adversarial fine-tuning methods often disrupt pre-trained cross-modal alignment, weakening visual-textual correspondence and degrading zero-shot performance. In this paper, we propose an Alignment-Guided Fine-Tuning (AGFT) framework that enhances zero-shot adversarial robustness while preserving the cross-modal semantic structure. Unlike label-based methods that rely on hard labels and fail to maintain the relative relationships between image and text, AGFT leverages the probabilistic predictions of the original model for text-guided adversarial training, which aligns adversarial visual features with textual embeddings via soft alignment distributions, improving zero-shot adversarial robustness. To address structural discrepancies introduced by fine-tuning, we introduce a distribution consistency calibration mechanism that adjusts the robust model output to match a temperature-scaled version of the pre-trained model predictions. Extensive experiments across multiple zero-shot benchmarks demonstrate that AGFT outperforms state-of-the-art methods while significantly improving zero-shot adversarial robustness.

Yubo Cui, Xianchao Guan, Zijun Xiong +1

Harbin Institute of Technology, Shenzhen, China · Shenzhen Loop Area Institute, Shenzhen, China

VLM Robustness Vision-Language Alignment

Ask PDF

May 17, 2026

TAME: Test-Time Adversarial Prompt Tuning via Mixture-of-Experts for Vision-Language Models

Large-scale pre-trained Vision-Language models (VLMs), such as CLIP, exhibit strong zero-shot generalization, yet remain highly vulnerable to imperceptible adversarial perturbations, raising serious safety concerns for open-world deployment. To enhance robustness without requiring downstream task-specific retraining, we propose TAME, a novel test-time defense. Building upon our prior Test-Time Adversarial Prompt Tuning (TAPT), TAME introduces an architectural reformulation by replacing TAPT's single adaptive prompt with an input-conditioned Mixture-of-Experts (MoE) framework, enabling more expressive and adaptive defense. Specifically, TAME maintains a bank of learnable expert prompts and employs an input-dependent routing mechanism to aggregate a customized prompt mixture for each unlabeled test sample at inference time. This test-time defense mechanism is driven by three unsupervised objectives: (1) multi-view prediction entropy minimization, (2) layer-wise alignment of visual token statistics to precomputed clean and adversarial reference distributions, and (3) MoE regularization for balanced expert utilization and prompt diversity. We evaluated TAME on 11 benchmark datasets, including ImageNet and 10 additional zero-shot datasets. The results show that TAME improves the zero-shot adversarial robustness of the original CLIP by at least 49.1% under AutoAttack while largely preserving generalization on clean samples. TAME also consistently outperforms existing adversarial prompt tuning methods across multiple prompt designs, yielding an average robustness gain of at least 30.2%.

Xin Wang, Yixu Wang, Jiaming Zhang +6

Adversarial Attacks on VLMs Test-Time Adaptation VLM Adaptation Test-Time Prompt Tuning

Ask PDF

Jul 10, 2026

Robustifying Vision-Language Models via Test-Time Prompt Adaptation

Pre-trained Vision-Language Models (VLMs) such as CLIP achieve strong zero-shot generalization, but their performance degrades sharply under adversarial perturbations. Existing test-time adaptation methods typically rely on sample-level confidence heuristics, overlooking the intrinsic distributional structure of the data. This sample-centric approach limits robustness, as it fails to distinguish confident adversarial mispredictions from true semantic consistency. In this work, we observe that adversarial distortion is structurally brittle: while holistic representations are corrupted, semantic integrity is often preserved in the distribution of augmented views. Motivated by this insight, we propose RITA, a Robust test-tIme prompt-TAdaptation framework that shifts from sample-level estimates to distribution-level alignment. Specifically, RITA employs optimal transport to align the distribution of augmented visual features with textual prototypes, mitigating adversarial outliers and rectifying cross-modal semantic misalignment. Furthermore, we introduce a dynamic cache to progressively accumulate reliable cues from the test stream for online refinement. Extensive experiments demonstrate that RITA significantly improves adversarial robustness without compromising clean accuracy.

Xingyu Zhu, Huanshen Wu, Shuo Wang +4

National University of Singapore · University of Science and Technology of China · The Hong Kong University of Science and Technology

VLM Robustness VLM Adaptation Test-Time Adaptation of VLMs Adversarial Robustness Test-Time Prompt Tuning

Ask PDF

Abstract

VLM Robustness

Explore similar work

AGFT: Alignment-Guided Fine-Tuning for Zero-Shot Adversarial Robustness of Vision-Language Models

TAME: Test-Time Adversarial Prompt Tuning via Mixture-of-Experts for Vision-Language Models

Robustifying Vision-Language Models via Test-Time Prompt Adaptation