Finetune Like You Pretrain: Boosting Zero-shot Adversarial Robustness in Vision-language Models

Songlong Xing; Weijie Wang; Zhengyu Zhao; Jindong Gu; Philip Torr; Nicu Sebe

cs.CVApr 13, 2026

Finetune Like You Pretrain: Boosting Zero-shot Adversarial Robustness in Vision-language Models

Authors: Songlong Xing, Weijie Wang, Zhengyu Zhao, Jindong Gu, Philip Torr, Nicu Sebe

Organizations: University of Trento, Italy · Fondazione Bruno Kessler, Italy · Xi’an Jiaotong University, China · University of Oxford, UK

Abstract

Despite their impressive zero-shot abilities, vision-language models such as CLIP have been shown to be susceptible to adversarial attacks. To enhance its adversarial robustness, recent studies finetune the pretrained vision encoder of CLIP with adversarial examples on a proxy dataset such as ImageNet by aligning adversarial images with correct class labels. However, these methods overlook the important roles of training data distributions and learning objectives, resulting in reduced zero-shot capabilities and limited transferability of robustness across domains and datasets. In this work, we propose a simple yet effective paradigm AdvFLYP, which follows the training recipe of CLIP's pretraining process when performing adversarial finetuning to the model. Specifically, AdvFLYP finetunes CLIP with adversarial images created based on image-text pairs collected from the web, and match them with their corresponding texts via a contrastive loss. To alleviate distortion of adversarial image embeddings of noisy web images, we further propose to regularise AdvFLYP by penalising deviation of adversarial image features. We show that logit- and feature-level regularisation terms benefit robustness and clean accuracy, respectively. Extensive experiments on 14 downstream datasets spanning various domains show the superiority of our paradigm over mainstream practices. Our code and model weights are released at https://github.com/Sxing2/AdvFLYP.

Figures & tables

Panel of Figure 1 — Figure 1: A basic illustration of current mainstream AFT methods and our AdvFLYP paradigm. Rather than performing AFT on a well-curated proxy dataset via a cross-entropy loss, AdvFLYP adversarially finetunes the vision encoder on web-scale image-text pairs via a contrastive loss.

Figure 2: Overview of the formulation of $\mathcal{L}_{CLIP}$ , logit-level regularisation $\mathcal{L}_{logit}$ and feature-level regularisation $\mathcal{L}_{feat}$ . $\mathcal{L}_{CLIP}$ is the main loss of AdvFLYP ( Eq. 7 ). $\mathcal{L}_{logit}$ and $\mathcal{L}_{feat}$ are only employed in the regularised variant of AdvFLYP ( Eq. 13 ), denoted as $\textrm{AdvFLYP}_{full}$ .

(%)		CIFAR10	CIFAR100	STL10	Caltech101	Caltech256	OxfordPets	Flowers102	Food101	StanfordCars	SUN397	Country211	FGVCAircraft	EuroSAT	DTD	avg.
PGD	CLIP	0.77	0.18	11.33	14.87	8.58	1.04	1.04	0.67	0.03	1.14	0.03	0.00	0.03	3.19	3.06
	FARE	21.44	10.05	59.82	54.51	44.73	37.18	16.21	15.90	6.47	15.05	0.74	1.65	6.00	17.13	21.92
	TeCoA	37.54	19.87	75.59	69.02	59.41	61.16	31.05	27.89	13.83	31.07	3.22	5.40	15.37	22.39	33.77
	PMG-AFT	42.06	21.71	77.34	71.91	60.88	63.83	33.81	32.84	17.83	32.19	3.25	6.09	14.71	23.40	35.85
	TGA-ZSR	37.73	19.01	78.89	75.64	65.13	73.34	34.41	37.27	19.29	37.38	3.42	8.97	14.36	26.17	37.93
	AdvFLYP	45.55	24.36	75.15	70.81	60.35	56.77	35.13	34.46	24.32	34.28	3.82	7.02	5.76	26.06	35.99

Table 1: Classification accuracy (%) on 14 downstream datasets tested with three adversarial attack algorithms. We highlight the best and second best result.

(%)	CIFAR10	CIFAR100	STL10	Caltech101	Caltech256	OxfordPets	Flowers102	Food101	StanfordCars	SUN397	Country211	FGVCAircraft	EuroSAT	DTD	avg.
CLIP	85.05	57.18	96.41	86.19	82.04	87.27	65.62	83.83	52.13	58.87	15.26	20.16	38.32	40.11	62.03
FARE	80.49	55.21	95.14	86.42	81.97	87.46	61.94	76.24	48.68	58.75	12.46	18.69	28.74	40.11	59.45
TeCoA	70.60	40.11	91.48	81.31	76.76	82.01	51.63	55.79	34.12	53.29	8.14	13.44	26.97	33.30	51.35
PMG-AFT	75.36	43.94	92.91	84.50	78.43	84.11	55.23	64.59	42.07	55.31	9.23	15.09	24.36	34.31	54.25
TGA-ZSR	77.22	41.77	91.60	78.46	78.09	81.36	52.92	65.74	37.81	52.87	9.86	12.27	30.03	35.96	53.28
AdvFLYP	74.15	48.86	90.70	83.82	76.98	79.69	52.50	64.72	44.98	55.58	10.54	16.08	23.79	37.34	54.27

Table 2: Accuracy (%) on clean images of 14 downstream datasets.

(%)		PGD	CW	AA	$\overline{\textrm{AVG}}$
$\epsilon=2/255$	TeCoA	17.89	18.67	16.57	17.71
	PMG-AFT	18.66	19.25	17.00	18.31
	AdvFLYP	19.94	21.27	19.01	20.07
	$\textrm{AdvFLYP}_{full}$	22.02	22.69	20.35	21.69
$\epsilon=4/255$	TeCoA	4.40	5.37	2.49	4.09
	PMG-AFT	4.44	5.30	2.26	4.00

Table 3: Adversarial robustness under stronger attack budget

\epsilon

. The reported values are accuracy averaged over 14 datasets.

(%)		AutoAttack ( $\epsilon=1/255$ )	Clean	Avg.
	CLIP	0.05	62.03	31.04
(a)	AdvFLYP	34.94	54.27	44.61
(b)	AdvFLYP + $\mathcal{L}_{logit}$	37.40	53.86	45.63
(c)	AdvFLYP + $\mathcal{L}_{feat}$	35.01	57.50	46.26
(d)	AdvFLYP + $\mathcal{L}_{logit}$ + $\mathcal{L}_{feat}$	37.74	55.84	46.79

Table 4: Adversarial robustness and clean accuracy of baselines adopting different combinations of finetuning objectives. Variant (d) amounts to

\textrm{AdvFLYP}_{full}

(%)		IN	-R	-A	-S	Net	Avg.
Adv.	CLIP	1.14	6.32	0.09	5.08	0.12	2.91
	PMG-AFT	39.42	40.95	3.01	25.07	9.61	19.66
	$\textrm{AdvFLYP}_{full}$	31.15	42.45	4.04	25.36	10.72	20.64
Clean	CLIP	59.75	65.10	29.71	39.47	33.28	41.89
	PMG-AFT	61.74	59.94	12.32	36.91	21.47	32.66
	$\textrm{AdvFLYP}_{full}$	52.43	60.40	13.32	37.07	22.36	33.29

Table 5: Comparison of PMG-AFT finetuned on ImageNet (IN) and AdvFLYP on web image-text data. The adversarial accuracy is tested under PGD-10 with

\epsilon=1/255

Figure 3: Performance of $\textrm{AdvFLYP}_{full}$ averaged over 14 downstream datasets versus the amount of image-text pairs from the web. The robust accuracy is evaluated under PGD-10 ( $\epsilon=1/255$ ).

(%)	PGD	CW	AA	clean
$f_{\theta}$ , $g_{\phi}$	35.67	34.72	33.89	53.81
all	35.71	34.63	33.76	53.97
$f_{\theta}$ (ours)	39.15	38.30	37.74	55.84

Table 6: We unfreeze more modules of CLIP in our

\textrm{AdvFLYP}_{full}

. We evaluate the finetuned models using the attack methods of PGD, CW, and AutoAttack with

\epsilon=1/255

(%)	PGD	AA	clean
$\textrm{AdvFLYP}_{2nd}$	32.71	31.25	49.01
$\textrm{AdvFLYP}_{full}$	33.69	31.79	53.66

Table 7: Adversarial robustness evaluated at

\epsilon=1/255

and clean accuracy averaged over 14 datasets. Both variants are trained with 100k image-text pairs.

(%)	PGD	CW	AA	clean
TeCoA	33.77	33.04	32.17	51.35
naive FLYP	32.89	32.28	31.23	51.66
PMG-AFT	35.85	34.87	33.89	54.25
naive FLYP + $\mathcal{L}_{logit}$	35.28	34.35	33.33	53.87

Table 8: We implement ‘naive FLYP’ in our adversarial finetuning context and compare to the previous standard AFT paradigm TeCoA [ 30 ] . PMG-AFT [ 51 ] is equivalent to TeCoA +

\mathcal{L}_{logit}

. We evaluate the robustness of the finetuned models using PGD, CW, and AutoAttack with

\epsilon=1/255

and report the average results over 14 downstream datasets.

(%)		AutoAttack ( $\epsilon=1/255$ )	Clean	Avg.
	CLIP	0.05	62.03	31.04
(a)	TeCoA	32.17	51.35	41.76
(b)	TeCoA + $\mathcal{L}_{logit}$	33.89	54.25	44.07
(c)	TeCoA + $\mathcal{L}_{feat}$	32.94	52.31	42.63
(d)	TeCoA + $\mathcal{L}_{logit}$ + $\mathcal{L}_{feat}$	33.88	54.22	44.05

Table 9: Results of the models finetuned with different combinations of regularisation levels on top of TeCoA [ 30 ] . The combination ( b ) is equivalent to PMG-AFT [ 51 ] . The reported results are the average accuracy over 14 downstream datasets.

(%)		CIFAR10	CIFAR100	STL10	Caltech101	Caltech256	OxfordPets	Flowers102	Food101	StanfordCars	SUN397	Country211	FGVCAircraft	EuroSAT	DTD	avg.
PGD	CLIP	0.00	0.00	0.71	1.71	0.84	0.00	0.02	0.01	0.00	0.02	0.00	0.00	0.00	0.21	0.25
	FARE	1.11	0.85	15.21	22.14	14.02	3.41	1.12	0.72	0.15	1.79	0.04	0.00	2.69	5.11	4.88
	TeCoA	11.66	6.75	46.08	51.03	37.80	31.32	14.15	9.03	3.44	13.14	0.92	1.08	9.85	14.20	17.89
	PMG-AFT	12.75	7.68	47.84	51.15	38.41	32.98	15.09	10.34	4.42	13.29	0.89	1.59	10.31	14.52	18.66
	TGA-ZSR	2.99	1.34	27.29	33.30	23.64	19.13	4.15	4.26	1.08	5.67	0.21	0.09	0.56	6.60	9.31
	AdvFLYP	19.32	9.63	52.24	52.96	41.13	27.61	18.13	12.40	9.15	16.51	0.99	1.80	0.60	16.70	19.94

Table 10: Classification accuracy (%) on 14 downstream datasets tested with three adversarial attack algorithms (

\epsilon=2/255

). We highlight the best and second best result.

(%)		CIFAR10	CIFAR100	STL10	Caltech101	Caltech256	OxfordPets	Flowers102	Food101	StanfordCars	SUN397	Country211	FGVCAircraft	EuroSAT	DTD	avg.
PGD	CLIP	0.00	0.00	0.04	0.61	0.11	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.11	0.06
	FARE	0.06	0.02	1.42	5.05	2.04	1.55	0.02	0.02	0.00	0.03	0.00	0.00	0.00	0.37	0.76
	TeCoA	0.72	0.79	9.39	20.42	12.46	3.08	2.00	0.63	0.06	2.06	0.10	0.00	5.25	4.63	4.40
	PMG-AFT	0.53	0.89	9.91	20.02	12.24	2.81	1.74	0.76	0.06	1.92	0.07	0.03	6.34	4.84	4.44
	TGA-ZSR	0.04	0.01	2.10	6.37	3.47	2.10	0.05	0.10	0.00	0.22	0.01	0.00	0.00	0.80	1.09
	AdvFLYP	2.08	1.92	19.50	27.69	17.20	2.94	3.27	1.23	0.85	3.30	0.06	0.03	0.01	6.44	6.18

Table 11: Classification accuracy (%) on 14 downstream datasets tested with three adversarial attack algorithms (

\epsilon=4/255

). We highlight the best and second best result.

(%)	PGD	AA	clean
PMG-AFT	31.59	28.92	54.46
$\textrm{AdvFLYP}_{full}$	35.13	33.62	56.07

Table 12: Robustness (

\epsilon=1/255

) and clean accuracy of CLIP ViT-B/16 averaged over 14 datasets.

(%)	ImageNet	CIFAR10	CIFAR100	STL10	Caltech101	Caltech256	OxfordPets	Flowers102	Food101	StanfordCars	SUN397	Country211	FGVCAircraft	EuroSAT	DTD	avg.
CLIP	59.75	85.05	57.18	96.41	86.19	82.04	87.27	65.62	83.83	52.13	58.87	15.26	20.16	38.32	40.11	62.03
web data	59.69	84.31	58.77	95.92	86.57	82.42	87.33	65.56	83.06	52.94	60.67	15.59	21.15	34.58	41.12	62.14
IN	67.19	84.33	58.14	96.52	85.02	82.06	87.16	62.29	77.99	46.61	59.27	13.69	17.43	35.83	40.11	60.46

Table 13: The clean accuracy of the models finetuned on 100k image-text web data and a 100k-subset of ImageNet, respectively. Both baselines are finetuned with clean non-adversarial images.

Panel of Figure 4 — Figure 4: An example of an image from ImageNet and its description generated by Qwen2.5-VL-3B-Instruct.

(%)	ImageNet	CIFAR10	CIFAR100	STL10	Caltech101	Caltech256	OxfordPets	Flowers102	Food101	StanfordCars	SUN397	Country211	FGVCAircraft	EuroSAT	DTD	avg.
PMG-AFT	37.17	39.23	19.40	76.79	71.14	59.34	62.42	30.69	29.86	14.48	29.26	2.59	4.65	12.83	21.76	33.89
$\textrm{AFT}_{capIN}$	34.34	49.71	26.08	82.36	71.82	60.37	64.95	34.33	37.27	19.28	28.96	2.71	4.77	11.12	23.30	36.93
$\mathrm{AdvFLYP}_{full}$	29.47	50.15	25.99	78.50	72.47	61.31	59.88	37.81	37.53	24.61	34.44	3.46	5.73	11.46	24.95	37.74

Table 14: Robustness of the models finetuned on class-labelled ImageNet (PMG-AFT), VLM-captioned ImageNet (

\textrm{AFT}_{capIN}

), and our

\textrm{AdvFLYP}_{full}

, evaluated with AutoAttack (

\epsilon=1/255

Panel of Figure 5 — Figure 5: t-SNE visualisation of adversarial and clean image features in the latent space.

VLM Robustness

Vision-language model robustness studies whether models remain reliable under visual, textual, and distribution shifts, adversarial inputs, and safety challenges.

See all 443 papers

AGFT: Alignment-Guided Fine-Tuning for Zero-Shot Adversarial Robustness of Vision-Language Models
Mar 31, 2026Yubo Cui, Xianchao Guan, Zijun Xiong +1VLM Robustness Vision-Language Alignment
Harbin Institute of Technology, Shenzhen, China · Shenzhen Loop Area Institute, Shenzhen, China
TAME: Test-Time Adversarial Prompt Tuning via Mixture-of-Experts for Vision-Language Models
May 17, 2026Xin Wang, Yixu Wang, Jiaming Zhang +6Adversarial Attacks on VLMs Test-Time Adaptation
Robustifying Vision-Language Models via Test-Time Prompt Adaptation
Jul 10, 2026Xingyu Zhu, Huanshen Wu, Shuo Wang +4VLM Robustness VLM Adaptation
National University of Singapore · University of Science and Technology of China · The Hong Kong University of Science and Technology

Mar 31, 2026J/K move · Enter open · S save

AGFT: Alignment-Guided Fine-Tuning for Zero-Shot Adversarial Robustness of Vision-Language Models

Yubo Cui, Xianchao Guan, Zijun Xiong, Zheng Zhang

Harbin Institute of Technology, Shenzhen, China · Shenzhen Loop Area Institute, Shenzhen, China

VLM Robustness Vision-Language Alignment

Open paper AskPDF arXiv

Pre-trained vision-language models (VLMs) exhibit strong zero-shot generalization but remain vulnerable to adversarial perturbations. Existing classification-guided adversarial fine-tuning methods often disrupt pre-trained cross-modal alignment, weakening visual-textual correspondence and degrading zero-shot performance. In this paper, we propose an Alignment-Guided Fine-Tuning (AGFT) framework that enhances zero-shot adversarial robustness while preserving the cross-modal semantic structure. Unlike label-based methods that rely on hard labels and fail to maintain the relative relationships between image and text, AGFT leverages the probabilistic predictions of the original model for text-guided adversarial training, which aligns adversarial visual features with textual embeddings via soft alignment distributions, improving zero-shot adversarial robustness. To address structural discrepancies introduced by fine-tuning, we introduce a distribution consistency calibration mechanism that adjusts the robust model output to match a temperature-scaled version of the pre-trained model predictions. Extensive experiments across multiple zero-shot benchmarks demonstrate that AGFT outperforms state-of-the-art methods while significantly improving zero-shot adversarial robustness.

Abstract

VLM Robustness

Explore similar work

AGFT: Alignment-Guided Fine-Tuning for Zero-Shot Adversarial Robustness of Vision-Language Models

TAME: Test-Time Adversarial Prompt Tuning via Mixture-of-Experts for Vision-Language Models

Robustifying Vision-Language Models via Test-Time Prompt Adaptation