AGFT: Alignment-Guided Fine-Tuning for Zero-Shot Adversarial Robustness of Vision-Language Models

Yubo Cui; Xianchao Guan; Zijun Xiong; Zheng Zhang

cs.CVMar 31, 2026

AGFT: Alignment-Guided Fine-Tuning for Zero-Shot Adversarial Robustness of Vision-Language Models

Authors: Yubo Cui, Xianchao Guan, Zijun Xiong, Zheng Zhang

Organizations: Harbin Institute of Technology, Shenzhen, China · Shenzhen Loop Area Institute, Shenzhen, China

Abstract

Pre-trained vision-language models (VLMs) exhibit strong zero-shot generalization but remain vulnerable to adversarial perturbations. Existing classification-guided adversarial fine-tuning methods often disrupt pre-trained cross-modal alignment, weakening visual-textual correspondence and degrading zero-shot performance. In this paper, we propose an Alignment-Guided Fine-Tuning (AGFT) framework that enhances zero-shot adversarial robustness while preserving the cross-modal semantic structure. Unlike label-based methods that rely on hard labels and fail to maintain the relative relationships between image and text, AGFT leverages the probabilistic predictions of the original model for text-guided adversarial training, which aligns adversarial visual features with textual embeddings via soft alignment distributions, improving zero-shot adversarial robustness. To address structural discrepancies introduced by fine-tuning, we introduce a distribution consistency calibration mechanism that adjusts the robust model output to match a temperature-scaled version of the pre-trained model predictions. Extensive experiments across multiple zero-shot benchmarks demonstrate that AGFT outperforms state-of-the-art methods while significantly improving zero-shot adversarial robustness.

Figures & tables

Panel of Figure 1 — Figure 1: The performance of AGFT is shown in Fig. 1 (a). Figures 1 (b) and 1 (c) compare classification-guided and alignment-guided adversarial fine-tuning. Unlike classification-guided methods, which rely on label supervision and can disrupt the pre-trained cross-modal alignment, AGFT leverages the probabilistic predictions of the original model to preserve the cross-modal semantic structure, enhancing zero-shot adversarial robustness.

Figure 2: The overall pipeline of AGFT. First, we obtain the probabilistic predictions of the pre-trained model and use the resulting distribution as the target for adversarial fine-tuning to encourage adversarial visual features to align with textual embeddings. To mitigate the discrepancies in visual–textual semantic structure, we calibrate the pre-trained output distribution through temperature adjustment, while maintaining the cross-modal similarity structure across images and textual descriptions.

Method	Caltech101	Caltech256	CIFAR10	CIFAR100	DTD	EuroSAT	FGVC	Flower102	Food101	ImageNet	OxfordPet	PCAM	Stanf.Cars	STL10	SUN397	Average
CLIP	21.27	11.69	10.31	4.69	2.55	0.02	0.00	1.00	4.06	1.13	2.37	0.13	0.21	33.10	1.08	6.24
TeCoA	71.83	61.23	59.85	34.47	23.56	15.06	4.85	30.60	29.01	41.29	61.90	14.42	14.06	83.33	32.12	38.51
PMG-AFT	77.78	63.15	65.93	39.41	26.70	15.05	5.93	33.36	39.37	35.05	65.34	18.99	20.33	86.31	32.21	41.66
TGA-ZSR	71.67	64.06	63.30	36.79	22.07	13.95	4.85	27.26	28.81	52.80	65.12	30.95	14.78	85.01	31.12	40.84
GLADIATOR	73.34	63.50	67.89	40.82	24.75	14.92	6.23	34.58	34.92	44.53	68.10	39.29	20.65	86.53	31.86	43.46
AGFT (Ours)	82.23	67.95	71.72	45.55	24.14	16.25	7.94	35.95	44.76	44.95	71.03	33.89	27.34	88.52	36.39	46.57

Table 1: Zero-shot robust accuracy (%). Adversarial examples are generated by PGD-20 attack with the perturbation budget

\epsilon=1/255

. Models are adversarially fine-tuned on ImageNet and evaluated across 15 datasets. Results of GLADIATOR are from the original paper.

Method	Caltech101	Caltech256	CIFAR10	CIFAR100	DTD	EuroSAT	FGVC	Flower102	Food101	ImageNet	OxfordPet	PCAM	Stanf.Cars	STL10	SUN397	Average
CLIP	92.03	82.81	87.19	61.15	41.17	45.40	19.25	65.34	83.61	59.53	87.32	58.26	54.10	96.89	59.01	66.20
TeCoA	85.30	77.34	78.75	50.02	34.68	27.22	12.62	51.31	56.24	63.50	82.01	53.44	33.88	93.23	54.45	56.93
PMG-AFT	90.25	78.33	83.24	57.26	35.42	25.93	13.97	52.79	70.03	53.57	82.58	54.22	42.21	94.73	53.96	59.23
TGA-ZSR	81.70	77.45	81.18	53.45	31.70	24.81	9.98	44.19	51.31	70.96	79.91	46.48	29.34	93.52	50.10	55.07
GLADIATOR	85.42	79.44	86.94	60.86	35.59	26.52	15.66	56.77	67.32	63.83	84.25	49.37	43.75	95.71	53.61	60.34
AGFT (Ours)	91.05	80.26	86.72	62.38	33.40	25.06	16.49	54.89	72.14	60.11	85.58	55.52	45.69	95.72	55.19	61.35

Table 2: Zero-shot clean accuracy (%). Clean images from 15 datasets are evaluated on adversarially fine-tuned CLIP models.

Method	Robust Accuracy
Method	1/255	2/255	3/255	4/255	Avg.
TeCoA	38.51	22.30	11.89	6.75	19.86
PMG-AFT	41.66	25.15	13.63	7.19	21.90
TGA-ZSR	40.84	26.61	16.15	10.30	23.47
GLADIATOR	43.46	26.41	15.29	8.59	23.43
AGFT (Ours)	46.57	29.80	16.61	8.88	25.46

Table 3: Average zero-shot robust accuracy (%) under PGD-20 attacks with varying perturbation budgets during inference .

Budget $\epsilon$	Method	Clean	PGD	C&W	AA
1/255	TeCoA	56.93	38.51	37.57	37.01
	PMG-AFT	59.23	41.66	40.55	40.09
	TGA-ZSR	55.07	40.84	40.26	39.90
	GLADIATOR	60.34	43.46	42.50	41.91
	AGFT (Ours)	61.35	46.57	43.06	42.81
2/255	TeCoA	49.69	26.79	25.92	25.14

Table 4: Average accuracy (%) under strong attacks with different perturbation budgets used during both fine-tuning and inference .

Architecture	Method	Clean	PGD	C&W	AA
ViT-B/32	TeCoA	56.93	38.51	37.57	37.01
	PMG-AFT	59.23	41.66	40.55	40.09
	TGA-ZSR	55.07	40.84	40.26	39.90
	AGFT (Ours)	61.35	46.57	43.06	42.81
ViT-B/16	TeCoA	60.12	42.05	41.15	40.50
	PMG-AFT	61.49	47.08	46.08	45.60

Table 5: Average performance (%) of different CLIP architectures with the perturbation budget

\epsilon=1/255

Method	PGD	MI	DI 2	TI	NI	PI	PI++	C&W	AA	PGD T	PGD TL	MI T	DI 2 T	TI T	NI T	Average
TeCoA	38.51	38.53	41.93	47.13	49.45	38.80	40.06	37.57	37.01	51.93	52.49	51.90	53.02	54.55	55.15	45.86
PMG-AFT	41.66	41.69	44.82	49.21	52.08	41.93	42.84	40.55	40.09	54.96	55.42	54.96	55.83	57.02	57.67	48.71
TGA-ZSR	40.84	40.88	43.43	47.18	48.92	41.07	41.97	40.26	39.90	52.36	52.92	52.33	53.00	53.85	54.14	46.87
AGFT (Ours)	46.57	46.56	48.99	53.26	55.68	46.73	47.93	43.06	42.81	57.99	58.41	58.01	58.87	59.92	60.42	52.34

Table 6: Zero-shot robust accuracy (%) under diverse unseen untargeted and targeted attacks with the perturbation budget

\epsilon=1/255

. Targeted attacks with random target selection are denoted by ATT T , and those with least-likely target selection are denoted by ATT TL .

Method	ImageNet		ImageNet-R		ImageNet-S
Method	Clean	Robust	Clean	Robust	Clean	Robust
TeCoA	63.50	41.29	55.62	37.25	34.41	22.92
PMG-AFT	53.57	35.05	59.33	41.08	35.38	24.07
TGA-ZSR	70.96	52.80	57.61	40.74	38.50	27.49
AGFT (Ours)	60.11	44.95	61.70	45.21	39.58	29.63

Table 7: Out-of-distribution performance on ImageNet variants under PGD-20 attack with perturbation budget

\epsilon=1/255

$1/\tau$	100	140	180	220
$\gamma=1.0$	60.97 / 45.07	59.78 / 45.60	58.96 / 45.81	58.36 / 45.97
$\gamma=0.8$	61.65 / 44.94	60.49 / 45.87	59.53 / 46.19	58.79 / 46.25
$\gamma=0.6$	62.05 / 44.72	61.31 / 46.11	60.39 / 46.57	59.55 / 46.79
$\gamma=0.4$	62.20 / 43.17	61.90 / 45.41	61.35 / 46.57	60.68 / 47.08
$\gamma=0.2$	63.23 / 33.29	62.19 / 40.41	62.00 / 43.00	61.77 / 44.38

Table 8: Ablation study of hyperparameters. The original pre-trained CLIP setting corresponds to

(\gamma=1.0,\tau=1/100)

. Entries are reported as clean accuracy / robust accuracy.

Figure 3: Trade-off between robust and clean accuracy across different methods. Each marker type denotes one method, and each point corresponds to a different trade-off configuration.

Method	clean images			adversarial examples
Method	$conf$	$s_{max}$	$s_{mean}$	$conf$	$s_{max}$	$s_{mean}$
CLIP	63.51	29.19	21.66	-	-	-
TeCoA	54.15	28.11	20.82	50.68	27.85	20.85
PMG-AFT	53.81	28.77	22.02	49.85	28.53	22.04
TGA-ZSR	59.98	26.64	18.13	58.15	26.48	18.16
AGFT (Ours)	23.72	17.96	15.02	20.75	17.75	15.04

Table 9: Empirical study of distribution consistency calibration.

conf

s_{max}

, and

s_{mean}

represent the prediction confidence, and the maximum and mean cosine similarity, respectively (

\times 100

	TeCoA	PMG-AFT	TGA-ZSR	AGFT (Ours)
Forward	1	3	3	2
Backward	1	2	2	1
Time	23.45	34.36	39.78	25.96

Table 10: Computational cost comparison: numbers of forward and backward propagation, and training time per epoch (min).

Panel of Figure 4 — Figure 4: T-SNE visualization of 7 image categories. (a) The original pre-trained CLIP evaluated on clean images. Both (b) AGFT and (c) TeCoA are evaluated on adversarial examples.

Method	Learning Rate	Clean Acc.	Robust Acc.
PMG-AFT	$lr=$ 5e-6	59.77	40.86
	$lr=$ 1e-5 (ours)	59.23	41.66
	$lr=$ 2e-5	57.67	42.08
	$lr=$ 4e-5	55.11	42.04
	$lr=$ 5e-5 (original)	53.87	41.54
TGA-ZSR	$lr=$ 1e-5	56.41	39.13

Table 11: Average accuracy (%) under different hyperparameters for PMG-AFT and TGA-ZSR.

Method	Caltech101	Caltech256	CIFAR10	CIFAR100	DTD	EuroSAT	FGVC	Flower102	Food101	ImageNet	OxfordPet	PCAM	Stanf.Cars	STL10	SUN397	Average
TeCoA	70.92	60.81	59.24	33.03	21.32	13.54	4.10	29.13	27.57	40.21	61.33	14.17	13.80	83.31	31.11	37.57
PMG-AFT	77.48	62.21	64.84	37.45	24.41	14.71	5.00	30.92	38.29	33.44	64.79	18.77	19.18	86.31	30.43	40.55
TGA-ZSR	71.60	63.82	62.90	35.85	20.26	12.86	4.16	26.45	27.90	52.39	64.98	30.61	14.67	84.92	30.56	40.26
AGFT (Ours)	79.14	64.66	69.66	41.49	21.86	13.52	5.60	30.69	38.86	39.81	67.24	33.57	21.36	87.69	30.73	43.06

Table 12: Zero-shot robust accuracy (%). Adversarial examples are generated by C&W attack with the perturbation budget

\epsilon=1/255

Method	Caltech101	Caltech256	CIFAR10	CIFAR100	DTD	EuroSAT	FGVC	Flower102	Food101	ImageNet	OxfordPet	PCAM	Stanf.Cars	STL10	SUN397	Average
TeCoA	70.62	60.22	58.95	32.80	21.32	12.75	3.68	28.53	26.63	39.04	60.81	14.56	12.06	83.13	30.08	37.01
PMG-AFT	77.06	61.89	64.75	37.07	24.41	14.11	4.58	30.61	37.16	32.83	64.41	19.22	17.42	86.17	29.67	40.09
TGA-ZSR	71.37	63.46	62.62	35.49	20.31	12.68	3.74	26.16	27.19	51.56	64.79	31.04	13.41	84.83	29.79	39.90
AGFT (Ours)	79.02	64.53	69.73	41.37	22.02	12.99	5.18	30.58	38.18	39.34	67.24	33.71	20.11	87.81	30.30	42.81

Table 13: Zero-shot robust accuracy (%). Adversarial examples are generated by AutoAttack with the perturbation budget

\epsilon=1/255

Method	Caltech101	Caltech256	CIFAR10	CIFAR100	DTD	EuroSAT	FGVC	Flower102	Food101	ImageNet	OxfordPet	PCAM	Stanf.Cars	STL10	SUN397	Average
TeCoA	76.71	66.03	60.51	35.74	25.37	15.57	5.48	36.27	30.62	48.86	66.15	19.43	19.72	88.03	36.20	42.05
PMG-AFT	83.34	69.42	71.82	43.52	26.38	10.29	7.28	38.01	42.18	44.87	69.04	48.44	24.26	90.92	36.47	47.08
TGA-ZSR	75.91	67.11	65.31	38.58	24.30	17.39	5.39	27.25	29.04	60.35	68.25	33.93	16.00	89.86	35.80	43.63
AGFT (Ours)	85.60	71.75	73.40	48.39	25.10	8.25	9.77	42.61	45.55	51.56	73.67	49.44	31.98	91.41	39.12	49.84

Table 14: Zero-shot robust accuracy (%) on ViT-B/16 . Adversarial examples are generated by PGD-20 attack with

{\epsilon=1/255}

Method	Caltech101	Caltech256	CIFAR10	CIFAR100	DTD	EuroSAT	FGVC	Flower102	Food101	ImageNet	OxfordPet	PCAM	Stanf.Cars	STL10	SUN397	Average
TeCoA	58.61	50.81	34.97	18.23	22.92	8.77	3.53	17.85	16.07	33.12	58.82	32.45	11.81	76.71	20.29	31.00
PMG-AFT	63.58	52.15	31.02	10.70	20.37	10.67	4.88	21.00	24.02	31.73	59.28	14.03	16.84	68.02	25.66	30.26
TGA-ZSR	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
AGFT (Ours)	74.09	59.43	35.31	14.69	22.44	9.84	5.15	25.36	36.72	39.00	64.93	16.72	22.37	76.07	28.19	34.69

Table 15: Zero-shot robust accuracy (%) on RN50

\bf{\times}

4 . Adversarial examples are generated by PGD-20 attack with

{\epsilon=1/255}

VLM Robustness

Vision-language model robustness studies whether models remain reliable under visual, textual, and distribution shifts, adversarial inputs, and safety challenges.

See all 443 papers

Hierarchically Robust Zero-shot Vision-language Models
Apr 20, 2026Junhao Dong, Yifei Zhang, Hao Zhu +2Hierarchical Classification VLM Robustness
1Nanyang Technological University · 2CFAR, IHPC, A*STAR · 3Northwest Polytechnical University +2
Robustifying Vision-Language Models via Test-Time Prompt Adaptation
Jul 10, 2026Xingyu Zhu, Huanshen Wu, Shuo Wang +4VLM Robustness VLM Adaptation
National University of Singapore · University of Science and Technology of China · The Hong Kong University of Science and Technology
Complementary Text-Guided Attention for Zero-Shot Adversarial Robustness
Mar 19, 2026Lu Yu, Haiyang Zhang, Changsheng XuAttention-Based Adversarial Attacks Contrastive Learning for VLMs
School of Computer Science and Engineering, Tianjin University of Technology, 300384, China · State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, 100190, China · School of Artificial Intelligence, University of the Chinese Academy of Sciences, Beijing, 100049, China

Apr 20, 2026J/K move · Enter open · S save

Hierarchically Robust Zero-shot Vision-language Models

Junhao Dong, Yifei Zhang, Hao Zhu, Yew-Soon Ong+1

1Nanyang Technological University · 2CFAR, IHPC, A*STAR · 3Northwest Polytechnical University+2

Hierarchical Classification VLM Robustness Zero-Shot Learning Fine-Tuning VLM Adaptation

Open paper AskPDF arXiv

Vision-Language Models (VLMs) can perform zero-shot classification but are susceptible to adversarial attacks. While robust fine-tuning improves their robustness, existing approaches align fixed text embeddings with an image embedding, sacrificing natural performance and robustness. A robustness degradation also occurs when a model faces adversarial attacks targeting superclasses (parent classes, e.g., mammal) in addition to their base (leaf) classes (e.g., cat). Thus, to enhance adversarial robustness and leverage the inherent hierarchical properties of class space, we propose a novel adversarial fine-tuning framework based on hierarchical embeddings and several levels of adversarially robust alignment of image-text modalities. Additional mechanisms place visual embeddings at the desired depth of hierarchy, and we provide a theoretical connection between the depth of embedding in the hierarchy and the maximum viable margin size. Our model naturally realizes several margin sizes, boosting generalization of adversaries for robustification. As various trees with different parent labels can share the same leaf labels, we also consider aligning over multiple trees to boost semantic variety. Experiments across several datasets are performed.

Abstract

VLM Robustness

Explore similar work

Hierarchically Robust Zero-shot Vision-language Models

Robustifying Vision-Language Models via Test-Time Prompt Adaptation

Complementary Text-Guided Attention for Zero-Shot Adversarial Robustness