Textual Response-Aware Guidance for Zero-Shot Adversarial Robustness of Vision-Language Models

Yubo Cui; Xianchao Guan; Zheng Zhang

cs.CVOct 29, 2026

Textual Response-Aware Guidance for Zero-Shot Adversarial Robustness of Vision-Language Models

Authors: Yubo Cui, Xianchao Guan, Zheng Zhang

Organizations: Harbin Institute of Technology, Shenzhen, China · Shenzhen Loop Area Institute, Shenzhen, China

Abstract

Vision-language models (VLMs) exhibit strong zero-shot generalization but remain vulnerable to adversarial attacks. Existing adversarial fine-tuning methods often use pre-trained model guidance to preserve image--text alignment while improving robustness. Our analysis of visual representation drift reveals complementary limitations of two representative forms of guidance. Matching visual features provides broad coverage by constraining all directions equally, but lacks semantic emphasis. Matching prediction distributions prioritizes semantically relevant directions, but remains limited by the candidate texts and prediction probabilities. These limitations can hinder preservation of pre-trained image--text relationships and degrade zero-shot performance. To address these issues, we propose Textual Response-Aware Guidance (TRAG), which constrains visual representation drift using the pre-trained text encoder's responses to local variations in input embeddings. Specifically, TRAG constructs anchor-level constraints from the response directions and magnitudes around semantic anchors, then aggregates them using pre-trained image--anchor relevance, yielding an image-adaptive constraint. Extensive experiments across architectures and tasks demonstrate state-of-the-art clean and robust performance. Further analysis shows that TRAG helps preserve pre-trained image--text relationships.

Figures & tables

Panel of Figure 1 — Figure 1: Directional constraints on visual representation drift. Thicker lower arrows indicate stronger penalties. (a) Feature matching constrains drift equally in all directions. (b) Prediction distribution matching weights directions associated with differences between candidate textual representations using pre-trained prediction probabilities. (c) TRAG constructs anchor-level constraints from local responses of the pre-trained text encoder and aggregates them using pre-trained image–anchor relevance.

Figure 2: Geometric diagnostics of CLIP ViT-B/32 before fine-tuning. Bars (left) show similarity to downstream class geometry; lines (right) show directional coverage. The best fixed FM–DM mix is selected by mean similarity per dataset.

Figure 3: Overview of TRAG. We construct anchor-level constraints $\mathbf{W}_{j}$ from the text encoder’s local response directions and magnitudes under a fixed-budget criterion. Pre-trained image–anchor relevance on clean images weights their aggregation. After global trace normalization, $\mathbf{M}_{x}$ constrains visual representation drift during adversarial fine-tuning.

Method	Caltech101	Caltech256	CIFAR10	CIFAR100	DTD	EuroSAT	FGVC	Flower102	Food101	ImageNet	ImageNet-R	ImageNet-S	OxfordPet	Stanf.Cars	STL10	SUN397	Average
CLIP	88.71	84.66	87.55	62.56	41.38	43.72	19.17	66.15	83.51	61.89	66.26	40.75	87.33	58.62	97.28	60.10	65.60
TeCoA	82.89	72.92	73.45	40.84	30.74	19.59	11.25	42.86	41.83	55.66	49.28	27.54	75.23	25.28	90.22	47.67	49.20
PMG-AFT	86.46	77.32	80.13	50.87	31.65	22.90	12.51	45.22	52.50	61.19	54.91	33.74	79.54	32.75	93.34	50.24	54.08
FARE	86.51	79.72	78.40	57.50	34.04	17.59	14.94	52.23	64.72	56.87	57.53	35.61	83.19	48.71	93.77	52.05	57.09
TGA-ZSR	82.84	76.13	80.75	49.01	28.72	14.10	9.78	43.92	45.16	70.53	57.33	38.16	77.82	25.74	93.19	47.88	52.57
QT-AFT	86.18	78.54	82.47	56.50	34.20	11.13	14.31	51.32	62.08	55.12	56.13	34.47	81.91	47.07	94.38	51.38	56.07

Table 1: Zero-shot classification with CLIP ViT-B/32. Models are fine-tuned on ImageNet with training/test budgets of

2/255

. Bold and underlined values mark the best and second-best fine-tuned results; Average spans all 16 datasets.

Method	Clean	Robust Accuracy
Method		1/255	2/255	3/255	4/255	Avg.
TeCoA	49.20	37.42	26.33	17.40	10.75	22.97
PMG-AFT	54.08	42.88	31.99	22.26	14.53	27.92
FARE	57.09	42.81	29.34	18.33	10.76	25.31
TGA-ZSR	52.57	40.50	29.32	19.47	11.69	25.24
QT-AFT	56.07	42.52	29.86	19.33	11.77	25.87

Table 2: Different test budgets. ViT-B/32 models are trained at

2/255

. Results are means over 16 datasets; Avg. further averages robust accuracy over all four listed test budgets.

Method	1/255		3/255		4/255
Method	Clean	Robust	Clean	Robust	Clean	Robust
TeCoA	56.31	38.43	42.87	18.87	37.45	13.95
PMG-AFT	59.58	42.72	46.29	22.63	42.01	17.59
FARE	61.90	38.66	52.46	22.80	47.74	17.88
TGA-ZSR	58.37	37.66	47.72	23.16	43.86	18.71
QT-AFT	61.03	39.55	52.06	23.40	48.00	18.66

Table 3: Different training budgets. Mean accuracy (%) over 16 datasets; ViT-B/32 uses matched training/test budgets.

Method	ViT-B/16		ViT-L/14		RN101
Method	Clean	Robust	Clean	Robust	Clean	Robust
TeCoA	53.38	30.18	64.12	39.99	31.65	16.45
PMG-AFT	55.40	30.83	63.85	43.40	47.01	27.69
FARE	60.28	32.69	69.06	45.03	51.87	20.26
TGA-ZSR	51.66	32.16	60.82	40.34	–	–
QT-AFT	60.05	33.35	69.05	45.36	45.46	21.64

Table 4: Different backbones. Mean accuracy (%) over 16 datasets; training and test budgets are

2/255

Method	Flickr30k								COCO
	Clean				Robust				Clean				Robust
	TR@1	TR@10	IR@1	IR@10	TR@1	TR@10	IR@1	IR@10	TR@1	TR@10	IR@1	IR@10	TR@1	TR@10	IR@1	IR@10
CLIP	82.60	99.10	62.08	92.02	0.30	1.40	0.16	1.08	53.18	85.14	33.32	68.95	0.10	0.36	0.04	0.30
TeCoA	69.20	92.90	55.96	87.44	45.00	75.40	34.44	71.26	41.32	76.00	28.02	64.09	21.90	52.80	15.09	43.73
PMG-AFT	76.00	95.50	59.44	89.16	54.70	83.20	42.20	77.08	45.58	79.24	30.68	66.93	29.72	62.22	19.10	51.55
FARE	78.70	96.90	61.62	90.42	52.50	84.80	39.34	76.24	45.04	80.62	31.28	67.16	25.66	58.90	17.41	48.40

Table 5: Zero-shot image–text retrieval after CC3M fine-tuning. TR/IR denote image-to-text/text-to-image recall (%). Robust results use image PGD at

2/255

. Bold/underlined values mark the best/second-best fine-tuned results.

Method	BERT-Attack		Sep-Attack		Co-Attack
Method	TR@m	IR@m	TR@m	IR@m	TR@m	IR@m
TeCoA	41.85	34.74	26.05	21.79	51.65	38.76
PMG-AFT	47.25	37.54	31.40	26.68	58.05	43.27
FARE	53.15	41.34	33.50	26.54	55.35	41.74
TGA-ZSR	46.85	38.12	28.50	24.34	53.65	41.65
QT-AFT	51.45	42.50	33.80	28.69	58.85	45.46

Table 6: Text and multimodal attacks. Flickr30k results for CLIP ViT-B/16 fine-tuned on CC3M. TR@m/IR@m average R@1 and R@10 in each direction.

Method	COCO			Flickr30k			TextVQA			VQAv2
Method	Clean	APGD	Ensemble	Clean	APGD	Ensemble	Clean	APGD	Ensemble	Clean	APGD	Ensemble
TeCoA	101.94	53.76	40.41	59.07	32.23	19.05	24.48	12.80	13.26	65.60	39.76	40.92
PMG-AFT	93.12	51.50	35.78	50.69	32.27	18.28	24.08	13.14	12.60	62.76	36.06	39.06
FARE	115.67	92.47	70.05	72.88	54.34	39.44	32.40	19.82	20.90	67.86	48.14	50.74
TGA-ZSR	100.14	57.60	42.12	53.69	33.80	20.86	23.68	14.18	13.34	63.10	38.88	41.54
QT-AFT	117.91	91.65	71.31	74.27	54.49	37.92	32.26	19.44	19.82	68.36	47.98	49.90

Table 7: Robustness transfer to LLaVA-1.5-7B. ImageNet-adapted ViT-L/14 encoders are evaluated using CIDEr for captioning and accuracy (%) for VQA. Both attacks use an

\ell_{\infty}

budget of

2/255

in pixel space.

Constraint	Clean	Robust
FM	58.69	34.44
DM	54.81	34.91
Quadratic mix ( $\lambda=5$ )	56.33	35.08
Quadratic mix ( $\lambda=10$ )	57.33	35.05
FM + DM	56.91	35.01
TRAG (Ours)	59.60	35.09

Table 8: Pre-trained constraint ablations. Mean accuracy (%) over 16 datasets with ViT-B/32 at

2/255

. Representative quadratic mixtures and a direct FM+DM loss are compared under the same adversarial fine-tuning pipeline and task objective.

Figure 4: Directional weight spectra before fine-tuning. Eigenvalues are sorted per image and averaged over 1,000 ImageNet images after normalizing each matrix to trace 512. The inset enlarges the first 50 ranks to show the leading directional weights.

Guidance	Feature drift		Relation error $\downarrow$		Retention $\uparrow$
Guidance	C	A	C	A	A
FM	0.184	0.214	8.11	13.36	41.84
DM	0.332	0.353	7.71	11.07	41.97
Quadratic mix	0.202	0.229	7.47	11.90	42.00
AGFT	0.605	0.619	9.05	11.28	41.30
FARE	0.166	0.200	7.39	14.90	37.13

Table 9: Post-fine-tuning relationship preservation. Means over 15 zero-shot datasets, with 64 images per dataset. C/A: clean/adversarial inputs. Relation error is scaled by

10^{5}

. Retention (%) is measured on images classified correctly by pre-trained CLIP before adaptation and averaged across tasks.

VLM Robustness

Vision-language model robustness studies whether models remain reliable under visual, textual, and distribution shifts, adversarial inputs, and safety challenges.

See all 455 papers

AGFT: Alignment-Guided Fine-Tuning for Zero-Shot Adversarial Robustness of Vision-Language Models
Mar 31, 2026Yubo Cui, Xianchao Guan, Zijun Xiong +1VLM Robustness Vision-Language Alignment
Harbin Institute of Technology, Shenzhen, China · Shenzhen Loop Area Institute, Shenzhen, China
Hierarchically Robust Zero-shot Vision-language Models
Apr 20, 2026Junhao Dong, Yifei Zhang, Hao Zhu +2Hierarchical Classification VLM Robustness
1Nanyang Technological University · 2CFAR, IHPC, A*STAR · 3Northwest Polytechnical University +2
Robustifying Vision-Language Models via Test-Time Prompt Adaptation
Jul 10, 2026Xingyu Zhu, Huanshen Wu, Shuo Wang +4VLM Robustness VLM Adaptation
National University of Singapore · University of Science and Technology of China · The Hong Kong University of Science and Technology

Mar 31, 2026J/K move · Enter open · S save

AGFT: Alignment-Guided Fine-Tuning for Zero-Shot Adversarial Robustness of Vision-Language Models

Yubo Cui, Xianchao Guan, Zijun Xiong, Zheng Zhang

Harbin Institute of Technology, Shenzhen, China · Shenzhen Loop Area Institute, Shenzhen, China

VLM Robustness Vision-Language Alignment

Open paper AskPDF arXiv

Pre-trained vision-language models (VLMs) exhibit strong zero-shot generalization but remain vulnerable to adversarial perturbations. Existing classification-guided adversarial fine-tuning methods often disrupt pre-trained cross-modal alignment, weakening visual-textual correspondence and degrading zero-shot performance. In this paper, we propose an Alignment-Guided Fine-Tuning (AGFT) framework that enhances zero-shot adversarial robustness while preserving the cross-modal semantic structure. Unlike label-based methods that rely on hard labels and fail to maintain the relative relationships between image and text, AGFT leverages the probabilistic predictions of the original model for text-guided adversarial training, which aligns adversarial visual features with textual embeddings via soft alignment distributions, improving zero-shot adversarial robustness. To address structural discrepancies introduced by fine-tuning, we introduce a distribution consistency calibration mechanism that adjusts the robust model output to match a temperature-scaled version of the pre-trained model predictions. Extensive experiments across multiple zero-shot benchmarks demonstrate that AGFT outperforms state-of-the-art methods while significantly improving zero-shot adversarial robustness.

Abstract

VLM Robustness

Explore similar work

AGFT: Alignment-Guided Fine-Tuning for Zero-Shot Adversarial Robustness of Vision-Language Models

Hierarchically Robust Zero-shot Vision-language Models

Robustifying Vision-Language Models via Test-Time Prompt Adaptation