Vision-language models (VLMs) exhibit strong zero-shot generalization but remain vulnerable to adversarial attacks. Existing adversarial fine-tuning methods often use pre-trained model guidance to preserve image--text alignment while improving robustness. Our analysis of visual representation drift reveals complementary limitations of two representative forms of guidance. Matching visual features provides broad coverage by constraining all directions equally, but lacks semantic emphasis. Matching prediction distributions prioritizes semantically relevant directions, but remains limited by the candidate texts and prediction probabilities. These limitations can hinder preservation of pre-trained image--text relationships and degrade zero-shot performance. To address these issues, we propose Textual Response-Aware Guidance (TRAG), which constrains visual representation drift using the pre-trained text encoder's responses to local variations in input embeddings. Specifically, TRAG constructs anchor-level constraints from the response directions and magnitudes around semantic anchors, then aggregates them using pre-trained image--anchor relevance, yielding an image-adaptive constraint. Extensive experiments across architectures and tasks demonstrate state-of-the-art clean and robust performance. Further analysis shows that TRAG helps preserve pre-trained image--text relationships.
Figures & tables
Figure 1: Directional constraints on visual representation drift. Thicker lower arrows indicate stronger penalties. (a) Feature matching constrains drift equally in all directions. (b) Prediction distribution matching weights directions associated with differences between candidate textual representations using pre-trained prediction probabilities. (c) TRAG constructs anchor-level constraints from local responses of the pre-trained text encoder and aggregates them using pre-trained image–anchor relevance.
Figure 2: Geometric diagnostics of CLIP ViT-B/32 before fine-tuning. Bars (left) show similarity to downstream class geometry; lines (right) show directional coverage. The best fixed FM–DM mix is selected by mean similarity per dataset.
Figure 3: Overview of TRAG. We construct anchor-level constraints Wj from the text encoder’s local response directions and magnitudes under a fixed-budget criterion. Pre-trained image–anchor relevance on clean images weights their aggregation. After global trace normalization, Mx constrains visual representation drift during adversarial fine-tuning.
Eval.
Method
Caltech101
Caltech256
CIFAR10
CIFAR100
DTD
EuroSAT
FGVC
Flower102
Food101
ImageNet
ImageNet-R
ImageNet-S
OxfordPet
Stanf.Cars
STL10
SUN397
Average
CLIP
88.71
84.66
87.55
62.56
41.38
43.72
19.17
66.15
83.51
61.89
66.26
40.75
87.33
58.62
97.28
60.10
65.60
TeCoA
82.89
72.92
73.45
40.84
30.74
19.59
11.25
42.86
41.83
55.66
49.28
27.54
75.23
25.28
90.22
47.67
49.20
PMG-AFT
86.46
77.32
80.13
50.87
31.65
22.90
12.51
45.22
52.50
61.19
54.91
33.74
79.54
32.75
93.34
50.24
54.08
FARE
86.51
79.72
78.40
57.50
34.04
17.59
14.94
52.23
64.72
56.87
57.53
35.61
83.19
48.71
93.77
52.05
57.09
TGA-ZSR
82.84
76.13
80.75
49.01
28.72
14.10
9.78
43.92
45.16
70.53
57.33
38.16
77.82
25.74
93.19
47.88
52.57
QT-AFT
86.18
78.54
82.47
56.50
34.20
11.13
14.31
51.32
62.08
55.12
56.13
34.47
81.91
47.07
94.38
51.38
56.07
Table 1: Zero-shot classification with CLIP ViT-B/32. Models are fine-tuned on ImageNet with training/test budgets of 2/255 . Bold and underlined values mark the best and second-best fine-tuned results; Average spans all 16 datasets.
Method
Clean
Robust Accuracy
1/255
2/255
3/255
4/255
Avg.
TeCoA
49.20
37.42
26.33
17.40
10.75
22.97
PMG-AFT
54.08
42.88
31.99
22.26
14.53
27.92
FARE
57.09
42.81
29.34
18.33
10.76
25.31
TGA-ZSR
52.57
40.50
29.32
19.47
11.69
25.24
QT-AFT
56.07
42.52
29.86
19.33
11.77
25.87
Table 2: Different test budgets. ViT-B/32 models are trained at 2/255 . Results are means over 16 datasets; Avg. further averages robust accuracy over all four listed test budgets.
Method
1/255
3/255
4/255
Clean
Robust
Clean
Robust
Clean
Robust
TeCoA
56.31
38.43
42.87
18.87
37.45
13.95
PMG-AFT
59.58
42.72
46.29
22.63
42.01
17.59
FARE
61.90
38.66
52.46
22.80
47.74
17.88
TGA-ZSR
58.37
37.66
47.72
23.16
43.86
18.71
QT-AFT
61.03
39.55
52.06
23.40
48.00
18.66
Table 3: Different training budgets. Mean accuracy (%) over 16 datasets; ViT-B/32 uses matched training/test budgets.
Method
ViT-B/16
ViT-L/14
RN101
Clean
Robust
Clean
Robust
Clean
Robust
TeCoA
53.38
30.18
64.12
39.99
31.65
16.45
PMG-AFT
55.40
30.83
63.85
43.40
47.01
27.69
FARE
60.28
32.69
69.06
45.03
51.87
20.26
TGA-ZSR
51.66
32.16
60.82
40.34
–
–
QT-AFT
60.05
33.35
69.05
45.36
45.46
21.64
Table 4: Different backbones. Mean accuracy (%) over 16 datasets; training and test budgets are 2/255 .
Model
Method
Flickr30k
COCO
Clean
Robust
Clean
Robust
TR@1
TR@10
IR@1
IR@10
TR@1
TR@10
IR@1
IR@10
TR@1
TR@10
IR@1
IR@10
TR@1
TR@10
IR@1
IR@10
CLIP
82.60
99.10
62.08
92.02
0.30
1.40
0.16
1.08
53.18
85.14
33.32
68.95
0.10
0.36
0.04
0.30
TeCoA
69.20
92.90
55.96
87.44
45.00
75.40
34.44
71.26
41.32
76.00
28.02
64.09
21.90
52.80
15.09
43.73
PMG-AFT
76.00
95.50
59.44
89.16
54.70
83.20
42.20
77.08
45.58
79.24
30.68
66.93
29.72
62.22
19.10
51.55
FARE
78.70
96.90
61.62
90.42
52.50
84.80
39.34
76.24
45.04
80.62
31.28
67.16
25.66
58.90
17.41
48.40
Table 5: Zero-shot image–text retrieval after CC3M fine-tuning. TR/IR denote image-to-text/text-to-image recall (%). Robust results use image PGD at 2/255 . Bold/underlined values mark the best/second-best fine-tuned results.
Method
BERT-Attack
Sep-Attack
Co-Attack
TR@m
IR@m
TR@m
IR@m
TR@m
IR@m
TeCoA
41.85
34.74
26.05
21.79
51.65
38.76
PMG-AFT
47.25
37.54
31.40
26.68
58.05
43.27
FARE
53.15
41.34
33.50
26.54
55.35
41.74
TGA-ZSR
46.85
38.12
28.50
24.34
53.65
41.65
QT-AFT
51.45
42.50
33.80
28.69
58.85
45.46
Table 6: Text and multimodal attacks. Flickr30k results for CLIP ViT-B/16 fine-tuned on CC3M. TR@m/IR@m average R@1 and R@10 in each direction.
Method
COCO
Flickr30k
TextVQA
VQAv2
Clean
APGD
Ensemble
Clean
APGD
Ensemble
Clean
APGD
Ensemble
Clean
APGD
Ensemble
TeCoA
101.94
53.76
40.41
59.07
32.23
19.05
24.48
12.80
13.26
65.60
39.76
40.92
PMG-AFT
93.12
51.50
35.78
50.69
32.27
18.28
24.08
13.14
12.60
62.76
36.06
39.06
FARE
115.67
92.47
70.05
72.88
54.34
39.44
32.40
19.82
20.90
67.86
48.14
50.74
TGA-ZSR
100.14
57.60
42.12
53.69
33.80
20.86
23.68
14.18
13.34
63.10
38.88
41.54
QT-AFT
117.91
91.65
71.31
74.27
54.49
37.92
32.26
19.44
19.82
68.36
47.98
49.90
Table 7: Robustness transfer to LLaVA-1.5-7B. ImageNet-adapted ViT-L/14 encoders are evaluated using CIDEr for captioning and accuracy (%) for VQA. Both attacks use an ℓ∞ budget of 2/255 in pixel space.
Constraint
Clean
Robust
FM
58.69
34.44
DM
54.81
34.91
Quadratic mix ( λ=5 )
56.33
35.08
Quadratic mix ( λ=10 )
57.33
35.05
FM + DM
56.91
35.01
TRAG (Ours)
59.60
35.09
Table 8: Pre-trained constraint ablations. Mean accuracy (%) over 16 datasets with ViT-B/32 at 2/255 . Representative quadratic mixtures and a direct FM+DM loss are compared under the same adversarial fine-tuning pipeline and task objective.
Figure 4: Directional weight spectra before fine-tuning. Eigenvalues are sorted per image and averaged over 1,000 ImageNet images after normalizing each matrix to trace 512. The inset enlarges the first 50 ranks to show the leading directional weights.
Guidance
Feature drift
Relation error ↓
Retention ↑
C
A
C
A
A
FM
0.184
0.214
8.11
13.36
41.84
DM
0.332
0.353
7.71
11.07
41.97
Quadratic mix
0.202
0.229
7.47
11.90
42.00
AGFT
0.605
0.619
9.05
11.28
41.30
FARE
0.166
0.200
7.39
14.90
37.13
Table 9: Post-fine-tuning relationship preservation. Means over 15 zero-shot datasets, with 64 images per dataset. C/A: clean/adversarial inputs. Relation error is scaled by 105 . Retention (%) is measured on images classified correctly by pre-trained CLIP before adaptation and averaged across tasks.
Vision-language model robustness studies whether models remain reliable under visual, textual, and distribution shifts, adversarial inputs, and safety challenges.
Pre-trained vision-language models (VLMs) exhibit strong zero-shot generalization but remain vulnerable to adversarial perturbations. Existing classification-guided adversarial fine-tuning methods often disrupt pre-trained cross-modal alignment, weakening visual-textual correspondence and degrading zero-shot performance. In this paper, we propose an Alignment-Guided Fine-Tuning (AGFT) framework that enhances zero-shot adversarial robustness while preserving the cross-modal semantic structure. Unlike label-based methods that rely on hard labels and fail to maintain the relative relationships between image and text, AGFT leverages the probabilistic predictions of the original model for text-guided adversarial training, which aligns adversarial visual features with textual embeddings via soft alignment distributions, improving zero-shot adversarial robustness. To address structural discrepancies introduced by fine-tuning, we introduce a distribution consistency calibration mechanism that adjusts the robust model output to match a temperature-scaled version of the pre-trained model predictions. Extensive experiments across multiple zero-shot benchmarks demonstrate that AGFT outperforms state-of-the-art methods while significantly improving zero-shot adversarial robustness.
Yubo Cui, Xianchao Guan, Zijun Xiong +1
Harbin Institute of Technology, Shenzhen, China · Shenzhen Loop Area Institute, Shenzhen, China
Vision-Language Models (VLMs) can perform zero-shot classification but are susceptible to adversarial attacks. While robust fine-tuning improves their robustness, existing approaches align fixed text embeddings with an image embedding, sacrificing natural performance and robustness. A robustness degradation also occurs when a model faces adversarial attacks targeting superclasses (parent classes, e.g., mammal) in addition to their base (leaf) classes (e.g., cat). Thus, to enhance adversarial robustness and leverage the inherent hierarchical properties of class space, we propose a novel adversarial fine-tuning framework based on hierarchical embeddings and several levels of adversarially robust alignment of image-text modalities. Additional mechanisms place visual embeddings at the desired depth of hierarchy, and we provide a theoretical connection between the depth of embedding in the hierarchy and the maximum viable margin size. Our model naturally realizes several margin sizes, boosting generalization of adversaries for robustification. As various trees with different parent labels can share the same leaf labels, we also consider aligning over multiple trees to boost semantic variety. Experiments across several datasets are performed.
Junhao Dong, Yifei Zhang, Hao Zhu +2
1Nanyang Technological University · 2CFAR, IHPC, A*STAR · 3Northwest Polytechnical University +2
Pre-trained Vision-Language Models (VLMs) such as CLIP achieve strong zero-shot generalization, but their performance degrades sharply under adversarial perturbations. Existing test-time adaptation methods typically rely on sample-level confidence heuristics, overlooking the intrinsic distributional structure of the data. This sample-centric approach limits robustness, as it fails to distinguish confident adversarial mispredictions from true semantic consistency. In this work, we observe that adversarial distortion is structurally brittle: while holistic representations are corrupted, semantic integrity is often preserved in the distribution of augmented views. Motivated by this insight, we propose RITA, a Robust test-tIme prompt-TAdaptation framework that shifts from sample-level estimates to distribution-level alignment. Specifically, RITA employs optimal transport to align the distribution of augmented visual features with textual prototypes, mitigating adversarial outliers and rectifying cross-modal semantic misalignment. Furthermore, we introduce a dynamic cache to progressively accumulate reliable cues from the test stream for online refinement. Extensive experiments demonstrate that RITA significantly improves adversarial robustness without compromising clean accuracy.
Xingyu Zhu, Huanshen Wu, Shuo Wang +4
National University of Singapore · University of Science and Technology of China · The Hong Kong University of Science and Technology