Organizations: Southern University of Science and Technology Shenzhen, China · City University of Hong Kong Hong Kong, China · Hong Kong University of Science and Technology Hong Kong, China
While adversarial prompt tuning can enhance robustness of vision-language models efficiently, we find that existing methods aggravate robust generalization overfitting on seen classes, leading to a rapid degradation in performance against adversarial examples of unseen classes as training progresses. We empirically identify that this degradation stems from the tendency of the model to learn pseudo-robust features (i.e., non-generalizable shortcuts). To mitigate this, we propose ADAPT (Adversarial Disentangled Prompt Tuning), a robust prompt tuning framework following the philosophy of ``Learning What Not to Learn''. Specifically, ADAPT uses a dual-prompt mechanism with a target prompt and a pool of decoy prompts. During training, the decoy prompts are guided to entrap diverse pseudo-robust features, while the target prompt is constrained to be orthogonal to the decoys in the embedding space to learn robust features. By disentangling the robust features from the pseudo-robust features, ADAPT effectively prevents robust generalization overfitting. We further provide an analysis showing that the orthogonal loss bounds the effect of shifts in pseudo-robust features on unseen classes, yielding a testing error guarantee. Empirically, extensive experiments demonstrate that ADAPT substantially improves the robustness of the target prompt on unseen classes. The code is available at https://github.com/cheny02/ADAPT-ACMMM2026.
Figures & tables
Figure 1. Illustration of robust generalization overfitting. Adversarial prompt tuning on Caltech101 with CLIP ViT-B/32 under PGD- l∞ attack ( ϵ=4/255 ). Top: Robust loss remains stable on base (seen) classes but increases on new (unseen) classes as training progresses. Bottom: The robust–clean loss gap remains nearly constant on base classes but grows steadily on new classes. A two-panel line graph illustrates loss metric changes over 200 training epochs. The vertical axis of the top panel ranges from 0.0 to 3.0. A solid line representing robust training loss on base classes steadily decreases. Dashed lines for clean test loss on base and new classes remain near zero at the bottom. The solid line for robust test loss on base classes is stable around 1.3. In contrast, the solid line for robust test loss on new classes rises continuously from 1.6 to 2.2. The vertical axis of the bottom panel plots the loss gap. The gap between robust and clean test loss on base classes is a horizontal line near 0.9. The gap for new classes climbs steadily from 0.9 to 1.3 and deviates significantly.
Figure 2. Toy example (bird vs. non-bird) . While standard adversarial prompt tuning (i.e., black arrow) gets trapped in non-generalizable shortcuts, our ADAPT employs a decoy prompt (i.e., red arrow) to absorb these shortcuts, guiding the target prompt (i.e., blue arrow) to learn robust features. A two-panel diagram demonstrates how the proposed method avoids shortcut learning in a 2D feature space. The left panel is a scatter plot with semantic features on the horizontal axis and shortcut features on the vertical axis. It shows four data clusters with two for base classes and two for new classes. These clusters are separated by a large vertical gap representing the shortcut direction. The right panel displays three vector arrows originating from the center coordinates. A black arrow representing standard adversarial training points almost entirely along the vertical shortcut axis. A red arrow for the decoy prompt also points vertically to absorb this shortcut. A blue arrow for the target prompt points along the horizontal semantic direction and is orthogonal to the shortcut.
Figure 3. Overview of the proposed ADAPT method. ADAPT operates on a dual-prompt mechanism consisting of a Target Prompt and a Decoy Prompt (uniformly sampled from a pool). The decoy prompt serves as a shortcut trap, optimized to absorb pseudo-robust features via Lced while deviating from the general semantic anchor via Ldis . Conversely, the target prompt is guided to learn robust features by enforcing orthogonality to the decoy prompt via Lorth and maintaining semantic consistency with the general semantic anchor via Lsem . A block diagram details the dual-prompt architecture and latent space loss constraints. On the left side, a sampled decoy prompt, a hand-crafted prompt, and a target prompt pass in parallel through a frozen text encoder. This generates three embeddings, including a decoy feature, a general semantic anchor, and a target feature. A dissimilarity loss pushes the decoy feature away from the semantic anchor. A semantic loss pulls the target feature toward the semantic anchor. An orthogonal loss forces the target feature away from the decoy feature. On the right side, an adversarial image passes through a frozen image encoder to generate an image feature. Independent cross-entropy losses then optimize this image feature against the decoy text feature and the target text feature.
Table 1. Performance under the setting of adversarial base-to-new generalization. All methods (except zero-shot TeCoA) are trained with 16 instances per base class. Here ϵ equals 4/255 . Results of ϵ=1/255 are provided in Appendix C.2 .
Figure 4. Average performance on the 11 datasets under the adversarial few-shot classification setting. Since accuracies are comparable across methods, we plot only robustness and ‘H’ for clarity. ϵ=4/255 . Full results are provided in Appendix C.3 . A line chart compares robustness and harmonic mean scores across one, four, and sixteen-shot training scales. The horizontal axis uses a logarithmic scale for the amount of training data per class. The vertical axis measures percentage scores from 10 to 30. The data form two distinct upward-sloping bands. The upper red band represents the harmonic mean, while the lower blue band represents robustness. Within both regions, the proposed multi-modal ADAPT method occupies the top position and achieves the highest scores at all data scales. It is closely followed by the single-modal ADAPT method, FAP, and APT. Two isolated star markers on the far left vertical axis indicate the zero-shot TeCoA baseline performance. These markers are positioned significantly lower than the starting points of the proposed methods at one shot.
Method
Average over 10 datasets
Base
New
Ha
Acc.
PGD
Acc.
PGD
CW
AA
TeCoA
40.54
13.59
38.08
14.90
14.59
13.18
17.86
APT
54.57
26.15
26.63
9.83
9.45
8.56
14.33
ADAPT
55.76
26.26
35.03
14.41
13.64
12.59
19.56
FAP
54.07
26.70
36.36
14.98
13.32
12.19
19.52
Table 2. Average performance under the adversarial base-to-new generalization setting with various attacks. ϵ=4/255 .
Methods
Base
New
Hb
Acc
Rob
Acc
Rob
TeCoA
44.51
14.62
42.16
16.14
22.66
APT
58.62
27.40
30.13
10.96
22.47
ADAPT
60.04
27.48
38.98
15.65
28.05
w/o Pd
59.95
27.55
36.43
14.75
26.99
w/o Ldis
59.80
27.49
38.25
14.99
27.41
Table 3. Ablation study of the proposed ADAPT framework. We report the average performance under the adversarial base-to-new generalization setting across 11 datasets. ϵ=4/255 .
Figure 5. The average performance of ADAPT on all datasets w.r.t. hyperparameters (i.e., K , α , β , and λ ) under the adversarial base-to-new generalization setting. A series of four line charts analyzes the impact of adjusting four hyperparameters on average accuracy. Chart A plots parameter K on a horizontal axis from 1 to 4 and shows accuracy peaking at K equals 2. Chart B plots parameter alpha from 0 to 0.1 with a gentle peak around 0.01. Chart C plots parameter beta from 0 to 40, where accuracy rises sharply from 0 to 30 before leveling off. Chart D plots parameter lambda from 0 to 0.5 and reaches a distinct peak at 0.05. Each subchart includes a light gray rectangular shaded background that visually highlights the parameter range producing the highest and most stable accuracy results.
Figure 6. t-SNE visualization of prompt embeddings on Caltech101. The target prompts cluster around general semantic anchors, while the decoy prompts lie in a separate subspace. Two side-by-side scatter plots display the 2D spatial clustering of prompt embeddings for base and new classes. The left chart for base classes and the right chart for new classes exhibit identical spatial layout patterns. On the left side of each chart, the target prompt markers cluster tightly and overlap directly with markers representing general semantic anchors. On the far right side of each chart, two distinct sets of decoy prompts form their own independent clusters. These decoy clusters are spatially isolated from the semantic anchors on the left. The two decoy clusters also maintain a large physical distance from each other, indicating they have converged into separate sub-spaces without interference.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Table A1 . Performance under the setting of adversarial base-to-new generalization on different VLM backbones. ‘hc’ denotes hand-crafted prompt, ‘tp’ denotes textual prompt tuning, and ‘mmp’ denotes multi-modal prompt tuning.
Method
Base
New
Acc.
Rob.
Acc.
Rob.
APT
69.78
35.40
39.11
13.69
ADAPT
71.79
35.57
47.13
19.09
Appendix
Table A2 . Adversarial base-to-new generalization with OpenCLIP ViT-B/32 pretrained on LAION2B. Results are averaged over 10 datasets with ϵ=4/255 .
Table A3 . Performance of various methods under the adversarial base-to-new generalization setting. All methods (except TeCoA) are trained with 16 instances per base class. ‘ Hb ’ denotes the harmonic mean accuracy between Acc.Base , Rob.Base , Acc.New , and Rob.New . ϵ=1/255 .
Dataset
ϵ=4/255
Zero-shot
1 shot
4 shot
16 shot
TeCoA
APT
FAP
ADAPT
ADAPT M
APT
FAP
ADAPT
ADAPT M
APT
FAP
ADAPT
ADAPT M
Average
Acc.
33.67
33.20
37.78
38.10
38.06
43.32
44.42
44.89
45.09
51.08
51.11
52.68
49.89
Rob.
10.79
11.79
13.19
12.69
13.53
14.66
15.31
15.80
17.09
20.26
19.65
20.51
21.56
H
16.34
17.40
19.55
19.04
19.96
21.91
22.77
23.37
24.78
29.02
28.39
29.53
30.11
ImageNet
Acc.
40.11
37.88
36.95
39.85
36.62
39.13
38.94
40.84
37.60
41.06
40.45
42.27
38.13
Rob.
10.14
10.79
11.41
10.96
11.05
11.49
11.86
11.15
11.83
12.02
12.03
11.64
12.19
Appendix
Table A4 . The performance on the 11 datasets for different shots under the adversarial few-shot classification setting.
Pre-trained Vision-Language Models (VLMs) such as CLIP achieve strong zero-shot generalization, but their performance degrades sharply under adversarial perturbations. Existing test-time adaptation methods typically rely on sample-level confidence heuristics, overlooking the intrinsic distributional structure of the data. This sample-centric approach limits robustness, as it fails to distinguish confident adversarial mispredictions from true semantic consistency. In this work, we observe that adversarial distortion is structurally brittle: while holistic representations are corrupted, semantic integrity is often preserved in the distribution of augmented views. Motivated by this insight, we propose RITA, a Robust test-tIme prompt-TAdaptation framework that shifts from sample-level estimates to distribution-level alignment. Specifically, RITA employs optimal transport to align the distribution of augmented visual features with textual prototypes, mitigating adversarial outliers and rectifying cross-modal semantic misalignment. Furthermore, we introduce a dynamic cache to progressively accumulate reliable cues from the test stream for online refinement. Extensive experiments demonstrate that RITA significantly improves adversarial robustness without compromising clean accuracy.
Xingyu Zhu, Huanshen Wu, Shuo Wang +4
National University of Singapore · University of Science and Technology of China · The Hong Kong University of Science and Technology
Vision-Language Models (VLMs), such as CLIP, have achieved significant zero-shot performance on downstream tasks with various fine-tuning adaptation methods. However, recent studies have proven that adversarial attacks can significantly degrade the inference ability of VLMs, posing substantial risks to their practical applications. Prevalent test-time adaptation methods typically rely on multi-view augmentation to implement various fine-tuning strategies, which struggle to identify semantic information and are prone to destroying discriminative regions in fine-grained scenarios. To address these limitations, we propose Attention-Guided Test-Time Prompt Tuning (A-TPT), a semantics-preserving method designed for test-time adaptation. We first refine the gradient attention rollout mechanism to identify semantically meaningful regions surviving under adversarial attacks. Furthermore, we leverage them to guide the spatially varying augmentation intensities and multi-view ensemble for prompt tuning and inference. Extensive experiments demonstrate that A-TPT outperforms existing test-time adaptation methods on both adversarial and clean data. Codes are available at https://github.com/SEU-VIPGroup/A-TPT .
Jia-Wei Hai, Yijun Wang, Xiu-Shen Wei
School of Computer Science and Engineering, and Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications, Southeast University, China. · School of Intelligence Science and Engineering, Southeast University, China.
Vision-language models (VLMs) such as CLIP achieve strong zero-shot recognition but remain highly fragile under adversarial perturbations. Recent test-time adaptation defenses improve robustness by leveraging many augmented views, but this leads to impractical slowdown and a clear robustness-throughput trade-off. To address this challenge, we present Stability and Suitability-guided Test-time Prompt Tuning (SS-TPT), evaluating the quality of each augmented view via two complementary scores: (1) stability, measuring prediction invariance to weak augmentations, and (2) suitability, measuring feature-space density among views. These stability and suitability (SS) scores guide both adaptation and inference through an SS-guided consistency loss and an SS-weighted prediction, amplifying trustworthy views while suppressing corrupted ones. Extensive experiments demonstrate that SS-TPT significantly outperforms prior state-of-the-art methods, achieving superior robustness-throughput trade-offs across diverse datasets and varying numbers of views, thereby demonstrating both strong practicality and generality. Our code is available at https://github.com/sunoh-kim/SS-TPT.
Sunoh Kim, Daeho Um
Dankook University, Yongin, South Korea · University of Seoul, Seoul, South Korea