Learning What Not to Learn: Adversarial Disentangled Prompt Tuning for Robust Vision-Language Models

Yang Chen; Zhan Zhuang; Yanbin Wei; Zebin Chen; Hua Liu; Yu Zhang

cs.CVAug 18, 2026

Learning What Not to Learn: Adversarial Disentangled Prompt Tuning for Robust Vision-Language Models

Authors: Yang Chen, Zhan Zhuang, Yanbin Wei, Zebin Chen, Hua Liu, Yu Zhang

Organizations: Southern University of Science and Technology Shenzhen, China · City University of Hong Kong Hong Kong, China · Hong Kong University of Science and Technology Hong Kong, China

Abstract

While adversarial prompt tuning can enhance robustness of vision-language models efficiently, we find that existing methods aggravate robust generalization overfitting on seen classes, leading to a rapid degradation in performance against adversarial examples of unseen classes as training progresses. We empirically identify that this degradation stems from the tendency of the model to learn pseudo-robust features (i.e., non-generalizable shortcuts). To mitigate this, we propose ADAPT (Adversarial Disentangled Prompt Tuning), a robust prompt tuning framework following the philosophy of ``Learning What Not to Learn''. Specifically, ADAPT uses a dual-prompt mechanism with a target prompt and a pool of decoy prompts. During training, the decoy prompts are guided to entrap diverse pseudo-robust features, while the target prompt is constrained to be orthogonal to the decoys in the embedding space to learn robust features. By disentangling the robust features from the pseudo-robust features, ADAPT effectively prevents robust generalization overfitting. We further provide an analysis showing that the orthogonal loss bounds the effect of shifts in pseudo-robust features on unseen classes, yielding a testing error guarantee. Empirically, extensive experiments demonstrate that ADAPT substantially improves the robustness of the target prompt on unseen classes. The code is available at https://github.com/cheny02/ADAPT-ACMMM2026.

Figures & tables

Figure 1. Illustration of robust generalization overfitting. Adversarial prompt tuning on Caltech101 with CLIP ViT-B/32 under PGD- $l_{\infty}$ attack ( $\epsilon=4/255$ ). Top: Robust loss remains stable on base (seen) classes but increases on new (unseen) classes as training progresses. Bottom: The robust–clean loss gap remains nearly constant on base classes but grows steadily on new classes. A two-panel line graph illustrates loss metric changes over 200 training epochs. The vertical axis of the top panel ranges from 0.0 to 3.0. A solid line representing robust training loss on base classes steadily decreases. Dashed lines for clean test loss on base and new classes remain near zero at the bottom. The solid line for robust test loss on base classes is stable around 1.3. In contrast, the solid line for robust test loss on new classes rises continuously from 1.6 to 2.2. The vertical axis of the bottom panel plots the loss gap. The gap between robust and clean test loss on base classes is a horizontal line near 0.9. The gap for new classes climbs steadily from 0.9 to 1.3 and deviates significantly.

Figure 2. Toy example (bird vs. non-bird) . While standard adversarial prompt tuning (i.e., black arrow) gets trapped in non-generalizable shortcuts, our ADAPT employs a decoy prompt (i.e., red arrow) to absorb these shortcuts, guiding the target prompt (i.e., blue arrow) to learn robust features. A two-panel diagram demonstrates how the proposed method avoids shortcut learning in a 2D feature space. The left panel is a scatter plot with semantic features on the horizontal axis and shortcut features on the vertical axis. It shows four data clusters with two for base classes and two for new classes. These clusters are separated by a large vertical gap representing the shortcut direction. The right panel displays three vector arrows originating from the center coordinates. A black arrow representing standard adversarial training points almost entirely along the vertical shortcut axis. A red arrow for the decoy prompt also points vertically to absorb this shortcut. A blue arrow for the target prompt points along the horizontal semantic direction and is orthogonal to the shortcut.

Figure 3. Overview of the proposed ADAPT method. ADAPT operates on a dual-prompt mechanism consisting of a Target Prompt and a Decoy Prompt (uniformly sampled from a pool). The decoy prompt serves as a shortcut trap, optimized to absorb pseudo-robust features via $\mathcal{L}_{\mathrm{ce}}^{\mathrm{d}}$ while deviating from the general semantic anchor via $\mathcal{L}_{\mathrm{dis}}$ . Conversely, the target prompt is guided to learn robust features by enforcing orthogonality to the decoy prompt via $\mathcal{L}_{\mathrm{orth}}$ and maintaining semantic consistency with the general semantic anchor via $\mathcal{L}_{\mathrm{sem}}$ . A block diagram details the dual-prompt architecture and latent space loss constraints. On the left side, a sampled decoy prompt, a hand-crafted prompt, and a target prompt pass in parallel through a frozen text encoder. This generates three embeddings, including a decoy feature, a general semantic anchor, and a target feature. A dissimilarity loss pushes the decoy feature away from the semantic anchor. A semantic loss pulls the target feature toward the semantic anchor. An orthogonal loss forces the target feature away from the decoy feature. On the right side, an adversarial image passes through a frozen image encoder to generate an image feature. Independent cross-entropy losses then optimize this image feature against the decoy text feature and the target text feature.

Table 1. Performance under the setting of adversarial base-to-new generalization. All methods (except zero-shot TeCoA) are trained with 16 instances per base class. Here

\epsilon

equals

4/255

. Results of

\epsilon=1/255

are provided in Appendix C.2 .

Figure 4. Average performance on the 11 datasets under the adversarial few-shot classification setting. Since accuracies are comparable across methods, we plot only robustness and ‘H’ for clarity.

\epsilon=4/255

. Full results are provided in Appendix C.3 . A line chart compares robustness and harmonic mean scores across one, four, and sixteen-shot training scales. The horizontal axis uses a logarithmic scale for the amount of training data per class. The vertical axis measures percentage scores from 10 to 30. The data form two distinct upward-sloping bands. The upper red band represents the harmonic mean, while the lower blue band represents robustness. Within both regions, the proposed multi-modal ADAPT method occupies the top position and achieves the highest scores at all data scales. It is closely followed by the single-modal ADAPT method, FAP, and APT. Two isolated star markers on the far left vertical axis indicate the zero-shot TeCoA baseline performance. These markers are positioned significantly lower than the starting points of the proposed methods at one shot.

Method	Average over 10 datasets
	Base		New				$\mathrm{H}_{\mathrm{a}}$
	Acc.	PGD	Acc.	PGD	CW	AA	$\mathrm{H}_{\mathrm{a}}$
TeCoA	40.54	13.59	38.08	14.90	14.59	13.18	17.86
APT	54.57	26.15	26.63	9.83	9.45	8.56	14.33
ADAPT	55.76	26.26	35.03	14.41	13.64	12.59	19.56
FAP	54.07	26.70	36.36	14.98	13.32	12.19	19.52

Table 2. Average performance under the adversarial base-to-new generalization setting with various attacks.

\epsilon=4/255

Methods	Base		New		$\mathrm{H}_{\mathrm{b}}$
Methods	Acc	Rob	Acc	Rob	$\mathrm{H}_{\mathrm{b}}$
TeCoA	44.51	14.62	42.16	16.14	22.66
APT	58.62	27.40	30.13	10.96	22.47
ADAPT	60.04	27.48	38.98	15.65	28.05
w/o $\mathcal{P}_{\mathrm{d}}$	59.95	27.55	36.43	14.75	26.99
w/o $\mathcal{L}_{\mathrm{dis}}$	59.80	27.49	38.25	14.99	27.41

Table 3. Ablation study of the proposed ADAPT framework. We report the average performance under the adversarial base-to-new generalization setting across 11 datasets.

\epsilon=4/255

Panel of Figure 5 — Figure 5. The average performance of ADAPT on all datasets w.r.t. hyperparameters (i.e., $K$ , $\alpha$ , $\beta$ , and $\lambda$ ) under the adversarial base-to-new generalization setting. A series of four line charts analyzes the impact of adjusting four hyperparameters on average accuracy. Chart A plots parameter K on a horizontal axis from 1 to 4 and shows accuracy peaking at K equals 2. Chart B plots parameter alpha from 0 to 0.1 with a gentle peak around 0.01. Chart C plots parameter beta from 0 to 40, where accuracy rises sharply from 0 to 30 before leveling off. Chart D plots parameter lambda from 0 to 0.5 and reaches a distinct peak at 0.05. Each subchart includes a light gray rectangular shaded background that visually highlights the parameter range producing the highest and most stable accuracy results.

Panel of Figure 6 — Figure 6. t-SNE visualization of prompt embeddings on Caltech101. The target prompts cluster around general semantic anchors, while the decoy prompts lie in a separate subspace. Two side-by-side scatter plots display the 2D spatial clustering of prompt embeddings for base and new classes. The left chart for base classes and the right chart for new classes exhibit identical spatial layout patterns. On the left side of each chart, the target prompt markers cluster tightly and overlap directly with markers representing general semantic anchors. On the far right side of each chart, two distinct sets of decoy prompts form their own independent clusters. These decoy clusters are spatially isolated from the semantic anchors on the left. The two decoy clusters also maintain a large physical distance from each other, indicating they have converged into separate sub-spaces without interference.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Table A1 . Performance under the setting of adversarial base-to-new generalization on different VLM backbones. ‘hc’ denotes hand-crafted prompt, ‘tp’ denotes textual prompt tuning, and ‘mmp’ denotes multi-modal prompt tuning.

Method	Base		New
Method	Acc.	Rob.	Acc.	Rob.
APT	69.78	35.40	39.11	13.69
ADAPT	71.79	35.57	47.13	19.09

Appendix

Table A2 . Adversarial base-to-new generalization with OpenCLIP ViT-B/32 pretrained on LAION2B. Results are averaged over 10 datasets with

\epsilon=4/255

Table A3 . Performance of various methods under the adversarial base-to-new generalization setting. All methods (except TeCoA) are trained with 16 instances per base class. ‘

\mathrm{H}_{\mathrm{b}}

’ denotes the harmonic mean accuracy between

\mathrm{Acc.}_{\mathrm{Base}}

\mathrm{Rob.}_{\mathrm{Base}}

\mathrm{Acc.}_{\mathrm{New}}

, and

\mathrm{Rob.}_{\mathrm{New}}

\epsilon=1/255

Dataset	$\epsilon=4/255$	Zero-shot	1 shot				4 shot				16 shot
Dataset	$\epsilon=4/255$	TeCoA	APT	FAP	ADAPT	ADAPT ${}_{\text{M}}$	APT	FAP	ADAPT	ADAPT ${}_{\text{M}}$	APT	FAP	ADAPT	ADAPT ${}_{\text{M}}$
Average	Acc.	33.67	33.20	37.78	38.10	38.06	43.32	44.42	44.89	45.09	51.08	51.11	52.68	49.89
	Rob.	10.79	11.79	13.19	12.69	13.53	14.66	15.31	15.80	17.09	20.26	19.65	20.51	21.56
	H	16.34	17.40	19.55	19.04	19.96	21.91	22.77	23.37	24.78	29.02	28.39	29.53	30.11
ImageNet	Acc.	40.11	37.88	36.95	39.85	36.62	39.13	38.94	40.84	37.60	41.06	40.45	42.27	38.13
	Rob.	10.14	10.79	11.41	10.96	11.05	11.49	11.86	11.15	11.83	12.02	12.03	11.64	12.19

Appendix

Table A4 . The performance on the 11 datasets for different shots under the adversarial few-shot classification setting.

Method	Source			Target
	ImageNet			ImageNet-V2			ImageNet-Sketch			ImageNet-A			ImageNet-R			Average
	Acc.	Rob.	H	Acc.	Rob.	H	Acc.	Rob.	H	Acc.	Rob.	H	Acc.	Rob.	H	Acc.	Rob.	H
TeCoA	40.11	10.14	16.19	33.11	7.49	12.22	17.59	7.24	10.26	4.04	0.28	0.52	37.52	12.51	18.76	23.07	6.88	10.60
APT	41.06	12.02	18.60	33.67	9.09	14.32	18.22	7.87	10.99	4.19	0.36	0.66	37.04	13.29	19.56	23.28	7.65	11.52
FAP	40.32	12.06	18.57	32.81	9.17	14.33	16.42	7.11	9.92	3.87	0.43	0.77	36.04	13.55	19.70	22.29	7.57	11.30
ADAPT	41.34	12.02	18.62	34.24	8.92	14.15	18.56	8.22	11.39	4.21	0.36	0.66	38.65	13.87	20.41	23.92	7.84	11.81

Appendix

Table A5 . Performance under adversarial domain generalization setting.

\epsilon=4/255

16 shot	Method	Source	Target
16 shot	Method	ImageNet	Caltech101	OxfordPets	StanfordCars	Flowers102	Food101	FGVCAircraft	SUN397	DTD	EuroSAT	UCF101	Avg.
Acc.	TeCoA	40.11	78.78	66.26	10.32	30.13	23.52	7.17	33.24	24.29	19.56	36.98	33.03
	APT	41.06	77.40	64.19	11.21	27.65	23.77	4.17	31.85	23.52	16.98	33.73	31.45
	FAP	40.32	79.19	63.67	9.64	30.25	22.82	5.43	31.92	23.23	16.79	32.25	31.52
	ADAPT	41.34	79.80	65.85	10.55	29.60	23.28	5.91	32.57	24.05	17.40	34.97	32.40
	ADAPT ${}_{\text{M}}$	38.20	78.86	60.92	9.23	29.92	20.34	6.30	28.90	22.28	22.74	30.43	30.99

Appendix

Table A6 . Performance under adversarial cross-dataset generalization setting.

\epsilon=4/255

Method	Train.Memory	Train.time (h)	Infer.time (ms)	Base		New		$\mathrm{H}_{\mathrm{b}}$
Method	Train.Memory	Train.time (h)	Infer.time (ms)	Acc	Rob	Acc	Rob	$\mathrm{H}_{\mathrm{b}}$
APT	11454M	5.83	1.70	44.54	13.78	38.56	12.47	19.89
ADAPT	27470M	9.65	1.72	47.07	13.49	43.36	13.71	20.90
FAP	19808M	11.22	1.97	43.29	13.89	37.43	13.05	20.16
ADAPT ${}_{\text{M}}$	27530M	13.81	1.69	41.67	13.88	39.47	14.49	21.01

Appendix

Table A7 . Computational cost and performance comparison on ImageNet under the adversarial base-to-new generalization setting with

\epsilon=4/255

	$\mathrm{Acc.}_{\mathrm{Base}}$	$\mathrm{Rob.}_{\mathrm{Base}}$	$\mathrm{Acc.}_{\mathrm{New}}$	$\mathrm{Rob.}_{\mathrm{New}}$
$z_{U}$	37.27	15.74	22.45	10.65
$z_{S}$	19.93	9.78	30.56	14.98

Appendix

Table A8 . Image-side projection analysis on DTD under adversarial base-to-new generalization.

Figure A1 . Robust generalization overfitting under different learning rates and training epochs. .

	Base			New
	Acc.	PGD	DGAA	Acc.	PGD	DGAA
ADAPT	60.04	27.48	23.62	38.98	15.65	13.66

Appendix

Table A9 . Robustness of ADAPT under the decoy-guided adaptive attack. Results are averaged over 11 datasets with

\epsilon=4/255

Prompt Tuning

Prompt tuning adapts pretrained models by learning compact prompts or prefixes instead of updating all model parameters.

See all 76 papers

Robustifying Vision-Language Models via Test-Time Prompt Adaptation
Jul 10, 2026Xingyu Zhu, Huanshen Wu, Shuo Wang +4VLM Robustness VLM Adaptation
National University of Singapore · University of Science and Technology of China · The Hong Kong University of Science and Technology
Towards Fine-Grained Robustness: Attention-Guided Test-Time Prompt Tuning for Vision-Language Models
May 19, 2026Jia-Wei Hai, Yijun Wang, Xiu-Shen WeiVLM Robustness Adversarial Attacks on VLMs
School of Computer Science and Engineering, and Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications, Southeast University, China. · School of Intelligence Science and Engineering, Southeast University, China.
SS-TPT: Stability and Suitability-Guided Test-Time Prompt Tuning for Adversarially Robust Vision-Language Models
Jun 5, 2026Sunoh Kim, Daeho UmVLM Robustness VLM Adaptation
Dankook University, Yongin, South Korea · University of Seoul, Seoul, South Korea

Jul 10, 2026J/K move · Enter open · S save

Robustifying Vision-Language Models via Test-Time Prompt Adaptation

Xingyu Zhu, Huanshen Wu, Shuo Wang, Beier Zhu+3

National University of Singapore · University of Science and Technology of China · The Hong Kong University of Science and Technology

VLM Robustness VLM Adaptation Test-Time Adaptation of VLMs Adversarial Robustness Test-Time Prompt Tuning

Open paper AskPDF arXiv

Pre-trained Vision-Language Models (VLMs) such as CLIP achieve strong zero-shot generalization, but their performance degrades sharply under adversarial perturbations. Existing test-time adaptation methods typically rely on sample-level confidence heuristics, overlooking the intrinsic distributional structure of the data. This sample-centric approach limits robustness, as it fails to distinguish confident adversarial mispredictions from true semantic consistency. In this work, we observe that adversarial distortion is structurally brittle: while holistic representations are corrupted, semantic integrity is often preserved in the distribution of augmented views. Motivated by this insight, we propose RITA, a Robust test-tIme prompt-TAdaptation framework that shifts from sample-level estimates to distribution-level alignment. Specifically, RITA employs optimal transport to align the distribution of augmented visual features with textual prototypes, mitigating adversarial outliers and rectifying cross-modal semantic misalignment. Furthermore, we introduce a dynamic cache to progressively accumulate reliable cues from the test stream for online refinement. Extensive experiments demonstrate that RITA significantly improves adversarial robustness without compromising clean accuracy.

Abstract

Prompt Tuning

Explore similar work

Robustifying Vision-Language Models via Test-Time Prompt Adaptation

Towards Fine-Grained Robustness: Attention-Guided Test-Time Prompt Tuning for Vision-Language Models

SS-TPT: Stability and Suitability-Guided Test-Time Prompt Tuning for Adversarially Robust Vision-Language Models