FVG-PT: Adaptive Foreground View-Guided Prompt Tuning for Vision-Language Models

Haoyang Li; Liang Wang; Siyu Zhou; Jiacheng Sun; Jing Jiang; Chao Wang; Guodong Long; Yan Peng

cs.CVMar 9, 2026

FVG-PT: Adaptive Foreground View-Guided Prompt Tuning for Vision-Language Models

Authors: Haoyang Li, Liang Wang, Siyu Zhou, Jiacheng Sun, Jing Jiang, Chao Wang, Guodong Long, Yan Peng

Organizations: University of Technology Sydney, NSW 2007, Australia · Shanghai University, Shanghai 200444, China

Abstract

CLIP-based prompt tuning enables pretrained Vision-Language Models (VLMs) to efficiently adapt to downstream tasks. Although existing studies have made significant progress, they pay limited attention to changes in the internal attention representations of VLMs during the tuning process. In this paper, we attribute the failure modes of prompt tuning predictions to shifts in foreground attention of the visual encoder, and propose Foreground View-Guided Prompt Tuning (FVG-PT), an adaptive plug-and-play foreground attention guidance module, to alleviate the shifts. Concretely, FVG-PT introduces a learnable Foreground Reliability Gate to automatically enhance the foreground view quality, applies a Foreground Distillation Compensation module to guide visual attention toward the foreground, and further introduces a Prior Calibration module to mitigate generalization degradation caused by excessive focus on the foreground. Experiments on multiple backbone models and datasets show the effectiveness and compatibility of FVG-PT. Codes are available at: https://github.com/JREion/FVG-PT

Figures & tables

Figure 1: Comparison of visual encoder attention maps generated by Grad-CAM [ 37 ] for the same image under (a) original CLIP [ 34 ] , (b) CoOp [ 55 ] , and (c) our FVG-PT . In the bad cases of (a) and (b), attention deviates from the foreground view, where FVG-PT effectively suppresses this shift and leads to a correct prediction.

Figure 2: Framework of our proposed FVG-PT . As a plug-and-play method, in (a) tuning stage, FVG-PT obtains the foreground view $x^{\mathrm{fg}}$ of image $x$ and fine-tunes the (b) Foreground Reliability Gate to learn a foreground trust score $r$ . Meanwhile, Foreground Distillation Compensation module inserts adapters after image-text alignment of frozen backbone model to guide visual attention toward the foreground. In parallel, independent Prior Calibration fine-tunes the (c) Backbone Reliability Gate on new branch (indicated by dashed lines ) to balance the tuned model and the CLIP prior.

Figure 3: Inference stage of FVG-PT on (a) base branch and (b) new branch. The design of Prior Calibration (Sec. 3.4 ) enables full decoupling between the two branches during inference, addressing the BNT problem.

	Average over 11			ImageNet			Caltech101			Food101
Method	Base	New	HM	Base	New	HM	Base	New	HM	Base	New	HM
CoOp [ 55 ]	80.94	70.03	75.09	75.96	68.93	72.27	97.81	94.65	96.20	90.14	91.48	90.81
+FVG-PT	82.40	73.62	77.76	76.65	70.20	73.28	98.13	94.76	96.42	89.43	91.74	90.57
KgCoOp [ 49 ]	80.28	73.75	76.88	76.18	70.26	73.10	98.19	94.21	96.16	90.59	91.65	91.12
+FVG-PT	82.06	74.51	78.10	77.14	70.28	73.55	98.26	94.76	96.48	90.07	91.63	90.84
PromptSRC [ 17 ]	80.81	73.98	77.24	77.00	70.60	73.66	98.00	94.32	96.12	90.80	91.74	91.27

Table 1: Base-to-new generalization performance of 4 backbone models w/ or w/o our FVG-PT on 11 datasets. FVG-PT achieves performance improvements on both base-class fine-tuning and new-class generalization across all backbones.

	Source	Target
Method	ImgNet	Avg.	Caltech	Pets	Cars	Flower	Food	Aircraft	SUN	DTD	EuroSAT	UCF
CoOp	71.55	64.57	93.67	90.11	64.23	70.00	86.52	22.11	65.54	43.97	41.49	68.07
+FVG-PT	72.09	65.72	94.16	89.75	65.56	71.66	86.77	24.75	66.54	45.39	43.77	68.89
KgCoOp	71.20	65.41	93.96	90.49	65.22	71.05	86.37	23.85	66.86	46.39	41.41	68.49
+FVG-PT	72.17	65.82	94.04	90.27	65.58	71.25	86.55	24.93	66.93	46.63	43.24	68.81
PromptSRC	72.43	64.76	92.13	90.90	65.34	68.57	86.12	23.46	66.80	45.04	41.25	67.94

Table 2: Cross-Dataset Transfer performance of 4 backbone models w/ or w/o our FVG-PT on 11 datasets. FVG-PT achieves performance improvements on both the ImageNet source and multiple target datasets across all backbones.

Panel of Figure 4 — Figure 4: Base-to-new performance comparison across (a) visual attention-related prompt tuning methods and (b) different weights of the FDC distillation loss $\lambda_{d}$ .

	FRG	FDC		PC		Average over 11
	$\mathcal{L}_{\text{FRG}}$	$\mathcal{L}_{\text{CE}}$	$\mathcal{L}_{\text{dist}}$	$\mathcal{L}_{\text{CE}}$	$\mathcal{L}_{\text{KL}}$	Base	New	HM	$\Delta$
	✗	✗	✗	✗	✗	80.94	70.03	75.09	(baseline)
(1)	✗	✓	✓	✗	✗	81.04	63.14	70.98	$-$ 4.11
(2)	✗	✗	✗	✓	✓	80.94	73.62	77.11	$+$ 2.02
(3)	✓	✓	✓	✗	✗	82.40	63.45	71.69	$-$ 3.40
(4)	✗	✓	✓	✓	✓	81.04	73.62	77.15	$+$ 2.06

Table 3: Ablation study of components in FVG-PT on base-to-new tasks. Detailed ablation of losses and segmentation module are in Supplementary Material 0.C.3 .

Table 8

	FRG indicators			BRG indicators		Average over 11
	Distribution	Similarity	Geometric	Distribution	Similarity	Base	New	HM	$\Delta$
	✓	✓	✓	✓	✓	82.40	73.62	77.76	(full)
(1)	✗	✓	✓	✓	✓	82.18	73.62	77.66	$-$ 0.10
(2)	✓	✗	✓	✓	✓	82.23	73.62	77.69	$-$ 0.07
(3)	✓	✓	✗	✓	✓	82.32	73.62	77.73	$-$ 0.03
(4)	✓	✓	✓	✗	✓	82.40	73.53	77.71	$-$ 0.05

Table 6: Ablation study of indicators in FVG-PT on base-to-new tasks. Detailed ablation of losses and segmentation module are in Supplementary Material 0.C.3 .

Stage	Learnable Params	Tuning FPS $\uparrow$	GPU Memory $\downarrow$	Inference FPS $\uparrow$	HM $\uparrow$
MMRL	4.99 M	87.52	2009.9 MB	610.6	80.10
FVG-PT	0.13 M	153.36	808.4 MB	847.6	80.75

Table 7: Computation cost of MMRL [ 8 ] backbone and FVG-PT on Flowers102 dataset.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: Trends of the foreground shift index and base-class accuracy for CLIP, fine-tuned CoOp, and our FVG-PT on the (a) Caltech101 and (b) Flowers102 datasets.

Figure 6: Failure cases of (a) the original CLIP compared with (b) CoOp and (c) our FVG-PT . Attention maps of visual encoders are generated by Grad-CAM [ 37 ] .

Figure 7: Failure cases of (b) CoOp compared with (a) the original CLIP and (c) our FVG-PT . Attention maps of visual encoders are generated by Grad-CAM [ 37 ] .

Params	CoOp	KgCoOp	PromptSRC	MMRL
Depth of text prompt	1	1	9	-
Depth of visual prompt	-	-	9	-
Text token length	4	4	4	5
Visual token length	-	-	4	5
Layers that insert learnable tokens	1	1	1-9	6-12
Optimizer	SGD	SGD	SGD	AdamW

Appendix

Table 8: Training settings of backbones on base-to-new tasks.

Dataset	SEEM Textual Prompt
ImageNet	The whole [CLASS] in the middle.
Caltech101	A huge [CLASS].
Food101	The whole [CLASS] in the middle.
StanfordCars	A huge and whole car.
OxfordPets	The pet in the middle.
Flowers102	The flower in the middle.

Appendix

Table 9: Textual prompts used in SEEM model for generating foreground masks.

Figure 8: Illustration of the foreground mask generated by SEEM [ 58 ] , where (a) shows the raw image and (b) presents the corresponding foreground mask visualization.

Method	Avg.	ImageNet	Caltech101	Food101	StanfordCars	OxfordPets
CoOp	75.09	72.27	96.20	90.81	69.43	96.07
+FVG-PT	77.16 (±0.20)	73.28 (±0.02)	96.42 (±0.02)	90.57 (±0.01)	74.66 (±0.22)	95.81 (±0.14)
KgCoOp	76.88	73.10	96.16	91.12	72.44	96.25
+FVG-PT	78.10 (±0.15)	73.55 (±0.04)	96.48 (±0.03)	90.84 (±0.03)	75.28 (±0.17)	96.28 (±0.04)
PromptSRC	77.24	73.66	96.12	91.27	73.73	96.47
+FVG-PT	77.89 (±0.12)	73.82 (±0.01)	96.32 (±0.02)	91.25 (±0.03)	75.03 (±0.07)	96.28 (±0.03)

Appendix

Table 10: Error bar analysis of FVG-PT on 4 backbone models.

		Avg. over 11
Method	Model	Base	New	HM
zero-shot	CLIP [ 34 ]	69.34	74.22	71.70
prompt optimization	CoCoOp [ 54 ]	80.47	71.69	75.83
	ProDA [ 28 ]	81.56	72.30	76.65
	MaPLe [ 16 ]	82.28	75.14	78.55
knowledge integration	KAPT [ 14 ]	78.41	70.45	74.22

Appendix

Table 11: Comparison with additional prompt tuning baselines.

	FRG	FDC		PC		Average over 11
	$\mathcal{L}_{\text{FRG}}$	$\mathcal{L}_{\text{CE}}$	$\mathcal{L}_{\text{dist}}$	$\mathcal{L}_{\text{CE}}$	$\mathcal{L}_{\text{KL}}$	Base	New	HM	$\Delta$
	✗	✗	✗	✗	✗	80.94	70.03	75.09	(baseline)
(6)	✓	✗	✓	✓	✓	81.01	73.62	77.14	+ 2.05
(7)	✓	✓	✗	✓	✓	81.56	73.62	77.38	+ 2.29
(8)	✓	✓	✓	✗	✓	82.40	72.58	77.18	+ 2.09
(9)	✓	✓	✓	✓	✗	82.40	71.11	76.33	+ 1.24

Appendix

Table 12: Ablation study of losses in FVG-PT on base-to-new tasks.

		Average over 11
Method	Branch	Base	New	HM	$\Delta$
CoOp	-	80.94	70.03	75.09	(baseline)
+FVG-PT	visual + textual	82.40	73.62	77.76	+2.67
+FVG-PT	visual only	81.80	73.62	77.49	+2.40

Appendix

Table 13: Ablation study of the branches of FDC adapters (Sec. 3.3 ) in FVG-PT on base-to-new tasks.

		Average over 11
Foundation CLIP	Method	Base	New	HM	$\Delta$
ViT-B/16	CoOp	80.94	70.03	75.09
ViT-B/16	+FVG-PT	82.40	73.62	77.76	+2.67
ViT-L/14	CoOp	83.40	75.05	79.01
ViT-L/14	+FVG-PT	83.98	80.39	82.15	+3.14

Appendix

Table 14: Ablation study of the ViT encoders of foundation CLIP used in CoOp [ 55 ] and FVG-PT on base-to-new tasks.

		EuroSAT
Method	epoch	Base	New	HM	$\Delta$
CoOp	10	88.43	45.87	60.41
+FVG-PT	10	72.62	58.15	64.58	+4.18
	15	86.98	58.21	69.74	+9.34
	20	89.76	58.28	70.67	+10.27

Appendix

Table 15: Failure case study of FVG-PT on EuroSAT dataset.

Figure 9: Data efficiency analysis using different proportions of the 11 datasets as the training set on base-to-new tasks.

Explore similar work

Towards Fine-Grained Robustness: Attention-Guided Test-Time Prompt Tuning for Vision-Language Models
May 19, 2026Jia-Wei Hai, Yijun Wang, Xiu-Shen WeiVLM Robustness Adversarial Attacks on VLMs
School of Computer Science and Engineering, and Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications, Southeast University, China. · School of Intelligence Science and Engineering, Southeast University, China.
Towards Calibrating Prompt Tuning of Vision-Language Models
Feb 22, 2026Ashshak Sharifdeen, Fahad Shamshad, Muhammad Akhtar Munir +6VLM Calibration
Mohamed bin Zayed University of AI · University of Colombo · Michigan State University
SS-TPT: Stability and Suitability-Guided Test-Time Prompt Tuning for Adversarially Robust Vision-Language Models
Jun 5, 2026Sunoh Kim, Daeho UmVLM Robustness VLM Adaptation
Dankook University, Yongin, South Korea · University of Seoul, Seoul, South Korea

May 19, 2026J/K move · Enter open · S save

Towards Fine-Grained Robustness: Attention-Guided Test-Time Prompt Tuning for Vision-Language Models

Jia-Wei Hai, Yijun Wang, Xiu-Shen Wei

School of Computer Science and Engineering, and Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications, Southeast University, China. · School of Intelligence Science and Engineering, Southeast University, China.

VLM Robustness Adversarial Attacks on VLMs Test-Time Adaptation Test-Time Adaptation of VLMs Test-Time Prompt Tuning

Open paper AskPDF arXiv

Vision-Language Models (VLMs), such as CLIP, have achieved significant zero-shot performance on downstream tasks with various fine-tuning adaptation methods. However, recent studies have proven that adversarial attacks can significantly degrade the inference ability of VLMs, posing substantial risks to their practical applications. Prevalent test-time adaptation methods typically rely on multi-view augmentation to implement various fine-tuning strategies, which struggle to identify semantic information and are prone to destroying discriminative regions in fine-grained scenarios. To address these limitations, we propose Attention-Guided Test-Time Prompt Tuning (A-TPT), a semantics-preserving method designed for test-time adaptation. We first refine the gradient attention rollout mechanism to identify semantically meaningful regions surviving under adversarial attacks. Furthermore, we leverage them to guide the spatially varying augmentation intensities and multi-view ensemble for prompt tuning and inference. Extensive experiments demonstrate that A-TPT outperforms existing test-time adaptation methods on both adversarial and clean data. Codes are available at https://github.com/SEU-VIPGroup/A-TPT .

Abstract

Explore similar work

Towards Fine-Grained Robustness: Attention-Guided Test-Time Prompt Tuning for Vision-Language Models

Towards Calibrating Prompt Tuning of Vision-Language Models

SS-TPT: Stability and Suitability-Guided Test-Time Prompt Tuning for Adversarially Robust Vision-Language Models