Visual prompting reimagined: The power of the Activation Prompts

Yihua Zhang; Hongkang Li; Yuguang Yao; Aochuan Chen; Shuai Zhang; Pin-Yu Chen; Meng Wang; Sijia Liu

cs.CVApr 7, 2026

Visual prompting reimagined: The power of the Activation Prompts

Authors: Yihua Zhang, Hongkang Li, Yuguang Yao, Aochuan Chen, Shuai Zhang, Pin-Yu Chen, Meng Wang, Sijia Liu

Organizations: Michigan State University · University of Pennsylvania · New Jersey Institute of Technology · IBM Research ·

Abstract

Visual prompting (VP) has emerged as a popular method to repurpose pretrained vision models for adaptation to downstream tasks. Unlike conventional model fine-tuning techniques, VP introduces a universal perturbation directly into the input data to facilitate task-specific fine-tuning rather than modifying model parameters. However, there exists a noticeable performance gap between VP and conventional fine-tuning methods, highlighting an unexplored realm in theory and practice to understand and advance the input-level VP to reduce its current performance gap. Towards this end, we introduce a generalized concept, termed activation prompt (AP), which extends the scope of the input-level VP by enabling universal perturbations to be applied to activation maps within the intermediate layers of the model. By using AP to revisit the problem of VP and employing it as an analytical tool, we demonstrate the intrinsic limitations of VP in both performance and efficiency, revealing why input-level prompting may lack effectiveness compared to AP, which exhibits a model-dependent layer preference. We show that AP is closely related to normalization tuning in convolutional neural networks and vision transformers, although each model type has distinct layer preferences for prompting. We also theoretically elucidate the rationale behind such a preference by analyzing global features across layers. Through extensive experiments across 29 datasets and various model architectures, we provide a comprehensive performance analysis of AP, comparing it with VP and parameter-efficient fine-tuning baselines. Our results demonstrate AP's superiority in both accuracy and efficiency, considering factors such as time, parameters, memory usage, and throughput.

Figures & tables

Figure 1: An illustration of the proposed activation prompt vs. the conventional input-based prompt.

Figure 2: Performance and efficiency comparison of VP , Norm-Tune and AP over different layers of ResNet-101 on OxfordPets.

Figure 3: Tunable parameter shape of Norm-Tune and AP . The same color indicates shared parameters across dimensions .

Figure 4: Layer preference of AP with different model architectures on OxfordPets ( Parkhi et al., 2012 ) . CNNs and ViTs exhibit opposite layer preferences. Results on more datasets are provided in Fig. A2 .

Figure 5: Features dissection to understand the layer effect. (Left) CKA-based feature similarity comparison between ViT-L/16 and ResNet-101. (Right) Average attention distances across all the heads and layers of ViT-L/16. Larger distance indicates global features.

Figure 6: Sample complexity study of AP .

	Benchmark	FGVC					VTAB									Others
Architecture		CUB200	StanfordDog	StanfordCars	NA-Birds	OxfordFlowers	CIFAR-100	Caltech-101	DTD	Flowers102	OxfordPets	SVHN	SUN397	Camelyon	EuroSAT	CIFAR-10	GTSRB	UCF101	Food101	Waterbirds	A verage
ResNet-101	$\bullet$ Full-Tune	88.91	90.13	87.76	84.45	99.98	92.24	99.13	79.97	99.81	90.49	97.14	79.19	91.13	99.13	97.24	97.68	88.32	82.72	96.69	91.69
	$\mathbf{\circ}$ Linear-Probe	63.76	86.63	49.62	52.09	82.01	73.87	90.58	61.35	93.14	91.17	66.30	54.51	83.36	95.84	92.25	79.64	71.03	64.31	88.11	75.76
	$\mathbf{\circ}$ Norm-Tune	66.39	87.59	67.64	56.72	66.50	82.58	91.32	63.53	92.85	89.81	95.26	54.56	84.42	96.14	93.90	96.43	69.44	72.54	88.95	79.81
	$\mathbf{\circ}$ VP	65.72	86.91	51.04	54.23	78.50	72.01	93.51	63.12	90.17	87.93	80.68	54.97	83.71	95.44	92.55	83.18	66.30	57.89	86.71	76.03
	$\mathbf{\circ}$ AP (ours)	69.42	87.79	59.06	58.31	85.14	76.94	94.85	69.80	95.13	91.31	87.30	56.83	84.91	97.21	94.08	90.43	73.96	68.12	88.13	80.45

Table 1: Performance comparison on 19 datasets from 3 benchmarks. Three parameter-efficient baselines (denoted by

\mathbf{\circ}

) are compared to AP , where the best performance is highlighted in bold . The most expensive Full-Tune (denoted by

\bullet

) serves as the performance reference. Each accuracy value is averaged over 5 trials, with variance omitted due to its negligible values (

\leq 0.3\%

). “Average” column averaged accuracies over all the datasets.

Method	Param. Efficiency	Train-Time Efficiency
Method	Parameter # (M) $\downarrow$	Memory Cost (G) $\downarrow$	Time Cost (s/epoch) $\downarrow$	Troughput (image/s) $\uparrow$
ResNet-101
Full-Tune	44.5	10.32	118	41.47
Linear-Probe	0.02	6.2	39	41.33
Norm-Tune	0.13	11.7	83	41.45
VP	0.12	12.2	72	40.59

Table 2: A performance overview. The efficiency analysis is on (ViT-L, CIFAR-10) with a batch size of 128, and time consumption is based on using a single RTX-A6000.

\uparrow

\downarrow

indicate whether a larger or smaller value is favored.

	Accuracy			Efficiency
	Full-Data				Train-Time Efficiency
	FGVC	VTAB	Others	Param. #	Memory	Time	Throughput
Number of tasks	5	9	5	-	-	-	-
Full-Tune	91.43	91.97	93.91	304.33	41.5	520	79.58
Linear-Probe	82.23	78.90	87.81	0.01	9.7	121	79.64
Bias	85.32	89.84	90.41	0.29	32.9	297	79.43

Table 3: Performance of AP and more SOTA PEFT methods on ViT-L/16. Settings follow Tab. 1 .

	Benchmark	VTAB-Natural							VTAB-Specialized				VTAB-Structured
Architecture		Caltech101	CIFAR-100	DTD	Flowers102	OxfordPets	Sun397	SVHN	Camelyon	EuroSAT	Resisc45	Retinopathy	Clevr-Count	Clevr-Dist	DMLab	dSpr-Loc	dSpr-Ori	KITTI-Dist	sNORB-Azim	sNORB-Elev	A verage
ResNet-101	$\bullet$ Full-Tune	89.99	45.17	63.78	84.29	89.82	41.09	67.79	84.92	74.57	91.37	74.14	58.11	60.99	43.61	67.05	40.45	78.34	33.64	36.38	64.50
	$\mathbf{\circ}$ Linear-Probe	83.87	39.13	53.09	70.89	85.15	28.14	43.44	78.65	69.43	90.78	69.31	35.91	36.48	35.75	34.76	19.51	65.68	16.91	23.39	51.12
	$\mathbf{\circ}$ Norm-Tune	85.61	35.78	47.71	56.64	78.10	10.10	68.67	83.16	61.10	90.50	72.44	37.54	55.24	40.04	60.89	20.33	65.54	24.86	25.96	53.70
	$\mathbf{\circ}$ VP	84.73	43.01	57.55	76.91	87.03	28.75	55.47	75.15	70.27	89.26	69.08	36.70	54.24	34.48	42.41	20.32	63.71	17.93	26.93	54.42
	$\mathbf{\circ}$ AP	87.49	39.80	63.62	81.44	88.74	34.83	65.92	78.91	74.19	91.44	71.18	40.20	55.26	38.95	54.68	21.98	72.86	26.24	28.77	58.76

Table 4: Performance comparison in the few-shot setting on the VTAB-

1K

benchmark. Other settings follow Tab. 1 .

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Train Size	Test Size	Class Number	Batch Size	Reference
Full-Data Setting
Flowers102	4093	2463	102	128	( Nilsback and Zisserman, 2008 )
DTD	2820	1692	47	128	( Cimpoi et al., 2014 )
UCF101	7639	3783	101	128	( Soomro et al., 2012 )
Food101	50500	30300	101	128	( Bossard et al., 2014 )
SVHN	73257	26032	10	128	( Netzer et al., 2011 )

Appendix

Table A1: Dataset attributes and training configs through 29 target image-classification datasets.

Figure A1: The layer effect of AP applied to a (Left) CLIP model and (Right) Swin-Transformer on the OxfordPets dataset.

Dataset	OxfordPets	DTD	EuroSAT	Flowers102	UCF101	Food101	Waterbirds
CLIP
VP	81.97	64.43	95.54	83.74	70.42	79.61	72.42
AP (Ours)	83.82	69.42	96.43	85.52	76.42	82.43	79.32
Swin-Transformer
VP	80.42	65.39	97.23	84.48	74.41	75.72	75.22
AP (Ours)	82.29	69.13	96.45	84.98	75.92	81.38	78.99

Appendix

Table A2: Performance comparison of VP and the proposed AP on CLIP and Swin-Transformer model with different datasets. CLIP with ViT-B/32 and Swin-B with 12 Swin-Transformer blocks pretrained on ImageNet are tested. Other settings follows Tab. 1 .

Panel of Figure 14 — Figure A2: Layer preference of AP with different model architectures on different datasets. CNNs and ViTs exhibit opposite layer preferences.

ViT-B/16 (85.8M)	Total Params	FGCV	VTAB-1k
ViT-B/16 (85.8M)	Total Params	FGCV	Natural	Specialized	Structured
Full-Finetune	24.02 $\times$	88.54	75.88	83.36	47.64
Linear-Probe	1.02 $\times$	79.32	68.93	77.16	26.84
VPT-Shallow	1.04 $\times$	84.62	76.81	74.66	46.98
VPT-Deep	1.18 $\times$	89.11	78.48	82.43	54.98
AP (Ours)	1.11 $\times$	87.33	76.59	79.32	49.98

Appendix

Table A3: Performance comparison of AP with other methods in the setting of VPT ( Jia et al., 2022 ) . Specifically, ViT-B/16 pretrained on supervised ImageNet-21k is adopted as the pretrained model. The numbers except AP are directly sourced from VPT ( Jia et al., 2022 ) .

	Accuracy			Efficiency
	Full-Data				Train-Time Efficiency
	FGVC	VTAB	Others	Param. #	Memory	Time	Throughput
Number of tasks	5	9	5	-	-	-	-
Full-Tune	91.43	91.97	93.91	304.33	41.5	520	79.58
Linear-Probe	82.23	78.90	87.81	0.01	9.7	121	79.64
Bias	85.32	89.84	90.41	0.29	32.9	297	79.48

Appendix

Table A4: Ablation study on AP with more prompt types. Specifically, instead of using additive prompt in the intermediate layer, AP-Product uses feature-wise product and AP-Concate adopts concatenating prompt.

	Accuracy			Efficiency
	Full-Data				Train-Time Efficiency
	FGVC	VTAB	Others	Param. #	Memory	Time	Throughput
Number of tasks	5	9	5	-	-	-	-
Full-Tune	91.43	91.97	93.91	304.33	41.5	520	79.58
Linear-Probe	82.23	78.90	87.81	0.01	9.7	121	79.64
Bias	85.32	89.84	90.41	0.29	32.9	297	79.48

Appendix

Table A5: Ablation study on the number of layers installed with AP. In particular, for AP-3 and AP-5, AP are installed on the input of the first 3 and 5 blocks of the pretrained ViT-L. Other experiment settings follow Tab. 1 , and Tab. 2 .

	Accuracy			Efficiency
	Full-Data				Train-Time Efficiency
	FGVC	VTAB	Others	Param. #	Memory	Time	Throughput
Number of tasks	5	9	5	-	-	-	-
Full-Tune	91.43	91.97	93.91	304.33	41.5	520	79.58
Linear-Probe	82.31	78.43	87.71	0.01	8.1	121	79.69
Bias	85.49	89.47	90.85	0.29	27.4	297	79.51

Appendix

Table A6: Performance comparison between AP and SOTA PEFT methods on ViT-Large/16 with re-initialized classification head. Experiment settings follow Tab. 1 , and Tab. 2 .

	Accuracy			Efficiency
	Full-Data				Train-Time Efficiency
	FGVC	VTAB	Others	Param. #	Memory	Time	Throughput
Number of tasks	5	9	5	-	-	-	-
Full-Tune	91.43	91.97	93.91	304.33	41.5	520	79.58
Linear-Probe	82.23	78.90	87.81	0.01	9.7	121	79.64
VPT -10	86.34	89.24	90.14	0.25	33.7	334	76.35

Appendix

Table A7: Performance comparison between AP and VPT with different prompt lengths on ViT-Large/16. Experiment settings follow Tab. 1 , and Tab. 3 .

Figure A3: Layer-wise performance comparison between AP and VPT on OxfordPets.

	Accuracy			Efficiency
	Full-Data				Train-Time Efficiency
	FGVC	VTAB	Others	Param. #	Memory	Time	Throughput
Number of tasks	5	9	5	-	-	-	-
Full-Tune	91.43	91.97	93.91	304.33	41.5	520	79.58
Linear-Probe	82.23	78.90	87.81	0.01	9.7	121	79.64
Bias	85.32	89.84	90.41	0.29	32.9	297	79.48

Appendix

Table A8: Performance comparison of AP with more PEFT methods (SSF ( Lian et al., 2022 ) ). Experiment settings follow Tab. 1 and Tab. 3 .

	Accuracy			Efficiency
	Full-Data				Train-Time Efficiency
	FGVC	VTAB	Others	Param. #	Memory	Time	Throughput
Number of tasks	5	9	5	-	-	-	-
Full-Tune	91.43	91.97	93.91	304.33	41.5	520	79.58
Linear-Probe	82.23	78.90	87.81	0.01	9.7	121	79.64
LoRA -1	84.43	88.21	90.07	0.04	10.43	139	79.43

Appendix

Table A9: Ablation study on performance of LoRA with different rank values. Experiment settings follow Tab. 1 and Tab. 3 .

Figure A4: Performance of ResNet101 trained with varying sizes of available training data on (a) CIFAR10, (b) Food101, and (c) Camelyon. All other experimental settings strictly follow those in Tab. 1 .

Visual Prompt Learning

Visual prompt learning adapts pretrained vision and vision-language models through learned tokens or visual cues, often without changing their backbones.

See all 123 papers

ES-VP : Energy-Shaped Dynamic Visual Prompting for Efficient Model Adaptation
Aug 21, 2026Can Jin, Ying Li, Jingchen Sun +5Visual Prompt Learning Vision Foundation Model Adaptation
Rutgers University · Zhejiang University · University at Buffalo, SUNY +3
A note on data-efficient finetuning of vision foundation models
Oct 27, 2026Niclas Hergenröther, Antonio OrvietoVision Foundation Model Adaptation Vision Foundation Models
ELLIS Institute Tübingen, Max Planck Institute for Intelligent Systems, Tübingen AI Center
Improving Calibration in Test-Time Prompt Tuning for Vision-Language Models via Data-Free Flatness-Aware Prompt Pretraining
Apr 30, 2026Hyeonseo Jang, Jaebyeong Jeon, Joong-Won Hwang +1Deep Learning Optimization Vision-Language Models
1Yonsei University · 2ETRI

Aug 21, 2026J/K move · Enter open · S save

ES-VP : Energy-Shaped Dynamic Visual Prompting for Efficient Model Adaptation

Can Jin, Ying Li, Jingchen Sun, Hongwu Peng+4

Rutgers University · Zhejiang University · University at Buffalo, SUNY+3

Visual Prompt Learning Vision Foundation Model Adaptation Parameter-Efficient Fine-Tuning VLM Adaptation

Open paper AskPDF arXiv

Visual prompting (VP) has emerged as a parameter-efficient method for adapting pre-trained models to downstream tasks. However, existing approaches encounter a trade-off between flexibility and efficiency. Some methods apply a fixed prompt to all images, ignoring individual image characteristics, while others introduce auxiliary networks to generate diverse prompts. Although the latter can improve performance, it also significantly increases parameter usage and the potential for overfitting to specific datasets. Furthermore, the auxiliary networks, combined with inherent biases in pre-trained models, limit scalability and generalization. In this paper, we propose Energy-Shaped Visual Prompting (ES-VP), a novel approach that generates image-specific prompts using low-rank initialization and energy-guided dynamic adaptation, achieving superior performance with fewer parameters compared to single-prompt methods. ES-VP directly utilizes the pre-trained model for adaptive prompt generation, ensuring both parameter efficiency and improved generalization. Extensive experiments conducted on five architectures across fifteen datasets demonstrate that ES-VP consistently outperforms current state-of-the-art (SOTA) single and diverse VP methods. For instance, using the CLIP architecture across four datasets, ES-VP outperforms the SOTA method DAM-VP by an average of 2.6% in accuracy while utilizing 590

\times

fewer VP parameters, thereby establishing a new benchmark for efficient and generalizable model adaptation.

Abstract

Visual Prompt Learning

Explore similar work

ES-VP : Energy-Shaped Dynamic Visual Prompting for Efficient Model Adaptation

A note on data-efficient finetuning of vision foundation models

Improving Calibration in Test-Time Prompt Tuning for Vision-Language Models via Data-Free Flatness-Aware Prompt Pretraining