Organizations: Michigan State University · University of Pennsylvania · New Jersey Institute of Technology · IBM Research · Rensselaer Polytechnic Institute
Visual prompting (VP) has emerged as a popular method to repurpose pretrained vision models for adaptation to downstream tasks. Unlike conventional model fine-tuning techniques, VP introduces a universal perturbation directly into the input data to facilitate task-specific fine-tuning rather than modifying model parameters. However, there exists a noticeable performance gap between VP and conventional fine-tuning methods, highlighting an unexplored realm in theory and practice to understand and advance the input-level VP to reduce its current performance gap. Towards this end, we introduce a generalized concept, termed activation prompt (AP), which extends the scope of the input-level VP by enabling universal perturbations to be applied to activation maps within the intermediate layers of the model. By using AP to revisit the problem of VP and employing it as an analytical tool, we demonstrate the intrinsic limitations of VP in both performance and efficiency, revealing why input-level prompting may lack effectiveness compared to AP, which exhibits a model-dependent layer preference. We show that AP is closely related to normalization tuning in convolutional neural networks and vision transformers, although each model type has distinct layer preferences for prompting. We also theoretically elucidate the rationale behind such a preference by analyzing global features across layers. Through extensive experiments across 29 datasets and various model architectures, we provide a comprehensive performance analysis of AP, comparing it with VP and parameter-efficient fine-tuning baselines. Our results demonstrate AP's superiority in both accuracy and efficiency, considering factors such as time, parameters, memory usage, and throughput.
Figures & tables
Figure 1: An illustration of the proposed activation prompt vs. the conventional input-based prompt.
Figure 2: Performance and efficiency comparison of VP , Norm-Tune and AP over different layers of ResNet-101 on OxfordPets.
Figure 3: Tunable parameter shape of Norm-Tune and AP . The same color indicates shared parameters across dimensions .
Figure 4: Layer preference of AP with different model architectures on OxfordPets ( Parkhi et al., 2012 ) . CNNs and ViTs exhibit opposite layer preferences. Results on more datasets are provided in Fig. A2 .
Figure 5: Features dissection to understand the layer effect. (Left) CKA-based feature similarity comparison between ViT-L/16 and ResNet-101. (Right) Average attention distances across all the heads and layers of ViT-L/16. Larger distance indicates global features.
Figure 6: Sample complexity study of AP .
Benchmark
FGVC
VTAB
Others
Architecture
CUB200
StanfordDog
StanfordCars
NA-Birds
OxfordFlowers
CIFAR-100
Caltech-101
DTD
Flowers102
OxfordPets
SVHN
SUN397
Camelyon
EuroSAT
CIFAR-10
GTSRB
UCF101
Food101
Waterbirds
A verage
ResNet-101
∙ Full-Tune
88.91
90.13
87.76
84.45
99.98
92.24
99.13
79.97
99.81
90.49
97.14
79.19
91.13
99.13
97.24
97.68
88.32
82.72
96.69
91.69
∘ Linear-Probe
63.76
86.63
49.62
52.09
82.01
73.87
90.58
61.35
93.14
91.17
66.30
54.51
83.36
95.84
92.25
79.64
71.03
64.31
88.11
75.76
∘ Norm-Tune
66.39
87.59
67.64
56.72
66.50
82.58
91.32
63.53
92.85
89.81
95.26
54.56
84.42
96.14
93.90
96.43
69.44
72.54
88.95
79.81
∘ VP
65.72
86.91
51.04
54.23
78.50
72.01
93.51
63.12
90.17
87.93
80.68
54.97
83.71
95.44
92.55
83.18
66.30
57.89
86.71
76.03
∘ AP (ours)
69.42
87.79
59.06
58.31
85.14
76.94
94.85
69.80
95.13
91.31
87.30
56.83
84.91
97.21
94.08
90.43
73.96
68.12
88.13
80.45
Table 1: Performance comparison on 19 datasets from 3 benchmarks. Three parameter-efficient baselines (denoted by ∘ ) are compared to AP , where the best performance is highlighted in bold . The most expensive Full-Tune (denoted by ∙ ) serves as the performance reference. Each accuracy value is averaged over 5 trials, with variance omitted due to its negligible values ( ≤0.3% ). “Average” column averaged accuracies over all the datasets.
Method
Param. Efficiency
Train-Time Efficiency
Parameter # (M) ↓
Memory Cost (G) ↓
Time Cost (s/epoch) ↓
Troughput (image/s) ↑
ResNet-101
Full-Tune
44.5
10.32
118
41.47
Linear-Probe
0.02
6.2
39
41.33
Norm-Tune
0.13
11.7
83
41.45
VP
0.12
12.2
72
40.59
Table 2: A performance overview. The efficiency analysis is on (ViT-L, CIFAR-10) with a batch size of 128, and time consumption is based on using a single RTX-A6000. ↑ or ↓ indicate whether a larger or smaller value is favored.
Accuracy
Efficiency
Full-Data
Train-Time Efficiency
FGVC
VTAB
Others
Param. #
Memory
Time
Throughput
Number of tasks
5
9
5
-
-
-
-
Full-Tune
91.43
91.97
93.91
304.33
41.5
520
79.58
Linear-Probe
82.23
78.90
87.81
0.01
9.7
121
79.64
Bias
85.32
89.84
90.41
0.29
32.9
297
79.43
Table 3: Performance of AP and more SOTA PEFT methods on ViT-L/16. Settings follow Tab. 1 .
Benchmark
VTAB-Natural
VTAB-Specialized
VTAB-Structured
Architecture
Caltech101
CIFAR-100
DTD
Flowers102
OxfordPets
Sun397
SVHN
Camelyon
EuroSAT
Resisc45
Retinopathy
Clevr-Count
Clevr-Dist
DMLab
dSpr-Loc
dSpr-Ori
KITTI-Dist
sNORB-Azim
sNORB-Elev
A verage
ResNet-101
∙ Full-Tune
89.99
45.17
63.78
84.29
89.82
41.09
67.79
84.92
74.57
91.37
74.14
58.11
60.99
43.61
67.05
40.45
78.34
33.64
36.38
64.50
∘ Linear-Probe
83.87
39.13
53.09
70.89
85.15
28.14
43.44
78.65
69.43
90.78
69.31
35.91
36.48
35.75
34.76
19.51
65.68
16.91
23.39
51.12
∘ Norm-Tune
85.61
35.78
47.71
56.64
78.10
10.10
68.67
83.16
61.10
90.50
72.44
37.54
55.24
40.04
60.89
20.33
65.54
24.86
25.96
53.70
∘ VP
84.73
43.01
57.55
76.91
87.03
28.75
55.47
75.15
70.27
89.26
69.08
36.70
54.24
34.48
42.41
20.32
63.71
17.93
26.93
54.42
∘ AP
87.49
39.80
63.62
81.44
88.74
34.83
65.92
78.91
74.19
91.44
71.18
40.20
55.26
38.95
54.68
21.98
72.86
26.24
28.77
58.76
Table 4: Performance comparison in the few-shot setting on the VTAB- 1K benchmark. Other settings follow Tab. 1 .
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Train Size
Test Size
Class Number
Batch Size
Reference
Full-Data Setting
Flowers102
4093
2463
102
128
( Nilsback and Zisserman, 2008 )
DTD
2820
1692
47
128
( Cimpoi et al., 2014 )
UCF101
7639
3783
101
128
( Soomro et al., 2012 )
Food101
50500
30300
101
128
( Bossard et al., 2014 )
SVHN
73257
26032
10
128
( Netzer et al., 2011 )
Appendix
Table A1: Dataset attributes and training configs through 29 target image-classification datasets.
Figure A1: The layer effect of AP applied to a (Left) CLIP model and (Right) Swin-Transformer on the OxfordPets dataset.
Dataset
OxfordPets
DTD
EuroSAT
Flowers102
UCF101
Food101
Waterbirds
CLIP
VP
81.97
64.43
95.54
83.74
70.42
79.61
72.42
AP (Ours)
83.82
69.42
96.43
85.52
76.42
82.43
79.32
Swin-Transformer
VP
80.42
65.39
97.23
84.48
74.41
75.72
75.22
AP (Ours)
82.29
69.13
96.45
84.98
75.92
81.38
78.99
Appendix
Table A2: Performance comparison of VP and the proposed AP on CLIP and Swin-Transformer model with different datasets. CLIP with ViT-B/32 and Swin-B with 12 Swin-Transformer blocks pretrained on ImageNet are tested. Other settings follows Tab. 1 .
Figure A2: Layer preference of AP with different model architectures on different datasets. CNNs and ViTs exhibit opposite layer preferences.
ViT-B/16 (85.8M)
Total Params
FGCV
VTAB-1k
Natural
Specialized
Structured
Full-Finetune
24.02 ×
88.54
75.88
83.36
47.64
Linear-Probe
1.02 ×
79.32
68.93
77.16
26.84
VPT-Shallow
1.04 ×
84.62
76.81
74.66
46.98
VPT-Deep
1.18 ×
89.11
78.48
82.43
54.98
AP (Ours)
1.11 ×
87.33
76.59
79.32
49.98
Appendix
Table A3: Performance comparison of AP with other methods in the setting of VPT ( Jia et al., 2022 ) . Specifically, ViT-B/16 pretrained on supervised ImageNet-21k is adopted as the pretrained model. The numbers except AP are directly sourced from VPT ( Jia et al., 2022 ) .
Accuracy
Efficiency
Full-Data
Train-Time Efficiency
FGVC
VTAB
Others
Param. #
Memory
Time
Throughput
Number of tasks
5
9
5
-
-
-
-
Full-Tune
91.43
91.97
93.91
304.33
41.5
520
79.58
Linear-Probe
82.23
78.90
87.81
0.01
9.7
121
79.64
Bias
85.32
89.84
90.41
0.29
32.9
297
79.48
Appendix
Table A4: Ablation study on AP with more prompt types. Specifically, instead of using additive prompt in the intermediate layer, AP-Product uses feature-wise product and AP-Concate adopts concatenating prompt.
Accuracy
Efficiency
Full-Data
Train-Time Efficiency
FGVC
VTAB
Others
Param. #
Memory
Time
Throughput
Number of tasks
5
9
5
-
-
-
-
Full-Tune
91.43
91.97
93.91
304.33
41.5
520
79.58
Linear-Probe
82.23
78.90
87.81
0.01
9.7
121
79.64
Bias
85.32
89.84
90.41
0.29
32.9
297
79.48
Appendix
Table A5: Ablation study on the number of layers installed with AP. In particular, for AP-3 and AP-5, AP are installed on the input of the first 3 and 5 blocks of the pretrained ViT-L. Other experiment settings follow Tab. 1 , and Tab. 2 .
Accuracy
Efficiency
Full-Data
Train-Time Efficiency
FGVC
VTAB
Others
Param. #
Memory
Time
Throughput
Number of tasks
5
9
5
-
-
-
-
Full-Tune
91.43
91.97
93.91
304.33
41.5
520
79.58
Linear-Probe
82.31
78.43
87.71
0.01
8.1
121
79.69
Bias
85.49
89.47
90.85
0.29
27.4
297
79.51
Appendix
Table A6: Performance comparison between AP and SOTA PEFT methods on ViT-Large/16 with re-initialized classification head. Experiment settings follow Tab. 1 , and Tab. 2 .
Accuracy
Efficiency
Full-Data
Train-Time Efficiency
FGVC
VTAB
Others
Param. #
Memory
Time
Throughput
Number of tasks
5
9
5
-
-
-
-
Full-Tune
91.43
91.97
93.91
304.33
41.5
520
79.58
Linear-Probe
82.23
78.90
87.81
0.01
9.7
121
79.64
VPT -10
86.34
89.24
90.14
0.25
33.7
334
76.35
Appendix
Table A7: Performance comparison between AP and VPT with different prompt lengths on ViT-Large/16. Experiment settings follow Tab. 1 , and Tab. 3 .
Figure A3: Layer-wise performance comparison between AP and VPT on OxfordPets.
Accuracy
Efficiency
Full-Data
Train-Time Efficiency
FGVC
VTAB
Others
Param. #
Memory
Time
Throughput
Number of tasks
5
9
5
-
-
-
-
Full-Tune
91.43
91.97
93.91
304.33
41.5
520
79.58
Linear-Probe
82.23
78.90
87.81
0.01
9.7
121
79.64
Bias
85.32
89.84
90.41
0.29
32.9
297
79.48
Appendix
Table A8: Performance comparison of AP with more PEFT methods (SSF ( Lian et al., 2022 ) ). Experiment settings follow Tab. 1 and Tab. 3 .
Accuracy
Efficiency
Full-Data
Train-Time Efficiency
FGVC
VTAB
Others
Param. #
Memory
Time
Throughput
Number of tasks
5
9
5
-
-
-
-
Full-Tune
91.43
91.97
93.91
304.33
41.5
520
79.58
Linear-Probe
82.23
78.90
87.81
0.01
9.7
121
79.64
LoRA -1
84.43
88.21
90.07
0.04
10.43
139
79.43
Appendix
Table A9: Ablation study on performance of LoRA with different rank values. Experiment settings follow Tab. 1 and Tab. 3 .
Figure A4: Performance of ResNet101 trained with varying sizes of available training data on (a) CIFAR10, (b) Food101, and (c) Camelyon. All other experimental settings strictly follow those in Tab. 1 .
Visual prompt learning adapts pretrained vision and vision-language models through learned tokens or visual cues, often without changing their backbones.
Visual prompting (VP) has emerged as a parameter-efficient method for adapting pre-trained models to downstream tasks. However, existing approaches encounter a trade-off between flexibility and efficiency. Some methods apply a fixed prompt to all images, ignoring individual image characteristics, while others introduce auxiliary networks to generate diverse prompts. Although the latter can improve performance, it also significantly increases parameter usage and the potential for overfitting to specific datasets. Furthermore, the auxiliary networks, combined with inherent biases in pre-trained models, limit scalability and generalization. In this paper, we propose Energy-Shaped Visual Prompting (ES-VP), a novel approach that generates image-specific prompts using low-rank initialization and energy-guided dynamic adaptation, achieving superior performance with fewer parameters compared to single-prompt methods. ES-VP directly utilizes the pre-trained model for adaptive prompt generation, ensuring both parameter efficiency and improved generalization. Extensive experiments conducted on five architectures across fifteen datasets demonstrate that ES-VP consistently outperforms current state-of-the-art (SOTA) single and diverse VP methods. For instance, using the CLIP architecture across four datasets, ES-VP outperforms the SOTA method DAM-VP by an average of 2.6% in accuracy while utilizing 590× fewer VP parameters, thereby establishing a new benchmark for efficient and generalizable model adaptation.
Can Jin, Ying Li, Jingchen Sun +5
Rutgers University · Zhejiang University · University at Buffalo, SUNY +3
Adapting foundation models to a new task within or beyond the training domain is the de facto standard for building specialized machine learning models. Recently, Visual Prompting (VP) has emerged as a promising technique for efficient knowledge transfer. As new foundation model families~(like Mamba) get introduced and VP pipelines such as AutoVP reach greater maturity, we find a growing need for a systematic evaluation of current approaches. In this work, we assess the performance of the latest models, comparing them to earlier architectures and alternative fine-tuning methods, to better understand the progress, challenges and opportunities in the field of efficient fine-tuning under resource limitations. Towards this goal, this paper provides a concise empirical overview of the interactions among foundation model families (Attention-, Convolution-, and Mamba-based) and transfer paradigms: VP, Linear Probing (LP), Full Fine-tuning (FFT) and Low Rank Adaptation (LoRA). Our work builds up on previous findings by broadening the selection of evaluated models and techniques while tuning hyperparameters. In the interest of delivering practical guidelines for the user, we also explore application of prevalent regularization techniques to boost performance in the context of VP.
Niclas Hergenröther, Antonio Orvieto
ELLIS Institute Tübingen, Max Planck Institute for Intelligent Systems, Tübingen AI Center
Test-time prompt tuning (TPT) has emerged as a promising technique for enhancing the adaptability of vision-language models by optimizing textual prompts using unlabeled test data. However, prior studies have observed that TPT often produces poorly calibrated models, raising concerns about the reliability of their predictions. Recent works address this issue by incorporating additional regularization terms that constrain model outputs, which improve calibration but often degrade performance. In this work, we reveal that these regularization strategies implicitly encourage optimization toward flatter minima, and that the sharpness of the loss landscape around adapted prompts is a key factor governing calibration quality. Motivated by this observation, we introduce Flatness-aware Prompt Pretraining (FPP), a simple yet effective pretraining framework for TPT that initializes prompts within flatter regions of the loss landscape prior to adaptation. We show that simply replacing the initialization in existing TPT pipelines--without modifying any other components--is sufficient to improve both calibration and performance. Notably, FPP requires no labeled data and incurs no additional computational costs during test-time tuning, making it highly practical for real-world deployment. The code is available at: https://github.com/YonseiML/fpp.