ES-VP : Energy-Shaped Dynamic Visual Prompting for Efficient Model Adaptation

Can Jin; Ying Li; Jingchen Sun; Hongwu Peng; Jiahui Zhao; Yang Zhou; Lei Li; Dimitris N. Metaxas

cs.CVAug 21, 2026

ES-VP : Energy-Shaped Dynamic Visual Prompting for Efficient Model Adaptation

Authors: Can Jin, Ying Li, Jingchen Sun, Hongwu Peng, Jiahui Zhao, Yang Zhou, Lei Li, Dimitris N. Metaxas

Organizations: Rutgers University · Zhejiang University · University at Buffalo, SUNY · Adobe Research ·

Abstract

Visual prompting (VP) has emerged as a parameter-efficient method for adapting pre-trained models to downstream tasks. However, existing approaches encounter a trade-off between flexibility and efficiency. Some methods apply a fixed prompt to all images, ignoring individual image characteristics, while others introduce auxiliary networks to generate diverse prompts. Although the latter can improve performance, it also significantly increases parameter usage and the potential for overfitting to specific datasets. Furthermore, the auxiliary networks, combined with inherent biases in pre-trained models, limit scalability and generalization. In this paper, we propose Energy-Shaped Visual Prompting (ES-VP), a novel approach that generates image-specific prompts using low-rank initialization and energy-guided dynamic adaptation, achieving superior performance with fewer parameters compared to single-prompt methods. ES-VP directly utilizes the pre-trained model for adaptive prompt generation, ensuring both parameter efficiency and improved generalization. Extensive experiments conducted on five architectures across fifteen datasets demonstrate that ES-VP consistently outperforms current state-of-the-art (SOTA) single and diverse VP methods. For instance, using the CLIP architecture across four datasets, ES-VP outperforms the SOTA method DAM-VP by an average of 2.6% in accuracy while utilizing 590 $\times$ fewer VP parameters, thereby establishing a new benchmark for efficient and generalizable model adaptation.

Figures & tables

Figure 1: Existing VP methods. Existing VP methods can be categorized into two groups: Single-VP and Diverse-VP methods. $\boldsymbol{g}(\boldsymbol{\delta})$ is the VP parameters.

Figure 2: Overview of ES-VP . We establish a single universal VP via a low-rank prior $\boldsymbol{\delta}$ to capture global information, and derive an image-specific dynamic adaptation $\boldsymbol{\mu}_{\boldsymbol{\delta}}(\mathbf{x})$ using an energy function to form the final prompted image $\mathcal{P}(\mathbf{x})$ . Both the low-rank prior and the dynamic adaptation are jointly optimized via meta-learning.

Model	Method	Tiny-ImageNet	EuroSAT	OxfordPets	Food101	DTD	Flowers102	CIFAR10	CIFAR100	SVHN	GTSRB	Average
ResNet-18	LP [CVPR22]	65.12	93.56	87.25	51.24	59.88	88.12	87.05	67.01	65.56	78.02	74.28
	ILM-VP [CVPR23]	14.11	85.12	65.32	24.94	35.23	27.85	66.31	25.32	75.12	53.05	47.24
	AutoVP [ICLR24]	59.72	93.11	82.77	54.24	54.67	85.32	87.55	63.75	83.89	81.73	74.68
	DAM-VP [CVPR23]	67.15	94.45	87.13	54.01	61.29	89.51	87.88	68.82	84.35	81.52	77.61
	SMM [ICML24]	39.81	92.15	74.01	17.78	33.56	38.84	72.56	38.89	84.14	80.56	57.23
	LoR-VP [ICLR25]	68.28	92.54	83.31	54.78	61.21	87.22	88.64	69.88	83.14	81.36	77.04

Table 1: Performance of ImageNet-1K pre-trained ResNet-18 and ImageNet-21K pre-trained ViT-B/32 on downstream datasets. Overview of the performance of ES-VP compared to five SOTA baseline methods. Results are averaged over three runs. ES-VP outperforms all baselines across models and datasets.

Panel of Figure 3 — Figure 3: Performance of ImageNet-1K pre-trained ResNet-50 and CLIP Pre-trained Models on Downstream Datasets. Overview of the performance of ES-VP compared to five baseline methods. The final performance of each method is indicated by the markers. ES-VP consistently outperforms all baselines across various models and datasets with fewer training epochs.

Table 5

Method	Output Transformatin	Tiny-ImageNet	CIFAR100	Average
LP [CVPR22]	LP	83.88	86.01	84.95
ILM-VP [CVPR23]	ILM	32.38	40.02	36.20
AutoVP [ICLR24]	FM	82.39	85.86	84.13
DAM-VP [CVPR23]	LP	84.90	88.10	86.50
SMM [ICML24]	ILM	79.72	82.27	81.00
ES-VP w. ILM	ILM	85.57	82.44	84.01

Table 4: The Impact of Output Transformation. Performance comparison of ES-VP using ILM, FM, and LP as output transformations and baseline methods. ES-VP achieves the best performance across all settings.

Method	Type	Model	Epochs	Time	#VP Params	Latency	Peak Memory	Acc.
ILM-VP [CVPR23]	Single-VP	ResNet-18	200	1.2h	147K	1.1ms	1.4GB	35.2
ILM-VP [CVPR23]	Single-VP	ResNet-50	200	1.8h	147K	2.5ms	2.2GB	40.5
AutoVP [ICLR24]	Single-VP	ResNet-18	100	0.5h	101K	1.1ms	1.4GB	54.7
AutoVP [ICLR24]	Single-VP	ResNet-50	100	0.8h	101K	2.6ms	2.2GB	65.5
SMM [ICML24]	Diverse-VP	ResNet-18	200	1.2h	177K	1.5ms	2.5GB	33.6
SMM [ICML24]	Diverse-VP	ResNet-50	200	1.9h	177K	3.3ms	3.5GB	41.5

Table 5: Efficiency Comparison. Comparison of the training and inference efficiency of ES-VP compared to the baselines using ResNet-18/ResNet-50 on DTD.

Visual Prompt Learning

Visual prompt learning adapts pretrained vision and vision-language models through learned tokens or visual cues, often without changing their backbones.

See all 123 papers

Visual prompting reimagined: The power of the Activation Prompts
Apr 7, 2026Yihua Zhang, Hongkang Li, Yuguang Yao +5Visual Prompt Learning Vision Foundation Model Adaptation
Michigan State University · University of Pennsylvania · New Jersey Institute of Technology +2
MMLoP: Multi-Modal Low-Rank Prompting for Efficient Vision-Language Adaptation
Feb 24, 2026Sajjad Ghiasvand, Haniyeh Ehsani Oskouie, Mahnoosh Alizadeh +1Multimodal Foundation Models Prompt Learning
University of California, Santa Barbara · University of California, Los Angeles
Guided Prompt Evolution for Vision-Language Models Adaptation
Mar 10, 2026Enming Zhang, Jiayang Li, Yanlong Wang +3VLM Adaptation Few-Shot Learning
Tsinghua Shenzhen International Graduate School, Tsinghua University · Sun Yat-sen University · Chinese University of Hong Kong, Shenzhen

Apr 7, 2026J/K move · Enter open · S save

Visual prompting reimagined: The power of the Activation Prompts

Yihua Zhang, Hongkang Li, Yuguang Yao, Aochuan Chen+4

Michigan State University · University of Pennsylvania · New Jersey Institute of Technology+2

Visual Prompt Learning Vision Foundation Model Adaptation Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Visual prompting (VP) has emerged as a popular method to repurpose pretrained vision models for adaptation to downstream tasks. Unlike conventional model fine-tuning techniques, VP introduces a universal perturbation directly into the input data to facilitate task-specific fine-tuning rather than modifying model parameters. However, there exists a noticeable performance gap between VP and conventional fine-tuning methods, highlighting an unexplored realm in theory and practice to understand and advance the input-level VP to reduce its current performance gap. Towards this end, we introduce a generalized concept, termed activation prompt (AP), which extends the scope of the input-level VP by enabling universal perturbations to be applied to activation maps within the intermediate layers of the model. By using AP to revisit the problem of VP and employing it as an analytical tool, we demonstrate the intrinsic limitations of VP in both performance and efficiency, revealing why input-level prompting may lack effectiveness compared to AP, which exhibits a model-dependent layer preference. We show that AP is closely related to normalization tuning in convolutional neural networks and vision transformers, although each model type has distinct layer preferences for prompting. We also theoretically elucidate the rationale behind such a preference by analyzing global features across layers. Through extensive experiments across 29 datasets and various model architectures, we provide a comprehensive performance analysis of AP, comparing it with VP and parameter-efficient fine-tuning baselines. Our results demonstrate AP's superiority in both accuracy and efficiency, considering factors such as time, parameters, memory usage, and throughput.

Abstract

Visual Prompt Learning

Explore similar work

Visual prompting reimagined: The power of the Activation Prompts

MMLoP: Multi-Modal Low-Rank Prompting for Efficient Vision-Language Adaptation

Guided Prompt Evolution for Vision-Language Models Adaptation