Diffusion Meta-Prompting and Steering for Generalizable Foundation Model Adaptation

Deepak Sridhar; Yi Li; Kartikeya Bhardwaj; Shuangjun Liu; Taotao Jing; Yuan Li; Shuai Zhang; Jiancheng Lyu

cs.CVOct 8, 2026

Diffusion Meta-Prompting and Steering for Generalizable Foundation Model Adaptation

Authors: Deepak Sridhar, Yi Li, Kartikeya Bhardwaj, Shuangjun Liu, Taotao Jing, Yuan Li, Shuai Zhang, Jiancheng Lyu, +2 more

Organizations: University of California, San Diego · Qualcomm AI Research

Abstract

Prompt learning is a popular method for adapting foundation models, but learned prompts are typically task-specific and fail to generalize to new classes, domains, or compositions of tasks. In this paper, we introduce a Diffusion Meta-Prompt (DMP) model , a framework that models the distribution of learned prompts using diffusion models. Given a repository of previously learned prompts, DMP is trained and sampled without access to the original task examples or task losses, and synthesizes new prompts conditioned on natural language task descriptions. To improve the sampling stability, we introduce a test-time steering strategy for DMP, which uses the best training-selected prompt in the repository as a latent anchor during diffusion sampling, without retraining the DMP or accessing test classes. DMP improves generalization across classification, retrieval and text-to-image generation tasks, supports concept composition and negative prompting without explicit training. It reduces storage and inference costs by over 90% compared to prompt retrieval methods. For composite classification, DMP achieves upto 2.0% average gain over prior meta-learning methods across 55 pairs of datasets with gains as high as 8.5% on specific pairs such as Eurosat and Flowers. DMP also enhances cross-task generalization with ~2-9% improvement for hierarchical classification task. We further provide a theoretical guarantee bounding the expected task loss of prompts sampled from a DMP. Code is available: https://github.com/DeepakSridhar/dmp

Figures & tables

Table 2

Table 3

		All		Base		New
Setting		Acc.	$\Delta$	Acc.	$\Delta$	Acc.	$\Delta$
$L_{2}$	0.00	63.10	–	69.40	–	70.80	–
$L_{2}$	0.25	63.01	$-0.09$	69.13	$-0.27$	70.77	$-0.03$
$L_{2}$	0.50	62.94	$-0.16$	69.13	$-0.27$	70.73	$-0.07$
$L_{2}$	1.00	63.09	$-0.01$	69.28	$-0.12$	70.90	$+0.10$
Fixed anchor		62.36	$-0.74$	69.12	$-0.28$	70.21	$-0.59$

Table 8: Robustness of test-time steering to perturbations of the anchor prompt. Results are averaged over dataset pairs, and over repeated noise samples within each pair.

\Delta

is relative to the unperturbed anchor. Fixed anchor : a randomly selected training prompt is used as the anchor. Resampled : the perturbation direction is resampled at every denoising step.

Replaced names	Condition $L_{2}$	w/o Steering			w/ Steering
		Prompt $L_{2}$	Caltech101	ImageNet	Prompt $L_{2}$	Caltech101	ImageNet
0%	0.0	0.000	92.0	64.7	0.000	93.0	64.9
40%	24.2	1.623	92.1	64.4	0.025	93.0	64.8
60%	29.8	1.606	92.6	64.3	0.049	93.1	64.5
80%	34.0	1.600	92.5	64.5	0.049	93.2	64.5

Table 9 : Sensitivity of DMP to the class-name condition, with and without test-time steering. A fraction of the Caltech101 class names in the condition is replaced with ImageNet class names.

L_{2}

distances of the condition embedding and of the sampled prompt are measured with respect to the original (0%) condition.

Appendix figures & tables28 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4 : Left: Diffusion Meta-Prompt framework for Text-to-Prompt synthesis. Right: Prompt Variation synthesis conditioned on learned Textual Inversion Prompts.

Method	Task Data/	Supports	Learns	Minimal	Core Idea
	Loss Free	Multiple Tasks	across Tasks	Storage
Stylus / Retrieval-based	✓	✗	✓	✗	Searches large repositories; limited generalization.
Diffusion-based Methods
Prompt Diffusion	✗	✗	✗	✗	Refines prompts for a specific task using data.
Diff-Prompt	✗	✗	✗	✗	Mask-supervised diffusion for a single task.
Neural Network Diffusion	✗	✗	✗	✓	Diffusion to generate model weights

Appendix

Table 10 : Conceptual differences between DMP and prior prompt learning methods. Unlike prior works that refine prompts using data for a specific task or example, DMP treats prompts as a distribution and trains a diffusion meta-model that amortizes prompt generation across many tasks and repositories. DMP does not require access to the original data used to train the prompts (L159–L161) and supports one-shot sampling, composition, and negative prompting without per-task optimization.

$\lambda$	All	Base	New	Avg.
$0.1$	62.5	68.5	69.8	66.9
$0.2$	63.1	69.4	70.8	67.8
$0.3$	62.8	69.1	70.4	67.4
$0.4$	62.2	68.4	69.6	66.7
$0.5$	61.7	67.8	68.9	66.1
$0.6$	61.6	67.9	69.1	66.2

Appendix

Table 11 : Ablation of steering strength

\lambda

for DMP on 55 composite dataset-pairs classification. Moderate steering gives the strongest average accuracy, while overly aggressive steering begins to over-constrain the sampling trajectory.

Table 10

Table 11

Method	All	Base	New
CoOp	58.7	65.1	65.1
DMPCoOp	63.1	69.4	70.8
$\Delta$	$+4.4$	$+4.3$	$+5.7$
MaPLe	57.2	64.3	63.0
DMPMaPLe	57.9	64.7	64.2
$\Delta$	$+0.7$	$+0.4$	$+1.2$

Appendix

Table 18 : Composite classification accuracy (%) averaged over 55 dataset pairs. DMP yields consistent improvements across prompt-learning methods. Gains are largest for CoOp, which only learns soft prompts, and smaller for MaPLe and TAC, which include additional high-dimensional learned components that are kept fixed in our current implementation.

Prompt pair	Accuracy (%)	Prediction disagreement (%)	Per-class accuracy correlation	Error-set overlap (%)
Seed 1 vs. Seed 2	59.4 / 59.5	1.01	0.9992	98.81
Seed 1 vs. Seed 3	59.4 / 59.5	1.12	0.9991	98.68
Seed 1 vs. Seed 4	59.4 / 59.3	1.52	0.9982	97.97
Seed 1 vs. Seed 5	59.4 / 59.5	0.97	0.9992	98.78
Seed 2 vs. Seed 3	59.5 / 59.5	0.79	0.9994	99.07
Seed 2 vs. Seed 4	59.5 / 59.3	1.43	0.9985	98.19

Appendix

Table 20 : Agreement statistics across different random seeds.

Figure 5 : t-SNE visualizations of prompt embeddings. Each figure shows real prompts (blue) and DMP-generated prompts (orange) with different noise seeds for different datasets: (a) SUN397, (b) FGVC-Aircraft, (c) Oxford Flowers, (d) Stanford Cars. Generated prompts broadly overlap with the real prompt manifolds while exhibiting greater spread, demonstrating both fidelity and diversity across domains.

Figure 6 : Prompt Sliders images synthesized with sliders sampled by DMPSlider when prompted for age and smiling. The prompt used for the SD-XL model is "A photo of a beautiful man" .

Figure 7 : Qualitative results of DMPSlider prompts depicting the concepts “long hair" and “chubby". The prompts used for the SD-XL model for the images shown from left to right are as follows. "A closeup photo of a person", "Professional headshot of a person" .

Panel of Figure 8 — Figure 8 : Qualitative comparison of image synthesis with TI and DMPMulti prompts. Left: groundtruth images for Identity-38 and Identity-96 in the training set. Right: images synthesized by SD for the original TI prompts (first row) and prompts sampled from the DMPMulti model (second row):

Figure 9 : Negative prompting samples from the SD Realistic Vision model when prompted by the DMPMulti, itself prompted with id-21 (leftmost) as positive and id-24 (second from left) as negative prompt. The generated identity has features opposing to id-24 (chubby cheeks, small eyes, wide nose, etc.)

Figure 10 : Qualitative comparison of images synthesized with baseline prompt sliders and concept sliders against DMPSlider. The prompt used for the SD-XL model is “A photo of a girl" and the slider trained to control the concept “curlyhair" .

Figure 11 : Additional Qualitative results: Nearest neighbors in the training set shown on the left for novel ids sampled by DMPMulti model to the right.

Figure 12 : Generalization of DMPVariation model . Left: groundtruth images. Second: images generated by SD for the original TI prompts and the last two columns are the images for variation prompts sampled by DMP. See Fig. 13 for additional results.

Figure 13 : Additional Qualitative results of generalization of DMPVariation model . Left: groundtruth images. Second: images generated by SD for the original TI prompts and the last two columns are the images for variation prompts sampled by DMP.

Figure 14 : Ablation study on Identity composition with Textual Inversion: images generated with TI prompts by composing the Stable Diffusion outputs for the composition of the two identities shown on the left for each row. TI composition produces distorted identities or repeats the same identities.

Figure 15 : DMP Variations of Textual Inversion (TI) Prompts : The leftmost image is the real groundtruth, the second is generated by TI, and the rest are DMP variations (each image represents a new identity) conditioned on the TI embedding. All images are generated with a fixed seed to the stable diffusion model. The FaceID similarity to the groundtruth image is listed on top of each image. Lower is better for diverse variations.

Figure 16 : Qualitative results of images generated from prompts synthesized by DMPVariation. Our Diffusion Meta-Prompts are more robust and less overfitted than the baseline textual inversion which fails to generalize.

Figure 17 : Identity composition: images generated with DMPVariation model prompts (right) for increasing guidance scales from 9 to 10 for the composition of the two identities shown on the left.

Replacement	Class-name condition supplied to DMP
0%	accordion, bonsai, chandelier, euphonium, menorah, saxophone, stapler, yin_yang, anchor, camera
20%	brambling, goldfinch, chandelier, euphonium, menorah, saxophone, stapler, yin_yang, anchor, camera
40%	brambling, goldfinch, house finch, junco, menorah, saxophone, stapler, yin_yang, anchor, camera
60%	brambling, goldfinch, house finch, junco, indigo bunting, American robin, stapler, yin_yang, anchor, camera
80%	brambling, goldfinch, house finch, junco, indigo bunting, American robin, bulbul, jay, anchor, camera

Appendix

Table 26 : Class-name conditions supplied to DMP at each replacement level.

Concept	DMP	TI	Delta
ettblackteapot	0.2905	0.2778	+0.0127
dicoo	0.2837	0.2700	+0.0137
goku	0.2830	0.2800	+0.0030
degodsheavy	0.2827	0.2754	+0.0073
spider-gwen	0.2825	0.2810	+0.0015
johnny-silverhand	0.2812	0.2760	+0.0052

Appendix

Table 29 : HPSv2 comparison for 75 random concepts where each concept is evaluated for 6 different prompts.

Panel of Figure 20 — Figure 20 : Qualitative results of image synthesis with DMPMulti prompts. Left: groundtruth images. Right: images synthesized by SD with prompts sampled from the DMPMulti model.

Figure 23 : Interpolating between two faces using DMPMulti with concept composition.

	DMPVariation	DMPMulti	DMPCoOp	Autoencoder
$z$ -shape	-	-	16 x 8	16 x 8
$x$ -shape	768 x 1	768 x 1	2048 x 1	2048 x 1
$\|Z\|$	-	-	128	128
$\|X\|$	768	768	2048	768
Diffusion steps	1000	1000	1000	1000
Optimizer	AdamW	AdamW	AdamW	AdamW

Appendix

Table 30 : Hyperparameter Settings for the DMP models trained across three different tasks namely personalization, concepts and classification.

Figure 24 : Qualitative results of generating novel identities during inference using random identity conditioning with DMPMulti model. The prompt used for the Stable Diffusion Realistic Vision model is "A photo of a id-x" where x is the id not present in the training set. The identities do not overfit ( mean face ID similarity of 0.0102 across training images ). Note that all the images use a fixed seed to the diffusion model.

Explore similar work

Concept-Constrained Prompt Learning for Few-Shot CLIP Adaptation
Jun 21, 2026Na Sang, Ding Ma, Rui Sang +1Prompt Learning VLM Adaptation
University of California, San Diego, La Jolla, CA, USA · Georgia Institute of Technology, Atlanta, GA, USA · Independent Researcher
PETR: Prompt Ensembling with Training-free Routing for Vision-Language Models
Sep 20, 2026Weihan Cai, Hao Tan, Xinping Gao +2Zero-Shot Learning VLM Adaptation
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, and also with the School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China · Purple Mountain Laboratories, Nanjing, China · Beijing University of Posts and Telecommunications, Beijing, China
General and Efficient Steering of Diffusion Models
Feb 11, 2026Qingsong Wang, Mikhail Belkin, Yusu WangDiffusion Model Guidance Activation Steering
Halıcıoğlu Data Science Institute, University of California San Diego, La Jolla, CA, USA

Jun 21, 2026, cs.LGJ/K move · Enter open · S save

Concept-Constrained Prompt Learning for Few-Shot CLIP Adaptation

Na Sang, Ding Ma, Rui Sang, Yuxuan Liu

University of California, San Diego, La Jolla, CA, USA · Georgia Institute of Technology, Atlanta, GA, USA · Independent Researcher

Prompt Learning VLM Adaptation Few-Shot Prompting

Open paper AskPDF arXiv

Few-shot prompt learning is an effective strategy for adapting CLIP to downstream tasks, but class-only prompt optimization can overfit base-class supervision and weaken transfer to unseen classes. We propose Concept-Constrained Prompt Learning (CCPL), a lightweight regularization framework that anchors learnable class prompts to frozen concept-level text prototypes without updating CLIP encoders. CCPL learns a set of shared context tokens, instantiates class prompts by appending class names, and constructs frozen concept prototypes from a class-level concept bank. During training, a text-space cosine consistency objective aligns learnable class-prompt embeddings with frozen concept prototypes; concept dropout provides additional regularization against over-reliance on fixed concept lists. At inference, CCPL optionally fuses class-prompt logits with concept-prototype logits using a controllable ensemble weight alpha. Our default configuration uses text-space concept regularization lambda = 0.5, concept dropout p = 0.3 and weak concept-guided fusion (alpha = 0.1), with no KL-based prediction consistency term. Experiments under identical automatically-generated fallback splits show that CCPL improves the base-to-new harmonic mean on DTD (+0.6) and EuroSAT (+2.9) compared with CoOp, while remaining near-neutral on OxfordPets (-0.1). Ablations indicate that text-space concept regularization is consistently beneficial, while the best concept-guided inference strength is dataset- and protocol-sensitive. These results suggest concept constraints are most effective when concept prototypes align naturally with dataset semantics, and identify fine-grained categories as a current boundary condition. The code is released at: https://github.com/richael-sang/concept-constrained-prompt-learning.

Abstract

Explore similar work

Concept-Constrained Prompt Learning for Few-Shot CLIP Adaptation

PETR: Prompt Ensembling with Training-free Routing for Vision-Language Models

General and Efficient Steering of Diffusion Models