ProCAP: Probabilistic Cross-Attentive Prompt Learning for Vision-Language Models

Hiwa Azeez Abbas; Fatemeh Daneshfar; Moloud Abdar

cs.CVSep 24, 2026

ProCAP: Probabilistic Cross-Attentive Prompt Learning for Vision-Language Models

Authors: Hiwa Azeez Abbas, Fatemeh Daneshfar, Moloud Abdar

Organizations: University of Kurdistan, Sanandaj, Iran · Department of Computer Engineering, University of Kurdistan, Sanandaj, Iran · CHIRP, Child Health Research Centre, The University of Queensland, Brisbane, Australia

Abstract

Pre-trained vision-language models such as CLIP can recognize new categories via prompting, but they often struggle when labeled data are scarce or the test distribution shifts. Prompt learning adapts only a small set of parameters while keeping the backbone frozen, yet many existing multimodal prompt learners couple the visual and textual branches weakly and can be brittle in low-shot regimes. We propose ProCAP, a probabilistic cross-attentive prompt learning framework that improves cross-modal interaction and training stability without updating any CLIP weights: it learns both visual and textual prompt tokens and links them through stacked bidirectional multi-head cross-attention so the two branches refine each other across prompt depth. To reduce overfitting under limited supervision, we parameterize prompt tokens with Gaussian means and variances and regularize them with lightweight KL and L2 penalties, and we further add a compact symmetric InfoNCE head that aligns cross-attended image features with class-level text representations in a shared low-dimensional space. Across few-shot base-to-novel generalization on 11 datasets, cross-dataset transfer, and domain generalization on ImageNet shift benchmarks, ProCAP achieves strong aggregate base-to-novel performance and competitive transfer performance while keeping the CLIP backbone unchanged.

Figures & tables

Panel of Figure 1 — Figure 1: Overview of prompt learning designs and their effect on base-to-novel generalization. (a) Multi-template prompting and MaPLe-style prompting. (b) ProCAP prompting and base-to-novel $\mathrm{HM}$ across benchmarks.

Figure 2: Overview of the proposed ProCAP framework. ProCAP builds on a frozen CLIP backbone and introduces cross-attentive multimodal prompts, probabilistic (Gaussian) prompt parameterization, and a symmetric InfoNCE head for improved few-shot and base-to-novel transfer.

Method	Venue	(a) Average			(b) ImageNet			(c) Caltech101			(d) OxfordPets
Method	Venue	Base	Novel	HM	Base	Novel	HM	Base	Novel	HM	Base	Novel	HM
CLIP	ICML’21	69.34	74.22	71.70	72.43	68.14	70.22	96.84	94.00	95.40	91.17	97.26	94.12
CoCoOp	CVPR’22	80.47	71.69	75.83	75.98	70.43	73.10	97.96	93.81	95.84	95.20	97.69	96.43
CoOp	IJCV’22	82.69	63.22	71.66	76.47	67.88	71.92	98.00	89.81	93.73	93.67	95.29	94.47
KgCoOp	CVPR’23	80.73	73.60	77.00	75.83	69.96	72.78	97.72	94.39	96.03	94.65	97.76	96.18
MaPLe	CVPR’23	82.28	75.14	78.55	76.66	70.54	73.47	97.74	94.36	96.02	95.43	97.76	96.58

Table 1: Base-to-novel generalization (Top-1 %, HM). In the Average block, HM is computed from the average Base and Novel accuracies using Eq. ( 10 ). Dashes indicate unreported metrics; ProCAP reports three-seed means.

Method	Venue	Source	Target				Target Mean
Method	Venue	ImageNet	ImageNetV2	ImageNet-Sketch	ImageNet-A	ImageNet-R	Avg.
CLIP	ICML’21	66.73	60.83	46.15	47.77	73.96	57.18
CoOp	IJCV’22	71.51	64.20	47.99	49.71	75.21	59.28
CoCoOp	CVPR’22	71.02	64.07	48.75	50.63	76.18	59.91
MaPLe	CVPR’23	70.72	64.07	49.15	50.90	76.98	60.28
PromptSRC	ICCV’23	71.27	64.35	49.55	50.90	77.80	60.65

Table 2: Domain generalization (Top-1 %). Train on ImageNet and evaluate directly on ImageNet distribution shifts. Dashes indicate unreported values. DCPL ∗ denotes the domain-generalization variant defined by Mei et al. (2026) : its image-specific prompt is used only as an auxiliary input, while CLIP’s original image classification token is retained.

Method	Source	Target
Method	ImageNet	Avg.	Caltech101	Pets	Cars	Flowers	Food	Aircraft	SUN397	DTD	EuroSAT	UCF101
CoOp	71.51	63.88	93.70	89.14	64.51	68.71	85.30	18.47	64.15	41.92	46.39	66.55
CoCoOp	71.02	65.74	94.43	90.14	65.32	71.88	86.06	22.94	67.36	45.73	45.37	68.21
MaPLe	70.72	66.30	93.53	90.49	65.57	72.23	86.20	24.74	67.01	46.49	48.06	68.69
PromptSRC	71.27	65.81	93.60	90.25	65.70	70.25	86.15	23.90	67.10	46.87	45.50	68.75
TCP	71.40	66.29	93.97	91.25	64.69	71.21	86.69	23.45	67.15	44.35	51.45	68.73

Table 3: Cross-dataset transfer (Top-1 %). Train on ImageNet (16-shot) and evaluate directly on 10 target datasets.

Figure 3: Few-shot learning curves (Top-1 accuracy, %) across benchmark datasets with $K\in\{1,2,4,8,16\}$ training shots per class (CLIP ViT-B/16). We compare ProCAP against standard prompt-learning baselines under the same protocol. The Average panel reports the mean accuracy over all evaluated datasets.

Variant	Cross-Attn	Gaussian	KL	$L_{2}$	InfoNCE	DTD	EuroSAT	OxfordPets	StanfordCars	Avg.
MaPLe-style ( Khattak et al., 2023a )	–	–	–	–	–	67.65	80.15	96.58	73.47	79.46
w/o Cross-Attn	–	✓	✓	✓	✓	72.86	90.22	96.67	77.10	84.21
w/o Gaussian	✓	–	–	–	✓	72.35	89.92	96.60	76.80	83.92
w/o KL	✓	✓	–	✓	✓	72.88	90.12	97.05	77.55	84.40
w/o $L_{2}$	✓	✓	✓	–	✓	73.05	90.55	96.44	77.90	84.49
w/o InfoNCE	✓	✓	✓	✓	–	72.32	89.46	97.13	77.30	84.05

Table 4: Ablation study of ProCAP components (CLIP ViT-B/16, 16-shot). We report Top-1 accuracy (%) on representative texture, remote-sensing, and fine-grained datasets.

Figure 4: Effect of prompt length ( $N_{\text{ctx}}$ ) on base-to-novel generalization (averaged over EuroSAT, Caltech101, OxfordPets, and UCF101). Left: average base and novel accuracies. Right: harmonic mean (HM). Performance peaks at $N_{\text{ctx}}{=}4$ , indicating that a compact context is sufficient, while longer prompts tend to overfit under low-shot supervision.

Figure 5: Hyperparameter sensitivity of ProCAP. $\mathrm{HM}$ averaged over EuroSAT, OxfordPets, Caltech101, DTD, and Flowers102. (a) Sensitivity to $\lambda$ and $\beta$ . (b) Sensitivity to InfoNCE temperature init $\tau_{\mathrm{p}}$ (shown as $\tau$ in the plotted sweep) with $\lambda,\beta$ fixed.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Qualitative prediction comparison of MaPLe and ProCAP on Oxford Flowers. Columns show the input, ground truth, MaPLe prediction, and ProCAP prediction; green/red denote correct/incorrect predictions.

Figure 7: EigenCAM comparison of MaPLe and ProCAP on Oxford Flowers. Warmer regions indicate stronger contributions to the prediction.

Figure 8: Learned text-prompt means ( $\mu$ ) on EuroSAT base classes: (a) random initialization, (b) the trained baseline (96.86% base accuracy), and (c) ProCAP (98.43%).

Dataset	Base	Novel	$\mathrm{HM}$
Average	85.43 $\pm$ 0.18	77.61 $\pm$ 0.31	81.12 $\pm$ 0.24
ImageNet	77.33 $\pm$ 0.12	70.00 $\pm$ 0.18	73.48 $\pm$ 0.15
Caltech101	98.97 $\pm$ 0.10	95.31 $\pm$ 0.21	97.11 $\pm$ 0.15
OxfordPets	96.49 $\pm$ 0.17	98.21 $\pm$ 0.12	97.34 $\pm$ 0.14
StanfordCars	78.65 $\pm$ 0.29	77.26 $\pm$ 0.34	77.95 $\pm$ 0.31
Flowers102	98.39 $\pm$ 0.22	75.60 $\pm$ 0.71	85.50 $\pm$ 0.49

Appendix

Table 5: Three-seed stability of ProCAP on base-to-novel generalization (mean

\pm

standard deviation). The Average-row HM is the arithmetic mean of the 11 dataset-wise HMs; Table 1 reports aggregate HM from averaged Base and Novel accuracy.

Dataset	Method	Time (s)	FPS	Acc (%)
Flowers102	MaPLe-style ( Khattak et al., 2023a )	112.6	415.9	95.25
Flowers102	ProCAP (Full)	169.1	278.8	97.39
UCF101	MaPLe-style ( Khattak et al., 2023a )	131.3	415.6	85.21
UCF101	ProCAP (Full)	171.4	320.9	87.88

Appendix

Table 6: Training time, inference throughput, and accuracy on two representative datasets. All methods use CLIP ViT-B/16 ( Radford et al., 2021 ) in the same 16-shot setting; runtime is measured over a fixed 10-epoch benchmark on one NVIDIA A100 GPU with AMP.

Model	Epochs	Base	Novel	$\mathrm{HM}$
ProCAP	10	84.05	76.40	80.04
ProCAP	25	84.28	77.15	80.56
ProCAP	50	85.43	77.61	81.33

Appendix

Table 7: Effect of training duration on ProCAP for base-to-novel generalization, averaged over 11 datasets. HM is computed from the listed average Base and Novel accuracies using Eq. ( 10 ).

Dataset	Classes	Train	Val	Test	Description
ImageNet	1,000	1.28M	–	50,000	Generic object recognition
Caltech101	100	4,128	1,649	2,465	Generic object recognition
OxfordPets	37	2,944	736	3,669	Fine-grained pets
StanfordCars	196	6,509	1,635	8,041	Fine-grained cars
Flowers102	102	4,093	1,633	2,463	Fine-grained flowers
Food101	101	50,500	20,200	30,300	Fine-grained food

Appendix

Table 8: Datasets and benchmark split sizes used in our experiments.

Dataset	Template 1	Template 2
ImageNet	a photo of a {}	a close-up photo of a {}
Caltech101	a photo of a {}	an image of a {}
OxfordPets	a photo of a {}	a close-up photo of a {}
StanfordCars	a photo of a {} car	a close-up photo of a {} car
Flowers102	a photo of a {} flower	a close-up photo of a {} flower
Food101	a photo of {}	a close-up photo of {}

Appendix

Table 9: Representative prompt templates used for each dataset. The placeholder {} is replaced by the class name before tokenization.

Explore similar work

MMLoP: Multi-Modal Low-Rank Prompting for Efficient Vision-Language Adaptation
Feb 24, 2026Sajjad Ghiasvand, Haniyeh Ehsani Oskouie, Mahnoosh Alizadeh +1Cross-Modal Learning Prompt Learning
University of California, Santa Barbara · University of California, Los Angeles
CAPT: Confusion-Aware Prompt Tuning for Reducing Vision-Language Misalignment
Mar 3, 2026Maoyuan Shao, Yutong Gao, Xinyang Huang +3VLM Adaptation Vision-Language Alignment
School of Information Engineering, Minzu University of China · School of Artificial Intelligence, Beijing University of Posts and Telecommunications · National Library of China, Beijing, China +1
Concept-Constrained Prompt Learning for Few-Shot CLIP Adaptation
Jun 21, 2026Na Sang, Ding Ma, Rui Sang +1Prompt Learning VLM Adaptation
University of California, San Diego, La Jolla, CA, USA · Georgia Institute of Technology, Atlanta, GA, USA · Independent Researcher

Feb 24, 2026J/K move · Enter open · S save

MMLoP: Multi-Modal Low-Rank Prompting for Efficient Vision-Language Adaptation

Sajjad Ghiasvand, Haniyeh Ehsani Oskouie, Mahnoosh Alizadeh, Ramtin Pedarsani

University of California, Santa Barbara · University of California, Los Angeles

Cross-Modal Learning Prompt Learning VLM Adaptation Multimodal Foundation Models Vision-Language Alignment Visual Prompt Learning

Open paper AskPDF arXiv

Prompt learning has become a dominant paradigm for adapting vision-language models (VLMs) such as CLIP to downstream tasks without modifying pretrained weights. While extending prompts to both vision and text encoders across multiple transformer layers significantly boosts performance, it dramatically increases the number of trainable parameters, with state-of-the-art methods requiring millions of parameters and abandoning the parameter efficiency that makes prompt tuning attractive. In this work, we propose MMLoP (Multi-Modal Low-Rank Prompting), a framework that achieves deep multi-modal prompting with only 11.5K trainable parameters, comparable to early text-only methods like CoOp. MMLoP parameterizes vision and text prompts at each transformer layer through a low-rank factorization that constrains prompts to a compact subspace, providing parameter efficiency while motivating the need for our complementary regularization components. To further close the accuracy gap with state-of-the-art methods, we introduce three complementary components: a self-regulating consistency loss that anchors prompted representations to frozen zero-shot CLIP features at both the feature and logit levels, a uniform drift correction that removes the global embedding shift induced by prompt tuning to preserve class-discriminative structure, and a shared up-projection that couples vision and text prompts through a common low-rank factor to enforce cross-modal alignment. Extensive experiments across three benchmarks and 11 diverse datasets demonstrate that MMLoP achieves a highly favorable accuracy-efficiency tradeoff, outperforming the majority of existing methods including those with orders of magnitude more parameters, while achieving a harmonic mean of 79.70% on base-to-novel generalization. Code is available at https://github.com/sajjad-ucsb/MMLoP.

Abstract

Explore similar work

MMLoP: Multi-Modal Low-Rank Prompting for Efficient Vision-Language Adaptation

CAPT: Confusion-Aware Prompt Tuning for Reducing Vision-Language Misalignment

Concept-Constrained Prompt Learning for Few-Shot CLIP Adaptation