Re-calibrated Contrastive Loss for Transformation-Aware Prompt Conditioning in Vision-Language Models

Seungmin Oh; Seunghun Kang; Jongbin Ryu

cs.CVSep 7, 2026

Re-calibrated Contrastive Loss for Transformation-Aware Prompt Conditioning in Vision-Language Models

Authors: Seungmin Oh, Seunghun Kang, Jongbin Ryu

Organizations: Department of Artificial Intelligence, Ajou University, Suwon, South Korea · Department of Computer Engineering, Ajou University, Suwon, South Korea

Abstract

Ensuring effective transfer learning for vision-language models without compromising their generalization performance is crucial. However, many existing methods overlook data characteristics and simply reuse the training strategies adopted during pre-training. Specifically, they treat same-class samples as distinct instances and transform images independently of their paired text prompts, which makes model learning more difficult. We address these limitations through transformation-aware prompt conditioning and a re-calibrated contrastive loss. Fixed text descriptors identify the transformations applied to paired images, providing transformation-level consistency without altering class semantics. This design aligns the image and text branches at the transformation level, enabling richer representations while preserving the models' ability to generalize. In addition, our loss function mitigates positive-gradient dilution in soft-target cross-entropy when each anchor has multiple valid positives. During transfer, our approach treats same-class samples as positives rather than distinct instances, enabling the model to learn domain-specific features more effectively. Experiments across distribution shift, transfer learning, and few-shot settings demonstrate consistent improvements over existing approaches. Source code for our method is available at https://github.com/SoongE/ReCalCon.

Figures & tables

Figure 1: ID and OOD accuracy (%) of state-of-the-art models fine-tuned on the ImageNet dataset using CLIP ViT-B/16. Ours (red star) achieves the best ID-OOD performance.

Figure 2: Overview of the proposed training scheme. Each clean image and prompt produces a transformed image and a transformation-conditioned prompt. The encoders extract four types of contrastive matches: clean features $(F_{I},F_{T})$ and augmented features $(F_{I^{\prime}},F_{T^{\prime}})$ . These features construct an augmented contrastive space. We apply a re-calibrated contrastive loss to update the networks. In learnable text prompts, $[v_{1},v_{2},\dots,v_{n}]$ and $[a_{1},a_{2},\dots,a_{k}]$ denote learnable context vectors and fixed transformation descriptor embeddings, respectively.

Panel of Figure 3 — Figure 3: Illustration of positive sampling used by different contrastive objectives. (a) CLIP uses only paired samples as positives. (b) CLIP with soft cross-entropy adds same-class positives. (c) Ours augments the positive samples in the contrastive space from $F_{I^{\prime}}$ and $F_{T^{\prime}}$ .

Algorithm 1. PyTorch-like pseudocode of the proposed re-calibrated contrastive loss function.

⬇ "" " similarity_i: Similarities of an image to text # 2B, 2B similarity_t: Similarities of text to image # 2B, 2B targets: ground-truth of positive pairs # 2B, 2B B: batch size The comment in the middle of the code specifies the shape when computing the loss of similarity_i and targets. " "" # compute: adaptive re-calibration factor def recal ( similarities , targets , eps =1 e -8): exp_sim = torch . exp ( similarities ) pos = torch . sum ( exp_sim * targets , dim =-1) neg = torch . sum ( exp_sim * (1 - targets ), dim =-1) return pos / ( neg + eps ) # compute: our re-calibrated contrastive loss def loss ( similarities , targets , eps =1 e -8): prob = torch . softmax ( similarities , dim =-1) prob = torch . log ( prob * recal ( similarities , targets , eps ) + eps ) each_loss = torch . sum (- targets * prob , dim =-1) / ( torch . sum ( targets , dim =-1) + eps ) return torch . mean ( each_loss ) # compute: total re-calibrated contrastive loss def loss_t ( similarity_i , similarity_t , targets , eps =1 e -8): return ( loss ( similarity_i , targets , eps ) + loss ( similarity_t , targets , eps )) / 2

Table 4

Backbone	Archi.	Model	ImageNet								iWildCam
Backbone	Archi.	Model	ID	R	A	V2	Ske.	Obj.	O.M.	H.M.	ID	OOD	H.M.
Update	B/16	FFT	81.3	65.6	36.6	70.7	45.1	50.7	53.7	64.7	50.3	34.1	40.6
		LP-FT	81.7	73.5	42.5	72.1	50.3	55.1	58.7	68.3	49.7	34.7	40.9
		WiseFT	82.4	72.6	44.4	72.6	49.8	54.5	58.8	68.6	48.5	37.2	42.1
		FLYP	82.6	71.6	48.9	72.8	49.6	55.5	59.7	69.3	50.1	36.3	42.1
		ARF	82.7	75.6	50.3	72.8	51.8	55.8	61.3	70.4	-

Table 1: Experimental results of the distribution shift on ImageNet and iWildCam. We compare Ours with previous methods of backbone update and freeze training. The ‘O.M.’ and ‘H.M.’ denote the OOD mean and harmonic mean of ID and OOD mean.

Figure 4: (a-b) Training and evaluation losses for ImageNet transfer learning with ViT-B/16. (c-d) ID accuracy and harmonic mean under extended training.

Model	Caltech101	PCam	ImageNet	Flowers102	StanfordCars	iWildCam	Total Mean
Zero-shot	88.7	54.0	68.3	71.0	64.7	8.7	59.3
LinearProb	94.8	82.6	79.9	95.9	83.1	44.5	80.1
FFT	97.2	89.1	81.4	90.4	84.4	50.3	82.1
LP-FT	96.9	89.0	81.8	97.9	89.4	49.7	84.1
FLYP	97.6	90.3	82.6	97.7	89.6	50.1	84.6
Ours	97.9	89.4	83.3	98.9	91.4	51.3	85.4

Table 2: Experimental results of transfer learning on six downstream datasets. We use ViT-B/16 as a backbone network for all methods.

Figure 5: Experimental results of few-shot learning using a ResNet50 backbone. We compare training-based adaptation methods with zero-shot CLIP on 11 few-shot benchmarks.

Image	Text	ID	R	A	V2	Ske.	Obj.	OOD Mean	H.M.
CLIP		63.4	69.3	31.6	56.0	42.3	44.4	48.7	55.1
Random Crop	Fix	78.0	59.6	29.8	66.0	41.5	47.6	48.9	60.1
Random Crop	Misalign.	71.2	56.6	27.1	60.5	38.0	43.4	45.1	55.2
RandAug	Fix	78.6	61.7	31.8	68.2	43.5	48.3	50.7	61.6
RandAug	Misalign.	73.1	58.4	29.4	63.0	40.3	44.9	47.2	57.3
RandAug	Align.	79.4	62.4	32.6	68.2	44.5	49.2	51.4	62.4

Table 3: Experimental results of transformation-aware prompt conditioning on ViT-B/32 backbone. A misaligned fixed descriptor consistently degrades performance, whereas the correct transformation descriptor improves performance.

Table 4: Experimental results of our re-calibrated contrastive loss and contrastive space on ViT-B/32 backbone. Performance is measured in terms of accuracy with respect to the applied loss function. I

\leftrightarrow

T denotes image-text and text-image match ( e.g , red region in Figure 2 ) and I ′

\leftrightarrow

T ′ is a transformation-conditioned augmented match ( e.g , blue region in Figure 2 ). [I I ′ ]

\leftrightarrow

[T T ′ ] represents the four contrastive matches of all clean and transformation-conditioned augmented image-text pairs ( e.g , whole region of the augmented contrastive space).

Objective / Configuration	ID	R	A	V2	Ske.	Obj.	OOD Mean	H.M.
InfoNCE	73.7	56.3	26.1	63.1	40.4	46.1	46.4	57.0
Focal	78.5	63.3	32.8	67.5	43.3	48.3	51.0	61.8
SoftCE (=SupCon)	78.4	63.1	32.4	67.6	43.4	48.4	51.0	61.8
$\kern 4.62497pt+$ Re-calibrated	79.0	62.6	32.6	67.8	44.1	48.6	51.1	62.1
$\;\;\kern 4.62497pt+$ Multi-sample fusion	79.4	62.4	32.6	68.2	44.5	49.2	51.4	62.4

Table 5: Objective and component comparison on ImageNet distribution using ViT-B/32. All configurations use transformation-aware prompt conditioning and the augmented contrastive space. The last two rows incrementally add re-calibration and multi-sample fusion to SoftCE.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Backbone	B/16	B/16 †	B/32	B/32 †
Batch size	640	2048	512	2048
Epoch	25	25	25	25
LR	$1.25e^{-6}$	-	$1.25e^{-6}$	-
Adapter LR	$1.0e^{-4}$	$1.0e^{-3}$	$1.0e^{-4}$	$1.0e^{-3}$
Weight decay	$1.0e^{-2}$	$1.0e^{-1}$	$1.0e^{-2}$	$1.0e^{-2}$

Appendix

Table 12: Details of hyperparameter settings for the distribution shift experiments on ImageNet.

Backbone	B/16	B/16 †	B/32	B/32 †
Batch size	640	2048	512	2048
Epoch	45	25	45	25
LR	$5.0e^{-6}$	-	$5.0e^{-6}$	-
Adapter LR	$1.0e^{-4}$	$1.0e^{-2}$	$1.0e^{-4}$	$1.0e^{-2}$
Weight decay	$1.0e^{-2}$	$1.0e^{-2}$	$1.0e^{-2}$	$1.0e^{-2}$

Appendix

Table 13: Details of hyperparameter settings for the distribution shift experiments on iWildCam.

$\|\bm{P}(i)\|$	1	2	4	8	16
Gradient ratio (ours / SoftCE)	1.46	1.68	1.79	1.89	1.81

Appendix

Table 14: Gradient-norm ratio between the re-calibrated objective and SoftCE.

Batch size $B$	8	16	32	64
Runtime (ours / FLYP)	$1.0\times$	$1.1\times$	$1.6\times$	$1.7\times$
Peak memory (ours / FLYP)	$1.0\times$	$1.2\times$	$1.4\times$	$1.6\times$

Appendix

Table 15: Per-step training overhead relative to FLYP.

$k$	ID	R	A	V2	Ske.	Obj.	OOD Mean	H.M.
1	75.8	61.6	31.5	64.9	41.9	48.7	49.7	60.0
2 (default)	79.4	62.4	32.6	68.2	44.5	49.2	51.4	62.4
3	79.4	63.2	32.8	68.6	44.6	49.3	51.7	62.6

Appendix

Table 16: Sensitivity to the number of sampled image transformations

k

Figure 6: Additional experimental results of extended training with ViT-B/16.

VLM Adaptation

Vision-language model adaptation tailors pretrained image-text and multimodal models to new tasks and domains while retaining transferable capabilities.

See all 538 papers

ProCAP: Probabilistic Cross-Attentive Prompt Learning for Vision-Language Models
Sep 24, 2026Hiwa Azeez Abbas, Fatemeh Daneshfar, Moloud AbdarVLM Adaptation Visual Prompt Learning
University of Kurdistan, Sanandaj, Iran · Department of Computer Engineering, University of Kurdistan, Sanandaj, Iran · CHIRP, Child Health Research Centre, The University of Queensland, Brisbane, Australia
Neutral-Reference Prompting for Vision-Language Models
May 15, 2026Senmao Tian, Xiang Wei, Shunli ZhangVision-Language Models VLM Adaptation
Beijing Jiaotong University, Beijing, China.
AdaBoosting Text Prompts for Vision-Language Models
Jul 1, 2026Seokhee Jin, Changhwan Sung, Sunung Mun +2Vision-Language Models Ensemble Learning
KT Corporation, Republic of Korea · Pohang University of Science and Technology (POSTECH), Republic of Korea · National AI Research Lab, Republic of Korea

Sep 24, 2026J/K move · Enter open · S save

ProCAP: Probabilistic Cross-Attentive Prompt Learning for Vision-Language Models

Hiwa Azeez Abbas, Fatemeh Daneshfar, Moloud Abdar

University of Kurdistan, Sanandaj, Iran · Department of Computer Engineering, University of Kurdistan, Sanandaj, Iran · CHIRP, Child Health Research Centre, The University of Queensland, Brisbane, Australia

VLM Adaptation Visual Prompt Learning Cross-Modal Learning Few-Shot Learning Domain Generalization

Open paper AskPDF arXiv

Pre-trained vision-language models such as CLIP can recognize new categories via prompting, but they often struggle when labeled data are scarce or the test distribution shifts. Prompt learning adapts only a small set of parameters while keeping the backbone frozen, yet many existing multimodal prompt learners couple the visual and textual branches weakly and can be brittle in low-shot regimes. We propose ProCAP, a probabilistic cross-attentive prompt learning framework that improves cross-modal interaction and training stability without updating any CLIP weights: it learns both visual and textual prompt tokens and links them through stacked bidirectional multi-head cross-attention so the two branches refine each other across prompt depth. To reduce overfitting under limited supervision, we parameterize prompt tokens with Gaussian means and variances and regularize them with lightweight KL and L2 penalties, and we further add a compact symmetric InfoNCE head that aligns cross-attended image features with class-level text representations in a shared low-dimensional space. Across few-shot base-to-novel generalization on 11 datasets, cross-dataset transfer, and domain generalization on ImageNet shift benchmarks, ProCAP achieves strong aggregate base-to-novel performance and competitive transfer performance while keeping the CLIP backbone unchanged.

Abstract

VLM Adaptation

Explore similar work

ProCAP: Probabilistic Cross-Attentive Prompt Learning for Vision-Language Models

Neutral-Reference Prompting for Vision-Language Models

AdaBoosting Text Prompts for Vision-Language Models