Re-calibrated Contrastive Loss for Transformation-Aware Prompt Conditioning in Vision-Language Models
Authors: Seungmin Oh, Seunghun Kang, Jongbin Ryu
Organizations: Department of Artificial Intelligence, Ajou University, Suwon, South Korea · Department of Computer Engineering, Ajou University, Suwon, South Korea
Ensuring effective transfer learning for vision-language models without compromising their generalization performance is crucial. However, many existing methods overlook data characteristics and simply reuse the training strategies adopted during pre-training. Specifically, they treat same-class samples as distinct instances and transform images independently of their paired text prompts, which makes model learning more difficult. We address these limitations through transformation-aware prompt conditioning and a re-calibrated contrastive loss. Fixed text descriptors identify the transformations applied to paired images, providing transformation-level consistency without altering class semantics. This design aligns the image and text branches at the transformation level, enabling richer representations while preserving the models' ability to generalize. In addition, our loss function mitigates positive-gradient dilution in soft-target cross-entropy when each anchor has multiple valid positives. During transfer, our approach treats same-class samples as positives rather than distinct instances, enabling the model to learn domain-specific features more effectively. Experiments across distribution shift, transfer learning, and few-shot settings demonstrate consistent improvements over existing approaches. Source code for our method is available at https://github.com/SoongE/ReCalCon.
Figures & tables
Figure 1: ID and OOD accuracy (%) of state-of-the-art models fine-tuned on the ImageNet dataset using CLIP ViT-B/16. Ours (red star) achieves the best ID-OOD performance.
Figure 2: Overview of the proposed training scheme. Each clean image and prompt produces a transformed image and a transformation-conditioned prompt. The encoders extract four types of contrastive matches: clean features (FI,FT) and augmented features (FI′,FT′) . These features construct an augmented contrastive space. We apply a re-calibrated contrastive loss to update the networks. In learnable text prompts, [v1,v2,…,vn] and [a1,a2,…,ak] denote learnable context vectors and fixed transformation descriptor embeddings, respectively.
Figure 3: Illustration of positive sampling used by different contrastive objectives. (a) CLIP uses only paired samples as positives. (b) CLIP with soft cross-entropy adds same-class positives. (c) Ours augments the positive samples in the contrastive space from FI′ and FT′ .
Algorithm 1. PyTorch-like pseudocode of the proposed re-calibrated contrastive loss function.
⬇ "" " similarity_i: Similarities of an image to text # 2B, 2B similarity_t: Similarities of text to image # 2B, 2B targets: ground-truth of positive pairs # 2B, 2B B: batch size The comment in the middle of the code specifies the shape when computing the loss of similarity_i and targets. " "" # compute: adaptive re-calibration factor def recal ( similarities , targets , eps =1 e -8): exp_sim = torch . exp ( similarities ) pos = torch . sum ( exp_sim * targets , dim =-1) neg = torch . sum ( exp_sim * (1 - targets ), dim =-1) return pos / ( neg + eps ) # compute: our re-calibrated contrastive loss def loss ( similarities , targets , eps =1 e -8): prob = torch . softmax ( similarities , dim =-1) prob = torch . log ( prob * recal ( similarities , targets , eps ) + eps ) each_loss = torch . sum (- targets * prob , dim =-1) / ( torch . sum ( targets , dim =-1) + eps ) return torch . mean ( each_loss ) # compute: total re-calibrated contrastive loss def loss_t ( similarity_i , similarity_t , targets , eps =1 e -8): return ( loss ( similarity_i , targets , eps ) + loss ( similarity_t , targets , eps )) / 2
Table 4
Backbone
Archi.
Model
ImageNet
iWildCam
ID
R
A
V2
Ske.
Obj.
O.M.
H.M.
ID
OOD
H.M.
Update
B/16
FFT
81.3
65.6
36.6
70.7
45.1
50.7
53.7
64.7
50.3
34.1
40.6
LP-FT
81.7
73.5
42.5
72.1
50.3
55.1
58.7
68.3
49.7
34.7
40.9
WiseFT
82.4
72.6
44.4
72.6
49.8
54.5
58.8
68.6
48.5
37.2
42.1
FLYP
82.6
71.6
48.9
72.8
49.6
55.5
59.7
69.3
50.1
36.3
42.1
ARF
82.7
75.6
50.3
72.8
51.8
55.8
61.3
70.4
-
Table 1: Experimental results of the distribution shift on ImageNet and iWildCam. We compare Ours with previous methods of backbone update and freeze training. The ‘O.M.’ and ‘H.M.’ denote the OOD mean and harmonic mean of ID and OOD mean.
Figure 4: (a-b) Training and evaluation losses for ImageNet transfer learning with ViT-B/16. (c-d) ID accuracy and harmonic mean under extended training.
Model
Caltech101
PCam
ImageNet
Flowers102
StanfordCars
iWildCam
Total Mean
Zero-shot
88.7
54.0
68.3
71.0
64.7
8.7
59.3
LinearProb
94.8
82.6
79.9
95.9
83.1
44.5
80.1
FFT
97.2
89.1
81.4
90.4
84.4
50.3
82.1
LP-FT
96.9
89.0
81.8
97.9
89.4
49.7
84.1
FLYP
97.6
90.3
82.6
97.7
89.6
50.1
84.6
Ours
97.9
89.4
83.3
98.9
91.4
51.3
85.4
Table 2: Experimental results of transfer learning on six downstream datasets. We use ViT-B/16 as a backbone network for all methods.
Figure 5: Experimental results of few-shot learning using a ResNet50 backbone. We compare training-based adaptation methods with zero-shot CLIP on 11 few-shot benchmarks.
Image
Text
ID
R
A
V2
Ske.
Obj.
OOD Mean
H.M.
CLIP
63.4
69.3
31.6
56.0
42.3
44.4
48.7
55.1
Random Crop
Fix
78.0
59.6
29.8
66.0
41.5
47.6
48.9
60.1
Random Crop
Misalign.
71.2
56.6
27.1
60.5
38.0
43.4
45.1
55.2
RandAug
Fix
78.6
61.7
31.8
68.2
43.5
48.3
50.7
61.6
RandAug
Misalign.
73.1
58.4
29.4
63.0
40.3
44.9
47.2
57.3
RandAug
Align.
79.4
62.4
32.6
68.2
44.5
49.2
51.4
62.4
Table 3: Experimental results of transformation-aware prompt conditioning on ViT-B/32 backbone. A misaligned fixed descriptor consistently degrades performance, whereas the correct transformation descriptor improves performance.
Table 4: Experimental results of our re-calibrated contrastive loss and contrastive space on ViT-B/32 backbone. Performance is measured in terms of accuracy with respect to the applied loss function. I ↔ T denotes image-text and text-image match ( e.g , red region in Figure 2 ) and I ′ ↔ T ′ is a transformation-conditioned augmented match ( e.g , blue region in Figure 2 ). [I I ′ ] ↔ [T T ′ ] represents the four contrastive matches of all clean and transformation-conditioned augmented image-text pairs ( e.g , whole region of the augmented contrastive space).
Objective / Configuration
ID
R
A
V2
Ske.
Obj.
OOD Mean
H.M.
InfoNCE
73.7
56.3
26.1
63.1
40.4
46.1
46.4
57.0
Focal
78.5
63.3
32.8
67.5
43.3
48.3
51.0
61.8
SoftCE (=SupCon)
78.4
63.1
32.4
67.6
43.4
48.4
51.0
61.8
+ Re-calibrated
79.0
62.6
32.6
67.8
44.1
48.6
51.1
62.1
+ Multi-sample fusion
79.4
62.4
32.6
68.2
44.5
49.2
51.4
62.4
Table 5: Objective and component comparison on ImageNet distribution using ViT-B/32. All configurations use transformation-aware prompt conditioning and the augmented contrastive space. The last two rows incrementally add re-calibration and multi-sample fusion to SoftCE.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Backbone
B/16
B/16 †
B/32
B/32 †
Batch size
640
2048
512
2048
Epoch
25
25
25
25
LR
1.25e−6
-
1.25e−6
-
Adapter LR
1.0e−4
1.0e−3
1.0e−4
1.0e−3
Weight decay
1.0e−2
1.0e−1
1.0e−2
1.0e−2
Appendix
Table 12: Details of hyperparameter settings for the distribution shift experiments on ImageNet.
Backbone
B/16
B/16 †
B/32
B/32 †
Batch size
640
2048
512
2048
Epoch
45
25
45
25
LR
5.0e−6
-
5.0e−6
-
Adapter LR
1.0e−4
1.0e−2
1.0e−4
1.0e−2
Weight decay
1.0e−2
1.0e−2
1.0e−2
1.0e−2
Appendix
Table 13: Details of hyperparameter settings for the distribution shift experiments on iWildCam.
∣P(i)∣
1
2
4
8
16
Gradient ratio (ours / SoftCE)
1.46
1.68
1.79
1.89
1.81
Appendix
Table 14: Gradient-norm ratio between the re-calibrated objective and SoftCE.
Batch size B
8
16
32
64
Runtime (ours / FLYP)
1.0×
1.1×
1.6×
1.7×
Peak memory (ours / FLYP)
1.0×
1.2×
1.4×
1.6×
Appendix
Table 15: Per-step training overhead relative to FLYP.
k
ID
R
A
V2
Ske.
Obj.
OOD Mean
H.M.
1
75.8
61.6
31.5
64.9
41.9
48.7
49.7
60.0
2 (default)
79.4
62.4
32.6
68.2
44.5
49.2
51.4
62.4
3
79.4
63.2
32.8
68.6
44.6
49.3
51.7
62.6
Appendix
Table 16: Sensitivity to the number of sampled image transformations k .
Figure 6: Additional experimental results of extended training with ViT-B/16.
Vision-language model adaptation tailors pretrained image-text and multimodal models to new tasks and domains while retaining transferable capabilities.
Pre-trained vision-language models such as CLIP can recognize new categories via prompting, but they often struggle when labeled data are scarce or the test distribution shifts. Prompt learning adapts only a small set of parameters while keeping the backbone frozen, yet many existing multimodal prompt learners couple the visual and textual branches weakly and can be brittle in low-shot regimes. We propose ProCAP, a probabilistic cross-attentive prompt learning framework that improves cross-modal interaction and training stability without updating any CLIP weights: it learns both visual and textual prompt tokens and links them through stacked bidirectional multi-head cross-attention so the two branches refine each other across prompt depth. To reduce overfitting under limited supervision, we parameterize prompt tokens with Gaussian means and variances and regularize them with lightweight KL and L2 penalties, and we further add a compact symmetric InfoNCE head that aligns cross-attended image features with class-level text representations in a shared low-dimensional space. Across few-shot base-to-novel generalization on 11 datasets, cross-dataset transfer, and domain generalization on ImageNet shift benchmarks, ProCAP achieves strong aggregate base-to-novel performance and competitive transfer performance while keeping the CLIP backbone unchanged.
Hiwa Azeez Abbas, Fatemeh Daneshfar, Moloud Abdar
University of Kurdistan, Sanandaj, Iran · Department of Computer Engineering, University of Kurdistan, Sanandaj, Iran · CHIRP, Child Health Research Centre, The University of Queensland, Brisbane, Australia
Efficient transfer learning of vision-language models (VLMs) commonly suffers from a Base-New Trade-off (BNT): improving performance on unseen (new) classes often degrades accuracy on known (base) classes. Addressing how to boost recognition of unseen classes without sacrificing known-class performance remains a central challenge. Existing work often simplistically attributes the BNT to overfitting on known classes. We observe an interesting phenomenon: VLMs frequently exhibit asymmetric confusion on certain downstream data, i.e., samples of class A are systematically mispredicted as class B, while the reverse confusion (B to A) rarely occurs. For known classes, this kind of bias can be mitigated by tuning using a cross-entropy loss, but for unseen classes, such pretraining-induced bias persists and harms generalization. Motivated by this, we propose NeRP, a plug-and-play prompting correction strategy that improves discrimination on unseen classes without modifying model parameters. NeRP leverages neutral text prompts and reference images to measure class-wise prior preferences along the pre-trained inter-class geometry, and combines them with the sample likelihood to obtain the model's surrogate score. If, for a given sample, the prior strongly favors the current prediction while the observed evidence is clearly insufficient, we perform a local flip between easily confusable class pairs, thereby correcting prior-dominated mispredictions. Extensive experiments across multiple backbones and 15 few-shot and cross-domain benchmarks show that NeRP substantially improves accuracy on unseen classes while preserving known-class prediction performance.
The classification accuracy of pretrained Vision-Language Models (VLMs) relies on the quality of the text prompts. Handcrafted templates and Large Language Model (LLM)-generated descriptions not only make predictions more interpretable, but also enable reuse of the same prompts across heterogeneous VLMs. Recent works construct task-adapted text prompts with a small number of labeled images. However, existing few-shot text prompting methods do not explicitly focus on misclassified examples during prompt construction, leading to only marginal improvements even as more shots become available. To fully exploit few-shot supervision, we propose Text Prompt Boosting (TPB), an AdaBoost-inspired framework that treats each text-prompt-based classifier as a weak learner and sequentially aggregates them into a strong ensemble by explicitly targeting hard, misclassified examples. Extensive experiments show that TPB preserves task-intrinsic, model-agnostic cues in text space, enabling robust cross-model transfer. Across eleven classification benchmarks, TPB improves accuracy on the source model and preserves shot-driven gains when transferred to larger, more capable VLMs, where existing methods struggle to sustain such improvements.
Seokhee Jin, Changhwan Sung, Sunung Mun +2
KT Corporation, Republic of Korea · Pohang University of Science and Technology (POSTECH), Republic of Korea · National AI Research Lab, Republic of Korea