Domain Recentering and Confidence-Weighted Prior Calibration for Vision-Language Models

Youngeun Seol; Jimin Shin; Heeseo Yoon; Uiwon Hwang

cs.CVSep 24, 2026

Domain Recentering and Confidence-Weighted Prior Calibration for Vision-Language Models

Authors: Youngeun Seol, Jimin Shin, Heeseo Yoon, Uiwon Hwang

Organizations: Department of Computer Science and Engineering, Ewha Womans University

Abstract

Vision-language models such as CLIP achieve strong zero-shot classification, yet under distribution shift, visual embeddings drift from fixed text embeddings. Training-free calibration avoids the per-sample optimization of prompt learning, but prior feature calibration gives each image the full bias of one hard cluster. We propose Domain Recentering with Confidence Calibration (DRC), a training-free method adapting CLIP from a set of unlabeled target images. DRC fits a Gaussian mixture once and subtracts from each embedding a posterior-weighted average of component means. It then removes residual class preference with a log-prior correction, estimating the prior from confidence-weighted predictions. Among compared methods, DRC achieves the highest average accuracy on cross-domain datasets, exceeding zero-shot CLIP by 4.13 and 5.07 points with ViT-B/16 and ResNet-50, with gains over CLIP also holding under ImageNet distribution shifts.

Figures & tables

Figure 1: Overview of DRC. DRC recenters CLIP image features with a GMM-based soft bias correction and calibrates the resulting logits with a confidence-weighted class prior.

Method	Aircraft	Caltech101	Cars	DTD	EuroSAT	Flower102	Food101	Pets	SUN397	UCF101	Average
CLIP-ResNet-50	16.11	87.26	55.89	40.37	25.79	62.77	74.82	82.97	60.85	59.48	56.63
CoOp	15.12	86.53	55.32	37.29	26.20	61.55	75.59	87.00	58.15	59.05	56.18
CoCoOp	14.61	87.38	56.22	38.53	28.73	65.57	76.20	88.39	59.61	57.10	57.23
TPT	17.58	87.02	58.46	40.84	28.33	62.69	74.88	84.49	61.46	60.82	57.66
DiffTPT	17.60	86.89	60.71	40.72	41.04	63.53	79.21	83.40	62.72	62.67	59.85
UMFC	17.76	85.48	56.25	42.85	37.63	65.57	77.47	86.10	60.49	63.39	59.30

Table 1: Results on the Cross-Domain Benchmark with CLIP-ResNet-50 and CLIP-ViT-B/16. CoOp and CoCoOp serve as supervised transfer baselines trained on ImageNet with 16 labeled samples per class. CLIP, CoOp, CoCoOp, and TPT results are taken from the original TPT paper [ 9 ] , and DiffTPT results are taken from the original DiffTPT paper [ 2 ] .

Method	ImageNet	-A	-V2	-R	-S	Average	OOD Average
CLIP-ResNet-50	59.81	23.24	52.91	60.72	35.48	46.43	43.09
CoOp	63.33	23.06	55.40	56.60	34.67	46.61	42.43
CoCoOp	62.81	23.32	55.72	57.74	34.48	46.81	42.82
TPT	60.74	26.67	54.70	59.11	35.09	47.26	43.89
DiffTPT	60.80	31.06	55.80	58.80	37.10	48.71	45.69
UMFC	59.70	23.47	52.85	60.94	35.82	46.56	43.27

Table 2: Results on the OOD Benchmark with CLIP-ResNet-50 and CLIP-ViT-B/16. CoOp and CoCoOp serve as supervised transfer baselines trained on ImageNet with 16 labeled samples per class. CLIP, CoOp, CoCoOp, and TPT results are taken from the original TPT paper [ 9 ] , and DiffTPT results are taken from the original DiffTPT paper [ 2 ] .

	Prior calibration
Recentering	without	with
None	64.59	68.35
Hard	66.89	68.57
Soft	67.01	68.72

Table 3: Ablation on the cross-domain benchmark with CLIP-ViT-B/16, reporting the average accuracy over the ten datasets. The bottom-right entry is DRC.

Explore similar work

Improving CLIP Adaptation by Breaking Tail Alignment for Source-Free Cross-Domain Few-Shot Learning
May 28, 2026Shuai Yi, Yixiong Zou, Yuhua Li +1Vision-Language Model Adaptation Contrastive Language-Image Pre-Training Model
Institute of Artificial Intelligence, Huazhong University of Science and Technology, Wuhan, China · School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan, China.
Respect Your Zero-Shot Uncertainty: Conservative Calibration for Test-Time-Adapted Vision-Language Models
Aug 6, 2026Jingyan Jiang, Yaru Sun, Xiao Chen +5Vision-Language Model Adaptation Zero-Shot Robustness
1Shenzhen Technology University · 3Shenzhen University · 2Tsinghua University +1
CoDoL: Conditional Domain Prompt Learning for Out-of-Distribution Generalization
Sep 18, 2025Min Zhang, Yuyin Wang, Zhongxiang Dai +4Contrastive Language-Image Pre-Training Model Multiple Prompt Learning Frameworks
East China Normal University · Xidian University · Zhejiang Key Laboratory of Artificial Intelligence of Things (AIoT) Network and Data Security +3

May 28, 2026, cs.CVJ/K move · Enter open · S save

Improving CLIP Adaptation by Breaking Tail Alignment for Source-Free Cross-Domain Few-Shot Learning

Shuai Yi, Yixiong Zou, Yuhua Li, Ruixuan Li

Institute of Artificial Intelligence, Huazhong University of Science and Technology, Wuhan, China · School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan, China.

Vision-Language Model Adaptation Contrastive Language-Image Pre-Training Model Few-Shot Learning Domain Adaptation Value Alignment

Open paper AskPDF arXiv

Vision-Language Models (VLMs) such as CLIP demonstrate strong zero-shot generalization, but their performance significantly degrades in cross-domain scenarios with scarce target-domain training data (Cross-Domain Few-Shot Learning, CDFSL). In this paper, we focus on the target-domain few-shot finetuning in the CLIP-based CDFSL task. Prevailing finetuning paradigms uniformly align all image patch tokens with their corresponding textual embeddings. However, we find a counterintuitive phenomenon: actively pushing away certain low-similarity image tokens, termed "tail tokens", from their textual embeddings consistently improves target-domain performance. We delve into this phenomenon and provide a novel interpretation: under great domain shifts and scarce training data, the model can hardly extract semantic information from visual inputs; therefore, the common belief of alignment is valid only for tokens already containing sufficient semantic information; for tail tokens, forcing the alignment would lead to excessive overfitting to the scarce training, while breaking the alignment is more useful. Motivated by this, we propose Adaptive Tail-Head Alignment (ATHA), a novel fine-tuning strategy for CLIP that transforms the conventional uniform alignment paradigm to an adaptive alignment paradigm, with both alignment strengthening and weakening. Extensive experiments on four challenging CDFSL benchmarks validate our state-of-the-art performance. Our code is available at https://github.com/shuaiyi308/ATHA.

Abstract

Explore similar work

Improving CLIP Adaptation by Breaking Tail Alignment for Source-Free Cross-Domain Few-Shot Learning

Respect Your Zero-Shot Uncertainty: Conservative Calibration for Test-Time-Adapted Vision-Language Models

CoDoL: Conditional Domain Prompt Learning for Out-of-Distribution Generalization