A-TPT: Angular Diversity Calibration Properties for Test-Time Prompt Tuning of Vision-Language Models
Authors: Shihab Aaqil Ahamed, Udaya S. K. P. Miriya Thanthrige, Ranga Rodrigo, Muhammad Haris Khan
Organizations: Dept. of Electronic and Telecommunication Engineering, University of Moratuwa, Sri Lanka · Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE
Test-time prompt tuning (TPT) has emerged as a promising technique for adapting large vision-language models (VLMs) to unseen tasks without relying on labeled data. However, the lack of dispersion between textual features can hurt calibration performance, which raises concerns about VLMs' reliability, trustworthiness, and safety. Current TPT approaches primarily focus on improving prompt calibration by either maximizing average textual feature dispersion or enforcing orthogonality constraints to encourage angular separation. However, these methods may not always have optimal angular separation between class-wise textual features, which implies overlooking the critical role of angular diversity. To address this, we propose A-TPT, a novel TPT framework that introduces angular diversity to encourage uniformity in the distribution of normalized textual features induced by corresponding learnable prompts. This uniformity is achieved by maximizing the minimum pairwise angular distance between features on the unit hypersphere. We show that our approach consistently surpasses state-of-the-art TPT methods in reducing the aggregate average calibration error while maintaining comparable accuracy through extensive experiments with various backbones on different datasets. Notably, our approach exhibits superior zero-shot calibration performance on natural distribution shifts and generalizes well to medical datasets. We provide extensive analyses, including theoretical aspects, to establish the grounding of A-TPT. These results highlight the potency of promoting angular diversity to achieve well-dispersed textual features, significantly improving VLM calibration during test-time adaptation. Our code will be made publicly available.
Figures & tables
Figure 1
Figure 3: t-SNE visualization of class-wise embedded textual features with CLIP RN50 model on the fine-grained classification dataset ( Fei-Fei et al., 2004 ) for (a) hard prompts and (b) tuned prompts. In both subfigures, each unique color represents a distinct prompt in the prompt visualization (left) and a distinct class in the class visualization (right). The legends belong to the prompt visualization (left) for both subfigures.
Table 3
Table 2: Comparison of methods across fine-grained datasets for Accuracy (Acc.) and Expected Calibration Error (ECE) with CLIP ViT-B/16 and CLIP RN50 pre-trained backbone for both N>∣D∣ and N<∣D∣ cases. The overall top best-performing result is in bold.
Table 7: Comparison of methods when using CoOp ( Zhou et al., 2022b ) and CoCoOp ( Zhou et al., 2022a ) as a baseline across fine-grained datasets for Accuracy (Acc.) and Expected Calibration Error (ECE) with CLIP ViT-B/16 pre-trained backbone. The overall best-performing result is in bold.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Regularization
Asymptotic
Time (s)
Memory
ECE
Complexity
/Batch
(MiB)
C-TPT
L2 Regularization
O(N⋅∣D∣)
1.055
21838
6.23
O-TPT
Orthogonality
O(N2⋅∣D∣)
1.064
23740
3.97
Constraints
A-TPT
Angular
(Ours)
Diversity
O(N2⋅∣D∣)
1.058
21840
2.96
Appendix
Table 8: Comparison of different regularization methods on ImageNet-V2 ( Recht et al., 2019 ) . The angular diversity achieves lower ECE with negligible computational overhead.
Table 8
Method
Metric
CLIP ViT-B/16
CLIP RN50
Baseline
Acc.
62.99
51.75
ECE
3.93
4.04
TPT
Acc.
64.84
54.00
ECE
10.23
11.48
C-TPT
Acc.
64.60
53.64
ECE
4.74
6.11
Appendix
Table 11: Comparison of Weighted Average Accuracy and Expected Calibration Error (ECE) with CLIP ViT-B/16 and CLIP RN50 backbones.
Table 13: Static Calibration Error (SCE) (10e-2) performance comparison across fine-grained datasets with CLIP ViT-B/16 and CLIP RN50 backbone. The overall best-performing result is in bold.
Figure 12
Figure 10: Reliability and confidence histogram diagrams for EuroSAT on fine-grained classification tasks with CLIP ViT-B/16 ( N<∣D∣ , left ) and CLIP RN50 ( N<∣D∣ , right ) backbone.
Figure 11: Reliability diagrams for (ImageNet-V2,K) on natural distribution shifts with CLIP ViT-B/16 ( N>∣D∣ ) and CLIP RN50 ( N<∣D∣ ) backbone, respectively.
Figure 12: Comparison of mean cosine similarity changes on a natural distribution dataset (ImageNet-V2) ( Recht et al., 2019 ) ) with CLIP ViT-B/16 backbone. When N>∣D∣ our A-TPT offers slightly higher but more consistent cosine similarity values among text features for all the data points.
Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE · Dept. of Electronic and Telecommunication Engineering, University of Moratuwa, Sri Lanka · Albukhary International University, Alor Setar, Malaysia
School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China · NLPR & MAIS, Institute of Automation, Chinese Academy of Sciences, Beijing, China · Independent Researcher