VLM Calibration

VLM: Vision-Language Model

Overview

Vision-language model calibration studies whether confidence in image-text predictions reflects their actual correctness, supporting safer decisions such as abstaining or requesting more evidence. Approaches include temperature and logit scaling, prompt and embedding calibration, conformal prediction, confidence probes, and training or decoding methods that use visual evidence to detect hallucinations. These methods are evaluated across zero-shot classification, visual question answering, detection, and specialized settings such as medical imaging, documents, and autonomous driving, including under distribution shift. Calibration can improve selective prediction, but standard metrics may not reveal whether confidence tracks visual grounding or sound reasoning, and reliability can break under severe shifts or deliberate perturbations.

Latest papers

91 papers11 papers in the last four weeks, up 83% on the four weeks before. 0.1% of all new papers.

Oct 30, 2025

A-TPT: Angular Diversity Calibration Properties for Test-Time Prompt Tuning of Vision-Language Models

Test-time prompt tuning (TPT) has emerged as a promising technique for adapting large vision-language models (VLMs) to unseen tasks without relying on labeled data. However, the lack of dispersion between textual features can hurt calibration performance, which raises concerns about VLMs' reliability, trustworthiness, and safety. Current TPT approaches primarily focus on improving prompt calibration by either maximizing average textual feature dispersion or enforcing orthogonality constraints to encourage angular separation. However, these methods may not always have optimal angular separation between class-wise textual features, which implies overlooking the critical role of angular diversity. To address this, we propose A-TPT, a novel TPT framework that introduces angular diversity to encourage uniformity in the distribution of normalized textual features induced by corresponding learnable prompts. This uniformity is achieved by maximizing the minimum pairwise angular distance between features on the unit hypersphere. We show that our approach consistently surpasses state-of-the-art TPT methods in reducing the aggregate average calibration error while maintaining comparable accuracy through extensive experiments with various backbones on different datasets. Notably, our approach exhibits superior zero-shot calibration performance on natural distribution shifts and generalizes well to medical datasets. We provide extensive analyses, including theoretical aspects, to establish the grounding of A-TPT. These results highlight the potency of promoting angular diversity to achieve well-dispersed textual features, significantly improving VLM calibration during test-time adaptation. Our code will be made publicly available.