VLM Calibration

VLM: Vision-Language Model

Vision-language model calibration studies whether confidence in image-text predictions reflects their actual correctness, supporting safer decisions such as abstaining or requesting more evidence. Approaches include temperature and logit scaling, prompt and embedding calibration, conformal prediction, confidence probes, and training or decoding methods that use visual evidence to detect hallucinations. These methods are evaluated across zero-shot classification, visual question answering, detection, and specialized settings such as medical imaging, documents, and autonomous driving, including under distribution shift. Calibration can improve selective prediction, but standard metrics may not reveal whether confidence tracks visual grounding or sound reasoning, and reliability can break under severe shifts or deliberate perturbations.

Latest papers

81 papers9 papers in the last four weeks, down 47% on the four weeks before. 0.1% of all new papers.

Improving Calibration in Test-Time Prompt Tuning for Vision-Language Models via Data-Free Flatness-Aware Prompt Pretraining
Apr 30, 2026Hyeonseo Jang, Jaebyeong Jeon, Joong-Won Hwang +1Deep Learning Optimization Vision-Language Models
1Yonsei University · 2ETRI
DPC-VQA: Decoupling Quality Perception and Residual Calibration for Video Quality Assessment
Apr 14, 2026Xinyue Li, Shubo Xu, Zhichao Zhang +3Video Quality Assessment VLM Calibration
Shanghai Jiao Tong University, Shanghai, China · Baidu Inc., Beijing, China · Xinjiang University, Urumqi, China
HTDC: Hesitation-Triggered Differential Calibration for Mitigating Hallucination in Large Vision-Language Models
Apr 13, 2026Xinyun LiuVLM Hallucination Mitigation VLM Calibration
Sichuan University, Chengdu, China
VL-Calibration: Decoupled Confidence Calibration for Large Vision-Language Models Reasoning
Apr 10, 2026Wenyi Xiao, Xinchi Xu, Leilei GanVLM Calibration Confidence Estimation for VLMs
Zhejiang University
When to Call an Apple Red: Humans Follow Introspective Rules, VLMs Don't
Apr 7, 2026Jonathan Nemitz, Carsten Eickhoff, Junyi Jessy Li +3Faithfulness in VLMs VLM Calibration
University of Tübingen · The University of Texas at Austin · Brown University
AICA-Bench: Holistically Examining the Capabilities of VLMs in Affective Image Content Analysis
Apr 7, 2026Dong She, Xianrong Yao, Liqun Chen +3VLM Evaluation VLM Calibration
School of Future Technology,South China University of Technology,Guangzhou
Bridging the Missing-Modality Gap: Improving Text-Only Calibration of Vision Language Models
Apr 3, 2026Mingyeong Kim, Jungwon Choi, Chaeyun Jang +1Modality Gap in VLMs VLM Inference
Kim Jaechul Graduate School of AI, KAIST
Overconfidence and Calibration in Medical VQA: Empirical Findings and Hallucination-Aware Mitigation
Apr 2, 2026Ji Young Byun, Young-Jin Park, Jean-Philippe Corbeil +1Vision-Language Models VLM Hallucination
Johns Hopkins University, School of Medicine · Massachusetts Institute of Technology · Microsoft Healthcare & Life Sciences
Prompts Without Evidence: How Neuroimaging Mentions Shift Clinical Vision-Language Model Predictions
Mar 30, 2026Doan Nam Long Vu, Simone BalloccuMedical VLMs Prompt Sensitivity
Natural Language Processing for Expert Domains (ExpNLP), Technical University of Darmstadt
Zero-shot Vision-Language Reranking for Cross-View Geolocalization
Mar 28, 2026Yunus Talha Erzurumlu, John E. Anderson, William J. Shuart +2Cross-View Geo-Localization Multimodal Reranking
Dept. of Electrical and Computer Engineering, The Ohio State University, 281 W Lane Ave, Columbus, Ohio, USA · US Army Corps of Engineers Geospatial Research Lab, Corbin Field Station, Woodford, Virginia, USA · Dept. of Civil Engineering, The Ohio State University, 281 W Lane Ave, Columbus, Ohio, USA
Edge Reliability Gap in Vision-Language Models: Quantifying Failure Modes of Compressed VLMs Under Visual Corruption
Mar 24, 2026Mehmet Kaan ErolCompositional Reasoning in VLMs VLM Evaluation
Marmara University, Institute of Pure and Applied Sciences
To Agree or To Be Right? The Grounding-Sycophancy Tradeoff in Medical Vision-Language Models
Mar 23, 2026OFM Riaz Rahman Aranya, Kevin DesaiSycophancy in VLMs Medical VLMs
Department of Computer Science, The University of Texas at San Antonio
SkinCLIP-VL: Consistency-Aware Vision-Language Learning for Multimodal Skin Cancer Diagnosis
Mar 22, 2026Zhixiang Lu, Shijie Xu, Kaicheng Yan +6Medical VLMs VLM Adaptation
Xi’an Jiaotong-Liverpool University, China · University of Liverpool, United Kingdom
Fine-Grained Post-Training Quantization for Large Vision Language Models with Quantization-Aware Integrated Gradients
Mar 18, 2026Ziwei Xiang, Fanhu Zeng, Hongjian Fang +6VLM Quantization Post-Training Quantization
State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA · School of Artificial Intelligence, UCAS · Beijing National Research Center for Information Science and Technology +3
Linking Perception, Confidence and Accuracy in MLLMs
Mar 12, 2026Yuetian Du, Yucheng Wang, Rongyu Zhang +5Multimodal Large Language Models VLM Calibration
Zhejiang University · University of Michigan · Alibaba Group +1
UrbanAlign: Post-hoc Semantic Calibration for VLM-Human Preference Alignment
Feb 23, 2026Yecheng Zhang, Rong Zhao, Zhizhou Sha +10VLM Calibration Vision-Language Alignment
Tsinghua University · University College London · Hong Kong University of Science and Technology +6
Towards Calibrating Prompt Tuning of Vision-Language Models
Feb 22, 2026Ashshak Sharifdeen, Fahad Shamshad, Muhammad Akhtar Munir +6VLM Calibration Prompt Learning for VLMs
Mohamed bin Zayed University of AI · University of Colombo · Michigan State University
Similarity-as-Evidence: Calibrating Overconfident VLMs for Interpretable and Label-Efficient Medical Active Learning
Feb 21, 2026Zhuofan Xie, Zishan Lin, Jinliang Lin +3Active Learning VLM Calibration
School of Electronic Technology and Engineering, Xiamen University, Xiamen, China · School of Informatics, Xiamen University, Xiamen, China · School of Engineering, Case Western Reserve University, Cleveland, USA
LATA: Laplacian-Assisted Transductive Adaptation for Conformal Uncertainty in Medical VLMs
Feb 19, 2026Behzad Bozorgtabar, Dwarikanath Mahapatra, Sudipta Roy +3Conformal Prediction VLM Calibration
Aarhus University (A3 Lab) · Khalifa University · Jio Institute +2
How (Mis)calibrated is your Federated CLIP and what to do about it?
Dec 3, 2025Mainak Singha, Masih Aminbeidokhti, Paolo Casari +3Vision-Language Models VLM Adaptation
University of Trento, Italy · ´Ecole de technologie sup´erieure, QC, Canada · AMIAD, Pˆole Recherche, Palaiseau, France +2
From Classification to Cross-Modal Understanding: Leveraging Vision-Language Models for Fine-Grained Renal Pathology
Nov 15, 2025Zhenhao Guo, Rachit Saluja, Tianyu Shi +13Medical VLMs VLM Adaptation
New York University, New York, NY 10012, USA · Cornell Tech, New York, NY 10044, USA · Vanderbilt University, Nashville, TN, 37235, USA +4

Apr 30, 2026J/K move · Enter open · S save

Improving Calibration in Test-Time Prompt Tuning for Vision-Language Models via Data-Free Flatness-Aware Prompt Pretraining

Hyeonseo Jang, Jaebyeong Jeon, Joong-Won Hwang, Kibok Lee

1Yonsei University · 2ETRI

Deep Learning Optimization Vision-Language Models VLM Adaptation VLM Calibration Test-Time Prompt Tuning

Open paper AskPDF arXiv

Test-time prompt tuning (TPT) has emerged as a promising technique for enhancing the adaptability of vision-language models by optimizing textual prompts using unlabeled test data. However, prior studies have observed that TPT often produces poorly calibrated models, raising concerns about the reliability of their predictions. Recent works address this issue by incorporating additional regularization terms that constrain model outputs, which improve calibration but often degrade performance. In this work, we reveal that these regularization strategies implicitly encourage optimization toward flatter minima, and that the sharpness of the loss landscape around adapted prompts is a key factor governing calibration quality. Motivated by this observation, we introduce Flatness-aware Prompt Pretraining (FPP), a simple yet effective pretraining framework for TPT that initializes prompts within flatter regions of the loss landscape prior to adaptation. We show that simply replacing the initialization in existing TPT pipelines--without modifying any other components--is sufficient to improve both calibration and performance. Notably, FPP requires no labeled data and incurs no additional computational costs during test-time tuning, making it highly practical for real-world deployment. The code is available at: https://github.com/YonseiML/fpp.