VLM Calibration

VLM: Vision-Language Model

Vision-language model calibration studies whether confidence in image-text predictions reflects their actual correctness, supporting safer decisions such as abstaining or requesting more evidence. Approaches include temperature and logit scaling, prompt and embedding calibration, conformal prediction, confidence probes, and training or decoding methods that use visual evidence to detect hallucinations. These methods are evaluated across zero-shot classification, visual question answering, detection, and specialized settings such as medical imaging, documents, and autonomous driving, including under distribution shift. Calibration can improve selective prediction, but standard metrics may not reveal whether confidence tracks visual grounding or sound reasoning, and reliability can break under severe shifts or deliberate perturbations.

Latest papers

81 papers9 papers in the last four weeks, down 47% on the four weeks before. 0.1% of all new papers.

PathJev: Decision-Native Vision-Language Modelling for Histopathology
Oct 23, 2026Cristian Tommasino, Antonio M. RinaldiComputational Pathology Histopathology Image Classification
DIETI, University of Naples Federico II, Naples, Italy
Query-Time Calibration of CLIP Features for Open-Vocabulary 3D Segmentation
Oct 22, 2026Dominik Engel, Yousef Rajeh, Leon Sick +1Open-Vocabulary 3D Segmentation VLM Calibration
King Abdullah University of Science and Technology (KAUST) · Ulm University
CARLAR Influence: A Structured Hallucination Risk in Multimodal Cheque Extraction and an Exploratory Compressed-Image Countermeasure
Oct 19, 2026Joong Ho, Choi, Aditya Paul +1Document Information Extraction Multimodal Large Language Models
Joshua · BNY, Pittsburgh, PA, USA
UniJev: Separating Accuracy from Calibration in Trimodal System One Decision Models
Oct 19, 2026Jianing Wang, Youwen Jian, Derong Wang +5Model Calibration VLM Calibration
Fudan University · Institute of Information Engineering, Chinese Academy of Sciences
How Murky Before a VLM Knows It Cannot See? Accuracy, Confidence and Abstention on a Physical Visibility Axis
Oct 16, 2026Mochamad Rizal Fauzan, Irgi SuryaVLM Evaluation VLM Robustness
National Taipei University of Technology, Taipei, Taiwan
DICL-Qwen3VL: Decoupled Confidence Calibration for Interpretable Workwear Detection
Oct 16, 2026Meng Wang, Wu Zeng, Mei Li +1VLM Calibration VLM Interpretability
School of Artificial Intelligence, China University of Geosciences (Beijing), Beijing 100083, China · Hebei Technology Innovation Center for Key Components of Climbing Robots, Tangshan Polytechnic University, Tangshan 063299, China · School of Automation, Shenyang Aerospace University, Shenyang, 110136, China
Localized Conformal Safety Monitoring with Vision-Language Models for Autonomous Driving
Oct 2, 2026Luís Marques, Rong Fang, Disha Kamale +1Autonomous Driving Safety Evaluation Autonomous Driving
University of Michigan, Ann Arbor, MI, USA
Towards Reliable Vision-Language Models for Autonomous Driving
Oct 1, 2026Manasa Mariam Mammen, Priyanka Mary Mammen, Zafer Kayatas +1VLM Robustness Image Corruption Robustness
Technical University of Munich, Heilbronn, Germany · Mercedes Benz AG, Sindelfingen, Germany · University of Massachusetts, Amherst, USA
SubRot: Signed Gradient Subspace Calibration for VLM Rotation Quantization
Sep 28, 2026Zhenhao Shang, Haizhao Jing, Haokui Zhang +4VLM Quantization Rotation-Based Quantization
Northwest Polytechnical University · Nanjing University of Science and Technology · Intellifusion +1
Domain Recentering and Confidence-Weighted Prior Calibration for Vision-Language Models
Sep 24, 2026Youngeun Seol, Jimin Shin, Heeseo Yoon +1Source-Free Domain Adaptation Unsupervised Domain Adaptation
Department of Computer Science and Engineering, Ewha Womans University
Perception, Layout, and Validation: Calibrated Confidence for Reliable Straight-Through Processing of Financial Documents
Sep 17, 2026Yichao Jin, Yushuo Wang, Yuxuan Han +7Document Information Extraction VLM Calibration
OCBC, Singapore
Calibrated Probabilistic Obstruction Reasoning with Vision-Language Models for Grasping in Clutter
Sep 16, 2026Thanh-Tuan Tran, Ngoc-Chien Chu, Thanh Nguyen Canh +3Robotic Grasping Robotic Manipulation
University of Engineering and Technology, Vietnam National University, Hanoi, 10000, Vietnam · School of Information Science, Japan Advanced Institute of Science and Technology, Ishikawa, 923-1211, Japan · Department of Robotics, Hanyang University, Gyeonggi-do, 15588, Korea
The Mirage of Calibrated Confidence: Trajectory-Independence of Verbalized Confidence in Vision-Language Models
Sep 16, 2026Jisoo Yang, Jaeho Han, Trung X. Pham +1VLM Evaluation Confidence Estimation in Language Models
Department of Artificial Intelligence, Chung-Ang University, Republic of Korea · Van Lang University, Ho Chi Minh City, Vietnam
Can VLMs Reliably Assess Sidewalk Accessibility Attributes from Pedestrian-Level Imagery?
Sep 15, 2026Seung Jae Lieu, Diego Morra, Chiara Cadoni +3VLM Evaluation Urban Planning
Senseable City Lab Massachusetts Institute of Technology Cambridge, MA 02139
Bridging the Confidence Gap: Temperature Scaling for Calibrating Test-Time Prompt Tuning
Sep 15, 2026Yuwei Liang, Jian Liang, Dapeng Hu +2Post-Hoc Calibration Test-Time Adaptation of VLMs
School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China · NLPR & MAIS, Institute of Automation, Chinese Academy of Sciences, Beijing, China · Independent Researcher
SAVOR: Self-Aware Visual Grounding via Confidence-Calibrated Reinforcement Learning for Multimodal Hallucination Mitigation
Sep 15, 2026Zixiu Ding, Zilin Zhao, Yingjie He +3Object Hallucination in VLMs Visual Grounding
Central South University · Peking University
Human-Grounded Calibration for Long-Text Image-Text Congruence in Vision-Language Models
Sep 14, 2026Alessandro Gambetti, Qiwei HanText-Image Alignment Vision-Language Retrieval
NOVA School of Science and Technology · Universidade NOVA de Lisboa · Caparica, Portugal +2
Vision-Language Models for Criterion-Level Grading of Handwritten Examinations in Outcome-Based Education
Sep 13, 2026Asif Hasan Tonmoy, Saad Ahmed, Md Khalid Syfullah +1VLM Evaluation Rubric-Based Evaluation
Learning to Adapt and Calibrate: Score Distribution Alignment for Few-Shot Uncertainty Prediction in Medical VLMs
Sep 9, 2026Xuan Cuong Ngo, Ngan LeVLM Adaptation Conformal Prediction
University of Arkansas Fayetteville, Arkansas, USA
Target-Checked Reliability Score Refinement for Video Question Answering
Sep 9, 2026Guoxiang Ren, Rohitash ChandraConfidence Estimation for VLMs Video QA
Transitional Artificial Intelligence Research Group, School of Mathematics and Statistics, UNSW Sydney, NSW 2052, Australia
Vision-language models see more about agriculture than they report, and test-time scaling remains bounded by parametric knowledge
Sep 8, 2026Earl Ranario, Jared Smith, Lars Lundqvist +3Agricultural Image Analysis Plant Disease Classification
University of California, Davis
Separating Capability from Confidence: Grounded Dual-State Calibration for GRPO-Trained Medical Vision-Language Models
Sep 6, 2026Yangyang Xie, Ke Hao, Jiaqi Liu +2Confidence Estimation for VLMs VLM Calibration
Shanghai Jiao Tong University, Shanghai, China · Medical Image Insights Co. Ltd., Shanghai, China
IntroConformal: Conformal Factuality Guarantees for Large Vision-Language Models via Introspective Signals
Sep 1, 2026Md. Atabuzzaman, Christian Alexander, Chris ThomasVLM Evaluation Large Vision-Language Models
Department of Computer Science Virginia Tech
Good Memory Has ECC: Evaluating the Memory of Vision-Language Models Beyond Accuracy
Aug 31, 2026Shmuel Berman, Jia DengVLM Evaluation Memory-Augmented VLMs
Department of Computer Science Princeton University
Rethinking the Test-Time Prompt Tuning Objective from the Perspective of Calibration
Aug 31, 2026Jungwon Choi, Hyeonseo Jang, Kibok Lee +1VLM Calibration Uncertainty Calibration
Department of Computer Science and Engineering, Chung-Ang University · Department of Statistics and Data Science, Yonsei University
Frontier vision-language models have overtaken young adults at detecting AI-generated portraits -- but not their calibration
Aug 31, 2026Sunwhi Kim, Sunyul Kim, Meounggun Jo +1VLM Evaluation Vision-Language Models
Hwasung Medi-Science University, Dept. of Bio-Healthcare · Yonsei University, Graduate School of Engineering, Dept. of Artificial Intelligence · Hoseo University +1
SpanCalib-VLM: Calibrated Hallucination Span Detection in Vision-Language Models
Aug 30, 2026Amanuel Gizachew Abebe, Yasmin MoslemHallucination Detection in VLMs Hallucination Detection
Shaggar Institute of Technology Shaggar City, Ethiopia · Trinity College Dublin Dublin, Ireland
Evidence-Order Calibration for Selective Visual Reasoning under Progressive Loss of Question-Critical Evidence
Aug 29, 2026Muhamathu Ameer Ali Aacaas MuhamathSelective Prediction for VLMs VLM Calibration
Department of Electrical Engineering University of Moratuwa Moratuwa, Sri Lanka
CODE: Cross-Modal Calibration and Dynamic Suppression for Open World Object Detection
Aug 27, 2026Hao Xu, Zhaoning Shi, Hehe Jin +1Open-Vocabulary Object Detection Multimodal Object Detection
Beijing Institute of Technology Beijing, China
MVC-Bench: Benchmarking Calibration of Medical Vision-Language Models
Aug 27, 2026Ashshak Sharifdeen, Shihab Aaqil Ahamed, Ufaq Khan +6VLM Evaluation VLM Robustness
1Mohamed bin Zayed University of AI · 2Sabaragamuwa University of Sri Lanka · 3Digital Platform Development, SLT PLC

Oct 23, 2026J/K move · Enter open · S save

PathJev: Decision-Native Vision-Language Modelling for Histopathology

Cristian Tommasino, Antonio M. Rinaldi

DIETI, University of Naples Federico II, Naples, Italy

Computational Pathology Histopathology Image Classification VLM Calibration

Open paper AskPDF arXiv

Vision-language models used for decisions in histopathology rarely return calibrated probabilities over the candidates a user cares about. Decision-native models write a question and a candidate set chosen at run time into the input and return a normalised distribution from a small decision head in one forward pass. We build PathJev-P, a decision-native model for histopathology: MedGemma-1.5-4B with LoRA adapters and a decision head, trained on 40,000 patch states from three public collections whose expert labels are turned into typed questions with organ-specific candidate sets and a "none of these" option. In a comparison over 3 seeds with an analysis plan fixed before testing, on test sets without documented overlap with the training data of the backbone, PathJev-P is more accurate than a generative model trained with the same backbone, data and selection rule, the zero-shot pathology model CONCH and a general medical decision model (balanced accuracy 0.541 against 0.492, 0.345 and 0.213); its advantage over the generative model comes from two organs absent from training. The calibration advantage depends on how the generative model's probabilities are read out. With the summed candidate log-likelihood used in previous work, PathJev-P has the lower calibration error (ECE 0.204 against 0.279); with a length-normalised read-out of the same generative model, examined post hoc, the two have similar mean calibration error (0.201) and the generative model is better calibrated on the unseen organs, where PathJev-P remains over-confident. A supervised linear probe on a pathology foundation model is better calibrated in every setting. In a qualitative whole-slide demonstration, PathJev-P labels unsupervised tissue clusters with abstention and separates carcinoma from other tissue while running the model on a fraction of the tiles.