Answer or Defer? Testing VLMs' Stated Confidence on Brain MRI
Organizations: Joint Georgia Institute of Technology / Georgia State University / Emory University Center for Translational Research in Data Science and Neuroimaging · Georgia State University
Abstract
Vision-language models (VLMs) could support medical-image interpretation, but their use requires deciding which interpretations to accept and which images to refer to a clinician. We test whether thresholding stated confidence can retain useful coverage while limiting errors among accepted answers, and how this depends on the population. Eight open-weight VLMs, including two reasoning models, assess 3,888 brain MRI slices from post-treatment glioma patients (BraTS 2024) and healthy volunteers (IXI) for tumor or gross abnormality. BraTS labels are mask-derived; IXI slices are treated as negatives. On BraTS, no threshold selects a nonempty set with at most 5% empirical error for any model, even when chosen retrospectively. This also holds for four models evaluated on a large-lesion, distant-negative subset. At a 10% error budget, Gemma-4-12B retains 24.4% of slices and every other model at most 2.5%. To investigate this limited selection ability, we separate confidence differences between predicted answers from differences within each answer. For four models with available outcome-level summaries, pooled mean confidence differs more between answers than between correct and incorrect responses within each answer. A covariance decomposition explains how this pattern can produce opposite confidence-correctness associations across populations. At a 20% error budget, accepting every "tumor" answer covers more slices than any confidence threshold for two models. These findings caution against using unvalidated stated confidence as a standalone deferral signal and support population-specific validation before deployment. Code is available at https://github.com/amir-sbg/VLM-Answer-or-Defer.
Figures & tables
| Stratum | Source | Subjects | Slices | Reference |
|---|---|---|---|---|
| BraTS+ | BraTS 2024 | 150 | 840 | positive |
| BraTS | BraTS 2024 | 366 | negative | |
| IXI | IXI | 100 | 2,682 | negative |
| Model | Type | Sens. | Spec. | Spec. | BA | Conf. |
|---|---|---|---|---|---|---|
| BraTS+ | BraTS | IXI | BraTS | |||
| InternVL2.5-8B | G | .423 | .615 | .957 | .519 | .88–.90 |
| Qwen2.5-VL-3B | G | .198 | .872 | .707 | .535 | .67–.73 |
| Qwen2.5-VL-7B | G | .042 | .984 | .837 | .513 | .67–.70 |
| Gemma-3-4B | G | .619 | .473 | .274 | .546 | .87–.89 |
| MedGemma-4B | M | .424 | .814 | .727 | .619 | .95 |
| Model | Acc. | |||
|---|---|---|---|---|
| InternVL2.5-8B | .481 | 0 | 0 | 0 |
| Qwen2.5-VL-3B | .402 | 0 | 0 | 0 |
| Qwen2.5-VL-7B | .328 | 0 | 0 | .034 (41) |
| Gemma-3-4B | .575 | 0 | 0 | 0 |
| MedGemma-4B | .542 | 0 | 0 | 0 |
| Gemma-4-12B | .642 | 0 | .244 (294) | .483 (582) |
| More-confident | Share right | Share of slices | Coverage at a 20% target | ||
|---|---|---|---|---|---|
| Model | answer | (PPV or NPV) | given that answer | confidence gate | answer-only rule |
| InternVL2.5-8B | no | .317 | .589 | 0 | 0 |
| Qwen2.5-VL-3B | yes † | .780 | .177 | 0 | 0 |
| Qwen2.5-VL-7B | yes † | .854 | .034 | .034 | .034 |
| Gemma-3-4B | no † | .351 | .409 | 0 | 0 |
| MedGemma-4B | no † | .381 | .648 | 0 | .352 |
| Measured on the subset | Specificity needed | Coverage , all BraTS subset | Answer-only | ||||
|---|---|---|---|---|---|---|---|
| Model | Sensitivity | Specificity | for a 5% target | , subset | |||
| Gemma-4-12B | 0.68 | 0.70 | 0.92 | 0.57 † | |||
| GLM-4.6V-Flash | 0.71 | 0.57 | 0.91 | 0 † | |||
| Qwen3.6-27B | 0.75 | 0.60 | 0.91 | 0.65 † | |||
| MedGemma-4B | 0.49 | 0.81 | 0.94 | 0.40 † | |||
Radiology VLMs
Radiology vision-language models (VLMs) align medical images and text to support grounded diagnosis, reporting, retrieval, and clinical reasoning.
See all 123 papers