SpatialUQ: Post-Hoc Uncertainty Quantification from Spatial Consistency in Black-Box Vision Models

Md Kawsher Mahbub; Milon Biswas; Mirza Niaz Morshed; Wei Yu

cs.CVOct 7, 2026

SpatialUQ: Post-Hoc Uncertainty Quantification from Spatial Consistency in Black-Box Vision Models

Authors: Md Kawsher Mahbub, Milon Biswas, Mirza Niaz Morshed, Wei Yu

Organizations: NPI University of Bangladesh · Towson University · University of Asia Pacific

Abstract

Clinical vision models are often deployed as frozen black boxes with no access to internals, retraining, or ground truth at inference time. We introduce \textbf{SpatialUQ}, a post-hoc uncertainty method using only output probabilities. It measures the Jensen-Shannon divergence between the global prediction and the mean of five fixed spatial crops in six deterministic forward passes. The premise is simple, trustworthy predictions are spatially consistent. On NIH ChestX-ray14 (DenseNet-121, $N{=}25{,}596$ ), our Multicrop Uncertainty Score (MUS) reaches $0.784$ failure-detection AUC versus $0.664$ for MC-Dropout ( $p{<}10^{-6}$ ) at one-fifth the compute, with native calibration ( $\text{SCE}{=}0.049$ vs.\ $0.127$ for $\ell_1$ ), the best-calibrated among methods above 0.78 AUC. A supervised fusion of MUS with entropy, confidence, and $\ell_1$ reaches $0.832$ , outperforming a five-member ensemble ( $0.813$ ). MUS scales with model quality, reaching $0.899$ with BiomedCLIP ( $ρ= 0.846$ ), while this relationship remains meaningful in-distribution ( $ρ= 0.523$ ) but breaks down under severe distribution shift (VinBigData, $ρ= 0.027$ ). MUS is well-suited to diffuse findings but is less dependable for small focal lesions such as nodules. Code and experimental materials are publicly available at https://huggingface.co/datasets/kawsher11/SpatialUQ.

Figures & tables

Panel of Figure 1 — Figure 1: (a) The five-step inference pipeline computing JSD between global and aggregated local predictions. (b) High spatial inconsistency (MUS) successfully flags prediction errors.

Figure 2: Failure detection ROC curve for the NIH test set, comparing SpatialUQ (MUS) against sampling-based and feature-based baselines.

Method	AUC	95% CI	SCE
Fusion (CV)	0.832	[0.827, 0.837]	0.246
Deep Ensemble $\dagger$	0.813	[0.808, 0.819]	0.138
$\ell_{1}$ Distance	0.790	[0.785, 0.796]	0.127
MUS (Unsupervised)	0.784	[0.778, 0.789]	0.049
Deep Ens + T-scaling $\dagger$	0.774	[0.768, 0.780]	0.090
ODIN	0.759	[0.752, 0.765]	0.484

Table 1: Failure detection on NIH ChestX-ray14 (

N{=}25{,}596

, failure: Brier

>

75th percentile).

\dagger

requires

>

6 passes.

\ddagger

requires training features. Full results in Table 9 .

Figure 3: Reliability diagram comparing MUS against baseline uncertainty methods on NIH ChestX-ray14 (DenseNet-121).

Removed Feature	AUC	$\boldsymbol{\Delta}$
Entropy	0.788	$-$ 0.044
$\ell_{1}$ Distance	0.825	$-$ 0.007
MUS	0.827	$-$ 0.005
Confidence	0.831	$-$ 0.001

Table 2: Leave-one-out ablation of the four-feature fusion (NIH DenseNet-121). Full fusion AUC

=0.832

\Delta

: AUC drop when each feature is removed.

Dataset	MUS AUC	Fusion AUC	MC-Dropout AUC	Spearman $\rho$
NIH (in-dist.)	0.784	0.832	0.664	0.523
CheXpert (zero-shot)	0.708	0.733	0.599	0.298
VinBigData (zero-shot)	0.614	0.750	0.764	0.027

Table 3: Zero-shot failure detection (NIH-trained DenseNet-121, no retraining). Spearman

\rho

= correlation between MUS and per-image Brier score. Full per-method tables in Appendix F .

Dataset	Model	Top-1 Acc.	MUS	Confidence -1	Entropy	Fusion	Spearman $\rho$
ImageNet	EfficientNet-B4	79.27%	0.717	0.913	0.896	0.917	0.392
ImageNet	ViT-B/16	81.07%	0.710	0.923	0.884	0.937	0.341
ImageNet	ConvNeXt-Tiny	82.13%	0.641	0.913	0.819	0.936	0.139
COCO	Faster R-CNN	-	0.800	0.602	0.884	0.885	0.613
COCO	RetinaNet	-	0.747	0.702	0.912	0.913	0.510

Table 4: Failure detection AUC on ImageNet-1k and MS COCO 2014. Full secondary metrics in Appendix G .

Figure 4: Mean per-image Brier score of the retained set (lower is better) versus rejection rate (NIH DenseNet-121, $N{=}25{,}596$ ), rejecting images in descending uncertainty order.

Method	AUC	Passes	Internals	Train data	Retrain	Frozen ckpt?
Fusion (Ours)	0.832	6 + logistic reg	✗	✗	✗	✓
Deep Ensemble ( Lakshminarayanan et al., 2017 )	0.813	$5\times$ model	✗	✗	✓	✗
MUS (Ours)	0.784	6	✗	✗	✗	✓
Deep Ensemble + T-scaling	0.774	$5\times$ model+cal	✗	✓	✓	✗
ODIN ( Liang et al., 2017 )	0.759	1+grad	✓	✗	✗	✓
DDU † ( Mukhoti et al., 2021 )	0.754	1	✓	✓	✗	✗

Table 5: Failure detection on NIH ChestX-ray14 (DenseNet-121,

N{=}25{,}596

). † Post-hoc approximation.

Figure 5: Top-to-bottom decile ratio of mean per-image Brier score by lesion type (NIH DenseNet-121, $N{=}25{,}596$ ). Error bars: 95% bootstrap CIs; n = 2,560 images per decile.

Appendix figures & tables38 assets

Supplementary material from the paper’s appendix.

Appendix

Class	AUC	Brier	AP
Atelectasis	0.7628	0.3189	0.3272
Cardiomegaly	0.8746	0.5565	0.3350
Effusion	0.8192	0.3081	0.4907
Infiltration	0.6984	0.3161	0.3969
Mass	0.7965	0.3576	0.2915
Nodule	0.7400	0.3869	0.2191

Appendix

Table 6: NIH ChestX-ray14 backbone classification performance - DenseNet-121 (7.0 M parameters, best checkpoint epoch 28, validation loss 2.0564). Mean AUC of 0.7885.

Class	AUC	Brier	AP
Atelectasis	0.7192	0.3321	0.2732
Cardiomegaly	0.7952	0.5746	0.2118
Effusion	0.7740	0.3250	0.4189
Infiltration	0.6790	0.3102	0.3733
Mass	0.7002	0.4114	0.1591
Nodule	0.6941	0.4039	0.1520

Appendix

Table 7: NIH ChestX-ray14 backbone classification performance - EfficientNet-B4 (17.6 M parameters, best checkpoint epoch 29, validation loss 2.0793). Mean AUC = 0.7347.

Class	AUC	Brier	AP
Atelectasis	0.7689	0.2882	0.3399
Cardiomegaly	0.8868	0.5685	0.3500
Effusion	0.8257	0.3037	0.5059
Infiltration	0.6978	0.3029	0.3983
Mass	0.8157	0.3817	0.3176
Nodule	0.7502	0.3712	0.2230

Appendix

Table 8: NIH ChestX-ray14 backbone classification performance - ViT-B/16 (85.8 M parameters, early stopping at epoch 17, best validation loss at epoch 9). Mean AUC = 0.7921.

Method	AUC	95% CI	SCE
Fusion CV (MUS+Ent+Conf+ $\ell_{1}$ )	0.8319	[0.8268, 0.8368]	0.2464
Auditor (3-feat MLP) $\dagger$	0.8250	[0.8201, 0.8299]	0.2408
Auditor (6-feat MLP) $\dagger$	0.8242	[0.8189, 0.8292]	0.2363
Deep Ensemble (5-member) $\dagger$	0.8130	[0.8077, 0.8187]	0.1379
$\ell_{1}$ Distance	0.7902	[0.7848, 0.7960]	0.1270
MUS (Ours)	0.7837	[0.7778, 0.7893]	0.0487

Appendix

Table 9: NIH ChestX-ray14 failure detection DenseNet-121, full 16-method comparison (

N=25{,}596

). Brier threshold = 0.5607 (75th percentile, 25% failure rate). SCE = Score Calibration Error (ECE applied to min-max normalised scores, 15 bins).

\dagger

requires

>

6 forward passes or training-time modification.

\ddagger

requires access to training-set features.

Seed	MUS AUC	$\ell_{1}$ Dist.	MC-Drop	Conf -1	TTA Var	Entr.	Pair JSD
42	0.7837	0.7902	0.6643	0.4011	0.5837	0.1774	0.6057
186	0.7866	0.7959	0.6466	0.5802	0.5723	0.1748	0.6023
456	0.7902	0.7976	0.6532	0.5816	0.5856	0.1672	0.5709
789	0.7862	0.7914	0.6498	0.5788	0.5797	0.1681	0.5806
1011	0.7826	0.7844	0.6536	0.5737	0.5848	0.1688	0.5660
Mean	0.7859	0.7919	0.6535	0.5431	0.5812	0.1713	0.5851

Appendix

Table 10: Seed MUS: Comparison across different random seeds for DenseNet121 on NIH ChestX-ray14.

Method	CLIP ViT-B/32		BiomedCLIP
Method	AUC [95% CI]	SCE	AUC [95% CI]	SCE
MUS (SpatialUQ)	0.825 [0.818, 0.830]	0.098	0.899 [0.895, 0.903]	0.133
Entropy	0.204 [0.199, 0.210]	0.574	0.394 [0.387, 0.402]	0.421
Confidence -1	0.095 [0.092, 0.099]	0.429	0.206 [0.201, 0.212]	0.334
$L_{1}$ Distance	0.837 [0.831, 0.842]	0.115	0.915 [0.911, 0.918]	0.232
Fusion	0.883 [0.879, 0.887]	0.234	0.925 [0.922, 0.928]	0.278

Appendix

Table 11: Failure detection on NIH ChestX-ray14 with frozen foundation models plus linear probe (25,596 images). MUS requires only output probabilities;

L_{1}

and Fusion use the same six forward passes. Fusion is a logistic regression trained on a small labeled hold-out (10% of the test set, remaining 90% for evaluation).

Panel of Figure 6 — Figure 6: ROC curves for failure detection on NIH ChestX-ray14 with frozen foundation models plus linear probe. Left: CLIP ViT-B/32 (AUC 0.825). Right: BiomedCLIP (AUC 0.899). MUS (SpatialUQ) outperforms entropy and confidence -1 baselines.

Method	AUC	95% CI	SCE
Fusion CV (MUS+Ent+Conf+ $\ell_{1}$ )	0.8714	[0.8675, 0.8756]	0.2713
MUS (Ours)	0.7842	[0.7777, 0.7900]	0.0666
Mahalanobis $\ddagger$	0.7804	[0.7747, 0.7863]	0.3243
$\ell_{1}$ Distance	0.7746	[0.7683, 0.7809]	0.0873
ODIN	0.7682	[0.7622, 0.7742]	0.2185
Max Disagreement	0.6722	[0.6644, 0.6794]	0.1391

Appendix

Table 12: NIH ChestX-ray14 failure detection EfficientNet-B4, 14-method comparison (

N=25{,}596

). Brier threshold = 0.5776 (75th percentile, 25% failure rate). SCE = Score Calibration Error (ECE applied to min-max normalised scores, 15 bins).

\dagger

requires

>

6 forward passes or training-time modification.

\ddagger

requires access to training-set features. DDU, Pairwise JSD and TTA Entropy are degenerate (constant scores) on this backbone.

Method	AUC	95% CI	SCE
Fusion CV (MUS+Ent+Conf+ $\ell_{1}$ )	0.8314	[0.8265, 0.8363]	0.2662
ODIN	0.8003	[0.7945, 0.8062]	0.4059
MUS (Ours)	0.7497	[0.7435, 0.7563]	0.0455
$\ell_{1}$ Distance	0.7324	[0.7255, 0.7390]	0.1412
Mahalanobis $\ddagger$	0.7282	[0.7223, 0.7344]	0.2030
Max Disagreement	0.7084	[0.7019, 0.7154]	0.1545

Appendix

Table 13: NIH ChestX-ray14 failure detection ViT-B/16, 14-method comparison (

N=25{,}596

). Brier threshold = 0.5538 (75th percentile, 25% failure rate). SCE = Score Calibration Error (ECE applied to min-max normalised scores, 15 bins).

\dagger

requires

>

6 forward passes or training-time modification.

\ddagger

requires access to training-set features. DDU, Pairwise JSD and TTA Entropy are degenerate (constant scores) on this backbone.

Architecture	Upsample AUC	Mask AUC	$\Delta$
DenseNet-121	0.7837	0.7697	$-0.0141$
EfficientNet-B4	0.7791	0.5540	$-0.2251$
ViT-B/16	0.7421	0.7465	$+0.0045$

Appendix

Table 14: Comparison of failure detection AUC using upsampling versus masking strategies across different architectures (separate ablation run; see Appendix C.4 ).

Method	AUC	95% CI	SCE	Spearman $\rho$
Fusion	0.838	[0.832, 0.843]	0.069	-
MUS (SpatialUQ)	0.533	[0.526, 0.541]	0.202	$-$ 0.187
$\ell_{1}$ Distance	0.533	[0.525, 0.541]	0.120	-
Entropy	0.242	[0.236, 0.248]	0.723	-
Confidence -1	0.166	[0.160, 0.171]	0.572	-

Appendix

Table 15: SpatialUQ on BioViL-T (NIH ChestX-ray14,

N{=}25{,}596

, failure: Brier

>

75th percentile). Near-chance MUS performance is correctly flagged by the

\rho\lesssim 0.15

diagnostic (

\rho{=}{-}0.187

Class	Backbone AUC	JSD AUC	N Pos
Pneumonia	0.7020	0.9789	555
Edema	0.8368	0.9710	925
Consolidation	0.7401	0.9564	1,815
Pleural_Thickening	0.7569	0.9496	1,143
Cardiomegaly	0.8746	0.9518	1,069
Atelectasis	0.7628	0.9145	3,279

Appendix

Table 16: Per-class JSD AUC for failure detection DenseNet-121 on NIH ChestX-ray14. Classes sorted by JSD AUC (descending). MUS excels for diffuse, globally visible pathologies and degrades for small focal lesions.

Class	Backbone AUC	JSD AUC
Pneumonia	0.6589	0.89
Edema	0.8093	0.88
Nodule	0.6941	0.8822
Consolidation	0.7022	0.87
Pleural_Thickening	0.7201	0.86
Cardiomegaly	0.7952	0.86

Appendix

Table 17: Per-class JSD AUC - EfficientNet-B4 on NIH. Notable: Nodule AUC = 0.8822 vs. 0.3680 for DenseNet-121.

Class	Backbone AUC	JSD AUC
Pneumonia	0.7111	0.88
Edema	0.8359	0.87
Cardiomegaly	0.8868	0.85
Consolidation	0.7521	0.84
Pleural_Thickening	0.7622	0.83
Atelectasis	0.7689	0.81

Appendix

Table 18: Per-class JSD AUC - ViT-B/16 on NIH. ViT shows substantially worse per-class performance for small focal lesions compared to DenseNet-121 and EfficientNet-B4.

Class	JSD AUC
Pneumonia	0.9678
Pleural_Thickening	0.9675
Consolidation	0.9589
Atelectasis	0.9218
Cardiomegaly	0.9216
Pneumothorax	0.9149

Appendix

Table 19: Per-class JSD AUC - zero-shot transfer to CheXpert (NIH-trained DenseNet-121,

N=44{,}399

, 10 mapped classes). Nodule degrades severely (0.1446) under domain shift.

Class	JSD AUC
Atelectasis	0.9467
Infiltration	0.9222
Pneumothorax	0.9181
Cardiomegaly	0.8845
Pleural_Thickening	0.6678
Effusion	0.4229

Appendix

Table 20: Per-class JSD AUC - zero-shot transfer to VinBigData (NIH-trained DenseNet-121,

N=15{,}000

, 10 mapped classes). Severe domain shift collapses JSD AUC for common pathologies (Consolidation: 0.3375; Mass: 0.1644; Nodule: 0.1608).

Crops	MUS AUC	Forward Passes
3	0.7764	4
4	0.7859	5
5	0.7837	6
6	0.7801	7
7	0.7816	8
9	0.7902	10

Appendix

Table 21: Crop count ablation MUS AUC on NIH DenseNet-121 (failure threshold = 0.5607). Inference cost scales linearly with crop count plus one global pass.

Figure 7: Geometric coverage of fixed spatial crops. (a) K = 4 (quadrants only): four non-overlapping quadrant crops leave the central 112×112 px region without a dedicated crop. (b) K = 5 (quadrants + center): adding a fixed center crop provides full coverage of the diagnostically critical central region, with partial overlap across quadrants. Despite nearly identical performance (AUC of 0.7859 vs. 0.7837; $\Delta$ = 0.0022, within bootstrap CI), K = 5 is preferred for its strictly stronger geometric coverage guarantees

Figure 8: Ablation studies on NIH DenseNet-121. Left : Failure detection AUC as a function of the number of spatial crops

K

. Five crops achieves the best cost-efficiency trade-off; nine crops yields a marginal

+0.007

AUC gain at

67\%

more inference cost, while three crops reduces AUC by

0.007

. Right : Normalised uncertainty score distributions stratified by prediction outcome (failure vs. success) for six methods. SpatialUQ (MUS) and Fusion CV show the clearest separation; Entropy is anti-correlated with failure (AUC

=0.177

) under the overconfident multi-label training regime.

Method	AUC@50th	AUC@75th	AUC@90th
Random	0.4985	0.4979	0.4960
Confidence -1	0.4079	0.4011	0.3886
Entropy	0.1936	0.1774	0.1319
Patch Variance	0.6045	0.6103	0.6236
Pairwise Disagr.	0.5910	0.5990	0.6124
Pairwise JSD	0.5997	0.6057	0.6184

Appendix

Table 22: Failure detection AUC across different Brier score failure thresholds on NIH ChestX-ray14. Thresholds correspond to the

50^{\text{th}}

(

0.5170

75^{\text{th}}

(

0.5607

), and

90^{\text{th}}

(

0.5993

) percentiles of the test set Brier scores.

Removed Feature	Fusion AUC	$\Delta$
Entropy	0.7881	$-$ 0.0437
$\ell_{1}$ Distance	0.8248	$-$ 0.0071
MUS	0.8265	$-$ 0.0053
Confidence	0.8309	$-$ 0.0010

Appendix

Table 23: Leave-one-out fusion ablation - NIH DenseNet-121. Full 4-feature fusion AUC = 0.8319.

Removed Feature	Fusion AUC	$\Delta$
Entropy	0.7534	$-$ 0.0780
$\ell_{1}$ Distance	0.8292	$-$ 0.0022
MUS	0.8299	$-$ 0.0015
Confidence	0.8328	$+$ 0.0014

Appendix

Table 24: Leave-one-out fusion ablation - NIH ViT-B/16. Full 4-feature fusion AUC = 0.8314.

Model	Removed	LOO AUC	$\Delta$
ConvNeXt-Tiny	Confidence	0.8422	$-$ 0.0934
ConvNeXt-Tiny	Entropy	0.9217	$-$ 0.0139
ConvNeXt-Tiny	MUS	0.9322	$-$ 0.0035
ViT-B/16	Confidence	0.8962	$-$ 0.0407
ViT-B/16	Entropy	0.9317	$-$ 0.0051
ViT-B/16	MUS	0.9324	$-$ 0.0045

Appendix

Table 25: Leave-one-out fusion ablation ImageNet-1k. Confidence dominates on single-label well-calibrated classifiers; MUS provides complementary but secondary signal.

Method	AUC	ECE	AUPR	FPR@80%	FPR@95%	AURC	E-AURC
Fusion CV	0.7330	0.0601	0.4248	0.4581	0.6279	0.11467	0.08343
$\ell_{1}$ Distance	0.7102	0.1372	0.4048	0.4889	0.6976	0.12686	0.09562
MUS (Ours)	0.7076	0.0446	0.4023	0.4967	0.6956	0.12739	0.09614
TTA Variance	0.6004	0.0353	0.3206	0.6773	0.8854	0.18897	0.15772
MC-Dropout	0.5991	0.1275	0.3004	0.6611	0.8691	0.18347	0.15223
Patch Variance	0.5634	0.0499	0.2883	0.7187	0.9174	0.21090	0.17965

Appendix

Table 26: Zero-shot failure detection on CheXpert (NIH-trained DenseNet-121,

N=44{,}399

, failure threshold = 0.5130). All metrics computed on the full test set without any retraining. AURC and E-AURC lower is better.

Method	AUC	ECE	AUPR	FPR@80%	FPR@95%	AURC	E-AURC
MC-Dropout	0.7641	0.1414	0.4304	0.3743	0.5991	0.10452	0.07327
Fusion CV	0.7502	0.1161	0.3931	0.3631	0.5938	0.10707	0.07582
TTA Variance	0.6127	0.0771	0.3057	0.6196	0.8381	0.17268	0.14143
MUS (Ours)	0.6141	0.1254	0.3589	0.7343	0.9479	0.20936	0.17811
$\ell_{1}$ Distance	0.5827	0.0712	0.3426	0.7764	0.9530	0.22480	0.19355
Patch Variance	0.5003	0.0925	0.2514	0.8006	0.9560	0.25270	0.22145

Appendix

Table 27: Zero-shot failure detection on VinBigData (NIH-trained DenseNet-121,

N=15{,}000

, failure threshold = 0.4117). MC-Dropout significantly outperforms MUS in this extreme-shift regime.

Model	Method	AUC	AUPR	FPR@80%	FPR@95%	AURC
EfficientNet-B4	MUS (Ours)	0.7171	0.4491	0.5062	0.7670	0.3890
EfficientNet-B4	Entropy	0.8961	0.7785	0.1468	0.4742	0.5252
EfficientNet-B4	Confidence-inv	0.9132	0.8424	0.1389	0.4398	0.5442
EfficientNet-B4	Fusion CV	0.9173	0.8496	0.1237	-	-
ViT-B/16	MUS (Ours)	0.7095	0.4066	0.4945	0.7555	0.3686
ViT-B/16	Entropy	0.8840	0.7856	0.1691	0.5967	0.5247

Appendix

Table 28: ImageNet-1k failure detection - full secondary metrics (50,000 images, failure = top-1 misclassification). Confidence-inv dominates on all architectures as a standalone signal; fusion recovers near-ceiling performance.

Model	Method	AUC	ECE	AUPR	FPR@80%	FPR@95%	AURC	E-AURC
Faster R-CNN	MUS (Ours)	0.8003	0.0581	0.5835	0.3701	0.6781	0.10044	0.06919
Faster R-CNN	Entropy	0.8839	0.0806	0.7412	0.2077	0.4512	0.06895	0.03770
Faster R-CNN	Confidence-inv	0.6017	0.2429	0.2984	0.6485	0.8512	0.17993	0.14868
Faster R-CNN	Fusion CV	0.8851	-	0.7426	0.2127	-	-	-
RetinaNet	MUS (Ours)	0.7469	0.0561	0.4501	0.4171	0.6901	0.11438	0.08313
RetinaNet	Entropy	0.9121	0.1032	0.8055	0.1568	0.3845	0.05981	0.02856

Appendix

Table 29: MS COCO 2014 failure detection - Faster R-CNN and RetinaNet (5,000 images, failure = multi-label Brier

>

75th percentile). Entropy dominates as standalone signal; MUS contributes significantly in fusion, especially for RetinaNet.

Threshold	FRCNN AUC	FRCNN AUPR	RetinaNet AUC	RetinaNet AUPR
0.05	0.8003	0.5835	0.7469	0.4501
0.10	0.8003	0.5835	0.7469	0.4501
0.30	0.7876	0.5644	0.7023	0.3929

Appendix

Table 30: MS COCO detection confidence threshold sensitivity for MUS. AUC is fully stable from 0.05 to 0.10 and degrades modestly at 0.30, confirming robustness to conservative thresholding.

Figure 9: MUS discriminability as a function of lesion spatial extent (NIH ChestX-ray14, DenseNet-121). Per-class JSD AUC for failure detection plotted against approximate ordinal lesion size rank. Diffuse pathologies (group mean JSD AUC = 0.944) substantially outperform localized lesions (0.603), consistent with the five-crop resolution limit of 3 cm.

Figure 10: MUS performance vs. disease prevalence (NIH ChestX-ray14, DenseNet-121). Per-class JSD AUC for failure detection plotted against test-set prevalence (%)

Method	Nodule JSD AUC
Baseline (5-crop coarse)	0.3680
Fine MUS (16-crop, $56\times 56$ )	0.5892
Multi-scale avg (coarse $+$ fine)	0.4945
Improvement (Fine vs. Baseline)	$+$ 0.2212

Appendix

Table 31: Multi-scale MUS for Nodule failure detection (NIH DenseNet-121). Fine crops use a 16-crop

56\times 56

grid. The fine-only result (0.589) is the primary reported improvement; the coarse+fine average (0.495) is lower because averaging with the weak coarse score (0.368) dilutes the signal.

Use MUS when	Do not rely on MUS when
Diffuse, globally visible conditions	Highly localized lesions (nodules)
Overconfident multi-label classifiers	Well-calibrated single-label models
Moderate distribution shift	Severe covariate shift ( $\rho\lesssim 0.15$ )
No retraining or internals access	Generative VLMs with global pooling

Appendix

Table 32: When to use SpatialUQ. This table summarizes findings already present in Sections 5–6

Recall	Precision	Images Referred	Budget	Errors Caught
10%	58.7%	1,091	4.3%	640 / 6,399
20%	54.5%	2,349	9.2%	1,280 / 6,399
30%	51.4%	3,732	14.6%	1,920 / 6,399

Appendix

Table 33: Selective prediction - NIH DenseNet-121 (MUS as uncertainty score). Images are referred in descending MUS order. Budget = fraction of full test set referred; Errors Caught = number of detected failures out of 6,399 total.

Figure 11: Selective prediction via Multi-view Uncertainty Score (MUS) for Pneumothorax detection. Performance (AUROC) shows a general upward trend as images with high spatial divergence are rejected, with local fluctuation reflecting sampling variability at intermediate rejection rates. The overall trajectory stabilizes above the baseline AUC of 0.847 at higher rejection rates, indicating that MUS effectively identifies atypical or spatially incoherent presentations whose removal improves downstream classification performance.

Subgroup	N	MUS AUC
Age $<$ 40	8,872	0.8199
Age 40–60	11,151	0.7714
Age 60–80	5,379	0.7512
Age $>$ 80	190	0.7558
Female	10,714	0.7838
Male	14,882	0.7837

Appendix

Table 34: Demographic fairness audit - MUS failure detection AUC by age group and gender, NIH DenseNet-121 (

N=25{,}596

). Gender parity is near-perfect (0.7838 vs. 0.7837). MUS AUC is slightly higher for younger patients, consistent with radiographic image quality and lesion acuity differences.

Subgroup	N	MUS AUC
Age $<$ 40	8,872	0.7723
Age 40–60	11,151	0.7814
Age 60–80	5,379	0.7943
Female	10,714	0.7813
Male	14,882	0.7863
Overall	25,596	0.7842

Appendix

Table 35: Demographic fairness audit - NIH EfficientNet-B4. Unlike DenseNet-121, EfficientNet-B4 shows slightly higher MUS AUC for older patients (Age 60–80: 0.7943 vs.

<

40: 0.7723), suggesting architectural differences in how spatial inconsistency manifests across patient demographics.

Figure 12: Spatial disagreement heatmaps. High MUS (top) shows strong per - crop JSD variation; low MUS (bottom) shows uniform consistency. Color scale: black (low) to yellow (high).

Figure 13: Grad - CAM validation. High MUS (red border) produces diffuse, non - lung attributions; low MUS (blue border) yields compact, localized activations.

Explore similar work

Uncertainty Is Not a Safety Net for Clinical VQA, but Can It Anticipate Model Failure?
Jun 15, 2026Arnisa Fazla, Alberto Testoni, Ameen Abu-Hanna +2Large Language Model Uncertainty Medical Visual Question Answering
Department of Medical Informatics, Amsterdam University Medical Center, University of Amsterdam, Amsterdam, The Netherlands. · Amsterdam Public Health, Methodology, Amsterdam, The Netherlands. · MaiNLP, Center for Information and Language Processing, LMU Munich, Germany. +1
SegWithU: Uncertainty as Perturbation Energy for Single-Forward-Pass Risk-Aware Medical Image Segmentation
Apr 16, 2026Tianhao Fu, Austin Wang, Charles Chen +2Semi-Supervised Medical Image Segmentation Uncertainty-Aware Classification
University of Toronto, Toronto, ON, Canada · Vector Institute, Toronto, ON, Canada · Project Neura, Toronto, ON, Canada +4
Rethinking Correctness for Uncertainty Estimation in Clinical Prediction with Vision-Language Models
Sep 14, 2026Mingcheng Zhu, Jinning Liang, Tingting ZhuClinical Prediction Correctness
University of Oxford, Oxford, United Kingdom

Jun 15, 2026, cs.CLJ/K move · Enter open · S save

Uncertainty Is Not a Safety Net for Clinical VQA, but Can It Anticipate Model Failure?

Arnisa Fazla, Alberto Testoni, Ameen Abu-Hanna, Barbara Plank+1

Department of Medical Informatics, Amsterdam University Medical Center, University of Amsterdam, Amsterdam, The Netherlands. · Amsterdam Public Health, Methodology, Amsterdam, The Netherlands. · MaiNLP, Center for Information and Language Processing, LMU Munich, Germany.+1

Large Language Model Uncertainty Medical Visual Question Answering Recent Vision-Language Models Uncertainty

Open paper AskPDF arXiv

Safe deployment of clinical vision-language models (VLMs) requires reliable uncertainty estimation (UE): a signal indicating when predictions should be trusted or escalated to a clinician. We test whether current UE methods actually deliver this signal. Benchmarking 8 methods across 12 VLMs on clinical visual question-answering (VQA), we find that UE quality is not an intrinsic property of the UE method: it tracks model accuracy, degrading precisely where the model performance is weakest, and therefore where reliability is most needed. When we stress-test models by hiding the correct option among the multiple-choice answers (NOTA perturbations), accuracy collapses while uncertainty barely changes, leaving models systematically miscalibrated. Yet, we find that uncertainty on the unperturbed input reliably anticipates which predictions will collapse under NOTA, indicating that UE in current VLMs carries diagnostic information about model fragility. Our results position UE as a diagnostic tool for identifying fragile predictions and motivate perturbation-based evaluation as a path toward safe clinical deployment.

Abstract

Explore similar work

Uncertainty Is Not a Safety Net for Clinical VQA, but Can It Anticipate Model Failure?

SegWithU: Uncertainty as Perturbation Energy for Single-Forward-Pass Risk-Aware Medical Image Segmentation

Rethinking Correctness for Uncertainty Estimation in Clinical Prediction with Vision-Language Models