LATA: Laplacian-Assisted Transductive Adaptation for Conformal Uncertainty in Medical VLMs

Behzad Bozorgtabar; Dwarikanath Mahapatra; Sudipta Roy; Muzammal Naseer; Imran Razzak; Zongyuan Ge

cs.CVFeb 19, 2026

LATA: Laplacian-Assisted Transductive Adaptation for Conformal Uncertainty in Medical VLMs

Authors: Behzad Bozorgtabar, Dwarikanath Mahapatra, Sudipta Roy, Muzammal Naseer, Imran Razzak, Zongyuan Ge

Organizations: Aarhus University (A3 Lab) · Khalifa University · Jio Institute · MBZUAI · Monash University

Abstract

Medical vision-language models (VLMs) are strong zero-shot recognizers for medical imaging, but their reliability under domain shift hinges on calibrated uncertainty with guarantees. Split conformal prediction (SCP) offers finite-sample coverage, yet prediction sets often become large (low efficiency) and class-wise coverage unbalanced-high class-conditioned coverage gap (CCV), especially in few-shot, imbalanced regimes; moreover, naively adapting to calibration labels breaks exchangeability and voids guarantees. We propose **LATA** (Laplacian-Assisted Transductive Adaptation), a training- and label-free refinement that operates on the joint calibration and test pool by smoothing zero-shot probabilities over an image-image k-NN graph using a small number of CCCP mean-field updates, preserving SCP validity via a deterministic transform. We further introduce a failure-aware conformal score that plugs into the vision-language uncertainty (ViLU) framework, providing instance-level difficulty and label plausibility to improve prediction set efficiency and class-wise balance at fixed coverage. **LATA** is black-box (no VLM updates), compute-light (windowed transduction, no backprop), and includes an optional prior knob that can run strictly label-free or, if desired, in a label-informed variant using calibration marginals once. Across three medical VLMs and nine downstream tasks, **LATA** consistently reduces set size and CCV while matching or tightening target coverage, outperforming prior transductive baselines and narrowing the gap to label-using methods, while using far less compute. Comprehensive ablations and qualitative analyses show that **LATA** sharpens zero-shot predictions without compromising exchangeability.

Figures & tables

Figure 1: LATA pipeline and coverage–efficiency trade-off. (a) LATA pipeline. Frozen vision/text encoders yield zero-shot scores $q(x)$ , optionally adjusted via calibration-informed priors. LATA then refines predictions on the joint unlabeled pool $\mathcal{U}$ using a sparse $k$ NN graph and CCCP updates, producing $\tilde{z}(x)$ . A frozen ViLU module estimates difficulty $u(x)$ and attention $\alpha(x)$ , forming a failure-aware score $S^{\star}$ , which is conformalized into calibrated prediction sets. (b) Coverage–efficiency frontier ( $\alpha{=}0.10$ , APS). LATA-LF ( $\beta{=}0$ ) achieves SCP-level coverage with lower set size and CCV. LATA-LI ( $\beta{=}0.2$ ) improves coverage further with minimal cost, outperforming SCA-T in both efficiency and balance.

	Group	Method	ACA $\uparrow$	$\boldsymbol{\alpha=0.10}$			$\boldsymbol{\alpha=0.05}$
	Group	Method	ACA $\uparrow$	Cov.	Size $\downarrow$	CCV $\downarrow$	Cov.	Size $\downarrow$	CCV $\downarrow$
LAC	Baseline	SCP	50.2	0.890	3.99	9.96	0.951	4.88	5.68
	Labeled-Cal.	Adapt+SCP	$67.1_{\color[rgb]{0,0.88,0}{+16.9}}$	0.842	$2.40_{\color[rgb]{0,0.88,0}{-1.59}}$	$11.17_{\color[rgb]{0.72,0,0}{+1.21}}$	0.921	$3.07_{\color[rgb]{0,0.88,0}{-1.81}}$	$6.87_{\color[rgb]{0.72,0,0}{+1.19}}$
	Labeled-Cal.	FCA [ 32 ]	$67.1_{\color[rgb]{0,0.88,0}{+16.9}}$	0.896	$2.91_{\color[rgb]{0,0.88,0}{-1.08}}$	$8.38_{\color[rgb]{0,0.88,0}{-1.58}}$	0.952	$3.56_{\color[rgb]{0,0.88,0}{-1.32}}$	$5.02_{\color[rgb]{0,0.88,0}{-0.66}}$
	UT	Conf-OT [ 28 ]	$53.1_{\color[rgb]{0,0.88,0}{+2.9}}$	0.899	$3.18_{\color[rgb]{0,0.88,0}{-0.81}}$	$9.07_{\color[rgb]{0,0.88,0}{-0.89}}$	0.950	$3.81_{\color[rgb]{0,0.88,0}{-1.07}}$	$5.34_{\color[rgb]{0,0.88,0}{-0.34}}$
	UT	SCA-T [ 29 ]	$55.2_{\color[rgb]{0,0.88,0}{+5.0}}$	0.898	$3.30_{\color[rgb]{0,0.88,0}{-0.69}}$	$7.47_{\color[rgb]{0,0.88,0}{-2.49}}$	0.952	$4.03_{\color[rgb]{0,0.88,0}{-0.85}}$	$4.03_{\color[rgb]{0,0.88,0}{-1.65}}$

Table 1: Conformal prediction results with 16-shot calibration across three nonconformity scores and two error levels,

\alpha\in\{0.05,0.10\}

. Results are averaged over tasks and modalities. “

\downarrow

” indicates smaller is better; red marks error violations. Subscripted margins denote absolute gains w.r.t. the SCP baseline.

Figure 2: SICAPv2 — coverage, efficiency, and set structure (LAC, $\alpha{=}0.10$ ). (a) Left: Coverage–efficiency trade-off as calibration shots increase ( $K\!\in\!\{4,8,16\}$ ; dot size encodes $K$ ). LATA (ours) defines the best label-free frontier—achieving smaller sets with equal or better coverage, approaching FCA without using labels at transfer. Right: Test-time coverage on $K{=}16$ splits (same seeds). Adapt+SCP under-covers; SCP/SCA-T reduce violations but remain dispersed; LATA concentrates near/above nominal with the lowest CCV. (b) Qualitative results at $K{=}16$ : per-class set-size distributions (left) and label co-occurrence (right). LATA focuses uncertainty on adjacent grades (G3–G4), reducing CCV while preserving coverage.

Method	ACA $\uparrow$	Runtime $\mathbf{T}$ $\downarrow$	GPU $\downarrow$	Cov.	Size $\downarrow$	CCV $\downarrow$
LAC	50.2	0.00	–	0.890	3.99	9.96
[.6pt/2pt] TIM [ 4 ]	53.5 +3.3	1.12	0.6	0.888	3.96 -0.03	8.08 -1.88
TransCLIP [ 40 ]	54.8 +4.6	0.47	1.1	0.726	2.16 -1.83	22.31 +12.35
Conf - OT [ 28 ]	53.1 +2.9	0.60	–	0.899	3.18 -0.81	9.07 -0.89
SCA - T [ 29 ]	55.2 +5.0	1.04	0.6	0.898	3.30 -0.69	7.47 -2.49
LATA-LF ( $\beta{=}0$ )	57.0 +6.8	0.05	0.8	0.900	3.07 -0.92	6.40 -3.56

Table 2: Transductive solvers at

\alpha{=}0.10

. Best and second-best results are shown in bold and underline , respectively. Red indicates violations of the target error rate.

Figure 3: Ablations on shots and window size (LAC, $\alpha{=}0.10$ ; averages across tasks). (a) Effect of calibration shots $K$ . (b) Effect of query/window size $W$ (dashed line marks the full-batch limit). Across both sweeps, LATA achieves smaller prediction sets and lower CCV than baselines while preserving nominal coverage.

Figure 4: Exchangeability and per-dataset $\Delta$ accuracy– $\Delta$ set-size (APS, $\alpha{=}0.10$ ). (a) Across datasets, LATA-LI yields $\Delta\text{Accuracy}{>}0$ and $\Delta\text{Set Size}{<}0$ vs. SCP, with a weak linear fit (small $R^{2}$ ), indicating efficiency gains are not merely due to accuracy increases. (b) Exchangeability sanity check: the invalid Probe@cal + SCP@same under-covers, while LATA-LF (shared, label-free transform) stays near the nominal $(1{-}\alpha)$ across random seed trials.

Variant	LATA	$u$	$\alpha$	ACA $\uparrow$	Cov.	Size $\downarrow$	CCV $\downarrow$	Cov.	Size $\downarrow$	CCV $\downarrow$
	Components				$\boldsymbol{\alpha=0.10}$			$\boldsymbol{\alpha=0.05}$
SCP	$\times$	$\times$	$\times$	50.2	0.890	3.99	9.96	0.951	4.88	5.68
$u$ -only	$\times$	✓	$\times$	50.6	0.900	3.95	9.15	0.952	4.80	5.35
$\alpha$ -only	$\times$	$\times$	✓	50.8	0.900	3.60	8.85	0.952	4.62	5.20
LATA (no ViLU)	✓	$\times$	$\times$	55.6	0.900	3.14	6.65	0.952	3.86	3.60
LATA (ours)	✓	✓	✓	57.0	0.900	3.07	6.40	0.952	3.76	3.35

Table 3: Component ablations (LAC) at

\alpha\!\in\!\{0.10,0.05\}

; averages across tasks. Columns LATA /

u

\alpha

indicate active modules. SCP : no LATA , no ViLU. LATA (no ViLU) : LATA with

\lambda{=}\eta{=}0

(both

u

and

\alpha

off). LATA (ours) : LATA +

u

\alpha

with

\beta{=}0

. Best and second-best in bold / underline .

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Labels at transfer?	Training?	Per-query refits	Dominant complexity	Memory	Time/img (s) †	SCP validity
LATA (ours)	No	No	None	$O(N\log N\cdot d)\;+\;O(T_{\text{iter}}\,k\,N\,C)\;+\;O(nC)$	$O(kN{+}NC)$	0.05–0.06	Yes
SCA-T	No	Yes (unsup.)	None	$O(T^{\prime}NC)\;+\;O(nC)$	logits+buffers	1.04–1.15	Yes
Conf-OT	No	No	None	$O(S\,K\,(N{+}m))\;+\;O(nC)$	OT matrices	0.60–0.70	Yes
FCA	Yes (cal.)	Yes	$O(C)$	$O(C\!\times\!\text{fit\_cost})\;+\;O(nC)$	probe heads+logits	— (solver-dep.)	Yes

Appendix

Table S1 : Complexity & compute profile . Asymptotics in

N{=}n{+}m

C

K

d

$\lambda$	$\eta$	Cov.	Size $\downarrow$ / CCV $\downarrow$
0.25	0.10	0.897	2.90 / 6.55
0.25	0.25	0.898	2.88 / 6.40
0.25	0.50	0.895	2.82 / 6.50
0.50	0.10	0.900	2.96 / 6.45
0.50	0.25	0.900	2.95 / 6.32
0.50	0.50	0.898	2.88 / 6.35

Appendix

Table S2: Sensitivity to failure-aware weights

(\lambda,\eta)

(APS,

\alpha=0.10

, LATA-LF). All configurations remain near the nominal coverage (

\approx 0.90

) and retain clear gains in set size and CCV compared to SCP (Size

=4.05

, CCV

=9.59

) and SCA-T (Size

=3.35

, CCV

=7.18

$\gamma$	Cov.	Size $\downarrow$	CCV $\downarrow$
0.20	0.899	3.00	6.55
0.35	0.900	2.95	6.32
0.50	0.901	2.98	6.30

Appendix

Table S3: Sensitivity to Laplacian weight

\gamma

(APS,

\alpha{=}0.10

, LATA-LF). All settings stay near nominal coverage and retain clear gains in Size/CCV vs. SCP (Size

=4.05

, CCV

=9.59

) and SCA-T (Size

=3.35

, CCV

=7.18

$k$	Cov.	Size $\downarrow$	CCV $\downarrow$
10	0.899	3.02	6.60
15	0.900	2.95	6.32
20	0.900	2.97	6.35

Appendix

Table S4: Sensitivity to graph degree

k

(APS,

\alpha{=}0.10

, LATA-LF ). All settings remain near nominal coverage and maintain clear gains over SCP (Size

=4.05

, CCV

=9.59

) and SCA-T (Size

=3.35

, CCV

=7.18

Figure S1 : Compute–accuracy trade-off for LATA at $\alpha{=}0.10$ . Time per image (x-axis) vs. average set size (y-axis) for $T_{\text{iter}}\!\in\!\{4,8,12\}$ . Colors denote LAC/APS/RAPS; marker size encodes CCV. Annotations show Cov., CCV, and GPU memory. Default $T_{\text{iter}}{=}8$ balances speed and reliability; $T_{\text{iter}}{=}4$ is faster with mild trade-offs; $T_{\text{iter}}{=}12$ yields limited gains.

Figure S2 : Temperature sensitivity at $\alpha{=}0.10$ . Average set size versus temperature for LAC, APS, and RAPS under Base (SCP) and LATA-LF . LATA-LF preserves nominal coverage and remains strictly more efficient than Base across $\tau\!\in\![0.6,1.4]$ .

Variant	ACA $\uparrow$	$\boldsymbol{\alpha=0.10}$			$\boldsymbol{\alpha=0.05}$
Variant	ACA $\uparrow$	Cov.	Size $\downarrow$	CCV $\downarrow$	Cov.	Size $\downarrow$	CCV $\downarrow$
SCP	50.2	0.900	4.05	9.59	0.952	4.88	5.54
$u$ -only	50.6	0.900	3.98	9.15	0.952	4.82	5.30
$\alpha$ -only	50.8	0.900	3.55	8.80	0.952	4.62	5.15
LATA (no ViLU)	55.6	0.900	3.05	6.60	0.954	3.88	3.70
LATA (ours)	57.1	0.900	2.95	6.32	0.954	3.78	3.55

Appendix

Table S5 : Component ablation (APS) at

\alpha\!\in\!\{0.10,0.05\}

. Setup/notation follow Table 3 in the main manuscript.

		LAC , $\alpha{=}0.10$			LAC , $\alpha{=}0.05$
$\beta$	ACA $\uparrow$	Cov.	Size $\downarrow$	CCV $\downarrow$	Cov.	Size $\downarrow$	CCV $\downarrow$
0.0	57.0	0.900	3.07	6.40	0.952	3.76	3.35
0.1	57.2	0.905	3.11	6.32	0.958	3.82	3.40
0.2	57.4	0.910	3.15	6.25	0.962	3.86	3.40
0.3	57.4	0.914	3.20	6.22	0.966	3.92	3.45

Appendix

Table S6 : Sensitivity to the label-informed prior (

\beta

). We apply the prior as

q\leftarrow\mathrm{Renorm}\!\big(q\odot m^{\beta}\big)

(equivalently, logits

z\leftarrow z+\beta\log m

), where

m

are Dirichlet-smoothed calibration marginals (fixed once) and the same transform is applied to calibration and test. Numbers are averaged across tasks.

		APS , $\alpha{=}0.10$			APS , $\alpha{=}0.05$
$\beta$	ACA $\uparrow$	Cov.	Size $\downarrow$	CCV $\downarrow$	Cov.	Size $\downarrow$	CCV $\downarrow$
0.0	57.1	0.900	2.95	6.32	0.954	3.78	3.55
0.1	57.3	0.905	2.99	6.28	0.959	3.83	3.50
0.2	57.5	0.910	3.03	6.25	0.963	3.88	3.45
0.3	57.5	0.914	3.08	6.23	0.966	3.93	3.44

Appendix

Table S7 : Prior sensitivity for APS . Same prior application and setup as Table S6 .

Setting	Cov.	Size $\downarrow$	CCV $\downarrow$	T (s/img) / GPU (GB)
LATA (ours) (matched ViLU)	0.900	3.07	6.40	0.05 / 0.80
LATA (ours) (mismatched ViLU)	0.900	3.12	6.52	0.05 / 0.80
LATA (ours) (no ViLU: $\lambda{=}\eta{=}0$ )	0.900	3.22	6.90	0.05 / 0.80

Appendix

Table S8 : Effect of ViLU pretraining source (LAC,

\alpha{=}0.10

). UT window:

N{=}256

k{=}15

T_{\text{iter}}{=}8

; averaged over random seeds.

$\tau_{u}$	Cov.	Size $\downarrow$	CCV $\downarrow$	T (s/img) / GPU (GB)
0.00 (no gating)	0.900	3.07	6.40	0.050 / 0.80
0.20	0.900	3.12	6.48	0.045 / 0.70
0.30	0.900	3.16	6.55	0.040 / 0.65

Appendix

Table S9 : Resource-aware

u

-gating under LATA (ours) (LAC,

\alpha{=}0.10

). Averaged over random seeds; UT:

N{=}256

k{=}15

T_{\text{iter}}{=}8

. We skip CCCP updates for inputs with

u(x)\!<\!\tau_{u}

. Raising

\tau_{u}

lowers time/memory (e.g.,

\tau_{u}{=}0.30

: T

-20\%

, GPU

-18.8\%

) with no coverage loss .

Figure S3 : Qualitative comparison of prediction sets on Gleason grading (SICAPv2, APS, $\alpha{=}0.10$ ). LATA (ours) produces smaller sets than FCA—especially for ambiguous G3/G4—while preserving strong diagonals and clinically plausible adjacency (G3 $\leftrightarrow$ G4) and further suppressing unlikely NC $\leftrightarrow$ G5 co-occurrences.

Figure S4 : Coverage analysis under LAC at $\alpha=0.10$ . The dashed line indicates the nominal coverage level. LATA (ours) consistently achieves coverage closest to the target across all datasets.

Figure S5 : Conformal prediction results across datasets using $\alpha=0.10$ and LAC [ 26 ] . Each point represents a performance, with bubble size indicating the number of adaptation shots $K\in\{4,8,16\}$ .

Explore similar work

Sep 9, 2026

Learning to Adapt and Calibrate: Score Distribution Alignment for Few-Shot Uncertainty Prediction in Medical VLMs

Uncertainty estimation for medical vision--language models (VLMs) using conformal prediction has gained increasing attention due to its distribution-free coverage guarantees. However, standard conformal prediction relies on exchangeability between calibration and test data and typically requires a sufficiently large calibration set to obtain reliable coverage. These assumptions are difficult to satisfy in few-shot transfer settings, where only a small labeled support set is available to adapt a pretrained VLM to a new medical task, while an unlabeled query set is used for evaluation. Supervised fine-tuning on the support set changes the model parameters and consequently shifts the nonconformity score distribution, breaking exchangeability between calibration and query samples and leading to unreliable coverage under distribution shift. Existing transductive conformal adaptation methods often preserve validity by avoiding supervised updates. While this helps maintain conformal assumptions, it underutilizes the scarce labeled support data and limits task adaptation, which is the primary objective in few-shot learning. In this setting, conformal prediction should serve as an uncertainty estimation layer that supports the adapted model, rather than preventing adaptation itself. To this end, we propose AlignCP, a framework that reconciles supervised few-shot adaptation with conformal uncertainty estimation under non-exchangeability. AlignCP learns a reweighted calibration distribution that reduces the score-level discrepancy between the labeled support set and the unlabeled query set. By aligning the one-dimensional nonconformity score distributions, AlignCP aims to close the coverage gap induced by adaptation without requiring query labels.

Xuan Cuong Ngo, Ngan Le

University of Arkansas Fayetteville, Arkansas, USA

VLM Adaptation Conformal Prediction Medical VLMs Adaptive Conformal Inference VLM Calibration

Ask PDF

Aug 19, 2026

Does Marginal Coverage Guarantee Class-Conditional Safety for Zero-Shot VLMs Under Shift?

Split-conformal prediction provides marginal coverage under exchangeability and is increasingly used as an abstention layer for zero-shot vision-language models (VLMs). We audit this practice under deployment shift for CLIP, OpenCLIP, and SigLIP across ImageNet and non-ImageNet settings. Marginal coverage can remain relatively high while class-conditional tail coverage collapses: on ImageNet-Sketch, worst-class coverage falls to

\approx 0

and 10-12% of classes lie below a finite-sample null floor, despite marginal coverage of about 0.86. The failure is aligned with target-domain class accuracy but is not predicted by the source-domain diagnostics we test. Source-side Mondrian calibration improves the in-distribution tail but does not transfer, while clustered conformal and Conf-OT improve marginal or average metrics without recovering the worst-class tail. Target-side class calibration substantially lifts the tail, but requires labels for every class and remains set-size-intensive. We further identify a 2-3

\times

cross-family efficiency gap and show that native SigLIP sigmoid scores remove APS's probability-mass interpretation. The findings persist across the tested model scale, pretraining corpus, prompt, miscoverage level

α

, and shifted non-ImageNet settings. Marginal conformal coverage should therefore be treated as an average reliability statistic, not as a safety guarantee for the class tail.

Jai Kumar Sharma, Amartya Dutta

Virginia Tech, Blacksburg, VA, USA

Conformal Prediction VLM Evaluation VLM Robustness Group-Conditional Conformal Prediction Zero-Shot Learning

Ask PDF

Jul 30, 2026

Mitigating Class-Tail Undercoverage in Medical Vision-Language Models under Clinical Shift

Medical vision-language models (VLMs) can retain high observed marginal coverage after clinical shift while substantially under-covering an individual disease class. The affected class varies with acquisition protocol and backbone geometry, so source prevalence does not reliably reveal the failure. Existing localized and tail-aware conformal methods respectively adapt to test neighborhoods and source-frequency tails, leaving held-out class-wise coverage failure unmodeled. We introduce Class-Tail Adaptive Localized Conformal Deferral (CALCoDe), a post-hoc reliability layer for frozen medical VLMs. Cross-fitted validation predictions identify classes at risk of undercoverage, and a disjoint calibration split estimates their class-conditional tail thresholds. CALCoDe combines each protected threshold with a localized conformal threshold using a one-sided maximum. The resulting set contains every label admitted by the localized rule, with additional protection confined to validation-identified classes. An independently calibrated support audit defers cases with insufficient inlier support. Under exchangeability among accepted examples within each protected class, CALCoDe provides finite-sample coverage at the prespecified guard level and contains the corresponding localized conformal sets; coverage on shifted external cohorts is evaluated empirically. Among standard conformal baselines and recent VLM-specific conformal methods evaluated across two dermatology shifts (HAM10000 to ISIC 2019 and HAM10000 to PAD-UFES-20) and four frozen VLM backbones (BiomedCLIP, OpenAI CLIP ViT-B/32, PubMedCLIP ViT-B/32, and MedSigLIP-448), CALCoDe is the only approach whose observed marginal and worst-class accepted coverage both reach 0.95 in all eight settings. On HAM10000 to ISIC 2019, its average worst-class accepted coverage is 0.970, compared with 0.926 for sTACP and 0.864 for LCP-VLM.

Mushir Akhtar, M. Tanveer

Department of Mathematics, Indian Institute of Technology Indore, India

Group-Conditional Conformal Prediction Conformal Prediction Medical VLMs VLM Calibration

Ask PDF