LATA: Laplacian-Assisted Transductive Adaptation for Conformal Uncertainty in Medical VLMs

Behzad Bozorgtabar; Dwarikanath Mahapatra; Sudipta Roy; Muzammal Naseer; Imran Razzak; Zongyuan Ge

cs.CVFeb 19, 2026

LATA: Laplacian-Assisted Transductive Adaptation for Conformal Uncertainty in Medical VLMs

Authors: Behzad Bozorgtabar, Dwarikanath Mahapatra, Sudipta Roy, Muzammal Naseer, Imran Razzak, Zongyuan Ge

Organizations: Aarhus University (A3 Lab) · Khalifa University · Jio Institute · MBZUAI · Monash University

Abstract

Medical vision-language models (VLMs) are strong zero-shot recognizers for medical imaging, but their reliability under domain shift hinges on calibrated uncertainty with guarantees. Split conformal prediction (SCP) offers finite-sample coverage, yet prediction sets often become large (low efficiency) and class-wise coverage unbalanced-high class-conditioned coverage gap (CCV), especially in few-shot, imbalanced regimes; moreover, naively adapting to calibration labels breaks exchangeability and voids guarantees. We propose **LATA** (Laplacian-Assisted Transductive Adaptation), a training- and label-free refinement that operates on the joint calibration and test pool by smoothing zero-shot probabilities over an image-image k-NN graph using a small number of CCCP mean-field updates, preserving SCP validity via a deterministic transform. We further introduce a failure-aware conformal score that plugs into the vision-language uncertainty (ViLU) framework, providing instance-level difficulty and label plausibility to improve prediction set efficiency and class-wise balance at fixed coverage. **LATA** is black-box (no VLM updates), compute-light (windowed transduction, no backprop), and includes an optional prior knob that can run strictly label-free or, if desired, in a label-informed variant using calibration marginals once. Across three medical VLMs and nine downstream tasks, **LATA** consistently reduces set size and CCV while matching or tightening target coverage, outperforming prior transductive baselines and narrowing the gap to label-using methods, while using far less compute. Comprehensive ablations and qualitative analyses show that **LATA** sharpens zero-shot predictions without compromising exchangeability.

Figures & tables

Figure 1: LATA pipeline and coverage–efficiency trade-off. (a) LATA pipeline. Frozen vision/text encoders yield zero-shot scores $q(x)$ , optionally adjusted via calibration-informed priors. LATA then refines predictions on the joint unlabeled pool $\mathcal{U}$ using a sparse $k$ NN graph and CCCP updates, producing $\tilde{z}(x)$ . A frozen ViLU module estimates difficulty $u(x)$ and attention $\alpha(x)$ , forming a failure-aware score $S^{\star}$ , which is conformalized into calibrated prediction sets. (b) Coverage–efficiency frontier ( $\alpha{=}0.10$ , APS). LATA-LF ( $\beta{=}0$ ) achieves SCP-level coverage with lower set size and CCV. LATA-LI ( $\beta{=}0.2$ ) improves coverage further with minimal cost, outperforming SCA-T in both efficiency and balance.

	Group	Method	ACA $\uparrow$	$\boldsymbol{\alpha=0.10}$			$\boldsymbol{\alpha=0.05}$
	Group	Method	ACA $\uparrow$	Cov.	Size $\downarrow$	CCV $\downarrow$	Cov.	Size $\downarrow$	CCV $\downarrow$
LAC	Baseline	SCP	50.2	0.890	3.99	9.96	0.951	4.88	5.68
	Labeled-Cal.	Adapt+SCP	$67.1_{\color[rgb]{0,0.88,0}{+16.9}}$	0.842	$2.40_{\color[rgb]{0,0.88,0}{-1.59}}$	$11.17_{\color[rgb]{0.72,0,0}{+1.21}}$	0.921	$3.07_{\color[rgb]{0,0.88,0}{-1.81}}$	$6.87_{\color[rgb]{0.72,0,0}{+1.19}}$
	Labeled-Cal.	FCA [ 32 ]	$67.1_{\color[rgb]{0,0.88,0}{+16.9}}$	0.896	$2.91_{\color[rgb]{0,0.88,0}{-1.08}}$	$8.38_{\color[rgb]{0,0.88,0}{-1.58}}$	0.952	$3.56_{\color[rgb]{0,0.88,0}{-1.32}}$	$5.02_{\color[rgb]{0,0.88,0}{-0.66}}$
	UT	Conf-OT [ 28 ]	$53.1_{\color[rgb]{0,0.88,0}{+2.9}}$	0.899	$3.18_{\color[rgb]{0,0.88,0}{-0.81}}$	$9.07_{\color[rgb]{0,0.88,0}{-0.89}}$	0.950	$3.81_{\color[rgb]{0,0.88,0}{-1.07}}$	$5.34_{\color[rgb]{0,0.88,0}{-0.34}}$
	UT	SCA-T [ 29 ]	$55.2_{\color[rgb]{0,0.88,0}{+5.0}}$	0.898	$3.30_{\color[rgb]{0,0.88,0}{-0.69}}$	$7.47_{\color[rgb]{0,0.88,0}{-2.49}}$	0.952	$4.03_{\color[rgb]{0,0.88,0}{-0.85}}$	$4.03_{\color[rgb]{0,0.88,0}{-1.65}}$

Table 1: Conformal prediction results with 16-shot calibration across three nonconformity scores and two error levels,

\alpha\in\{0.05,0.10\}

. Results are averaged over tasks and modalities. “

\downarrow

” indicates smaller is better; red marks error violations. Subscripted margins denote absolute gains w.r.t. the SCP baseline.

Figure 2: SICAPv2 — coverage, efficiency, and set structure (LAC, $\alpha{=}0.10$ ). (a) Left: Coverage–efficiency trade-off as calibration shots increase ( $K\!\in\!\{4,8,16\}$ ; dot size encodes $K$ ). LATA (ours) defines the best label-free frontier—achieving smaller sets with equal or better coverage, approaching FCA without using labels at transfer. Right: Test-time coverage on $K{=}16$ splits (same seeds). Adapt+SCP under-covers; SCP/SCA-T reduce violations but remain dispersed; LATA concentrates near/above nominal with the lowest CCV. (b) Qualitative results at $K{=}16$ : per-class set-size distributions (left) and label co-occurrence (right). LATA focuses uncertainty on adjacent grades (G3–G4), reducing CCV while preserving coverage.

Method	ACA $\uparrow$	Runtime $\mathbf{T}$ $\downarrow$	GPU $\downarrow$	Cov.	Size $\downarrow$	CCV $\downarrow$
LAC	50.2	0.00	–	0.890	3.99	9.96
[.6pt/2pt] TIM [ 4 ]	53.5 +3.3	1.12	0.6	0.888	3.96 -0.03	8.08 -1.88
TransCLIP [ 40 ]	54.8 +4.6	0.47	1.1	0.726	2.16 -1.83	22.31 +12.35
Conf - OT [ 28 ]	53.1 +2.9	0.60	–	0.899	3.18 -0.81	9.07 -0.89
SCA - T [ 29 ]	55.2 +5.0	1.04	0.6	0.898	3.30 -0.69	7.47 -2.49
LATA-LF ( $\beta{=}0$ )	57.0 +6.8	0.05	0.8	0.900	3.07 -0.92	6.40 -3.56

Table 2: Transductive solvers at

\alpha{=}0.10

. Best and second-best results are shown in bold and underline , respectively. Red indicates violations of the target error rate.

Figure 3: Ablations on shots and window size (LAC, $\alpha{=}0.10$ ; averages across tasks). (a) Effect of calibration shots $K$ . (b) Effect of query/window size $W$ (dashed line marks the full-batch limit). Across both sweeps, LATA achieves smaller prediction sets and lower CCV than baselines while preserving nominal coverage.

Figure 4: Exchangeability and per-dataset $\Delta$ accuracy– $\Delta$ set-size (APS, $\alpha{=}0.10$ ). (a) Across datasets, LATA-LI yields $\Delta\text{Accuracy}{>}0$ and $\Delta\text{Set Size}{<}0$ vs. SCP, with a weak linear fit (small $R^{2}$ ), indicating efficiency gains are not merely due to accuracy increases. (b) Exchangeability sanity check: the invalid Probe@cal + SCP@same under-covers, while LATA-LF (shared, label-free transform) stays near the nominal $(1{-}\alpha)$ across random seed trials.

Variant	LATA	$u$	$\alpha$	ACA $\uparrow$	Cov.	Size $\downarrow$	CCV $\downarrow$	Cov.	Size $\downarrow$	CCV $\downarrow$
	Components				$\boldsymbol{\alpha=0.10}$			$\boldsymbol{\alpha=0.05}$
SCP	$\times$	$\times$	$\times$	50.2	0.890	3.99	9.96	0.951	4.88	5.68
$u$ -only	$\times$	✓	$\times$	50.6	0.900	3.95	9.15	0.952	4.80	5.35
$\alpha$ -only	$\times$	$\times$	✓	50.8	0.900	3.60	8.85	0.952	4.62	5.20
LATA (no ViLU)	✓	$\times$	$\times$	55.6	0.900	3.14	6.65	0.952	3.86	3.60
LATA (ours)	✓	✓	✓	57.0	0.900	3.07	6.40	0.952	3.76	3.35

Table 3: Component ablations (LAC) at

\alpha\!\in\!\{0.10,0.05\}

; averages across tasks. Columns LATA /

u

\alpha

indicate active modules. SCP : no LATA , no ViLU. LATA (no ViLU) : LATA with

\lambda{=}\eta{=}0

(both

u

and

\alpha

off). LATA (ours) : LATA +

u

\alpha

with

\beta{=}0

. Best and second-best in bold / underline .

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Labels at transfer?	Training?	Per-query refits	Dominant complexity	Memory	Time/img (s) †	SCP validity
LATA (ours)	No	No	None	$O(N\log N\cdot d)\;+\;O(T_{\text{iter}}\,k\,N\,C)\;+\;O(nC)$	$O(kN{+}NC)$	0.05–0.06	Yes
SCA-T	No	Yes (unsup.)	None	$O(T^{\prime}NC)\;+\;O(nC)$	logits+buffers	1.04–1.15	Yes
Conf-OT	No	No	None	$O(S\,K\,(N{+}m))\;+\;O(nC)$	OT matrices	0.60–0.70	Yes
FCA	Yes (cal.)	Yes	$O(C)$	$O(C\!\times\!\text{fit\_cost})\;+\;O(nC)$	probe heads+logits	— (solver-dep.)	Yes

Appendix

Table S1 : Complexity & compute profile . Asymptotics in

N{=}n{+}m

C

K

d

$\lambda$	$\eta$	Cov.	Size $\downarrow$ / CCV $\downarrow$
0.25	0.10	0.897	2.90 / 6.55
0.25	0.25	0.898	2.88 / 6.40
0.25	0.50	0.895	2.82 / 6.50
0.50	0.10	0.900	2.96 / 6.45
0.50	0.25	0.900	2.95 / 6.32
0.50	0.50	0.898	2.88 / 6.35

Appendix

Table S2: Sensitivity to failure-aware weights

(\lambda,\eta)

(APS,

\alpha=0.10

, LATA-LF). All configurations remain near the nominal coverage (

\approx 0.90

) and retain clear gains in set size and CCV compared to SCP (Size

=4.05

, CCV

=9.59

) and SCA-T (Size

=3.35

, CCV

=7.18

$\gamma$	Cov.	Size $\downarrow$	CCV $\downarrow$
0.20	0.899	3.00	6.55
0.35	0.900	2.95	6.32
0.50	0.901	2.98	6.30

Appendix

Table S3: Sensitivity to Laplacian weight

\gamma

(APS,

\alpha{=}0.10

, LATA-LF). All settings stay near nominal coverage and retain clear gains in Size/CCV vs. SCP (Size

=4.05

, CCV

=9.59

) and SCA-T (Size

=3.35

, CCV

=7.18

$k$	Cov.	Size $\downarrow$	CCV $\downarrow$
10	0.899	3.02	6.60
15	0.900	2.95	6.32
20	0.900	2.97	6.35

Appendix

Table S4: Sensitivity to graph degree

k

(APS,

\alpha{=}0.10

, LATA-LF ). All settings remain near nominal coverage and maintain clear gains over SCP (Size

=4.05

, CCV

=9.59

) and SCA-T (Size

=3.35

, CCV

=7.18

Figure S1 : Compute–accuracy trade-off for LATA at $\alpha{=}0.10$ . Time per image (x-axis) vs. average set size (y-axis) for $T_{\text{iter}}\!\in\!\{4,8,12\}$ . Colors denote LAC/APS/RAPS; marker size encodes CCV. Annotations show Cov., CCV, and GPU memory. Default $T_{\text{iter}}{=}8$ balances speed and reliability; $T_{\text{iter}}{=}4$ is faster with mild trade-offs; $T_{\text{iter}}{=}12$ yields limited gains.

Figure S2 : Temperature sensitivity at $\alpha{=}0.10$ . Average set size versus temperature for LAC, APS, and RAPS under Base (SCP) and LATA-LF . LATA-LF preserves nominal coverage and remains strictly more efficient than Base across $\tau\!\in\![0.6,1.4]$ .

Variant	ACA $\uparrow$	$\boldsymbol{\alpha=0.10}$			$\boldsymbol{\alpha=0.05}$
Variant	ACA $\uparrow$	Cov.	Size $\downarrow$	CCV $\downarrow$	Cov.	Size $\downarrow$	CCV $\downarrow$
SCP	50.2	0.900	4.05	9.59	0.952	4.88	5.54
$u$ -only	50.6	0.900	3.98	9.15	0.952	4.82	5.30
$\alpha$ -only	50.8	0.900	3.55	8.80	0.952	4.62	5.15
LATA (no ViLU)	55.6	0.900	3.05	6.60	0.954	3.88	3.70
LATA (ours)	57.1	0.900	2.95	6.32	0.954	3.78	3.55

Appendix

Table S5 : Component ablation (APS) at

\alpha\!\in\!\{0.10,0.05\}

. Setup/notation follow Table 3 in the main manuscript.

		LAC , $\alpha{=}0.10$			LAC , $\alpha{=}0.05$
$\beta$	ACA $\uparrow$	Cov.	Size $\downarrow$	CCV $\downarrow$	Cov.	Size $\downarrow$	CCV $\downarrow$
0.0	57.0	0.900	3.07	6.40	0.952	3.76	3.35
0.1	57.2	0.905	3.11	6.32	0.958	3.82	3.40
0.2	57.4	0.910	3.15	6.25	0.962	3.86	3.40
0.3	57.4	0.914	3.20	6.22	0.966	3.92	3.45

Appendix

Table S6 : Sensitivity to the label-informed prior (

\beta

). We apply the prior as

q\leftarrow\mathrm{Renorm}\!\big(q\odot m^{\beta}\big)

(equivalently, logits

z\leftarrow z+\beta\log m

), where

m

are Dirichlet-smoothed calibration marginals (fixed once) and the same transform is applied to calibration and test. Numbers are averaged across tasks.

		APS , $\alpha{=}0.10$			APS , $\alpha{=}0.05$
$\beta$	ACA $\uparrow$	Cov.	Size $\downarrow$	CCV $\downarrow$	Cov.	Size $\downarrow$	CCV $\downarrow$
0.0	57.1	0.900	2.95	6.32	0.954	3.78	3.55
0.1	57.3	0.905	2.99	6.28	0.959	3.83	3.50
0.2	57.5	0.910	3.03	6.25	0.963	3.88	3.45
0.3	57.5	0.914	3.08	6.23	0.966	3.93	3.44

Appendix

Table S7 : Prior sensitivity for APS . Same prior application and setup as Table S6 .

Setting	Cov.	Size $\downarrow$	CCV $\downarrow$	T (s/img) / GPU (GB)
LATA (ours) (matched ViLU)	0.900	3.07	6.40	0.05 / 0.80
LATA (ours) (mismatched ViLU)	0.900	3.12	6.52	0.05 / 0.80
LATA (ours) (no ViLU: $\lambda{=}\eta{=}0$ )	0.900	3.22	6.90	0.05 / 0.80

Appendix

Table S8 : Effect of ViLU pretraining source (LAC,

\alpha{=}0.10

). UT window:

N{=}256

k{=}15

T_{\text{iter}}{=}8

; averaged over random seeds.

$\tau_{u}$	Cov.	Size $\downarrow$	CCV $\downarrow$	T (s/img) / GPU (GB)
0.00 (no gating)	0.900	3.07	6.40	0.050 / 0.80
0.20	0.900	3.12	6.48	0.045 / 0.70
0.30	0.900	3.16	6.55	0.040 / 0.65

Appendix

Table S9 : Resource-aware

u

-gating under LATA (ours) (LAC,

\alpha{=}0.10

). Averaged over random seeds; UT:

N{=}256

k{=}15

T_{\text{iter}}{=}8

. We skip CCCP updates for inputs with

u(x)\!<\!\tau_{u}

. Raising

\tau_{u}

lowers time/memory (e.g.,

\tau_{u}{=}0.30

: T

-20\%

, GPU

-18.8\%

) with no coverage loss .

Figure S3 : Qualitative comparison of prediction sets on Gleason grading (SICAPv2, APS, $\alpha{=}0.10$ ). LATA (ours) produces smaller sets than FCA—especially for ambiguous G3/G4—while preserving strong diagonals and clinically plausible adjacency (G3 $\leftrightarrow$ G4) and further suppressing unlikely NC $\leftrightarrow$ G5 co-occurrences.

Figure S4 : Coverage analysis under LAC at $\alpha=0.10$ . The dashed line indicates the nominal coverage level. LATA (ours) consistently achieves coverage closest to the target across all datasets.

Figure S5 : Conformal prediction results across datasets using $\alpha=0.10$ and LAC [ 26 ] . Each point represents a performance, with bubble size indicating the number of adaptation shots $K\in\{4,8,16\}$ .

Explore similar work

Learning to Adapt and Calibrate: Score Distribution Alignment for Few-Shot Uncertainty Prediction in Medical VLMs
Sep 9, 2026Xuan Cuong Ngo, Ngan LeVLM Adaptation Conformal Prediction
University of Arkansas Fayetteville, Arkansas, USA
Does Marginal Coverage Guarantee Class-Conditional Safety for Zero-Shot VLMs Under Shift?
Aug 19, 2026Jai Kumar Sharma, Amartya DuttaConformal Prediction VLM Evaluation
Virginia Tech, Blacksburg, VA, USA
Mitigating Class-Tail Undercoverage in Medical Vision-Language Models under Clinical Shift
Jul 30, 2026Mushir Akhtar, M. TanveerGroup-Conditional Conformal Prediction Conformal Prediction
Department of Mathematics, Indian Institute of Technology Indore, India

Sep 9, 2026J/K move · Enter open · S save

Learning to Adapt and Calibrate: Score Distribution Alignment for Few-Shot Uncertainty Prediction in Medical VLMs

Xuan Cuong Ngo, Ngan Le

University of Arkansas Fayetteville, Arkansas, USA

VLM Adaptation Conformal Prediction Medical VLMs Adaptive Conformal Inference VLM Calibration

Open paper AskPDF arXiv

Uncertainty estimation for medical vision--language models (VLMs) using conformal prediction has gained increasing attention due to its distribution-free coverage guarantees. However, standard conformal prediction relies on exchangeability between calibration and test data and typically requires a sufficiently large calibration set to obtain reliable coverage. These assumptions are difficult to satisfy in few-shot transfer settings, where only a small labeled support set is available to adapt a pretrained VLM to a new medical task, while an unlabeled query set is used for evaluation. Supervised fine-tuning on the support set changes the model parameters and consequently shifts the nonconformity score distribution, breaking exchangeability between calibration and query samples and leading to unreliable coverage under distribution shift. Existing transductive conformal adaptation methods often preserve validity by avoiding supervised updates. While this helps maintain conformal assumptions, it underutilizes the scarce labeled support data and limits task adaptation, which is the primary objective in few-shot learning. In this setting, conformal prediction should serve as an uncertainty estimation layer that supports the adapted model, rather than preventing adaptation itself. To this end, we propose AlignCP, a framework that reconciles supervised few-shot adaptation with conformal uncertainty estimation under non-exchangeability. AlignCP learns a reweighted calibration distribution that reduces the score-level discrepancy between the labeled support set and the unlabeled query set. By aligning the one-dimensional nonconformity score distributions, AlignCP aims to close the coverage gap induced by adaptation without requiring query labels.

Abstract

Explore similar work

Learning to Adapt and Calibrate: Score Distribution Alignment for Few-Shot Uncertainty Prediction in Medical VLMs

Does Marginal Coverage Guarantee Class-Conditional Safety for Zero-Shot VLMs Under Shift?

Mitigating Class-Tail Undercoverage in Medical Vision-Language Models under Clinical Shift