Gated Prototype Transfer for Zero-Support Medical Vision Language Model

Sumaiya Fazal; Rizwan Qureshi; Laleh Behjat; Samira Ebrahimi Kahou

cs.CVOct 20, 2026

Gated Prototype Transfer for Zero-Support Medical Vision Language Model

Authors: Sumaiya Fazal, Rizwan Qureshi, Laleh Behjat, Samira Ebrahimi Kahou

Organizations: FAST-NUCES · Massachusetts General Hospital, Harvard University · University of Calgary · University of Calgary/Mila

Abstract

Realistic few-shot adaptation of medical vision-language models (VLMs) can leave low-prevalence classes without any labeled support. The training-free SS- Text+ solver remains operational in this setting, but its zero-support prototypes receive no visual adaptation and fall back to a purely textual class prototype. We in- troduce Gated Knowledge-Guided Prototype Transfer (GProtoT), a pre-repulsion module that transfers supported-class visual residuals to missing classes using VLM similarity and an ontology-based clinical distance. A validation-free evidence gate scales transfer based on semantic relevance and residual agreement, thereby limiting unsupported extrapolation. In 50 paired realistic one-shot trials on MESSIDOR with FLAIR features, severe and proliferative diabetic retinopathy were absent from every support set. Relative to SS-Text+, gated ProtoT improved missing-class recall by 5.23 points, average class accuracy by 1.37 points, and the missing/supported recall harmonic mean by 1.83 points. It reduced negative- transfer frequency from 46-48% for ungated transfer to 2%, while at the cost of reductions of 1.20 percentage points in supported-class recall and 0.45 percentage points in top-1 accuracy. These results indicate that evidence-based gating provides a safety mechanism for the knowledge source. GProtoT is based on SS- Text+ and requires no gradients, validation data, or additional model inference, of- fering a low-compute path toward accountable adaptation under incomplete clini- cal supervision. Code is available: https://github.com/sumaiyaafridi/GProtoT

Figures & tables

Method	Top-1	ACA	MCR	SCR	$H$	NTR	Sev.
SS-Text+	57.45	54.64	54.90	54.46	52.87	–	–
ProtoT-VLM	54.83	51.91	52.73	51.37	50.25	46	38
	$(-2.62)$	$(-2.72)$	$(-2.16)$	$(-3.10)$	$(-2.63)$
ProtoT-Ont	54.92	51.90	52.47	51.52	50.20	48	34
	$(-2.53)$	$(-2.74)$	$(-2.43)$	$(-2.94)$	$(-2.67)$
Gated-ProtoT	57.00	56.01	60.12	53.27	54.71	2	2

Table 1: Fifty-seed realistic one-shot MESSIDOR performance. Parentheses report paired changes from SS-Text+ in percentage points. NTR and severe NTR denote the percentages of seeds with

\Delta\mathrm{MCR}<0

and

\Delta\mathrm{MCR}<-5

points, respectively

Method	Top-1	ACA	MCR	SCR	$H$	NTR	Sev.
SS-Text+	77.577	57.166	12.276	84.100	21.344	–	–
Gated-ProtoT	79.444	63.193	28.709	83.884	42.729	0	0
$\Delta$ (points)	$+1.867$	$+6.027$	$+16.433$	$-0.216$	$+21.385$	–	–

Table 2: Skin histology results with CONCH ( Lu et al., 2024 ) over 50 paired realistic nominal

K=1

seeds. Performance values are mean percentages;

\Delta

reports paired changes from SS-Text+ in percentage points. NTR and Sev. denote the percentages of seeds with

\Delta\mathrm{MCR}<0

and

\Delta\mathrm{MCR}<-5

points, respectively.

VLM Adaptation

Vision-language model adaptation tailors pretrained image-text and multimodal models to new tasks and domains while retaining transferable capabilities.

See all 545 papers

What Do Medical Vision-Language Models Learn in Radiology? Transfer, Alignment, and Source-Proxy Leakage Under Distribution Shift
Aug 26, 2026Ayoub Louaye Bouaziz, Lokmane Chebouba, Yassine HimeurMedical VLMs Radiology VLMs
LaTIM, Inserm, University of Western Brittany, Brest, France · University of Constantine, Algeria · University of Dubai, UAE
Semi-Supervised Few-Shot Adaptation of Vision-Language Models
Mar 3, 2026Julio Silva-Rodríguez, Ender KonukogluVLM Adaptation Few-Shot Image Classification
Computer Vision Lab, ETH Zurich, Zurich, Switzerland
TCLA: Training-Free Class-wise Logit Adaptation for Medical Vision-Language Models
Jul 10, 2026Tianyou Jiang, Ziyu ZhouFew-Shot Domain Adaptation Logit Adjustment
University of Bern, Bern, Switzerland · Shanghai Jiao Tong University, Shanghai, China

Aug 26, 2026J/K move · Enter open · S save

What Do Medical Vision-Language Models Learn in Radiology? Transfer, Alignment, and Source-Proxy Leakage Under Distribution Shift

Ayoub Louaye Bouaziz, Lokmane Chebouba, Yassine Himeur

LaTIM, Inserm, University of Western Brittany, Brest, France · University of Constantine, Algeria · University of Dubai, UAE

Medical VLMs Radiology VLMs VLM Robustness Domain Generalization Shortcut Learning Image-Text Retrieval

Open paper AskPDF arXiv

Medical vision-language models (VLMs) can appear reliable in-domain while failing when acquisition domain, paired supervision, or evaluation protocol changes. We study this failure mode as a representation-level blind spot relevant to epistemic intelligence, without claiming a formal estimator of epistemic uncertainty. Using NIH ChestXray14 and CheXpert, we first isolate source-only cross-dataset visual transfer from unsupervised domain-adaptation diagnostics. Using PadChest and OpenI, we then evaluate multimodal alignment under strict pair-index retrieval and quantify metadata-derived source-proxy information retained in frozen embeddings. Self-supervised visual initialization improves NIH-to-CheXpert transfer over supervised ImageNet initialization in matched ResNet-18 comparisons, whereas adversarial adaptation is useful only in a narrow regime and becomes unstable as adversarial pressure increases. Multimodal exact-pair retrieval remains low under external OpenI stress testing, and source-proxy information remains recoverable from learned representations. Qualitative nearest-neighbor and Grad-CAM analyses show clinically plausible cross-dataset structure and thoracic attention patterns in many cases, while device-heavy and false-positive cases remain ambiguous. Auxiliary architecture checks are task-dependent and do not support a universal backbone ranking. Overall, the study shows that apparent competence under a single protocol can conceal transfer, alignment, and shortcut-related failure modes, motivating stress-tested evaluation of medical VLMs under distribution shift.

Abstract

VLM Adaptation

Explore similar work

What Do Medical Vision-Language Models Learn in Radiology? Transfer, Alignment, and Source-Proxy Leakage Under Distribution Shift

Semi-Supervised Few-Shot Adaptation of Vision-Language Models

TCLA: Training-Free Class-wise Logit Adaptation for Medical Vision-Language Models