What Do Medical Vision-Language Models Learn in Radiology? Transfer, Alignment, and Source-Proxy Leakage Under Distribution Shift

Ayoub Louaye Bouaziz; Lokmane Chebouba; Yassine Himeur

cs.CVAug 26, 2026

What Do Medical Vision-Language Models Learn in Radiology? Transfer, Alignment, and Source-Proxy Leakage Under Distribution Shift

Authors: Ayoub Louaye Bouaziz, Lokmane Chebouba, Yassine Himeur

Organizations: LaTIM, Inserm, University of Western Brittany, Brest, France · University of Constantine, Algeria · University of Dubai, UAE

Abstract

Medical vision-language models (VLMs) can appear reliable in-domain while failing when acquisition domain, paired supervision, or evaluation protocol changes. We study this failure mode as a representation-level blind spot relevant to epistemic intelligence, without claiming a formal estimator of epistemic uncertainty. Using NIH ChestXray14 and CheXpert, we first isolate source-only cross-dataset visual transfer from unsupervised domain-adaptation diagnostics. Using PadChest and OpenI, we then evaluate multimodal alignment under strict pair-index retrieval and quantify metadata-derived source-proxy information retained in frozen embeddings. Self-supervised visual initialization improves NIH-to-CheXpert transfer over supervised ImageNet initialization in matched ResNet-18 comparisons, whereas adversarial adaptation is useful only in a narrow regime and becomes unstable as adversarial pressure increases. Multimodal exact-pair retrieval remains low under external OpenI stress testing, and source-proxy information remains recoverable from learned representations. Qualitative nearest-neighbor and Grad-CAM analyses show clinically plausible cross-dataset structure and thoracic attention patterns in many cases, while device-heavy and false-positive cases remain ambiguous. Auxiliary architecture checks are task-dependent and do not support a universal backbone ranking. Overall, the study shows that apparent competence under a single protocol can conceal transfer, alignment, and shortcut-related failure modes, motivating stress-tested evaluation of medical VLMs under distribution shift.

Figures & tables

Panel of Figure 1 — Figure 1: Overview of the corrected leakage-aware evaluation framework. The pipeline separates three questions that must not be conflated: source-only visual transfer, multimodal alignment under an external dataset shift, and recoverability of metadata-derived source proxies. OpenI is considered held out only for models whose pretraining did not use OpenI. The OpenI-pretrained CLIP initialization is therefore excluded from held-out OpenI quantitative claims.

Initialization	Linear probe	Partial fine-tune
ImageNet	0.82	0.85
BYOL	0.87	0.88

Table 1: NIH ChestXray14

\rightarrow

CheXpert transfer AUC with matched ResNet-18 encoders. The archived manuscript contains point estimates but not the seed-level records required to reconstruct mean

\pm

std; no significance claim is made.

Figure 2: DANN sensitivity on CheXpert. Target AUC across training epochs under a scheduled gradient-reversal weight. The trajectory is used as a stability diagnostic, not as evidence for a universal benefit of domain adaptation.

Dataset	Initialization	R@1	R@5	R@10	External held-out?
OpenI, $N=6800$ entries	Chance	0.000147	0.000735	0.001471	–
OpenI, $N=6800$ entries	ImageNet + Text	0.0006	0.0010	0.0020	Yes
OpenI, $N=6800$ entries	BYOL + Text	0.0002	0.0010	0.0022	Yes
PadChest, $N=15000$ entries	Chance	0.000067	0.000333	0.000667	–
PadChest, $N=15000$ entries	ImageNet + Text	0.0015	0.0019	0.0024	In-domain
PadChest, $N=15000$ entries	BYOL + Text	0.0007	0.0015	0.0026	In-domain

Table 2: Strict pair-index image-to-text retrieval. OpenI and PadChest pool sizes refer to candidate entries in the processed evaluation index. These numbers should not be interpreted as semantic retrieval accuracy when multiple indices can share equivalent report text. OpenI CLIP-init is intentionally omitted from the held-out OpenI comparison because that initialization was pretrained on OpenI.

Figure 3: Cross-dataset nearest neighbors in frozen image-representation space. For each NIH query, top CheXpert neighbors are shown for an SSL ResNet-50, ViT-S, Swin-T, and a CLIP-style ResNet-50 image encoder. Clean cases often preserve plausible anatomy/view similarity, while harder cases expose ambiguity and view sensitivity.

Figure 4: Grad-CAM for consolidation classification on representative CheXpert cases. The maps frequently concentrate attention within thoracic regions, while device-heavy and false-positive cases are more ambiguous. These visualizations are qualitative attention diagnostics, not a localization benchmark.

Panel of Figure 5 — Figure 5: Utility–source-proxy leakage trade-off. InstanceNorm gives the largest reduction in proxy accuracy and also the largest utility loss. Adversarial unlearning is intermediate, while CORAL produces a milder change.

Backbone	AUC	Note
ResNet-18	0.9257	Stable
ResNet-50	0.9163	Stable
DenseNet-121	0.9200	Stable
EfficientNet-B0	0.9203	Stable
ResNet-50 CLIP	0.9155	Stable
ViT-S	0.8774	Lower on this task

Table 3: Auxiliary consolidation-classifier AUC. These results are a sensitivity check and are not directly comparable with the primary NIH-to-CheXpert transfer table.

Medical VLMs

Medical vision-language models connect clinical images with text for diagnosis, reporting, question answering, and image grounding.

See all 458 papers

HarMoE: Multi-Source Chest Radiograph Pretraining with Dataset-Disentangled Experts
Aug 3, 2026Haozhe Luo, Ziyu Zhou, Shelley Zixin Shu +1Multi-Source Domain Adaptation Chest X-Ray Classification
ARTORG Center for Biomedical Engineering Research, University of Bern · Kaiko.AI, Zurich, Switzerland · Shanghai Jiao Tong University, Shanghai, China
MC-CXR: A Multi-Context Chest X-ray Benchmark for Context-Induced Disruption in Vision-Language Models
Aug 25, 2026Junhyeok Lee, Songsoo Kim, Kyu Sung ChoiVLM Evaluation Radiology VLMs
Interdisciplinary Program in Cancer Biology, Seoul National University College of Medicine · Department of Radiology, Seoul National University Hospital · Department of Radiology, Seoul National University College of Medicine +1
When Background Matters: Breaking Medical Vision Language Models by Transferable Attack
Apr 19, 2026Akash Ghosh, Subhadip Baidya, Sriparna Saha +1VLM Robustness Adversarial Attacks on VLMs
Indian Institute of Technology Patna1 · Work done as Visiting Researcher at MBZUAI · MBZUAI3 +1

Aug 3, 2026J/K move · Enter open · S save

HarMoE: Multi-Source Chest Radiograph Pretraining with Dataset-Disentangled Experts

Haozhe Luo, Ziyu Zhou, Shelley Zixin Shu, Mauricio Reyes

ARTORG Center for Biomedical Engineering Research, University of Bern · Kaiko.AI, Zurich, Switzerland · Shanghai Jiao Tong University, Shanghai, China

Multi-Source Domain Adaptation Chest X-Ray Classification Radiology VLMs Mixture of Experts Multimodal Pretraining

Open paper AskPDF arXiv

Recent vision-language models for chest X-ray understanding are largely built on image-report alignment and therefore rely heavily on MIMIC-CXR as the dominant pretraining source. While effective at scale, this paradigm underexplores an important alternative source of supervision: a range of existing multi-label classification datasets, which provide cleaner and more explicit disease signals than free-text reports, and can offer broader pathology coverage when combined across sources. However, learning from such heterogeneous datasets is nontrivial, as differences in label ontologies, annotation protocols, acquisition pipelines, and report styles can cause models to entangle clinical semantics with dataset identity, leading to poor transfer despite increased scale. In this work, we revisit radiology VLM construction from the perspective of harmonized multi-source learning. We propose HarMoE, a dataset-aware mixture-of-experts framework that learns shared cross-dataset medical semantics while confining source-specific variation to lightweight residual experts in deeper decoder layers. To further exploit clean supervision from labeled datasets, we train in a unified disease vocabulary with masked multi-dataset supervision, enabling the model to leverage complementary annotations without introducing false negatives. Experiments on large-scale chest X-ray benchmarks show that HarMoE consistently improves zero-shot classification, out-of-distribution transfer, and grounding over strong baselines. Our results suggest that building robust radiology VLMs requires moving beyond single-source image-report alignment toward structured knowledge construction from heterogeneous datasets with cleaner supervision and broader coverage. Code and the 873k harmonized dataset will be released at https://github.com/Roypic/harmoe.

Abstract

Medical VLMs

Explore similar work

HarMoE: Multi-Source Chest Radiograph Pretraining with Dataset-Disentangled Experts

MC-CXR: A Multi-Context Chest X-ray Benchmark for Context-Induced Disruption in Vision-Language Models

When Background Matters: Breaking Medical Vision Language Models by Transferable Attack