Meta-CLIP: Metadata-Conditioned Contrastive Learning for Zero-Shot Disease Inference

Haozhe Luo; Nirosh Sivanesan; Shelley Zixin Shu; Francesco Nitti; Mauricio Reyes

cs.AIOct 24, 2026

Meta-CLIP: Metadata-Conditioned Contrastive Learning for Zero-Shot Disease Inference

Authors: Haozhe Luo, Nirosh Sivanesan, Shelley Zixin Shu, Francesco Nitti, Mauricio Reyes

Organizations: University of Bern, Bern, Switzerland · ARTORG Center for Biomedical Engineering Research, University of Bern, Bern, Switzerland

Abstract

Medical vision--language pretraining learns transferable representations from paired images and radiology reports, yet differences in scanners and acquisition protocols can impair transfer. Existing approaches primarily enrich clinical semantics, leaving structured acquisition information in DICOM headers underused. We introduce Meta-CLIP, a metadata-conditioned contrastive framework that uses acquisition similarity to reweight cross-modal negatives during pretraining. A kernel over numerical and categorical DICOM fields defines a bounded pairwise margin. This margin increases the contribution of acquisition-similar negatives, emphasizing distinctions among studies that share imaging conditions. The kernel remains fixed during image--text optimization, and inference follows the standard dual-encoder architecture without requiring metadata. We evaluate Meta-CLIP on chest radiography and volumetric chest CT through zero-shot classification, appearance perturbations, grounding, and representation analyses. On CT-RATE, Meta-CLIP improves macro-AUROC over CT-CLIP by 1.41 points, with a 95% paired bootstrap confidence interval of 0.96--1.85 points. It also achieves the highest macro-AUROC on four of five CXR benchmarks. An exploratory BrgSA evaluation extends the analysis to another CT backbone. These findings support acquisition metadata as a practical training-time prior for medical vision--language learning. Code will be made publicly available upon publication.

Figures & tables

Fig. 1: Motivation and overview of Meta-CLIP. Left: (a) Acquisition protocols and scanner vendors influence image appearance. (b) Standard image–report pretraining leaves structured acquisition metadata unused. (c) Acquisition variation can shape the learned representation. Right: Meta-CLIP constructs a pairwise acquisition kernel from numerical and categorical DICOM fields and uses it to modulate negative logits in the contrastive objective. Metadata is used during pretraining and is unnecessary at inference.

Fig. 2: (a) Disease-wise AUC comparison on RAD-ChestCT and CT-RATE. (b) Volume-to-volume retrieval measured by label similarity at different top- $k$ levels. (c) Robustness under synthetic perturbations; the light bars show clean inputs and the dark bars show perturbed inputs.

Fig. 3: Embedding structure. UMAP visualizations and silhouette scores for (a) CT and (b) CXR. Meta-CLIP shows stronger disease-related clustering and, on CT, lower separation by dataset than CT-CLIP.

Dataset	Available fields
MIMIC-CXR-JPG	Mfr., KVP, ExpIndex, ExpTime, TubeCurrent, RelExp
CheXpert+	Mfr., View, KVP, Exp., ExpTime, TubeCurrent, RelExp, PixelSpace, PhotoInterp
VinDr	PhotoInterp
RSNA	View, PixelSpace, PhotoInterp
PadChest	Mfr., View, Exp., ExpTime, TubeCurrent, RelExp, PhotoInterp

TABLE I: Availability of representative DICOM fields across public CXR datasets.

Method	ChestXray14	CheXpert
KAD	76.49	89.23
Meta-CLIP (Ours)	80.65	91.31

TABLE II: Zero-shot CXR classification compared with KAD. Values are macro-AUROC (%).

Fig. 4: CT backbone portability. Zero-shot macro-AUROC of BrgSA and Meta-CLIP + BrgSA on CT-RATE, RAD-ChestCT, and INSPECT.

Metadata field	Coverage	Domain lift	Disease AUROC
Tube current	100.0%	0.131	0.536
Exposure	100.0%	0.126	0.536
Exposure time	100.0%	0.331	0.502
Source–detector distance	47.9%	0.351	0.524
CTDIvol	99.8%	0.183	0.542
Reconstruction diameter	100.0%	0.040	0.519

TABLE III: Characterization of selected CT metadata. Values report coverage, acquisition-domain lift, and disease AUROC.

Metadata set	CT-RATE	RAD-ChestCT
Alternative-5	73.21 [72.55,73.89]	65.45 [64.76,66.12]
Physics-5	73.30 [72.65,73.97]	65.33 [64.65,66.00]
Default + XYSpacing	73.76 [73.11,74.43]	64.92 [64.24,65.60]
Default compact-7	74.39 [73.75,75.04]	65.50 [64.82,66.17]

TABLE IV: Effect of metadata selection on CT zero-shot classification. Values are macro-AUROC (%) with 95% bootstrap confidence intervals.

Subgroup	$n$	CT-CLIP	Meta-CLIP	$\Delta$
All cases	3039	73.08	74.49	+1.41
Philips	1928	74.06	75.04	+0.97
Siemens	971	70.65	72.67	+2.02
PNMS	140	73.85	76.21	+2.37
Highest CTDIvol	759	71.01	72.51	+1.50
Highest tube current	756	72.78	74.68	+1.90

TABLE V: Acquisition-stratified CT-RATE validation results. Values are macro-AUROC (%) over evaluable findings.

Dataset	ChestXray-14			CheXpert			PadChest-seen			RSNA Pneumonia			SIIM-ACR
Method	AUC $\uparrow$	F1 $\uparrow$	ACC $\uparrow$	AUC $\uparrow$	F1 $\uparrow$	ACC $\uparrow$	AUC $\uparrow$	F1 $\uparrow$	ACC $\uparrow$	AUC $\uparrow$	F1 $\uparrow$	ACC $\uparrow$	AUC $\uparrow$	F1 $\uparrow$	ACC $\uparrow$
ConVIRT [ 2 ]	53.15	12.38	57.88	52.10	35.61	57.43	63.72	14.56	73.47	79.21	55.67	75.08	64.25	42.87	53.42
GLoRIA [ 3 ]	55.92	14.20	59.47	54.84	37.86	60.70	64.09	14.83	73.86	70.37	48.19	70.54	54.71	40.39	47.15
BioViL [ 4 ]	57.82	15.64	61.33	60.01	42.10	66.13	60.35	10.63	70.48	84.12	54.59	74.43	70.28	46.45	68.22
BioViL-T [ 4 ]	60.43	17.29	62.12	70.93	47.21	69.96	65.78	15.37	77.52	86.03	62.56	80.04	75.56	60.18	73.72
CheXzero [ 21 ]	66.99	19.95	65.38	87.90	61.90	81.17	73.24	19.53	83.49	83.13	61.49	78.34	84.60	65.97	77.34

TABLE VI: Zero-shot disease classification across five CXR datasets for findings represented during pretraining. Values are point estimates (%). Best results are bolded and second-best results are underlined.

Fig. 5: Robustness and grounding study on CXR. (a) Zero-shot AUC under metal artifacts, brightness shifts, and noise on ChestXray14, RSNA, and SIIM-ACR (light: clean; dark: perturbed). (b) ChestXray14-Det grounding: cross-attention maps for five findings (blue: ground-truth boxes) and metrics (Pointing Game, mIoU, and Dice).

Medical VLMs

Medical vision-language models connect clinical images with text for diagnosis, reporting, question answering, and image grounding.

See all 475 papers

Learning from Acquisition: Metadata-driven Multimodal Pre-training for Cardiac MRI
Jun 27, 2026Xueyi Fu, Liwei Hu, Zi Wang +1Medical Image Segmentation Multimodal Contrastive Learning
Department of Surgery & Cancer, Imperial College London, London, UK · Bioengineering Department and Imperial-X, Imperial College London, London, UK · National Heart and Lung Institute, Imperial College London, London, UK +2
PRIMA: Pre-Training with Risk-Integrated Image--Metadata Alignment for Medical Diagnosis with LLM-Based Feature Aggregation
Feb 26, 2026Yiqing Wang, Chunming He, Ziyun Yang +5Cross-Modal Alignment Multimodal Pretraining
Department of Biomedical Engineering, Duke University 101 Science Drive, Durham, NC 27705 USA · Department of Ophthalmology and Visual Sciences, University of Michigan 1000 Wall Street, Ann Arbor, MI, USA
ProtoCLIP: Prototype-Aligned Latent Refinement for Robust Zero-Shot Chest X-Ray Classification
Apr 20, 2026Florian Kittler, Sheethal Bhat, Andreas MaierVLM Robustness Chest X-Ray Classification
Friedrich-Alexander University Erlangen-Nuremberg, Erlangen 91054, Germany

Jun 27, 2026J/K move · Enter open · S save

Learning from Acquisition: Metadata-driven Multimodal Pre-training for Cardiac MRI

Xueyi Fu, Liwei Hu, Zi Wang, Guang Yang

Department of Surgery & Cancer, Imperial College London, London, UK · Bioengineering Department and Imperial-X, Imperial College London, London, UK · National Heart and Lung Institute, Imperial College London, London, UK+2

Medical Image Segmentation Multimodal Contrastive Learning Medical Image Representation Learning Cardiac MRI

Open paper AskPDF arXiv

Cardiac magnetic resonance imaging (CMR) routinely records structured acquisition metadata, yet most CMR foundation models rely primarily on image-only pre-training and leave this naturally available source of weak semantic supervision largely underexplored. We propose MetaCLIP-CMR, a metadata-driven framework based on Contrastive Language--Image Pre-training (CLIP), which converts imaging modality, anatomical view, scanner vendor, field strength, and scanner model into textual supervision for CMR representation learning. The pretrained image encoder is evaluated on imaging modality classification, cine view classification, and cardiac segmentation. MetaCLIP-CMR achieves 86.8% modality accuracy and 86.5% cine view accuracy, clearly outperforming ImageNet and masked reconstruction initialisations. For downstream cardiac segmentation, MetaCLIP-CMR consistently obtains the highest Dice score across the evaluated ACDC and M&Ms cine short-axis (SAX) settings under both full-data and 20% fine-tuning regimes. Compared with recent image-focused large-scale CMR pre-training models, MetaCLIP-CMR achieves comparable ACDC segmentation performance, while requiring less than 1% of their pre-training image scale. These results suggest that metadata learning offers a natural and easy-to-use strategy for transforming routinely recorded acquisition information into effective supervision for foundation-level CMR representation learning, highlighting the promise of metadata-driven multimodal pre-training.

Abstract

Medical VLMs

Explore similar work

Learning from Acquisition: Metadata-driven Multimodal Pre-training for Cardiac MRI

PRIMA: Pre-Training with Risk-Integrated Image--Metadata Alignment for Medical Diagnosis with LLM-Based Feature Aggregation

ProtoCLIP: Prototype-Aligned Latent Refinement for Robust Zero-Shot Chest X-Ray Classification