K-MaT: Knowledge-Anchored Manifold Transport for Cross-Modal Prompt Learning in Medical Imaging

Jiajun Zeng; Shadi Albarqouni

cs.CVMar 6, 2026

K-MaT: Knowledge-Anchored Manifold Transport for Cross-Modal Prompt Learning in Medical Imaging

Authors: Jiajun Zeng, Shadi Albarqouni

Organizations: University of Bonn, University Hospital Bonn, Clinic for Diagnostic and Interventional Radiology, 53127 Bonn, Germany

Abstract

Large-scale biomedical vision-language models (VLMs) adapted on high-end imaging (e.g., CT) often fail to transfer to frontline low-end modalities (e.g., radiography), collapsing into modality-specific shortcuts. We propose K-MaT (Knowledge-Anchored Manifold Transport), a prompt-learning framework that transfers decision structures to low-end modalities without requiring low-end training images. K-MaT factorizes prompts, anchors them to clinical text descriptions, and aligns the low-end prompt manifold to the visually-grounded high-end space using Fused Gromov-Wasserstein optimal transport. We evaluate K-MaT on four cross-modal benchmarks, including dermoscopy, mammography to ultrasound, and CT to chest X-ray. K-MaT achieves state-of-the-art results, improving the average harmonic mean of accuracy to 44.1% (from BiomedCoOp's 42.0%) and macro-F1 to 36.2%. Notably, on the challenging breast imaging task, it mitigates the catastrophic forgetting seen in standard methods like CoOp (which drops to 27.0% accuracy on the low-end), preserving robust performance across modalities. Aligning prompt manifolds via optimal transport provides a highly effective route for the zero-shot cross-modal deployment of medical VLMs.

Figures & tables

Figure 1: Overall architecture of the proposed K-MaT framework. During training, high-end textual prompts are optimized using high-end imaging data ( $\mathcal{L}_{ce}$ ), while prompts from both modalities are anchored to LLM-generated clinical descriptions ( $\mathcal{L}_{anc}$ ). The low-end prompt manifold is strictly aligned to the visually grounded high-end space via $\mathcal{L}_{fgw}$ . During inference, the frozen visual encoder extracts features from unseen low-end images, and predictions are computed via visual-textual similarity with the learned low-end embeddings, bypassing low-end visual training entirely.

Task	Type	Dataset	Modality	Classes	Train	Val	Test
Skin I	High-end	MRA-MIDAS [ 2 ]	Dermoscopy	6	513	91	135
Skin I	Low-end	MRA-MIDAS [ 2 ]	Clinical	6	501	89	132
Skin II	High-end	Derm7pt [ 4 ]	Dermoscopy	5	413	203	395
Skin II	Low-end	Derm7pt [ 4 ]	Clinical	5	413	203	395
Chest	High-end	CC-CCII [ 14 ]	CT	3	10670	3538	1678
Chest	Low-end	ChestXray [ 9 ]	X-Ray	3	4595	923	914

Table 1: Dataset statistics for the five cross-modal tasks.

Dataset		BiomedCLIP	CoOp	KgCoOp	CoCoOp	BiomedCoOp	K-MaT
Skin I	Hi	34.6 (24.2)	55.5 ( 42.3 )	56.6 (41.6)	44.6 (32.0)	58.3 (39.3)	56.8 ( 43.1 )
	Lo	31.7 (21.0)	40.8 (27.1)	40.5 ( 28.5 )	29.8 (24.2)	42.2 ( 29.1 )	46.4 (27.4)
	H	37.9 (22.9)	47.0 (33.0)	47.2 ( 33.9 )	35.7 (27.5)	49.0 ( 33.4 )	50.9 (33.3)
Skin II	Hi	32.6 (26.9)	34.8 (32.1)	34.6 (32.7)	40.7 ( 36.7 )	35.6 (31.8)	34.8 ( 32.8 )
	Lo	17.4 (13.6)	28.5 ( 26.4 )	28.5 ( 27.3 )	18.4 (18.0)	27.0 (24.6)	31.6 (25.0)
	H	22.7 (18.1)	31.4 ( 29.0 )	31.3 ( 29.8 )	25.4 (24.2)	30.7 (27.7)	33.1 (28.3)

Table 2: Cross-modal generalization performance on 4 medical image pairs with ACC and macro-F1 shown together. Each entry is reported as average ACC (F1) over 3 seeds. Hi: High-End (source), Lo: Low-End (target), H: Harmonic Mean. Best ACC/F1 results are highlighted in bold / underline respectively.

CSC	MSC	$\mathcal{L}_{\text{anc}}$	$\mathcal{L}_{\text{fgw}}$	High-end		Low-end		Harmonic Mean		Rel. Impr. (%)
CSC	MSC	$\mathcal{L}_{\text{anc}}$	$\mathcal{L}_{\text{fgw}}$	ACC	F1	ACC	F1	ACC	F1	ACC	F1
				53.36	47.53	35.27	25.72	40.08	31.85	0.00	0.00
✓				54.02	48.49	27.55	22.71	35.26	30.03	-12.03	-5.71
✓	✓			54.69	49.17	34.73	18.68	37.95	25.83	-5.31	-18.90
✓	✓	✓		54.80	49.34	30.95	20.73	35.15	27.82	-12.30	-12.65
✓	✓		✓	54.69	49.17	36.99	26.58	41.94	32.65	4.64	2.51

Table 3: Complete ablation study results across all dataset pairs. Relative improvement (%) is computed on the Harmonic Mean ACC/F1 with respect to the first row.

Figure 2: (a). Visualization of textual embeddings w/ and w/o $\mathcal{L}_{fgw}$ on the Chest cross-modal generalization task. (b). The breakdown of Accuracy for each class.

Cross-Modal Learning

Cross-modal learning connects information across different data types, enabling models to align, transfer, and reason across modalities.

See all 144 papers

BioVLM: Routing Prompts, Not Parameters, for Cross-Modality Generalization in Biomedical VLMs
Apr 19, 2026Mainak Singha, Tanisha Gupta, Ankit Jha +3Cross-Modal Learning Prompt Learning
University of Trento, Italy · Carnegie Mellon University, USA · LNMIIT Jaipur, India +3
What Do Medical Vision-Language Models Learn in Radiology? Transfer, Alignment, and Source-Proxy Leakage Under Distribution Shift
Aug 26, 2026Ayoub Louaye Bouaziz, Lokmane Chebouba, Yassine HimeurMedical VLMs Radiology VLMs
LaTIM, Inserm, University of Western Brittany, Brest, France · University of Constantine, Algeria · University of Dubai, UAE
Expandable Modality Experts for Medical Vision-Language Learning
Oct 21, 2026DM BAPPY, Eunsu Kim, Hanjoo Lee +2Medical VLMs Continual Learning for VLMs
Korea University · The Catholic University of Korea

Apr 19, 2026J/K move · Enter open · S save

BioVLM: Routing Prompts, Not Parameters, for Cross-Modality Generalization in Biomedical VLMs

Mainak Singha, Tanisha Gupta, Ankit Jha, Muhammad Haris Khan+2

University of Trento, Italy · Carnegie Mellon University, USA · LNMIIT Jaipur, India+3

Cross-Modal Learning Prompt Learning VLM Adaptation Few-Shot Learning Cross-Domain Generalization Medical VLMs+1

Open paper AskPDF arXiv

Pretrained biomedical vision-language models (VLMs) such as BioMedCLIP perform well on average but often degrade on challenging modalities where inter-class margins are small and acquisition-specific variations are pronounced, especially under few-shot supervision and when modality priors differ from pretraining corpora substantially. We propose BioVLM, a prompt-learning framework that improves cross-domain generalization without extensive backbone fine-tuning. BioVLM learns a diverse prompt bank and introduces dynamic prompt selection: for each input, it selects the most discriminative prompts via a low-entropy criterion on the predictive distribution, effectively coupling sparse few-shot evidence with rich LLM semantic priors. To strengthen this coupling, we distill high-confidence LLM-derived attributes and enforce robust knowledge transfer through strong/weak augmentation consistency. At test time, BioVLM adapts by choosing modality-appropriate prompts, enabling transfer to unseen categories and domains, while keeping training lightweight and inference efficient. On 11 MedMNIST+ 2D datasets, BioVLM achieves new state of the art across three distinct generalization settings. Codes are available at https://github.com/mainaksingha01/BioVLM.

Abstract

Cross-Modal Learning

Explore similar work

BioVLM: Routing Prompts, Not Parameters, for Cross-Modality Generalization in Biomedical VLMs

What Do Medical Vision-Language Models Learn in Radiology? Transfer, Alignment, and Source-Proxy Leakage Under Distribution Shift

Expandable Modality Experts for Medical Vision-Language Learning