Expandable Modality Experts for Medical Vision-Language Learning

DM BAPPY; Eunsu Kim; Hanjoo Lee; Sung Hak Lee; Sangjeong Ahn

cs.CVOct 21, 2026

Expandable Modality Experts for Medical Vision-Language Learning

Authors: DM BAPPY, Eunsu Kim, Hanjoo Lee, Sung Hak Lee, Sangjeong Ahn

Organizations: Korea University · The Catholic University of Korea

Abstract

Learning across medical imaging modalities requires making use of images without paired text and accommodating new domains while retaining existing representations. We propose a medical vision-language technique built around an expandable bank of modality experts. Its Multiway Transformer combines shared visual computation with deterministically routed feed-forward experts and a separate text pathway. A modality-aware visual tokenizer supplies targets for masked pretraining on 19.4 million unpaired image rows across seven modalities; contrastive alignment then uses 3.23 million image-text pairs. Additional fundus and optical coherence tomography experts learn from paired data while the original image and text embedding paths remain frozen. With a ViT-Large backbone, the proposed technique achieves the best or tied-best result in 24 of 25 benchmark settings across nine modalities: 10 of 11 zero-shot classification benchmarks, all 10 linear probes, and all four retrieval benchmarks. Mean balanced accuracy is 0.674 for zero-shot classification and 0.856 for linear probing, and mean bidirectional retrieval recall is 20.70%. A separate three-seed sequential experiment with matched adaptation data yields fundus and OCT retrieval gains of 14.95 and 9.31 percentage points, with no measured loss on preceding fundus or pathology tasks. The preservation argument requires unchanged existing computation paths and inference conditions, and feature-identity checks verify retention on evaluated inputs. The proposed technique provides a modular approach to learning and expanding medical vision-language representations.

Figures & tables

Panel of Figure 1 — Figure 1: Proposed technique: pipeline, training data, and evaluation breadth. (a) A modality-aware tokenizer supplies discrete targets for masked pretraining; contrastive alignment then connects the separate image and text pathways. New modalities use paired data to train new routed experts while existing paths remain frozen. (b) Each spoke is a source dataset grouped by modality. On a shared $\log_{10}$ scale, wide bars show Stage-1 image rows and narrow bars show Stage-2 pairs; fundus and OCT use paired data only. The core pair count is approximately $17\%$ of the masked-pretraining image-row count. (c) Raw mean performance by modality across applicable task families, with retrieval percentages divided by $100$ . Each axis uses its own range; markers are data and curves are visual guides.

Benchmark	Modality	OURS	BMC	BMCA	ConceptCLIP	UniMed-CLIP
BreaKHis	pathology	.665	.532	.553	.637	.601
Pneumonia	X-ray	.722	.732	.911	.859	.786
CASIA-CXR	X-ray	.720	.630	.590	.680	.670
Thyroid	ultrasound	.744	.604	.579	.612	.599
EDD-polyp	endoscopy	.824	.752	.644	.714	.623
HAM10000	dermatology	.819	.300	.365	.389	.178

Table 1: Zero-shot classification , balanced accuracy. Best , second . The proposed technique wins ten of eleven rows outright. BMCA leads on Pneumonia. ‡ PanNuke is a

19

-class tissue panel on which every model scores well below saturation.

Benchmark	OURS	BMC	BMCA	ConceptCLIP	MerMED	UniMed-CLIP
HEST (pathology)	.994	.978	.986	.981	.984	.979
BrainTumor (MRI)	.972	.866	.894	.926	.917	.903
CT-Kidney (CT)	.993	.950	.967	.986	.993	.964
TN5000 (ultrasound)	.802	.761	.768	.783	.782	.762
RadImageNet (MRI)	.956	.940	.942	.953	.932	.949
EDD2020 (endoscopy)	.825	.767	.808	.820	.824	.763

Table 2: Linear probe on frozen features,

5

-fold cross-validation (seed

0

), balanced accuracy. Best , second . The proposed technique is best or tied-best on all ten rows and has the highest mean over all ten benchmarks. Ties are CT-Kidney, where the proposed technique and MerMED-FM both reach

0.993

, and ColorFundus10, where the proposed technique and ConceptCLIP both reach

0.712

. † MerMED-FM has no reported fundus/OCT probe; its mean is computed over the original seven benchmarks, where the proposed technique scores

0.906

on the same subset.

Benchmark	Modality	OURS	BMC	BMCA	ConceptCLIP	UniMed-CLIP
OpenI	X-ray	35.88	2.60	4.23	5.47	21.97
MR-RATE	MRI	11.52	4.72	5.68	5.12	5.12
derm7pt	dermatology	9.64	6.00	7.37	8.74	3.30
PathGen-600	pathology	25.75	17.64	19.39	25.61	17.97
Mean (n=4)		20.70	7.74	9.17	11.24	12.09

Table 3: Cross-modal retrieval , mean

R@\{1,10,50\}

over image

\to

text and text

\to

image, with one fixed gallery per benchmark for all models. All benchmarks use cosine similarity; PathGen-600 treats rows with identical captions as positives. Best , second . The proposed technique leads all four rows and the mean.

Model	Fundus gain	OCT gain	Fundus forgetting	PathGen forgetting
BMC	13.95	11.58	9.85	6.11
BMCA	15.47	10.53	7.59	3.22
ConceptCLIP	13.78	9.01	5.33	5.39
UniMed-CLIP	11.69	10.50	15.29	2.97
Proposed technique ViT-L	$14.95$	$9.31$	0.00	0.00

Table 4: Learning and forgetting under sequential fundus

\to

OCT adaptation. All methods use the same

20{,}000

fundus pairs followed by the same

16{,}000

OCT pairs, three training seeds, and the same validation-based checkpoint-selection rules. The baselines use LoRA, whereas the proposed technique trains new experts while keeping its existing pathways frozen. Values are mean percentage-point changes across the three seeds. Fundus and OCT gain measure improvement after training on each new modality. Fundus forgetting is the decrease after OCT training, and PathGen forgetting is the decrease from the initial model to the final model.

Figure 2: Learning new modalities while retaining existing capabilities. Models use

20{,}000

fundus pairs followed by

16{,}000

OCT pairs. Left: fundus learning and retention. Middle: OCT learning. Right: retention on Derm7pt. Points show mean bidirectional recall and error bars show standard deviation across three training seeds. Baselines use LoRA; the proposed technique trains new experts with existing pathways frozen. This experiment uses training subsets, not the full training corpus.

Model (adaptation)	$\Delta$ PathGen
Proposed technique ViT-L (routed expert)	0.0
BMCA $+$ LoRA	$-3.22$
BMC $+$ LoRA	$-6.11$
UniMed-CLIP $+$ LoRA	$-2.97$
ConceptCLIP $+$ LoRA	$-5.39$

Table 5: PathGen pathology retention after fundus

\to

OCT adaptation. Baselines use sequential LoRA and report the mean over three seeds; the proposed technique uses one assembled-expert sequence. Full trajectories are in Figure 3 .

Appendix figures & tables39 assets

Supplementary material from the paper’s appendix.

Appendix

Modality	Stage 1 rows	Stage 2 pairs	Sources: Stage 1 $\rightarrow$ Stage 2
Pathology	5,956,670 $\dagger$	1,139,291	TCGA patches ( Weinstein et al., 2013 ) $\rightarrow$ Quilt-1M ( Ikezogwo et al., 2023 ) ; PathCap ( Sun et al., 2024 )
Endoscopy	4,864,701	158,672	Endo-FM private ( Wang et al., 2023b ) ; Hyper-Kvasir ( Borgli et al., 2020 ) ; Kvasir-Capsule ( Smedsrud et al., 2021 ) ; CholecT45 ( Nwoye et al., 2022 ) ; LDPolypVideo ( Ma et al., 2021 ) ; Colonoscopic $\rightarrow$ Kvasir-VQA-x1 ( Gautam et al., 2025 )
MRI	3,877,303	463,588	MedTrinity MRI ( Xie et al., 2025 ) , including BraTS ( Menze and others, 2015 ) $\rightarrow$ MedTrinity MRI ( Xie et al., 2025 )
CT	3,336,658	347,842	MedTrinity CT ( Xie et al., 2025 ) $\rightarrow$ MedTrinity CT ( Xie et al., 2025 )
X-ray	621,616	212,498	MIMIC-CXR-JPG ( Johnson et al., 2024 ) ; CheXpert ( Irvin et al., 2019 ) ; NIH ChestX-ray14 ( Wang et al., 2017 ) $\rightarrow$ CheXpert ( Irvin et al., 2019 )
Ultrasound	193,308	364,362	MIMIC-IV-ECHO ( Gow and others, 2026 ) ; brachial plexus ( Tyagi and others, 2024 ) ; STMUS ( Marzola and others, 2021 ) ; EchoCP ( Wang et al., 2021 ) $\rightarrow$ US-365K ( Jin et al., 2026 )

Appendix

Table 6: Training data for the ViT-L core and modality extensions. Stage 1 uses image rows for masked pretraining, and Stage 2 uses image–text pairs for alignment. The pathology Stage 1 count is measured in extracted patches. A dash indicates that no additional Stage 1 training is used for an extension.

Model	Vision backbone	Input	Image path (M)	Total (M)
Proposed technique ViT-L (9-mod)	Multiway ViT-L/16 + DA-MoE	384	355.40	1,280.00
BMC	ViT-B/16	224	86.19	195.90
BMCA	ViT-L/14	224	303.97	454.16
ConceptCLIP	SigLIP SO400M/14	384	428.23	542.87
UniMed-CLIP	ViT-B/16	224	86.19	195.90
MerMED (inference backbone)	ViT-B/16	224	85.80	85.80

Appendix

Table 7: Architecture and parameter counts in millions (

10^{6}

). Image path denotes selected-path image parameters for the proposed technique and visual-module parameters for the baselines. Total denotes the full stored inference model. MerMED is vision-only.

Setting	Fundus	OCT
Training pairs	88,143	19,396
Initialization	Seven-modality base	Seven-modality base
Additional Stage 1	None	None
Learning rate	$10^{-4}$	$10^{-4}$
Global batch size	112	112
Per-GPU batch size	16	16

Appendix

Table 8: Full-data ViT-L fundus and OCT expert training. Configured epochs are the training budgets; selected epochs identify the checkpoints used in the final full-data assembly.

Setting	Value
Training pairs	Fundus: $20{,}000$ ; OCT: $16{,}000$
Seeds	$0,1,2$
Optimizer	AdamW; weight decay $0.01$
Batch size	$32$ image–text pairs
Candidate learning rates	$10^{-5}$ and $3\times 10^{-5}$
Training length	Minimum $8$ , maximum $20$ epochs per candidate

Appendix

Table 9: Settings for the proposed technique’s matched-data expansion experiment. The

20{,}000

fundus pairs are selected from the available paired fundus training dataset; the

16{,}000

OCT pairs are selected from the available paired OCT training dataset.

Setting	Value
Shared initialization	Identical randomly initialized shared weights; no pretrained backbone checkpoint
Expert initialization	Shared-FFN initialization with zero output projections; routed model only
Visual tokenizer	Same pretrained, frozen VQ-KD tokenizer for both models
Stage-1 images	$69{,}902$ across seven modalities; 98 validation-overlapping image paths excluded
Stage-1 text	$3{,}229{,}073$ unpaired texts, sampled separately from images
Stage-1 duration	Five image-data epochs; $1{,}560$ optimizer updates

Appendix

Table 10: Settings for the ViT-B comparison with and without DA-MoE (Section C ).

Benchmark	Modality	Proposed technique ViT-B	no-MoE ViT-B	$\Delta$
Zero-shot (n=8 benchmarks)
HAM10000	dermatology	.1401	.1276	$+.0125$
LC25000	pathology	.2000	.0652	$+.1348$
Thyroid	ultrasound	.5091	.4608	$+.0483$
BreaKHis	pathology	.5000	.5000	$.0000$
CASIA-CXR	X-ray	.3200	.2000	$+.1200$

Appendix

Table 11: Comparison with and without DA-MoE, the proposed technique (ViT-B) vs. no-MoE ViT-B. Zero-shot and linear probe are balanced accuracy; retrieval is mean

R@\{1,10,50\}

over both directions. All retrieval benchmarks use cosine similarity. PathGen-600 treats rows with identical captions as positives. Both variants share the same random initialization, training data, optimization settings, and training seed (Table 10 ); best per row.

Task	Benchmark	Multi-teacher	Single-teacher
Zero-shot	HAM10000	14.29	14.11
	LC25000	20.00	20.00
	Thyroid	53.85	50.51
	BreaKHis	50.00	50.00
	CASIA-CXR	24.00	22.00
	Pneumonia	50.00	50.00

Appendix

Table 12: Comparison of multi-teacher and single-teacher visual tokenization. Zero-shot and linear-probe results are balanced accuracy (%). Retrieval results are mean bidirectional Recall@

\{1,10,50\}

(%). Means are recomputed from the benchmark values shown in the table. The best result in each row is in bold.

Figure 3: PathGen retention through fundus–OCT adaptation. Mean bidirectional Recall@ $\{1,10,50\}$ at the base checkpoint, after fundus, and after OCT. Baseline error bars show standard deviation over three seeds. The proposed technique’s independently trained experts leave the pathology pathway unchanged. Training history, data exposure, and adaptation capacity differ between the proposed technique and the sequential-LoRA baselines.

Benchmark	Modality	Proposed technique	BMC	BMCA	ConceptCLIP	UniMed
OpenI	X-ray	0.00	$-1.71$	$-0.70$	$-1.85$	$-14.70$
MR-RATE	MRI	0.00	$-0.20$	$-0.42$	$-0.23$	$+0.57$
Derm7pt	dermatology	0.00	$-2.46$	$-1.92$	$-2.59$	$-0.58$
PathGen-600	pathology	0.00	$-6.11$	$-3.22$	$-5.39$	$-2.97$
ROCO-CT	CT	0.00	$-33.26$	$-13.42$	$-10.86$	$-22.58$
Liver-US	ultrasound	0.00	$-0.44$	$-0.44$	$-0.54$	$-0.96$

Appendix

Table 13: Full base-modality retention ,

\Delta

from original checkpoint to after fundus+OCT integration (negative = forgetting, positive = improvement). Baseline values are the mean sequential-LoRA run over three seeds; ViT-L values for the proposed technique come from one checkpoint sequence (these values do not estimate training-seed variance). All seven rows use bidirectional retrieval recall.

Figure 4: X-ray (OpenI) qualitative example. Top-three image-to-report results for the same query radiograph before expansion (original checkpoint) and after fundus and OCT integration. Baseline results use seed $0$ . Green marks the reference report. The rank of the benchmark-positive report is: the proposed technique $2\!\to\!2$ (unchanged; saved representations identical); BMC $3\!\to\!210$ ; BMCA $2\!\to\!7$ ; ConceptCLIP $14\!\to\!213$ ; UniMed-CLIP $1\!\to\!46$ .

Figure 5: MRI (MR-RATE) qualitative example. We use the same before-and-after format as Figure 4 and apply MR-RATE scoring. The rank of the benchmark-positive report is: the proposed technique $1\!\to\!1$ ; BMC $440\!\to\!489$ ; BMCA $382\!\to\!237$ ; ConceptCLIP $252\!\to\!318$ ; UniMed-CLIP $217\!\to\!222$ . For this query, no baseline ranks the correct report among the top three at either checkpoint, whereas the proposed technique ranks it first at both checkpoints.

Figure 6: Dermatology (Derm7pt) qualitative example. We use the same before-and-after format as Figure 4 and apply Derm7pt scoring. The rank of the benchmark-positive report is: the proposed technique $1\!\to\!1$ ; BMC $21\!\to\!46$ ; BMCA $10\!\to\!238$ ; ConceptCLIP $1\!\to\!35$ ; UniMed-CLIP $391\!\to\!42$ . For this query, ConceptCLIP is the only baseline that ranks the correct report among the top three before adaptation; no baseline does so afterward.

Figure 7: Pathology (PathGen-600) qualitative example. We use the same before-and-after format as Figure 4 and apply PathGen-600 scoring. The rank of the benchmark-positive report is: the proposed technique $1\!\to\!1$ ; BMC $2\!\to\!5$ ; BMCA $2\!\to\!24$ ; ConceptCLIP $1\!\to\!6$ ; UniMed-CLIP $1\!\to\!7$ . All four baselines place the correct report in the top three before adaptation; none do afterward.

Figure 8: Additional CT-Kidney qualitative example. This is a zero-shot classification example, not a retrieval example, and it is not included in Table 13 . We show the top three predictions among four classes (kidney cyst, stone, tumor, and normal kidney) for the same query before and after integration. Baseline results use seed $0$ . Green marks the ground-truth class (cyst). Its rank is: the proposed technique $1\!\to\!1$ ; BMC $1\!\to\!3$ ; BMCA $1\!\to\!3$ ; ConceptCLIP $1\!\to\!2$ ; UniMed-CLIP $1\!\to\!4$ . At the original checkpoint, every model ranks the ground-truth class first. After adaptation, the rank decreases for every baseline on this query. However, on the full CT-Kidney benchmark, BMCA’s mean balanced accuracy improves after adaptation. CT-Kidney is a separate zero-shot evaluation and is not the ROCO-CT retrieval benchmark in Table 13 . This difference shows why a single example should not be used to infer aggregate performance.

Figure 9: Ultrasound (Liver-US) qualitative example. We use the same before-and-after format as Figure 4 and apply Liver-US scoring. The rank of the benchmark-positive report is: the proposed technique $1\!\to\!1$ ; BMC $208\!\to\!587$ ; BMCA $466\!\to\!364$ ; ConceptCLIP $359\!\to\!430$ ; UniMed-CLIP $597\!\to\!670$ . No baseline is within the top three at either checkpoint on this query.

Figure 10: Endoscopy (EndoReport) qualitative example. We use the same before-and-after format as Figure 4 and apply EndoReport scoring. The rank of the benchmark-positive report is: the proposed technique $1\!\to\!1$ ; BMC $14\!\to\!19$ ; BMCA $73\!\to\!130$ ; ConceptCLIP $59\!\to\!40$ ; UniMed-CLIP $13\!\to\!32$ . No baseline is within the top three at either checkpoint on this query.

Figure 11: Qualitative saliency across nine modalities. Similarity-gradient Grad-CAM through each modality’s routed expert; top row original, bottom row saliency. Left to right: pneumonia, renal tumour, thyroid nodule, glioma on FLAIR with the ground-truth box, macular scar, colorectal polyp, macular edema on OCT, melanocytic lesion, and malignant gland on H&E. Fundus and OCT use experts added to a frozen model.

Panel of Figure 12 — Figure 12: APTOS 2019 zero-shot classification. Examples: (a) proliferative diabetic retinopathy, (b) moderate diabetic retinopathy, and (c) no diabetic retinopathy. In panel (a), the proposed technique highlights several locations within the retinal field, including the lower portion of the image. BMC and BMCA also respond in this area, while ConceptCLIP includes peripheral foci and UniMed-CLIP concentrates much of its response near the lower edge. The comparison shows how a common retinopathy prompt produces different spatial distributions. Heatmaps use ground-truth-class descriptions. Saliency construction follows Section J.1 .

Panel of Figure 13 — Figure 13: BreakHis zero-shot classification. Examples: (a) malignant breast tumor, (b) malignant breast tumor, and (c) benign breast tumor. In panel (a), the proposed technique produces multiple foci across the cellular tissue, whereas BMC and BMCA emphasize a smaller set of locations. ConceptCLIP shows few prominent foci in this example, and UniMed-CLIP includes a response near the lower-right boundary. The overlays describe spatial responses to the tissue-class prompt without delineating a tumor boundary. Heatmaps use ground-truth-class descriptions. Saliency construction follows Section J.1 .

Panel of Figure 14 — Figure 14: CASIA-CXR zero-shot classification. Examples: (a) pneumothorax, (b) cardiomegaly, and (c) pleural effusion. In panel (a), the proposed technique, BMC, and BMCA show prominent foci within the lung field on the left side of the displayed image. ConceptCLIP distributes its foci differently, and UniMed-CLIP includes responses higher in the thorax. The differing maps illustrate sensitivity to the same finding description; they do not delineate the extent of pneumothorax. Heatmaps use ground-truth-class descriptions. Saliency construction follows Section J.1 .

Panel of Figure 15 — Figure 15: Color Fundus zero-shot classification. Examples: (a) diabetic retinopathy, (b) myopia, and (c) healthy. In panel (a), the proposed technique produces several localized retinal responses, while BMC shows broader responses toward the upper and peripheral retinal field. BMCA includes a prominent focus near the optic disc, ConceptCLIP shows several scattered foci, and UniMed-CLIP emphasizes lower peripheral locations. The figure illustrates that a shared class description can elicit different responses within the same fundus image. Heatmaps use ground-truth-class descriptions. Saliency construction follows Section J.1 .

Panel of Figure 16 — Figure 16: EDD2020 zero-shot classification. Examples: (a) polyp-positive, (b) polyp-positive, and (c) polyp-negative. In panel (a), the proposed technique shows a compact response at the raised structure on the right side of the image. BMCA, ConceptCLIP, and UniMed-CLIP also include a response in this area, while BMC’s overlay extends across more of the surrounding mucosa. This example contrasts localized and distributed responses to the polyp-related prompt. Heatmaps use ground-truth-class descriptions. Saliency construction follows Section J.1 .

Panel of Figure 17 — Figure 17: HAM10000 zero-shot classification. Examples: (a) melanoma, (b) basal cell carcinoma, and (c) melanocytic nevus. In panel (a), the proposed technique shows a concentrated response over part of the pigmented lesion. BMC also highlights a lesion-associated area, while BMCA distributes responses over both the lesion and surrounding skin. ConceptCLIP and UniMed-CLIP show different, more limited foci. The overlays compare responses to the melanoma description without supplying a lesion segmentation. Heatmaps use ground-truth-class descriptions. Saliency construction follows Section J.1 .

Panel of Figure 18 — Figure 18: LC25000 zero-shot classification. Examples: (a) colon tissue, (b) lung tissue, and (c) colon tissue. In panel (a), the proposed technique’s responses are distributed among the visible tissue structures. BMC, BMCA, and UniMed-CLIP place prominent foci at different positions within the same field, while ConceptCLIP highlights fewer locations. The comparison concerns the spatial response to the colon-tissue description, which can depend on tissue context rather than a single lesion. Heatmaps use ground-truth-class descriptions. Saliency construction follows Section J.1 .

Panel of Figure 19 — Figure 19: PanNuke zero-shot classification. Examples: (a) thyroid, (b) breast, and (c) prostate. In panel (a), the proposed technique highlights several areas along the cellular band in the lower portion of the tissue image. BMC and BMCA also respond along this band, with different spatial extents, while ConceptCLIP and UniMed-CLIP emphasize other locations. These are responses to a tissue-class description, rather than nucleus-instance localization maps. Heatmaps use ground-truth-class descriptions. Saliency construction follows Section J.1 .

Panel of Figure 20 — Figure 20: Pneumonia CXR zero-shot classification. Examples: (a) pneumonia, (b) pneumonia, and (c) normal. In panel (a), the proposed technique includes a prominent focus in the lower portion of the thorax on the right side of the displayed image. BMC, BMCA, ConceptCLIP, and UniMed-CLIP distribute their foci differently across the lung fields and upper thorax. These differences concern the spatial response to the pneumonia prompt and do not measure localization accuracy. Heatmaps use ground-truth-class descriptions. Saliency construction follows Section J.1 .

Panel of Figure 21 — Figure 21: SARS-CoV-2 CT zero-shot classification. Examples: (a) COVID-positive, (b) COVID-positive, and (c) non-COVID. In panel (a), the proposed technique produces responses at several locations within the lung fields. BMC and BMCA include peripheral foci, ConceptCLIP highlights a different set of sparse locations, and UniMed-CLIP also responds near central thoracic structures. The overlays show prompt-conditioned responses on a CT slice without marking a validated infection boundary. Heatmaps use ground-truth-class descriptions. Saliency construction follows Section J.1 .

Panel of Figure 22 — Figure 22: Thyroid US zero-shot classification. Examples: (a) malignant thyroid nodule, (b) malignant thyroid nodule, and (c) benign thyroid nodule. In panel (a), the proposed technique, BMC, and BMCA emphasize different parts of the ultrasound tissue band. ConceptCLIP and UniMed-CLIP also produce localized foci, but at different positions. The comparison shows variation in responses to a thyroid-nodule description across the same ultrasound image; it does not establish which response delineates the nodule most accurately. Heatmaps use ground-truth-class descriptions. Saliency construction follows Section J.1 .

Panel of Figure 23 — Figure 23: Brain Tumor MRI linear-probe classification. Examples: (a) pituitary tumor, (b) glioma, and (c) no tumor. In panel (a), the proposed technique, BMC, BMCA, and UniMed-CLIP show a prominent response around the sellar region. BMC additionally highlights a lateral area, while ConceptCLIP shows a different spatial distribution. These maps explain similarity to the pituitary-related text prompt, independently of the image-only linear classifier. Heatmaps explain image–prompt similarity, not the linear-probe class score. Saliency construction follows Section J.1 .

Panel of Figure 24 — Figure 24: CT-Kidney linear-probe classification. Examples: (a) tumor, (b) cyst, and (c) normal. In panel (a), the proposed technique and UniMed-CLIP show a localized response in the central abdominal region on the left side of the displayed image. BMC and BMCA show responses with a larger spatial extent, while ConceptCLIP emphasizes different locations. The maps describe similarity to the tumor-related text prompt rather than contributions to the linear-probe score. Heatmaps explain image–prompt similarity, not the linear-probe class score. Saliency construction follows Section J.1 .

Panel of Figure 25 — Figure 25: HEST-320 linear-probe classification. Examples: (a) tissue class 1, (b) tissue class 4, and (c) tissue class 7. In panel (a), the proposed technique’s responses include several locations within and around the darker cellular tissue. BMC, BMCA, and UniMed-CLIP emphasize different portions of the field, while ConceptCLIP shows fewer prominent foci. Because the target is a tissue-class description, these responses concern tissue context and should not be interpreted as cell-level segmentation. Heatmaps explain image–prompt similarity, not the linear-probe class score. Saliency construction follows Section J.1 .

Panel of Figure 26 — Figure 26: IDRiD linear-probe classification. Examples: (a) moderate diabetic retinopathy, (b) proliferative diabetic retinopathy, and (c) no diabetic retinopathy. In panel (a), the proposed technique distributes responses across several retinal locations. BMC includes prominent foci around the central bright retinal markings, while BMCA, ConceptCLIP, and UniMed-CLIP produce different spatial patterns. The maps illustrate text-conditioned responses for a retinopathy example; the accompanying linear probe uses image features without a text prompt. Heatmaps explain image–prompt similarity, not the linear-probe class score. Saliency construction follows Section J.1 .

Panel of Figure 27 — Figure 27: OCT-DL linear-probe classification. Examples: (a) age-related macular degeneration, (b) diabetic macular edema, and (c) normal OCT. In panel (a), the proposed technique’s response extends along portions of the visible retinal profile and into the region beneath it. BMC, BMCA, ConceptCLIP, and UniMed-CLIP emphasize different locations within the same cross-section. This comparison shows how the disease-related text prompt interacts with retinal structure, without supplying retinal-layer or lesion segmentation. Heatmaps explain image–prompt similarity, not the linear-probe class score. Saliency construction follows Section J.1 .

Panel of Figure 28 — Figure 28: RadImageNet linear-probe classification. Examples: (a) spine, (b) abdomen, and (c) shoulder. In panel (a), the proposed technique includes responses around the central spinal region and the lower image periphery. BMC, BMCA, and UniMed-CLIP emphasize different central and lateral locations, while ConceptCLIP shows a more limited pattern. The target describes an anatomical site, so these maps concern anatomy-related similarity rather than disease localization or the linear-probe decision. Heatmaps explain image–prompt similarity, not the linear-probe class score. Saliency construction follows Section J.1 .

Panel of Figure 29 — Figure 29: TN5000 linear-probe classification. Examples: (a) malignant thyroid nodule, (b) malignant thyroid nodule, and (c) benign thyroid nodule. In panel (a), the proposed technique shows several responses within and around the central nodule appearance. BMC has a prominent central focus, while BMCA, ConceptCLIP, and UniMed-CLIP highlight different central or peripheral locations. The comparison explains similarity to the nodule-related prompt and remains separate from the image-only classifier’s score. Heatmaps explain image–prompt similarity, not the linear-probe class score. Saliency construction follows Section J.1 .

Panel of Figure 30 — Figure 30: Derm7pt image-to-text retrieval. Examples: (a) Clark nevus, (b) basal cell carcinoma, and (c) Clark nevus. In panel (a), the proposed technique, BMC, and UniMed-CLIP include responses within the pigmented lesion, with different spatial extents. BMCA produces several responses outside the lesion, while ConceptCLIP shows limited lesion-associated emphasis. These are responses to the paired text; their visual plausibility alone does not determine whether that text is ranked correctly in the retrieval gallery. Heatmaps use the paired caption or report. Saliency construction follows Section J.1 .

Panel of Figure 31 — Figure 31: MR-RATE image-to-text retrieval. Examples: (a) ischemic changes and atrophy, (b) bone/dural metastases, and (c) cranial MRI within normal limits. In panel (a), the proposed technique includes responses within the posterior portion of the displayed cranial image. BMC emphasizes more anterior and peripheral locations, while BMCA, ConceptCLIP, and UniMed-CLIP highlight different posterior or lateral areas. The target is the paired report, so the map should be read as a response to its full description rather than to one diagnostic term. Heatmaps use the paired caption or report. Saliency construction follows Section J.1 .

Panel of Figure 32 — Figure 32: OpenI image-to-text retrieval. Examples: (a) cardiomegaly and basilar opacity, (b) pulmonary nodules/mass, and (c) normal chest report. In panel (a), the proposed technique includes a prominent response near the central thorax. BMC emphasizes upper and central locations, BMCA includes responses lower in the thorax, ConceptCLIP shows a peripheral focus, and UniMed-CLIP distributes responses across several thoracic areas. The comparison illustrates different spatial responses to the same reference report without validating individual findings in that report. Heatmaps use the paired caption or report. Saliency construction follows Section J.1 .

Panel of Figure 33 — Figure 33: PathGen-600 image-to-text retrieval. Examples: (a) papilloma / HPV-associated changes, (b) granulomatous inflammation, and (c) normal brain tissue. In panel (a), the proposed technique distributes responses among several cellular regions in the histology field. BMC and ConceptCLIP emphasize different groups of locations, while BMCA and UniMed-CLIP include prominent peripheral foci. These maps explain similarity to the paired pathology text and should not be interpreted as instance-level annotation of the reported cellular changes. Heatmaps use the paired caption or report. Saliency construction follows Section J.1 .

Medical VLMs

Medical vision-language models connect clinical images with text for diagnosis, reporting, question answering, and image grounding.

See all 475 papers

NEARL: Interacted Query Adaptation with Orthogonal Regularization for Medical Vision-Language Understanding
Aug 6, 2025Zelin Peng, Yichen Zhao, Yu Huang +5Cross-Modal Learning VLM Adaptation
MoE Key Lab of Artificial Intelligence, AI Institute, School of Computer Science, Shanghai Jiao Tong University · Department of Radiology, The First Affiliated Hospital, School of Medicine, Zhejiang University · Mohamed bin Zayed University of Artificial Intelligence +1
Can Experts Adapt Without Training? On Test-Time Modality Generalization in MVLMs
Jul 18, 2026Raza Imam, Darakshan Rashid, Yutong Xie +3VLM Robustness VLM Adaptation
Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE · Indian Institute of Technology Delhi, India · Khalifa University, Abu Dhabi, UAE
Two-Stage Mixture-of-LoRA for Multi-Task Medical Vision-Language Learning
Sep 13, 2026Zhanghao Chen, Yuanyuan Li, Zhenyu Lu +3VLM Adaptation Medical VLMs
School of Computer Science and Engineering, Southeast University, Nanjing 211189, China · School of Biological Science & Medical Engineering, Southeast University, Nanjing 211189, China

Aug 6, 2025J/K move · Enter open · S save

NEARL: Interacted Query Adaptation with Orthogonal Regularization for Medical Vision-Language Understanding

Zelin Peng, Yichen Zhao, Yu Huang, Piao Yang+4

MoE Key Lab of Artificial Intelligence, AI Institute, School of Computer Science, Shanghai Jiao Tong University · Department of Radiology, The First Affiliated Hospital, School of Medicine, Zhejiang University · Mohamed bin Zayed University of Artificial Intelligence+1

Cross-Modal Learning VLM Adaptation Cross-Modal Alignment Medical VLMs Cross-Modal Attention Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Computer-aided medical image analysis is crucial for disease diagnosis and treatment planning. While vision-language models (VLMs) such as CLIP exhibit strong generalization ability, their direct application to medical imaging remains hindered by a substantial domain gap. Existing methods for bridging this gap, including prompt learning and unidirectional modality interaction, typically introduce domain knowledge into only one modality. However, such approaches fail to fully exploit CLIP's inherent dual-modality structure and overlook the synergistic effect of bidirectional cross-modal interaction, resulting in persistent modality misalignment. In this paper, we propose NEARL (iNteracted quEry Adaptation with oRthogonaL Regularization), a novel parameter-efficient VLM framework for bidirectional cross-modal interaction. NEARL consists of two key components: (1) the Unified Synergy Embedding Transformer (USEformer), which dynamically generates compact cross-modal queries to facilitate interaction; and (2) the Orthogonal Cross-Attention Adapter (OCA), which decouples new knowledge into truly novel and incremental components through orthogonal regularization. This design reduces interference from incremental components, enabling more focused learning of novel information and improving modality interaction in VLMs. Notably, NEARL introduces only 1.46M learnable parameters. Extensive experiments on three medical imaging modalities demonstrate state-of-the-art performance (e.g., a 2.3% relative improvement on the pneumonia dataset), along with fast inference and low memory overhead, highlighting its effectiveness for real-world medical vision-language understanding.

Abstract

Medical VLMs

Explore similar work

NEARL: Interacted Query Adaptation with Orthogonal Regularization for Medical Vision-Language Understanding

Can Experts Adapt Without Training? On Test-Time Modality Generalization in MVLMs

Two-Stage Mixture-of-LoRA for Multi-Task Medical Vision-Language Learning