Multimodal Hierarchical Mixture-of-Experts for 3D Medical Vision-Language Modeling

Gizem Mert; Szymon Płotka; Pedro R. A. S. Bassi; Wenxuan Li; Zongwei Zhou; Jarosław B. Ćwikła; Arkadiusz Sitek; Ewa Szczurek

cs.CVNov 5, 2026

Multimodal Hierarchical Mixture-of-Experts for 3D Medical Vision-Language Modeling

Authors: Gizem Mert, Szymon Płotka, Pedro R. A. S. Bassi, Wenxuan Li, Zongwei Zhou, Jarosław B. Ćwikła, Arkadiusz Sitek, Ewa Szczurek

Organizations: Helmholtz Munich · Technical University of Munich · Munich Center for Machine Learning · Jagiellonian University ·

Abstract

Medical 3D scans and reports are inherently hierarchical, with diagnostic cues concentrated in specific anatomical regions across multiple levels of organization. Yet, most medical vision-language models ignore this structure and fail to focus on clinically relevant anatomy. We introduce MedHiVE, which combines hierarchical vision-language Soft Mixture-of-Experts (SMoE) with contrastive learning to jointly model fine-grained anatomy and global scan context. To our knowledge, MedHiVE is the first framework to route both vision and text tokens through a two-level SMoE hierarchy organized by anatomical scale, coupling this with report-derived anatomy-level contrastive alignment for 3D medical imaging. This replaces the direct region-text alignment of statically pooled features and vision-only hierarchical routing of prior work with dynamic, slot-based cross-modal information flow across anatomical scales. MedHiVE operates via a two-level, learnable token-routing mechanism, where local image and text features are first routed through a local SMoE and then fused by a global SMoE into unified multimodal representations. Contrastive learning aligns both anatomy-level and global vision-language representations in a unified space. Pre-trained on 25,000+ CT scans with anatomical supervision, MedHiVE consistently surpasses 11 state-of-the-art methods across 6 datasets on 3 tasks: segmentation, classification, and detection, demonstrating the importance of hierarchical modeling. The code for MedHiVE is at github.com/szczurek-lab/MedHiVE.

Figures & tables

Figure 1: Overview of the MedHiVE framework. A hierarchical encoder processes a 3D CT scan and anatomy masks alongside a full report representation $z_{i}^{T}$ and anatomy-specific snippet representations $z_{i,a}^{T}$ , extracted through a frozen text encoder. The global vision representation $z_{i}^{I}$ is contrasted with $z_{i}^{T}$ , while local vision representations $z_{i,a}^{I}$ are aligned with $z_{i,a}^{T}$ for each anatomical structure $a$ . This yields anatomy-aware vision-language representations that enhance downstream segmentation, classification, and detection.

Figure 2: Overview of the HiVEL layer. Global and anatomy-level vision tokens and text tokens are routed to $E$ global experts $\mathrm{FFN}^{G}$ and $F$ local experts $\mathrm{FFN}^{L}$ via first-level routers. The resulting slots are concatenated and processed by a second-level router. Global vision and text tokens are fused with $H$ shared experts $\mathrm{FFN}^{S}$ outputs via residual connection, yielding enhanced vision tokens $\hat{x}$ and text tokens $\hat{t}$ . Vision tokens are used at all stages $s\in\{1,2,3,4\}$ , while text tokens are enabled at $s\in\{3,4\}$ .

Method	Params	VRAM	Speed	Segmentation (mDSC %)				PDAC Cls. & Det.
	(M)	(GB)	(MV/s)	PANO	AMOS	KiTS	GEPAR3D	Sens. (%)	Spec. (%)	mAP (%)
Hermes [ 13 ]	44.5	4.1	12.9	77.6	85.3	65.7	84.0	75.4	79.1	62.3
Swin SMT [ 35 ]	170.8	3.6	14.7	73.4	81.6	47.2	84.1	77.2	80.5	64.1
VSmTrans [ 30 ]	47.6	3.1	26.2	74.7	84.8	54.6	84.6	78.8	81.3	65.8
uC 3DU-Net [ 21 ]	21.7	3.1	50.2	75.0	82.9	60.6	83.8	73.5	77.2	60.4
Swin UNETR [ 44 ]	72.8	4.1	27.5	72.6	83.8	48.3	82.6	80.1	82.8	67.5

Table 1: Comprehensive performance comparison across segmentation, classification, and detection tasks. Segmentation performance is reported in mDSC (%), while PDAC classification and detection are reported via Sensitivity (%), Specificity (%), and mAP (%). VRAM is measured in gigabytes (GB) (

\downarrow

), and speed in million voxels per second (MV/s) (

\uparrow

). The best results are bolded .

Figure 3: Qualitative segmentation results on four datasets, from top to bottom: PANORAMA, AMOS, KiTS and GEPAR3D. Comparison of MedHiVE with the top five baselines. Red boxes highlight significant artifacts or missing components in baseline predictions. MedHiVE shows superior consistency with the ground truth in these challenging regions.

#	Configuration	PANO	AMOS	KiTS	GEPAR3D	mDSC
1	w/o text & CL	74.1	85.5	60.1	86.1	76.5
2	w/o CL	75.6	86.7	63.2	88.6	78.5
3	w/o SMoE	72.8	84.8	56.8	86.7	75.3
4	w/o global CL	71.2	86.1	50.0	86.2	73.4
5	w/o 2nd level routing	58.2	67.1	38.7	54.1	54.5
6	w/o local routing	74.2	86.8	63.7	83.9	77.2

Table 2: Ablation studies of the key and individual architectural components. Segmentation performance (mDSC (%)) on PANORAMA (PANO), AMOS, KiTS, and GEPAR3D. CL denotes contrastive loss. The best results are bolded and second best results are underlined .

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: An overview of the 2D projections of anatomical segmentation masks from the Merlin dataset [ 6 ] . We consider 14 abdominal anatomical groups to facilitate analysis and model training (see Table 3 ). These projections illustrate the spatial distribution of the anatomies across representative slices.

Figure 5: Visualization of the 3D projections of anatomical segmentation masks from the Merlin dataset [ 6 ] , shown from the front, left, right, and back views. These projections highlight the global structural consistency and anatomical coherence of the segmented structures. Consistent with the 2D setting, we consider 14 abdominal anatomical groups for clarity and downstream processing (see Table 3 ). The multi-view renderings provide a qualitative assessment of segmentation quality across the full 3D volume.

No.	Anatomy	No.	Anatomy
1	Spleen	8	Adrenal gland right
2	Kidney right	9	Adrenal gland left
3	Kidney left	10	Small bowel
4	Gallbladder	11	Duodenum
5	Liver	12	Colon
6	Stomach	13	Urinary bladder

Appendix

Table 3: List of selected anatomies.

Panel of Figure 6 — Figure 6: Text preprocessing pipeline.

	mDSC (%) $\uparrow$							mHD95 $\downarrow$		Detection $\uparrow$
Method	NIH (Pancreas)	MSD (PDAC)	MSD (Pancreas)	In-house (PDAC)	In-house (Pancreas)	Overall (PDAC)	Overall (Pancreas)	Overall (PDAC)	Overall (Pancreas)	Sensitivity	Specificity
Hermes	93.6	34.8	89.4	64.1	82.5	52.8	89.1	24.5	5.1	89.0	87.9
Swin SMT	90.2	31.0	82.3	48.0	80.4	44.5	86.1	35.8	6.8	86.6	83.5
VSmTrans	91.0	39.5	85.4	60.2	77.7	47.2	85.4	29.4	7.8	88.0	87.2
SegMamba	92.1	41.1	87.9	47.2	80.2	36.2	87.6	41.1	6.3	84.8	82.4
uC 3DU-Net	92.5	41.8	88.5	53.9	80.7	46.8	88.2	30.0	5.5	87.8	86.7

Appendix

Table 4: Quantitative segmentation and detection performance on the PANORAMA test sets (NIH, MSD, and In-house). Evaluation metrics include mDSC (%) and mHD95 (mm), while PDAC detection efficacy is measured via Sensitivity and Specificity (%). The best results are highlighted in bold . MedHiVE demonstrates state-of-the-art performance, particularly in the challenging task of PDAC segmentation, where it significantly outperforms existing Transformer and Mamba-based architectures. Notably, our method achieves a substantial boost in mDSC for the In-house PDAC dataset (65.8%) and the Overall PDAC average (54.2%), representing a major improvement over the second-best performing models. Additionally, MedHiVE exhibits the lowest mHD95 and the highest sensitivity for patient-level detection, underscoring its clinical utility in identifying low-contrast ductal adenocarcinomas across diverse multi-center cohorts.

Method	Kidneys	Tumor	Cyst	mDSC $\uparrow$	mHD95 $\downarrow$
Hermes	88.0	53.0	47.9	65.7	49.5
Swin SMT	78.4	30.0	21.8	47.2	94.2
VSmTrans	78.6	39.0	39.4	54.6	75.3
uC 3DU-Net	80.0	45.8	52.4	60.6	59.6
Swin UNETR	78.1	31.5	24.6	48.3	92.1
SuPreM	76.2	47.4	39.7	57.0	64.2

Appendix

Table 5: Quantitative segmentation results on the KiTS dataset. Performance is evaluated across three distinct labels: Kidneys, Tumor, and Cyst. Results are reported in DSC (%) for individual classes, with mDSC (%) and mHD95 (mm) representing the overall average performance. MedHiVE achieves the highest performance in mDSC and mHD95, consistently outperforming both CNN-based and advanced sequence-modeling baselines. Specifically, the model shows a marked improvement in the segmentation of small, irregular structures compared to pure Mamba or Transformer architectures. The achievement of the lowest mHD95 further validates our method’s precision in boundary localization, demonstrating its robustness in handling the high anatomical variability and complex pathologies inherent in kidney tumor imaging. The best results are bolded .

	Organ Classes (%) $\uparrow$															Overall
Method	Spleen	R. Kidney	L. Kidney	Gallbladder	Esophagus	Liver	Stomach	Aorta	Inferior Vena Cava	Pancreas	R. Adrenal Gland	L. Adrenal Gland	Duodenum	Bladder	Prostate/Uterus	mDSC $\uparrow$	mHD95 $\downarrow$
Hermes	95.0	94.1	95.0	76.7	81.9	95.9	88.2	93.2	88.7	82.0	74.0	70.7	78.4	84.7	80.3	85.3	4.9
Swin SMT	94.3	93.7	93.1	71.1	77.4	94.4	82.7	92.6	86.4	77.4	71.0	69.5	72.1	74.0	74.3	81.6	26.1
VSmTrans	95.7	94.4	93.8	76.6	80.1	95.2	86.2	92.9	87.9	79.9	74.2	72.5	76.4	85.9	80.5	84.8	19.3
uC 3DU-Net	94.1	93.2	94.5	68.2	80.7	87.7	86.0	92.8	87.2	79.4	74.2	71.9	72.6	82.7	78.1	82.9	9.6
Swin UNETR	92.5	90.9	93.3	73.5	80.8	94.2	84.9	93.0	86.6	82.7	74.0	72.6	75.6	82.6	78.9	83.8	25.9

Appendix

Table 6: Quantitative multi-organ segmentation performance on the AMOS-CT dataset. The evaluation encompasses 15 distinct anatomical structures across a wide range of scales and morphologies. mDSC is reported in (%) and mHD95 in (mm), with the best results per category indicated in bold . MedHiVE achieves superior overall performance, demonstrating a high degree of robustness across both large, high-contrast organs (e.g., Liver, Spleen) and smaller, more complex structures (e.g., Adrenal Glands, Duodenum). Notably, our method establishes a new state-of-the-art for the Pancreas and Duodenum, which are traditionally challenging due to their low-contrast boundaries and proximity to the gastrointestinal tract. Furthermore, the significant reduction in mHD95 compared to both Transformer-based and pure Mamba-based baselines indicates a superior ability to preserve global topological consistency while maintaining fine-grained local details. MedHiVE exhibits competitive or superior Dice scores across the majority of classes, underscoring the generalizability of our proposed architecture.

	Upper Right (Quadrant 1)								Upper Left (Quadrant 2)								Lower Left (Quadrant 3)								Lower Right (Quadrant 4)								Overall
Method	18	17	16	15	14	13	12	11	21	22	23	24	25	26	27	28	38	37	36	35	34	33	32	31	41	42	43	44	45	46	47	48	mDSC $\uparrow$	mHD95 $\downarrow$
Hermes	77.8	86.6	93.0	86.0	83.6	80.5	91.8	88.9	85.0	89.3	86.1	80.5	89.9	91.2	90.5	85.9	63.5	86.8	90.3	90.1	86.8	85.8	83.0	81.4	78.2	86.6	92.4	87.5	93.8	85.4	88.7	53.9	85.2	2.1
Swin SMT	74.4	86.4	89.8	85.0	84.6	89.6	92.4	93.1	89.2	86.0	87.5	83.0	90.1	89.7	89.6	76.4	67.2	87.1	87.3	87.2	86.4	81.7	83.4	81.9	77.8	85.9	86.7	83.9	89.5	91.7	90.5	53.2	84.8	3.4
VSmTrans	71.4	86.3	89.1	86.8	85.4	90.8	91.9	93.0	92.9	90.0	85.8	79.9	90.7	90.6	92.0	79.1	63.3	85.8	87.8	88.6	86.1	85.5	83.1	79.4	72.4	87.7	91.7	84.5	90.4	89.1	90.0	53.8	85.0	5.0
uC 3DU-Net	78.8	88.7	89.4	81.7	81.5	85.1	91.8	91.9	88.8	85.3	85.1	83.7	87.7	90.5	90.2	79.6	61.9	83.4	86.7	86.0	86.2	85.9	86.6	82.1	78.5	89.4	89.6	83.2	87.6	90.1	89.5	47.5	84.3	3.3
Swin UNETR	73.7	81.7	88.9	86.1	81.6	86.3	92.1	86.2	77.1	87.2	84.2	83.5	87.8	90.4	90.9	75.2	63.8	87.1	86.7	84.9	85.4	80.6	76.2	73.5	72.2	84.8	88.5	87.1	89.6	86.3	87.2	48.2	82.5	3.7

Appendix

Table 7: Detailed quantitative evaluation of tooth segmentation on the GEPAR3D test set. Results are categorized by individual teeth following the FDI World Dental Federation notation across all four quadrants. mDSC is reported in (%) and mHD95 in (mm), with the best results per category indicated in bold . MedHiVE achieves the highest overall segmentation accuracy and the lowest mHD95 among all compared methods. The performance is notably consistent across different tooth types, effectively bridging the gap between large molars and smaller incisors. In particular, our model demonstrates superior precision in the upper and lower left quadrants, where it maintains high fidelity for complex occlusal surfaces and inter-proximal spaces. The significant reduction in the average Hausdorff distance suggests that MedHiVE is less prone to the isolated voxel noise and topological errors that often degrade the clinical utility of baseline segmentation outputs.

	Upper Right (Quadrant 1)								Upper Left (Quadrant 2)								Lower Left (Quadrant 3)								Lower Right (Quadrant 4)								Overall
Method	18	17	16	15	14	13	12	11	21	22	23	24	25	26	27	28	38	37	36	35	34	33	32	31	41	42	43	44	45	46	47	48	mDSC	mHD95
Hermes	39.9	80.1	90.6	88.6	88.2	88.2	85.9	89.2	92.3	92.2	91.3	89.9	90.3	93.0	78.4	42.8	65.0	78.8	87.0	78.5	70.3	78.6	87.6	89.1	89.3	87.4	88.7	91.3	86.9	81.8	69.7	59.5	81.7	22.1
Swin SMT	47.5	92.5	85.5	85.6	86.6	93.7	87.1	93.3	93.9	91.4	93.0	92.5	92.4	92.9	84.0	40.2	62.9	80.4	81.6	84.8	77.9	68.5	83.6	80.2	87.6	89.2	90.5	89.9	93.3	85.4	70.5	65.1	82.9	23.0
VSmTrans	45.7	91.9	85.6	88.8	85.3	93.5	86.9	94.3	93.5	90.5	93.2	92.7	92.8	92.6	85.4	37.7	68.2	82.6	74.5	88.4	90.7	84.1	86.2	81.3	88.7	89.1	92.2	93.2	93.8	81.4	72.1	65.8	83.9	21.0
uC 3DU-Net	47.8	90.9	84.7	90.0	85.8	91.4	82.5	93.2	94.1	92.5	92.6	91.5	90.8	91.8	84.4	43.5	70.1	71.1	75.5	86.0	85.4	79.9	89.1	84.1	89.9	89.6	87.9	87.0	87.4	81.6	68.9	64.5	82.8	29.9
Swin UNETR	42.4	91.8	85.8	88.4	92.1	91.8	86.2	90.8	93.6	92.0	90.7	92.9	91.8	89.8	72.7	41.3	65.6	75.2	76.3	89.4	90.3	82.8	86.0	81.3	87.1	86.8	89.8	91.4	93.3	80.0	71.9	64.2	82.8	26.2

Appendix

Table 8: Quantitative segmentation performance on the ToothFairy2 dataset subset. Evaluation follows the FDI World Dental Federation notation across the upper and lower dental arches. mDSC is reported in (%) and mHD95 in (mm), with the best results highlighted in bold . MedHiVE demonstrates a clear state-of-the-art advantage, particularly in maintaining structural continuity across the most challenging anatomical sites. Our method significantly outperforms all baselines in the posterior extremities, specifically the third molars (FDI 18, 28, 38, 48), where standard models typically struggle with severe under-segmentation or total loss of connectivity. Furthermore, MedHiVE attains the second-best mHD95 among all methods. These results validate the robustness of our approach in handling the high morphological variability and lower image quality characteristic of the ToothFairy2 benchmarks, ensuring precise tooth-by-tooth delineation across the entire oral cavity.

	Upper Right (Quadrant 1)								Upper Left (Quadrant 2)								Lower Left (Quadrant 3)								Lower Right (Quadrant 4)								Overall
Method	18	17	16	15	14	13	12	11	21	22	23	24	25	26	27	28	38	37	36	35	34	33	32	31	41	42	43	44	45	46	47	48	mDSC	mHD95
Hermes	64.2	84.4	92.1	87.0	85.1	83.2	89.7	89.0	87.7	90.3	87.9	83.8	90.1	91.8	86.3	70.5	64.1	84.0	89.3	86.1	81.1	83.3	84.6	84.0	82.1	86.9	91.1	88.8	91.3	84.2	82.1	56.0	84.0	9.0
Swin SMT	65.2	88.5	88.3	85.2	85.3	91.0	90.5	93.1	90.9	87.9	89.4	86.4	90.9	90.8	87.6	64.3	65.5	84.8	85.3	86.4	83.5	77.1	83.5	81.3	81.3	87.0	88.0	86.0	90.9	89.5	83.6	57.7	84.1	10.2
VSmTrans	62.8	88.2	87.9	87.5	85.4	91.8	90.1	93.4	93.1	90.2	88.4	84.3	91.4	91.3	89.7	64.4	65.1	84.7	83.2	88.5	87.7	85.0	84.2	80.1	78.1	88.2	91.9	87.5	91.6	86.5	83.7	58.4	84.6	10.5
uC 3DU-Net	67.7	89.5	87.7	84.6	83.0	87.3	88.5	92.3	90.7	87.9	87.7	86.5	88.8	91.0	88.2	66.4	64.9	79.0	82.8	86.0	85.9	83.8	87.5	82.8	82.6	89.5	89.0	84.5	87.5	87.1	82.4	53.6	83.8	12.5
Swin UNETR	62.5	85.3	87.8	86.9	85.2	88.2	90.0	87.9	83.0	88.9	86.5	86.8	89.2	90.2	84.6	63.6	64.4	82.9	83.3	86.5	87.1	81.4	79.6	76.2	77.4	85.5	88.9	88.6	90.8	84.1	81.9	54.0	82.6	11.5

Appendix

Table 9: Comparative quantitative analysis of tooth segmentation on the full dataset, including GEPAR3D [ 42 ] and ToothFairy2 [ 8 , 9 , 31 ] subsets. Evaluation is conducted for individual teeth using FDI World Dental Federation notation across all four quadrants. mDSC is reported in (%) and mHD95 in (mm), with the best results bolded . MedHiVE achieves the highest overall mDSC, particularly in the segmentation of molars and premolars. MedHiVE achieves the second-best mHD95, indicating precise boundary localization. Notably, the performance gap is most pronounced in the posterior regions, where complex root morphologies and proximity to distal anatomical landmarks typically challenge standard segmentation models. These results underscore the effectiveness of our architectural design in maintaining local precision while capturing the global context of the complete dental arch.

				mDSC (%) $\uparrow$							mHD95 $\downarrow$		Detection $\uparrow$
#	B	T	CL	NIH (Pancreas)	MSD (PDAC)	MSD (Pancreas)	In-house (PDAC)	In-house (Pancreas)	Overall (PDAC)	Overall (Pancreas)	Overall (PDAC)	Overall (Pancreas)	Sensitivity	Specificity
1	✓	✗	✗	91.3	41.3	88.3	56.8	80.5	49.2	86.7	31.3	7.1	84.6	85.3
2	✓	✓	✗	92.0	44.6	88.7	61.6	81.2	51.4	88.1	28.3	5.7	87.9	87.2
3	✓	✓	✓	93.1	45.5	89.0	65.8	81.4	54.2	88.9	22.1	4.8	89.5	88.6

Appendix

Table 10: Ablation study on the PANORAMA and in-house datasets demonstrating the impact of multi-modal supervision. We evaluate our proposed method across three diverse test sets: NIH (public), MSD (public), and a multi-institutional In-house dataset. Segmentation quality is measured through mDSC (%) and mHD95 (mm). Additionally, we report patient-level PDAC detection performance using Sensitivity/Specificity (%). The results indicate that while text guidance (T) significantly improves tumor localization, the addition of contrastive learning (CL) is crucial for refining boundary transitions, as evidenced by the reduction in mHD95. The best results are bolded .

B	T	CL	Kidneys	Tumor	Cyst	mDSC $\uparrow$	mHD95 $\downarrow$
✓	✗	✗	83.5	47.0	41.6	60.1	55.4
✓	✓	✗	86.1	48.0	49.1	63.2	51.0
✓	✓	✓	89.6	53.6	48.2	66.7	48.2

Appendix

Table 11: Ablation study on the KiTS dataset demonstrating the impact of multi-modal supervision. We evaluate the incremental contribution of text-based prompts (T) and contrastive learning (CL) over the baseline (B) architecture. Segmentation performance is reported via class-specific DSC (%) for Kidneys, Tumors, and Cysts, alongside aggregate mDSC (%) and mHD95 (mm) metrics. The results indicate that text guidance (T) primarily benefits cyst segmentation, while the addition of contrastive learning (CL) yields the largest gain for tumors and further reduces mHD95. The best results are bolded .

				Organ Classes (DSC %)															Overall
#	B	T	CL	Spleen	R. Kidney	L. Kidney	Gallbladder	Esophagus	Liver	Stomach	Aorta	IVC	Pancreas	R. Adr. Gland	L. Adr. Gland	Duodenum	Bladder	Pr/Ut	mDSC	mHD95
	✓	✗	✗	95.6	93.4	92.6	75.9	81.9	95.9	88.6	93.3	87.6	83.7	74.2	75.0	76.7	86.2	81.1	85.5	5.0
	✓	✓	✗	96.1	95.6	93.5	79.2	83.0	96.7	89.7	93.8	89.4	85.0	75.3	75.6	80.1	86.6	80.4	86.7	4.5
	✓	✓	✓	96.3	94.4	95.3	84.2	82.7	97.0	90.4	93.5	88.6	86.7	77.2	76.8	82.6	89.7	80.8	87.8	3.6

Appendix

Table 12: Ablation study on the AMOS-CT validation set across 15 organ classes. We evaluate the impact of the Text (T) supervision and Contrastive Loss (CL) components on segmentation accuracy (mDSC) and boundary precision (mHD95). The results demonstrate that incorporating textual representations (Config #2) consistently improves performance for anatomically complex organs, while the full MedHiVE framework (Config #3) achieves the highest overall accuracy and a significant reduction in Hausdorff Distance (mHD95 from 5.0 to 3.6 mm). This suggests that the joint contrastive objective effectively regularizes the latent space, particularly benefiting small-scale structures such as the adrenal glands and duodenum. B, T, and CL indicate baseline, text-conditioned representations, and contrastive loss, respectively. The best results are bolded .

				Upper Right (Quadrant 1)								Upper Left (Quadrant 2)								Lower Left (Quadrant 3)								Lower Right (Quadrant 4)								Overall
#	B	T	CL	18	17	16	15	14	13	12	11	21	22	23	24	25	26	27	28	38	37	36	35	34	33	32	31	41	42	43	44	45	46	47	48	mDSC $\uparrow$	mHD95 $\downarrow$
1	✓	✗	✗	78.1	88.4	88.3	84.5	77.2	90.5	92.4	90.9	86.9	89.5	86.9	84.0	90.4	90.7	91.6	89.2	70.0	90.1	87.2	89.9	87.3	84.2	81.6	77.9	74.6	85.6	91.9	85.6	89.7	84.5	91.1	55.8	85.4	1.7
2	✓	✓	✗	85.7	89.3	90.4	87.3	87.0	91.1	93.4	94.1	93.9	91.2	88.3	84.7	91.5	94.8	93.5	92.0	78.3	93.4	88.3	90.6	87.7	87.1	87.5	84.7	81.3	91.2	93.8	91.1	90.9	91.3	93.3	80.5	89.5	0.9
3	✓	✓	✓	82.3	89.2	89.4	86.5	86.7	94.4	93.4	94.1	93.7	89.0	88.8	83.5	91.3	95.0	93.8	89.0	83.3	90.7	90.6	89.8	87.5	85.1	85.5	79.8	74.0	84.5	88.8	87.5	93.2	92.1	91.4	56.0	87.6	0.5

Appendix

Table 13: Ablation study on the GEPAR3D test set, excluding ToothFairy2 subset across 32 dental classes (FDI notation). We quantify the influence of Textual Supervision (T) and Contrastive Loss (CL) on fine-grained segmentation. The inclusion of text-conditioned representations (Config #2) yields substantial gains in semantic recognition across the dental arch. Crucially, the full MedHiVE framework (Config #3) achieves the highest boundary fidelity, reaching sub-millimeter precision (mHD95: 0.5 mm). This indicates that the contrastive objective mitigates surface errors effectively in anatomically complex regions, such as the lingual surfaces of incisors (FDI 11, 21) and the occlusal pits of molars (FDI 36, 46), where baselines exhibit higher spatial variance. As this dataset covers dental anatomy absent from our pre-training inventory, the anatomy-level contrastive objective is expected to contribute less here. B, T, and CL denote baseline, text-conditioned representations, and contrastive loss, respectively. The best results are bolded .

				Upper Right (Quadrant 1)								Upper Left (Quadrant 2)								Lower Left (Quadrant 3)								Lower Right (Quadrant 4)								Overall
#	B	T	CL	18	17	16	15	14	13	12	11	21	22	23	24	25	26	27	28	38	37	36	35	34	33	32	31	41	42	43	44	45	46	47	48	mDSC $\uparrow$	mHD95 $\downarrow$
1	✓	✗	✗	69.7	90.6	91.2	86.5	84.2	93.9	91.1	93.9	91.4	92.0	89.9	86.8	90.9	94.4	90.1	69.6	66.8	84.0	83.5	88.9	87.4	85.8	85.9	81.7	84.4	89.4	92.3	89.8	93.5	85.4	84.2	60.0	86.1	6.5
2	✓	✓	✗	72.3	90.9	88.8	87.7	89.1	92.0	91.4	94.2	94.0	92.0	90.1	87.6	92.3	94.7	91.2	76.1	78.5	90.9	85.6	91.1	89.3	86.2	88.7	84.7	84.5	91.2	93.6	92.0	92.0	87.5	87.1	82.1	88.6	2.0
3	✓	✓	✓	79.5	90.7	88.0	87.5	88.9	94.0	91.0	94.3	94.0	90.4	90.4	86.7	91.8	92.7	90.3	80.0	80.2	89.7	88.7	90.3	89.0	85.5	86.9	83.3	79.6	86.5	90.2	89.5	93.3	88.1	86.1	64.9	87.7	3.4

Appendix

Table 14: Ablation study on the GEPAR3D dataset, including ToothFairy2 subset across 32 individual tooth classes (FDI notation). We evaluate the contribution of Text (T) conditioning and Contrastive Loss (CL) on fine-grained dental segmentation. While the baseline (Config #1) performs adequately on centralized teeth, the integration of textual representations (Config #2) provides a significant performance boost in boundary precision, reducing mHD95 from 6.5 to 2.0 mm. Notably, the full MedHiVE framework (Config #3) demonstrates superior robustness in segmenting distal molars (e.g., FDI 18, 28, 38), which are typically prone to severe imaging artifacts and morphological variation. This confirms that the contrastive objective effectively regularizes the feature space for anatomically rare or obscured structures. This indicates that the benefit of the anatomy-level contrastive objective can depend on overlap between the pre-training anatomy inventory and the target structures. B, T, and CL denote baseline, text-conditioned representations, and contrastive loss, respectively. The best results are bolded .

Figure 7: Qualitative segmentation results on the PANORAMA test sets. We compare the proposed MedHiVE against the ground truth and four state-of-the-art baselines. MedHiVE demonstrates superior spatial and semantic consistency, accurately capturing long-range dependencies in large-scale structures (e.g., the pancreas parenchyma) while maintaining high-fidelity boundaries for fine-grained objects like PDAC lesions and pancreatic ducts. Unlike competing methods that suffer from topological discontinuities or oversegmentation, our approach consistently preserves structural integrity across varying object scales.

Figure 8: Qualitative multi-organ segmentation performance on the AMOS-CT validation set. We compare the proposed MedHiVE against ground truth labels and four state-of-the-art baselines across 15 abdominal organs. MedHiVE demonstrates superior spatial and semantic consistency, leveraging long-range dependency modeling to maintain the global anatomical context of large-scale organs (e.g., liver, stomach) while ensuring high-fidelity boundary localization for small, low-contrast structures (e.g., adrenal glands, gallbladder). While competing methods frequently suffer from inter-class leakage and topological discontinuities in anatomically crowded regions, our approach preserves structural integrity and inter-organ spatial relationships. The highlighted regions (red boxes) illustrate MedHiVE’s robustness against oversegmentation at organ interfaces, where baseline models often fail to resolve clear semantic boundaries due to similar intensity profiles in non-contrast CT.

Figure 9: Qualitative multi-class segmentation results on the KiTS 2023 internal test set. We evaluate the proposed MedHiVE against ground truth annotations and four state-of-the-art baselines for the joint segmentation of the kidney parenchyma, cysts, and tumor masses. MedHiVE demonstrates superior spatial and semantic consistency by effectively modeling the hierarchical dependencies between nested structures; it captures the global morphology of the kidney while maintaining high-fidelity boundaries for small, low-contrast cysts and heterogeneous tumor regions. Unlike competing methods, which frequently exhibit topological fragmentation or semantic leakage (e.g., misclassifying tumor voxels as healthy tissue), our approach preserves structural integrity across varying object scales. The zoomed-in insets highlight MedHiVE’s ability to resolve ambiguous boundaries in complex cases of multi-focal cystic disease, where baseline models suffer from oversegmentation and loss of connectivity.

Figure 10: Qualitative evaluation of tooth segmentation on the GEPAR3D test set. We benchmark MedHiVE against ground truth labels and four state-of-the-art baselines. MedHiVE exhibits superior spatial and semantic consistency, leveraging its ability to capture long-range dependencies for large-scale dental arches while preserving high-fidelity boundaries in smaller, individual tooth structures. While competing methods frequently suffer from topological discontinuities (e.g., fragmented roots) or oversegmentation in high-density regions, our approach maintains global structural integrity and local connectivity. The zoomed-in regions (red boxes) highlight MedHiVE’s robustness in resolving ambiguous inter-proximal boundaries where baseline models typically fail to maintain clear separation.

Figure 11: Qualitative comparison of visual grounding performance on external in-house abdominal CT scans with kidney lesions. Each row displays a representative abdominal CT slice paired with its corresponding part of radiology report describing kidney tumors or cysts; in the absence of voxel-wise annotations, the report serves as the reference. Grounding predictions are shown for MedHiVE (left), Merlin (center), and fVLM (right). All models are fine-tuned on KiTS 2023. MedHiVE segments lesions consistent with the radiology reports, whereas Merlin and fVLM overshoot the lesion extent or place predictions in a kidney the report leaves unremarkable (red boxes).

Medical VLMs

Medical vision-language models connect clinical images with text for diagnosis, reporting, question answering, and image grounding.

See all 490 papers

Feb 27, 2026

3D Modality-Aware Pre-training for Vision-Language Model in MRI Multi-organ Abnormality Detection

Vision-language models (VLMs) show strong potential for complex diagnostic tasks in medical imaging. However, applying VLMs to multi-organ medical imaging introduces two principal challenges: (1) modality-specific vision-language alignment and (2) cross-modal feature fusion. In this work, we propose MedMAP, a Medical Modality-Aware Pretraining framework that enhances vision-language representation learning in 3D MRI. MedMAP comprises a modality-aware vision-language alignment stage and a fine-tuning stage for multi-organ abnormality detection. During the pre-training stage, the modality-aware encoders implicitly capture the joint modality distribution and improve alignment between visual and textual representations. We then fine-tune the pre-trained vision encoders (while keeping the text encoder frozen) for downstream tasks. To this end, we curated MedMoM-MRI3D, comprising 7,392 3D MRI volume-report pairs spanning twelve MRI modalities and nine abnormalities tailored for various 3D medical analysis tasks. Extensive experiments on MedMoM-MRI3D demonstrate that MedMAP significantly outperforms existing VLMs in 3D MRI-based multi-organ abnormality detection. Our code is available at https://github.com/RomantiDr/MedMAP.

Haowen Zhu, Ning Yin, Xiaogen Zhou

School of Computer Science and Engineering, Southeast University, China · Department of Medical Imaging, Suzhou Traditional Chinese Medicine Hospital, China · School of Electronic, Electrical Engineering and Physics, Fujian University of Technology

Medical VLMs Vision-Language Pretraining Medical Image Anomaly Detection Modality Gap in VLMs 3D Medical Imaging

Ask PDF

Jan 14, 2026

MedVL-SAM2: A unified 3D medical vision-language model for multimodal reasoning and prompt-driven segmentation

Recent progress in medical vision-language models (VLMs) has achieved strong performance on image-level text-centric tasks such as report generation and visual question answering (VQA). However, achieving fine-grained visual grounding and volumetric spatial reasoning in 3D medical VLMs remains challenging, particularly when aiming to unify these capabilities within a single, generalizable framework. To address this challenge, we proposed MedVL-SAM2, a unified 3D medical multimodal model that concurrently supports report generation, VQA, and multi-paradigm segmentation, including semantic, referring, and interactive segmentation. MedVL-SAM2 integrates image-level reasoning and pixel-level perception through a cohesive architecture tailored for 3D medical imaging, and incorporates a SAM2-based volumetric segmentation module to enable precise multi-granular spatial reasoning. The model is trained in a multi-stage pipeline: it is first pre-trained on a large-scale corpus of 3D CT image-text pairs to align volumetric visual features with radiology-language embeddings. It is then jointly optimized with both language-understanding and segmentation objectives using a comprehensive 3D CT segmentation dataset. This joint training enables flexible interaction via language, point, or box prompts, thereby unifying high-level visual reasoning with spatially precise localization. Our unified architecture delivers state-of-the-art performance across report generation, VQA, and multiple 3D segmentation tasks. Extensive analyses further show that the model provides reliable 3D visual grounding, controllable interactive segmentation, and robust cross-modal reasoning, demonstrating that high-level semantic reasoning and precise 3D localization can be jointly achieved within a unified 3D medical VLM.

Yang Xing, Jiong Wu, Savas Ozdemir +4

Department of Biomedical Engineering, University of Florida, Gainesville, FL, USA · Department of Radiology, University of Florida, Jacksonville, FL, USA · Research Computing, University of Florida, Gainesville, FL, USA +2

Interactive Medical Image Segmentation Unified Multimodal Models Medical VLMs Medical Image Segmentation 3D Medical Image Segmentation 3D Visual Grounding

Ask PDF

Jun 24, 2026

Disease-Centric Vision-Language Pretraining with Hybrid Visual Encoding for 3D Computed Tomography

Vision-language pre-training (VLP) holds great promise for general-purpose medical AI by leveraging radiology reports as rich textual supervision, yet existing methods struggle with 3D CT imaging due to inefficient visual backbones and coarse semantic alignment. To address these issues, we propose a tailored VLP framework featuring three key components: (1) a CNN-ViT hybrid encoder that replaces ViT's patch embedding with a 3D CNN backbone to efficiently capture local anatomical details while preserving global attention and compatibility with pre-trained cross-modal priors; (2) a disease-level contrastive learning mechanism using learnable query tokens to dynamically extract disease-specific semantics from full reports and align them with corresponding visual features, thereby disentangling distinct diseases within the same anatomical region; and (3) a diagnosis-aware prompt strategy that employs real clinical phrases and aggregated disease prototypes to bridge the pre-training-inference gap and enhance zero-shot diagnostic reliability. Our model achieves state-of-the-art performance on CT-RATE (84.4% AUC, +5.1%) and Rad-ChestCT (75.4% AUC, +5.4%), with even larger gains (+9.8% AUC) on a challenging 60-disease benchmark, and demonstrates strong transferability to radiology report generation, underscoring the generality and clinical utility of our approach.

Bowen Shi, Weiwei Cao, Ruifeng Yuan +5

DAMO Academy, Alibaba Group · Hupan Lab, 310023, Hangzhou, China · Shanghai Jiao Tong University, China +2

Contrastive Learning 3D Medical Imaging Medical Image Classification Medical VLMs Zero-Shot Medical Image Segmentation 3D VLMs

Ask PDF

Abstract

Medical VLMs

Explore similar work

3D Modality-Aware Pre-training for Vision-Language Model in MRI Multi-organ Abnormality Detection

MedVL-SAM2: A unified 3D medical vision-language model for multimodal reasoning and prompt-driven segmentation

Disease-Centric Vision-Language Pretraining with Hybrid Visual Encoding for 3D Computed Tomography