MedPlex: Deep Vision-Language Co-Adaptation for Clinically Grounded Medical Segmentation

Rafi Ibn Sultan; Hui Zhu; Chengyin Li; Dongxiao Zhu

cs.CVAug 13, 2026

MedPlex: Deep Vision-Language Co-Adaptation for Clinically Grounded Medical Segmentation

Authors: Rafi Ibn Sultan, Hui Zhu, Chengyin Li, Dongxiao Zhu

Organizations: Department of Computer Science Wayne State University · Department of Radiation Oncology Henry Ford Health · Institute for AI and Data Science Wayne State University

Abstract

Medical image segmentation is still largely treated as a vision-only problem, although clinical interpretation often relies on textual knowledge of anatomy, location, appearance, and surrounding context. Existing text-guided segmentation methods within the Vision-Language Model (VLM) paradigm often use language only as a late conditioning signal, limiting its influence on visual representation learning. We introduce MedPlex (Medical Plexus of Vision and Language), an end-to-end VLM framework that makes text guidance a continuous, clinically grounded component of segmentation learning. Through Bi-Fusion (Bidirectional Fusion), visual and textual representations evolve jointly across the encoding hierarchy. MedPlex further introduces class-level and region-level concept alignment to organize the shared representation at complementary granularities. Class-level alignment anchors each anatomical target to an aggregated clinical concept profile, while region-level alignment preserves individual concepts, such as shape, location, appearance, and texture, through class-specific visual evidence. In this way, language provides structured supervision throughout the encoder rather than serving only as a late-stage cue. MedPlex achieves state-of-the-art performance across CT and MR benchmarks for multi-organ, cardiac substructure, and tumor segmentation, including settings with real free-text clinical supervision. Code: https://github.com/rafiibnsultan/MedPlex.

Figures & tables

Figure 1: (a) MedPlex , an end-to-end training pipeline that integrates vision and language for medical image segmentation. (b) Bidirectional fusion between visual and text encoders through BiFusion blocks at each stage. (c) $\mathcal{L}_{\mathrm{ClassEmbed}}$ : mapping vision encoder embeddings to class-specific embeddings with decoder supervision. (d) $\mathcal{L}_{\mathrm{alignment}}$ : concept-grounded contrastive alignment operating at two granularities. Class-level concept alignment aggregates concept embeddings into a single textual representation matched to class-wise visual features, while region-level concept alignment aligns individual concept embeddings with class-specific visual evidence.

\toprule \multirow 3*Method	AMOS22 (CT)			MM-WHS (CT)			MM-WHS (MR)			MSD-Brain (MR)
	DSC $\uparrow$	HD95 $\downarrow$	NSD $\uparrow$	DSC $\uparrow$	HD95 $\downarrow$	NSD $\uparrow$	DSC $\uparrow$	HD95 $\downarrow$	NSD $\uparrow$	DSC $\uparrow$	HD95 $\downarrow$	NSD $\uparrow$
U-Net [ Ronneberger et al.(2015)Ronneberger, Fischer, and Brox ]	80.64	8.88	87.99	89.19	3.94	93.62	76.33	28.44	76.61	68.73	15.70	73.33
UNet++ [ Zhou et al.(2018)Zhou, Rahman Siddiquee, Tajbakhsh, and Liang ]	84.30	11.46	91.14	90.29	4.14	93.91	79.37	42.85	78.42	74.10	12.05	76.98
nnU-Net [ Isensee et al.(2021)Isensee, Jaeger, Kohl, Petersen, and Maier-Hein ]	86.33	9.05	92.01	90.05	4.03	93.72	80.71	46.37	79.21	75.10	12.38	78.29
STU-Net [ Huang et al.(2023)Huang, Wang, Deng, Ye, Su, Sun, He, Gu, et al. ]	84.21	8.88	91.42	90.85	3.88	94.22	77.11	37.51	77.25	72.05	11.95	77.97
MedNeXt [ Roy et al.(2023)Roy, Koehler, Ulrich, Baumgartner, et al. ]	84.88	8.32	91.92	90.39	3.62	94.11	77.54	36.51	77.11	71.84	14.59	76.90

Table 1: Comparison of MedPlex with benchmark vision-only models and VLMs for medical image segmentation. Vision-only models appear first, followed by VLMs (separated by a double horizontal line). The best results are bolded and the second-best results are underlined .

Figure 2: Qualitative visualizations of randomly selected image slices from AMOS22, MM-WHS (CT), MM-WHS (MR), and MSD-Brain (MR). Comparison of MedPlex against other benchmark models, with red boxes indicating areas where MedPlex outperforms.

Figure 3: t-SNE visualization shows improved concept separation in MedPlex’s text encoder (right) compared to ClinicalBERT (left), demonstrating the benefit of concept-grounded contrastive alignment.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

\toprule Title	Shape	Location	Appearance/Density	Contour/Symmetry	Internal Texture
\midrule Spleen in a CT Image	Ovoid or crescent-shaped with a smooth outer surface.	Located in the left upper quadrant, lateral to the stomach and superior to the left kidney.	Soft-tissue density with homogeneous enhancement after contrast administration.	Well-defined smooth contour, slightly convex laterally and concave medially.	Homogeneous fine-grained internal texture under normal conditions.
\midrule Right Kidney in a CT Image	Bean-shaped with a slightly concave medial border at the hilum.	Positioned in the right retroperitoneum, inferior to the liver and lateral to the inferior vena cava.	Intermediate soft-tissue density with a hypodense central collecting system.	Smooth and well-defined outer contour with subtle lobulations in some cases.	Cortex appears homogeneous, while the medulla may show striated texture due to renal pyramids.
\midrule Left Kidney in a CT Image	Bean-shaped with a slightly concave medial border at the hilum.	Positioned in the left retroperitoneum, inferior to the spleen and lateral to the aorta.	Intermediate soft-tissue density with a hypodense central collecting system.	Smooth and well-defined outer contour with subtle lobulations in some cases.	Cortex appears homogeneous, while the medulla may show striated texture due to renal pyramids.
\midrule Left Ventricle in a CT Image	Oval-like shape with thick walls tapering slightly toward the apex.	Positioned on the left side of the heart, below the left atrium and anterior to the descending aorta.	High-density muscular wall visible with contrast enhancement; the lumen appears lower in density when filled with blood or contrast agent.	Smooth and well-defined contour with relatively symmetric walls around the cavity.	Relatively homogeneous muscular structure, with myocardial striations sometimes visible.
\midrule Right Ventricle in a CT Image	Crescent-like shape with thinner walls compared with the left ventricle.	Positioned on the right side of the heart, anterior and slightly medial to the left ventricle.	Lower-density walls compared with the left ventricle, with the lumen often appearing darker due to blood or contrast agent.	Less symmetric contour than the left ventricle, with variable wall thickness.	Slightly heterogeneous texture due to trabeculae carneae along the inner surface.
\midrule Left Atrium in a CT Image	Elliptical or spherical chamber with relatively thin walls.	Superior to the left ventricle and posterior to the right atrium.	Thin, low-density walls surrounding a central lumen that appears darker due to blood or contrast.	Smooth rounded contour with relatively uniform wall thickness.	Homogeneous texture without prominent internal structures under normal conditions.

Appendix

Table 2: Examples of structured clinical concept descriptions used as textual inputs for concept-grounded alignment.

\toprule Dataset	Voxel Spacing (mm)	Intensity Range	Normalization
\midrule AMOS22 (CT)	$1.5\times 1.5\times 2.0$	$[-175,250]$ HU	$[0,1]$
BTCV (CT)	$1.5\times 1.5\times 2.0$	$[-175,250]$ HU	$[0,1]$
MM-WHS (CT)	$1.5\times 1.5\times 2.0$	$[0,400]$ HU	$[0,1]$
MM-WHS (MR)	$1.5\times 1.5\times 2.0$	Not applicable	Non-zero voxel normalization
MSD-Brain (MR)	$1.0\times 1.0\times 1.0$	Not applicable	Non-zero voxel normalization
ReXGroundingCT (CT)	$1.0\times 1.0\times 1.0$	$[-1000,1000]$ HU	$[-1,1]$

Appendix

Table 3: Dataset-specific preprocessing settings.

\toprule Parameter	Value
\midrule $\alpha_{1}$ , $\alpha_{2}$	1, 1
$\beta_{1}$ , $\beta_{2}$ , $\beta_{3}$	Learnable during training
Temperature $\tau$	0.2
Swin UNETR base feature dimension	48
ClinicalBERT feature dimension	768
Patch size	$96\times 96\times 96$

Appendix

Table 4: Key hyperparameters and model dimensions used in MedPlex.

Explore similar work

MedCLIPSeg: Probabilistic Vision-Language Adaptation for Data-Efficient and Generalizable Medical Image Segmentation
Feb 23, 2026Taha Koleilat, Hojat Asgariandehkordi, Omid Nejati Manzari +3Medical Image Segmentation Text-Guided Image Segmentation
Concordia University, Montreal, Canada
MedSIGHT: Towards Grounded Visual Comprehension in Medical Large Vision-Language Models
Jun 4, 2026Aofei Chang, Le Huang, Alex James Boyd +4Medical VLMs Medical Image Segmentation
GE HealthCare · College of Information Sciences and Technology, The Pennsylvania State University, State College, PA, USA · GE Healthcare, Bellevue, WA, USA.
MedPixel: A Unified Pixel-Language Model for Medical Reasoning and Segmentation
Aug 10, 2026Haoyu Yang, Meixing Shi, Zengjie Chen +5Medical VLMs VLM Reasoning
Zhejiang University · Fudan University · Kuaishou +2

Feb 23, 2026J/K move · Enter open · S save

MedCLIPSeg: Probabilistic Vision-Language Adaptation for Data-Efficient and Generalizable Medical Image Segmentation

Taha Koleilat, Hojat Asgariandehkordi, Omid Nejati Manzari, Berardino Barile+2

Concordia University, Montreal, Canada

Medical Image Segmentation Text-Guided Image Segmentation Medical VLMs VLM Robustness Segmentation Uncertainty Estimation

Open paper AskPDF arXiv

Medical image segmentation remains challenging due to limited annotations for training, ambiguous anatomical features, and domain shifts. While vision-language models such as CLIP offer strong cross-modal representations, their potential for dense, text-guided medical image segmentation remains underexplored. We present MedCLIPSeg, a novel framework that adapts CLIP for robust, data-efficient, and uncertainty-aware medical image segmentation. Our approach leverages patch-level CLIP embeddings through probabilistic cross-modal attention, enabling bidirectional interaction between image and text tokens and explicit modeling of predictive uncertainty. Together with a soft patch-level contrastive loss that encourages more nuanced semantic learning across diverse textual prompts, MedCLIPSeg effectively improves data efficiency and domain generalizability. Extensive experiments across 16 datasets spanning five imaging modalities and six organs demonstrate that MedCLIPSeg outperforms prior methods in accuracy, efficiency, and robustness, while providing interpretable uncertainty maps that highlight local reliability of segmentation results. This work demonstrates the potential of probabilistic vision-language modeling for text-driven medical image segmentation.

Abstract

Explore similar work

MedCLIPSeg: Probabilistic Vision-Language Adaptation for Data-Efficient and Generalizable Medical Image Segmentation

MedSIGHT: Towards Grounded Visual Comprehension in Medical Large Vision-Language Models

MedPixel: A Unified Pixel-Language Model for Medical Reasoning and Segmentation