MedCLIPSeg: Probabilistic Vision-Language Adaptation for Data-Efficient and Generalizable Medical Image Segmentation

Taha Koleilat; Hojat Asgariandehkordi; Omid Nejati Manzari; Berardino Barile; Yiming Xiao; Hassan Rivaz

cs.CVFeb 23, 2026

MedCLIPSeg: Probabilistic Vision-Language Adaptation for Data-Efficient and Generalizable Medical Image Segmentation

Authors: Taha Koleilat, Hojat Asgariandehkordi, Omid Nejati Manzari, Berardino Barile, Yiming Xiao, Hassan Rivaz

Organizations: Concordia University, Montreal, Canada

Abstract

Medical image segmentation remains challenging due to limited annotations for training, ambiguous anatomical features, and domain shifts. While vision-language models such as CLIP offer strong cross-modal representations, their potential for dense, text-guided medical image segmentation remains underexplored. We present MedCLIPSeg, a novel framework that adapts CLIP for robust, data-efficient, and uncertainty-aware medical image segmentation. Our approach leverages patch-level CLIP embeddings through probabilistic cross-modal attention, enabling bidirectional interaction between image and text tokens and explicit modeling of predictive uncertainty. Together with a soft patch-level contrastive loss that encourages more nuanced semantic learning across diverse textual prompts, MedCLIPSeg effectively improves data efficiency and domain generalizability. Extensive experiments across 16 datasets spanning five imaging modalities and six organs demonstrate that MedCLIPSeg outperforms prior methods in accuracy, efficiency, and robustness, while providing interpretable uncertainty maps that highlight local reliability of segmentation results. This work demonstrates the potential of probabilistic vision-language modeling for text-driven medical image segmentation.

Figures & tables

Figure 1: (Top) : Comparison between deterministic and probabilistic cross-modal fusion techniques in CLIP adaptation for text-driven segmentation. Probabilistic formulation models variability in visual–textual representations as distributions, enabling more robust feature alignment. (Bottom) : Robustness and Reliability plots over ID and OOD data show improved generalization, with smaller out-of-domain performance drops and better calibration of predicted confidence, reflected by lower Brier scores.

Figure 2: Overview of the proposed MedCLIPSeg framework for text-driven medical image segmentation. The model extends CLIP with vision and language encoders connected via PVL Adapters , which perform confidence-weighted image–text fusion at multiple deep layers. Segmentation and uncertainty maps arise from the mean and entropy of posterior samples, with a soft patch-level contrastive loss.

Figure 3: Illustrations of PVL Adapter and $\texttt{Attn}_{\texttt{PVL}}$ .

Method	10% Data		25% Data		50% Data		100% Data
Method	DSC $\uparrow$	NSD $\uparrow$	DSC $\uparrow$	NSD $\uparrow$	DSC $\uparrow$	NSD $\uparrow$	DSC $\uparrow$	NSD $\uparrow$
Unimodal Approaches
UNet [ 63 ]	60.95	64.43	62.74	66.16	71.61	75.14	78.49	82.07
UNet++ [ 86 ]	63.72	67.08	65.86	69.21	73.15	76.31	78.44	81.79
DeepLabv3 [ 12 ]	61.32	64.84	65.39	69.10	68.58	72.57	73.28	77.42
AttnUNet [ 56 ]	62.78	66.25	64.97	68.53	71.34	74.96	76.30	79.77

Table 1: Data-efficiency evaluation: This table reports the average DSC and NSD (%) when varying the fraction of training data across different segmentation methods. Best results are in bold , and second-best are underlined .

Method	Breast Ultrasound					Polyp Endoscopy					Brain MRI		Skin Dermatoscopy
	Source	Target				Source	Target				Source	Target	Source	Target
	BUSI	BUSBRA	BUSUC	BUID	UDIAT	Kvasir-SEG	ColonDB	ClinicDB	CVC300	BKAI	BTMRI	BRISC	ISIC	UWaterloo
LViT [ 50 ]	75.32	59.41	67.95	53.51	65.60	85.29	60.01	75.27	70.22	67.17	81.41	71.86	91.21	58.87
CLIPSeg [ 51 ]	80.95	63.66	75.03	68.43	56.67	81.98	59.93	71.49	72.74	66.46	86.33	77.61	90.55	80.19
DenseCLIP [ 61 ]	71.85	53.34	70.97	63.53	54.93	79.32	56.38	68.08	64.71	61.63	70.30	34.12	89.29	53.39
ZegCLIP [ 85 ]	72.08	61.08	73.57	71.75	52.41	78.46	53.46	69.75	60.73	65.60	76.65	66.31	81.45	38.60

Table 2: Domain generalization: Models are trained on a source dataset and evaluated on OOD target datasets without adaptation. DSC (%) values are reported where the best results are in bold , and second-best are underlined .

Probabilistic Vision-Language Adapters
Method	Domain Generalization
Method	ID DSC (%)	OOD DSC (%)	HM DSC (%)
MedCLIPSeg (Ours)	89.11	79.02	83.76
w/o PVL Adapters	81.23 (-7.88) $\downarrow$	55.23 (-23.79) $\downarrow$	65.75 (-18.01) $\downarrow$
w/o Gating	87.55 (-1.56) $\downarrow$	76.79 (-2.23) $\downarrow$	81.82 (-1.94) $\downarrow$
w/o $\texttt{Attn}_{\texttt{PVL}}$	86.21 (-2.90) $\downarrow$	74.13 (-4.89) $\downarrow$	79.71 (-4.05) $\downarrow$

Table 3: Effect of different key components.

Text Prompt Design	ID DSC (%)	OOD DSC (%)	HM DSC (%)
Contradictory	68.60	63.21	65.79
Missing Location	86.98	77.75	82.11
Overdescriptive	82.93	74.49	78.48
Underdescriptive	66.91	49.38	56.82
Original	89.11	79.02	83.76

Table 4: Effect of text prompt design.

Figure 4: Segmentation and uncertainty visualizations. Uncertainty peaks along lesion boundaries and remains consistent across diverse datasets, indicating reliable calibration and generalization. ID data are in blue while OOD data are in red .

Figure 5: Layer-wise interventions ( left ) and confidence weighting ( $\beta$ ) ( right ) ablations averaged on ID and OOD data.

Pre-trained Model	ID DSC (%)	OOD DSC (%)	HM DSC (%)
CLIP [ 60 ]	88.48	74.81	81.07
PubMedCLIP [ 22 ]	86.67	73.05	79.28
BiomedCLIP [ 76 ]	88.70	77.08	82.48
UniMedCLIP [ 42 ]	89.11	79.02	83.76

Table 5: Effect of pre-trained vision–language models.

Dataset	Train	Validation	Test	Modality	Organ
Data-Efficiency Evaluation
BUSI [ 1 ]	(62, 156, 312)	(7, 19, 39)	78	Ultrasound	Breast
BTMRI [ 14 ]	(273, 684, 1,369)	(132, 330, 660)	1,005	MRI	Brain
ISIC [ 66 , 17 ]	(80, 202, 404)	(9, 22, 45)	379	Dermatoscopy	Skin
Kvasir-SEG [ 36 ]	(80, 200, 400)	(10, 25, 50)	100	Endoscopy	Colon
QaTa-COV19 [ 18 ]	(571, 1,429, 2,858)	(142, 357, 714)	2,113	X-ray	Chest

Table S1 : Summary of medical datasets: Overview of datasets used in the data-efficiency, fully supervised, and domain generalization benchmarks. For data-efficiency experiments, values in parentheses under Train and Validation indicate the number of samples corresponding to (10%, 25%, 50%) of the full splits; other sections report full counts.

Model	Text?	Params. (M)	FLOPs (G)	Inf. Time (s)
UNet [ 63 ]	✗	14.8	50.3	0.55
UNet++ [ 86 ]	✗	74.5	94.6	0.81
DeepLabv3 [ 12 ]	✗	57.6	38.4	1.16
AttnUNet [ 56 ]	✗	34.9	101.9	0.77
nnUNet [ 34 ]	✗	19.1	412.7	1.55
Swin-UNet [ 9 ]	✗	82.3	67.3	1.38

Table S2 : Comparison of computational complexity between different methods. Models using text or multimodal supervision are marked with a ✓ in the “Text?” column.

Design Type	Description Style	Example (Normal)	Example (Tumor)
Original	Balanced, accurate	“ The breast appears normal with no signs of lesions. ”	“ A malignant tumor is present in the upper-left region of the breast. ”
Underdescriptive	Minimal, label-only	“ Normal breast. ”	“ Tumor present. ”
Overdescriptive	Verbose, redundant	“ The breast tissue appears entirely healthy, with homogeneous echotexture throughout. ”	“ A clearly defined malignant tumor with irregular boundaries located in the upper-left quadrant. ”
Contradictory	Incorrect/Conflicting info	“ Normal breast tissue with a visible lesion in the image. ”	“ Malignant tumor detected, but breast appears completely normal. ”
Missing Location	No spatial info	“ The breast appears normal with no signs of lesions. ”	“ A malignant tumor is detected in the breast. ”

Table S3 : Prompt design taxonomy with examples. Each configuration illustrates how wording choices (conciseness, spatial detail, contradictions, and noise) affect the semantics supplied to the model.

Samples	Runtime (s/batch)	FPS	HM DSC (%)	HM Spearman (%)	3D Runtime (s/vol)
5	1.78	24.92	83.52	83.07	0.98
10	2.20	14.35	83.66	83.44	1.03
20	4.08	7.64	83.71	83.52	1.97
30	6.01	5.23	83.76	83.84	2.90

Table S4 : Performance-cost tradeoff under MC sampling

Figure S1 : Effect of $\lambda_{SoftCon}$ on Domain Generalization

Gating Init. ( $g$ )	ID DSC (%)	OOD DSC (%)	HM DSC (%)
sigmoid(-0.5)	88.79	74.65	81.11
sigmoid(0)	89.11	79.02	83.76
sigmoid(0.5)	88.93	77.51	82.83

Table S5 : Effect of the gating initialization

Two-way Mechanism	ID DSC (%)	OOD DSC (%)	HM DSC (%)
None	88.71	77.71	82.85
Text First	88.55	76.99	82.37
Vision First	89.11	79.02	83.76

Table S6 : Effect of the two-way attention mechanism

Contrastive Pooling	ID DSC (%)	OOD DSC (%)	HM DSC (%)
[CLS]	88.89	78.28	83.25
Attention Pooling	88.73	75.60	81.64
Average Pooling	89.11	79.02	83.76

Table S7 : Effect of the pooling strategies

Num. Upscale	ID DSC (%)	OOD DSC (%)	HM DSC (%)
1	88.73	75.74	81.72
2	89.11	79.02	83.76
3	88.64	74.99	81.24

Table S8 : Effect of the number of upscaling layers

Adapter Dim. ( $D_{s}$ )	ID DSC (%)	OOD DSC (%)	HM DSC (%)
64	87.76	74.63	80.66
128	88.68	76.44	82.11
192	88.93	76.01	81.96
256	89.11	79.02	83.76
512	88.56	77.96	82.92

Table S9 : Effect of the shared dimension in the PVL adapters

Figure S2 : FP/FN comparison between deterministic and probabilistic MedCLIPSeg .

Figure S3 : Breast Tumor Ultrasound Segmentation and uncertainty visualizations. Uncertainty peaks along lesion boundaries and remains consistent across breast ultrasound datasets, indicating reliable calibration and generalization.

Figure S4 : Polyp Endoscopy Segmentation and uncertainty visualizations. Uncertainty peaks along lesion boundaries and remains consistent across polyp endoscopy datasets, indicating reliable calibration and generalization.

Mechanism	ID DSC (%)	OOD DSC (%)	HM DSC (%)
None	88.97	77.49	82.83
Scaling	88.92	78.78	83.54
Difference	89.11	79.02	83.76

Table S10 : Effect of confidence-weighted attention mechanism

Figure S5 : Relationship between predictive uncertainty and pixel-level segmentation error across imaging domains.

$\mathcal{L}_{Seg}$ ?	ID DSC (%)	OOD DSC (%)	HM DSC (%)
✗	19.84	12.68	15.47
✓	89.11	79.02	83.76

Table S11 : Effect of the segmentation annotations

Method	ID		OOD		HM
Method	DSC	Spearman	DSC	Spearman	DSC	Spearman
Ariadne’s Thread [ 81 ]	68.25	–	27.24	–	38.94	–
EviVLM [ 58 ]	84.06	–	54.47	–	66.10	–
VLSM-Ensemble [ 20 ]	87.36	–	63.24	–	73.37	–
nnUNet + Ensembling [ 79 ]	86.50	66.74	74.20	55.22	79.80	60.44
MedCLIPSeg (Evidential)	88.18	78.49	76.61	76.79	81.99	77.63

Table S12 : Domain generalization with more baselines (%)

Method	ID	OOD	HM
SAN [ 68 ]	(86.9, 88.9, 84.5)	(74.3, 83.9, 69.9)	(80.1, 86.3, 76.5)
VLSM-Adapter [ 18 ]	(88.5, 87.9, 85.8)	(80.3, 80.8, 73.3)	(84.2, 84.2, 79.0)
CAT-Seg [ 15 ]	(87.2, 89.1, 86.1)	(81.1, 82.6, 74.6)	(84.0, 85.7, 79.9)
MedCLIPSeg (Ours)	(89.9, 90.8, 89.1)	(86.2, 80.7, 79.0)	(88.0, 85.5, 83.8)

Table S13 : Domain generalization with different metrics (%)

Method	BUSI		BTMRI		ISIC		Kvasir-SEG		QaTa-COV19		EUS
	DSC $\uparrow$	NSD $\uparrow$	DSC $\uparrow$	NSD $\uparrow$	DSC $\uparrow$	NSD $\uparrow$	DSC $\uparrow$	NSD $\uparrow$	DSC $\uparrow$	NSD $\uparrow$	DSC $\uparrow$	NSD $\uparrow$
Unimodal Approaches
UNet [ 63 ]	49.33	52.13	64.49	69.12	79.43	81.93	40.66	44.16	71.41	77.18	60.40	62.07
UNet++ [ 86 ]	53.80	57.58	62.23	66.11	82.83	85.54	46.27	49.28	69.22	74.99	67.96	69.00
DeepLabv3 [ 12 ]	42.45	45.64	61.96	66.47	84.62	87.72	45.14	48.32	68.51	74.38	65.22	66.50
AttnUNet [ 56 ]	54.66	58.17	58.68	62.77	85.16	88.08	42.46	45.68	70.86	76.37	64.85	66.44

Table S14 : Per-dataset segmentation with 10% Labeled Data: This table reports DSC and NSD values (%) across six medical image segmentation benchmarks. All baseline methods are trained using 10% of the ground-truth annotations. Best results are in bold , and second-best are underlined .

Method	BUSI		BTMRI		ISIC		Kvasir-SEG		QaTa-COV19		EUS
	DSC $\uparrow$	NSD $\uparrow$	DSC $\uparrow$	NSD $\uparrow$	DSC $\uparrow$	NSD $\uparrow$	DSC $\uparrow$	NSD $\uparrow$	DSC $\uparrow$	NSD $\uparrow$	DSC $\uparrow$	NSD $\uparrow$
Unimodal Approaches
UNet [ 63 ]	56.28	59.90	70.40	74.98	82.44	85.68	41.88	44.93	68.23	73.05	57.23	58.45
UNet++ [ 86 ]	56.68	60.36	76.46	81.07	84.35	87.05	62.23	65.56	43.36	48.05	72.07	73.15
DeepLabv3 [ 12 ]	55.03	58.85	73.16	78.77	87.01	90.08	54.34	58.04	66.98	72.75	55.79	56.10
AttnUNet [ 56 ]	62.55	66.75	64.24	68.11	85.72	88.59	55.17	58.94	55.00	60.14	67.16	68.65

Table S15 : Per-dataset segmentation with 25% Labeled Data: This table reports DSC and NSD values (%) across six medical image segmentation benchmarks. All baseline methods are trained using 25% of the ground-truth annotations. Best results are in bold , and second-best are underlined .

Method	BUSI		BTMRI		ISIC		Kvasir-SEG		QaTa-COV19		EUS
	DSC $\uparrow$	NSD $\uparrow$	DSC $\uparrow$	NSD $\uparrow$	DSC $\uparrow$	NSD $\uparrow$	DSC $\uparrow$	NSD $\uparrow$	DSC $\uparrow$	NSD $\uparrow$	DSC $\uparrow$	NSD $\uparrow$
Unimodal Approaches
UNet [ 63 ]	60.46	64.38	81.47	86.06	86.85	89.93	72.12	75.51	66.36	71.15	62.38	63.80
UNet++ [ 86 ]	63.63	67.16	80.21	84.61	88.29	91.23	74.53	77.72	63.80	67.71	68.42	69.43
DeepLabv3 [ 12 ]	55.83	60.21	77.82	83.56	86.14	89.06	67.72	71.90	64.86	70.29	59.12	60.39
AttnUNet [ 56 ]	59.81	63.46	72.96	77.24	87.28	90.48	74.79	78.66	64.71	69.95	68.52	69.99

Table S16 : Per-dataset segmentation with 50% Labeled Data: This table reports DSC and NSD values (%) across six medical image segmentation benchmarks. All baseline methods are trained using 50% of the ground-truth annotations. Best results are in bold , and second-best are underlined .

Method	BUSI		BTMRI		ISIC		Kvasir-SEG		QaTa-COV19		EUS
	DSC $\uparrow$	NSD $\uparrow$	DSC $\uparrow$	NSD $\uparrow$	DSC $\uparrow$	NSD $\uparrow$	DSC $\uparrow$	NSD $\uparrow$	DSC $\uparrow$	NSD $\uparrow$	DSC $\uparrow$	NSD $\uparrow$
Unimodal Approaches
UNet [ 63 ]	70.04	73.88	86.06	90.71	89.10	92.06	80.26	83.53	76.97	82.32	68.53	69.92
UNet++ [ 86 ]	67.54	71.15	83.30	87.94	89.36	92.17	84.81	88.07	73.52	78.23	72.08	73.15
DeepLabv3 [ 12 ]	63.02	67.18	83.49	89.31	76.34	80.24	82.00	85.73	69.63	75.67	65.21	66.37
AttnUNet [ 56 ]	62.65	66.21	85.24	89.76	89.00	92.03	78.59	81.79	73.57	78.71	68.76	70.13

Table S17 : Per-dataset segmentation with 100% Labeled Data: This table reports DSC and NSD values (%) across six medical image segmentation benchmarks. All baseline methods are trained in a fully supervised manner using ground-truth annotations. Best results are in bold , and second-best are underlined .

Explore similar work

BiCLIP: Bidirectional and Consistent Language-Image Processing for Robust Medical Image Segmentation
Feb 25, 2026Saivan Talaei, Fatemeh Daneshfar, Abdulhady Abas Abdullah +1Multimodal Robustness Medical VLMs
Department of Computer Engineering, University of Kurdistan, Sanandaj, Kurdistan, Iran · Artificial Intelligence and Innovation Center, University of Kurdistan, Erbil, Kurdistan, Iraq · Center for Machine Vision and Signal Analysis, University of Oulu, Oulu, Finland
SwinTextUNet: Integrating CLIP-Based Text Guidance into Swin Transformer U-Nets for Medical Image Segmentation
Apr 11, 2026Ashfak Yeafi, Parthaw Goswami, Md Khairul Islam +1Medical Image Segmentation Text-Guided Image Segmentation
Department of Electrical and Electronic Engineering Khulna University of Engineering & Technology Khulna-9203, Bangladesh · Department of Electronics and Communication Engineering Khulna University of Engineering & Technology Khulna-9203, Bangladesh · Department of Mathematics and Computer Science Hobart and William Smith Colleges Geneva, NY, USA
DistMedVL: Distributional Vision-Language Alignment for Uncertainty-Aware Medical Image Segmentation
Aug 6, 2026Jiaxuan Li, Qing Xu, Xiangjian He +4Cross-Modality Medical Image Segmentation Medical VLMs
School of Computer Science, University of Nottingham Ningbo China, Ningbo, Zhejiang, China · School of Computer Science, University of Nottingham, UK

Feb 25, 2026J/K move · Enter open · S save

BiCLIP: Bidirectional and Consistent Language-Image Processing for Robust Medical Image Segmentation

Saivan Talaei, Fatemeh Daneshfar, Abdulhady Abas Abdullah, Mourad Oussalah

Department of Computer Engineering, University of Kurdistan, Sanandaj, Kurdistan, Iran · Artificial Intelligence and Innovation Center, University of Kurdistan, Erbil, Kurdistan, Iraq · Center for Machine Vision and Signal Analysis, University of Oulu, Oulu, Finland

Multimodal Robustness Medical VLMs Medical Image Segmentation Few-Shot Medical Image Segmentation

Open paper AskPDF arXiv

Medical image segmentation is a cornerstone of computer-assisted diagnosis and treatment planning. While recent multimodal vision-language models have shown promise in enhancing semantic understanding through textual descriptions, their resilience in "in-the-wild" clinical settings-characterized by scarce annotations and hardware-induced image degradations-remains under-explored. We introduce BiCLIP (Bidirectional and Consistent Language-Image Processing), a framework engineered to bolster robustness in medical segmentation. BiCLIP features a bidirectional multimodal fusion mechanism that enables visual features to iteratively refine textual representations, ensuring superior semantic alignment. To further stabilize learning, we implement an augmentation consistency objective that regularizes intermediate representations against perturbed input views. Evaluation on the QaTa-COV19 and MosMedData+ benchmarks demonstrates that BiCLIP consistently surpasses state-of-the-art image-only and multimodal baselines. Notably, BiCLIP maintains high performance when trained on as little as 1% of labeled data and exhibits significant resistance to clinical artifacts, including motion blur and low-dose CT noise.

Abstract

Explore similar work

BiCLIP: Bidirectional and Consistent Language-Image Processing for Robust Medical Image Segmentation

SwinTextUNet: Integrating CLIP-Based Text Guidance into Swin Transformer U-Nets for Medical Image Segmentation

DistMedVL: Distributional Vision-Language Alignment for Uncertainty-Aware Medical Image Segmentation